Evidence-based Global Software Engineering Risks Extracted from Systematic Literature Review

Transcription

1 Evidence-ased Gloal Software Engineering Risks Extracted from Systematic Literature Review J. Verner, O. P. Brereton, B. Kitchenham, M. Turner, M. K. Niazi TR ISSN Feruary 2012 School of Computing and Mathematics Keele University Keele, Staffordshire, ST5 5BG UK

2 Evidence-ased Gloal Software Engineering Risks Extracted from Systematic Literature Reviews J. Verner, O. P. Brereton, B. Kitchenham, M. Turner Keele University Staffordshire ST5 5BG, United Kingdom M. K. Niazi Keele University, UK and KFUPM, Saudi Araia ABSTRACT Gloal software development (GSD) projects are often largescale, tend to e complex, and have an increased risk of failure. An understanding of common GSD risks can help project managers control the risks and thus increase the likelihood of project success. We extend an earlier tertiary GSD systematic literature review (SLR) to extract data from the SLR studies identified to discover, 1) what are the most common risks descried in the SLRs, 2) what empirical evidence is there for the risks, and 3) who is responsile for controlling the risk. The 123 risks we found were categorized under four major themes 1) GSD outsourcing rationale, 2) software development, 3) human resources, and 4) project management. Most risks (32%) were related to project management. The risk with the most empirical support was a high level strategy risk. Mapping the research rather than providing industry guidance was the focus of the majority of the GSD SLRs. Categories and Suject Descriptors K.6.1 Project and People Management; lifecycle; K.6.3 Software Management; software development. General Terms Management Keywords Gloal software development, risk, risk management, systematic literature reviews 1. INTRODUCTION The aim of this research is to find the most common GSD risks (and the supporting evidence for those risks), from the gloal software development (GSD) systematic literature reviews (SLRs) we had identified in previous research [19]. An understanding of the most frequently occurring GSD project risks can help project managers control those risks and thus increase the likelihood of a successful project outcome. By a successful project we mean that the client of a development project elieves that the project has met their expectations. The software development paradigm is changing with the increasing use of GSD and a numer of different GSD usiness models have een identified e.g. [3, 13]. GSD can e used to descrie one or more of the following situations: organizations that have shifted all or part of their software development offshore to independent vendors located in lower cost destinations, or to destinations where the required skills are more readily availale, or organizations that have distriuted their software development activities across multiple sites which are in different countries, e.g., IBM, Bosch, and Siemens [4]. Software may e developed gloally y an organization to use itself, or to sell, or to incorporate into a device that the company sells (in-sourcing), e.g., Siemens. A client company may outsource software development to a vendor who then develops the required software for the company. The vendor may e located in the same country (outsourcing) or in a different country (GSD). GSD projects are often large-scale, and the gloal aspect of development leads to significantly increased complexity; GSD complexity leads to increased risk. Suggestions have een made that a 50% failure rate for GSD projects is not uncommon [17]. Offshore projects tend to e unsuccessful ecause physical time, cultural, organizational, and stakeholder distances negatively influence communication and knowledge exchange etween onshore and offshore project team memers [11]. When a software project is developed in multiple countries, the vendor project manager must address additional execution risks, including those related to project distriution, time zone differences, communication, coordination and control, project contextual issues, and infrastructure [5, 13, 17]. To deal with the increased risk the client must monitor the development closely [1, 18]; an understanding of the most widespread risks can help alleviate prolems efore they occur. Our previous research identified 24 GSD SLR studies descried in 37 papers [19]; we now extend that research y using the SLR studies to discover: RQ1 What are the most common GSD project risks identified in the SLRs? RQ2 What empirical evidence is there for these risks? RQ3 Who is responsile for controlling the risks? In the next section we riefly descrie our research methodology; in Section 3 we present our results. Section 4 provides a discussion of the limitations of our research, while Section 5 presents a discussion and conclusions.

3 RESEARCH METHODOLOGY An SLR of any kind is a method of identifying, evaluating and interpreting all availale research relevant to a particular research question or topic area or phenomenon of interest [14]. The aim is to synthesize existing evidence in a fair, rigorous, and open manner. After specifying research questions a review protocol is developed; this includes definitions of: (1) the search process, (2) inclusion and exclusion criteria, (3) the selection process, (4) the data extraction process, and (5) data synthesis. 2.1 Search Process The search process employed is descried in detail in [19]; thus we will provide only an outline here. Our search, which identified SLRs addressing GSD, was road, comining automated and manual searches. We identified peer-reviewed articles pulished (or availale on-line) up to Octoer Seven search engines and indexing systems plus manual searches of two conference proceedings were used; in addition iliographies of all selected papers were scanned for additional papers (snowalling). 2.2 Inclusion and Exclusion criteria Articles were included if they: Reported SLRs or meta-analyses in GSD Were pulished in, or sumitted to, a conference or journal or were technical reports or ook chapters Were written in English Articles were excluded if they were: Masters and PhD studies not pulished in refereed conferences or journals Informal literature surveys (no defined search questions, no search process, no defined data extraction or data analysis process) Papers discussing the process of performing SLRs, or metaanalyses SLRs relating to topics such as evaluating installed systems and applications, IT services, software applications and IT operations, open source software development, teaching and education. 2.3 Paper Selection Our search resulted in the identification of 24 studies reported in 37 papers. When we compared the papers we selected with those found in previous tertiary SLR studies that included GSD research [7, 14, 15] we found that we had not missed any appropriate papers. We reference the studies included in our research and listed in Tale 1 with a study numer, e.g., [S12]. If there is more than one paper reporting the same study we follow (the study numer) with a letter referring to the particular paper, e.g., [S12]. 2.5 Data extraction The following details had een previously extracted from the SLR studies we had identified: paper title, author names, affiliations, pulication venue, main GSD topic, the numer of primary studies included in the SLR [19], and details related to practitioner advice [20]. A data extraction form was provided to each of the researchers. Details to e extracted included paper ID, specific risk details, main risk category (ased on Tale 5 in [20]; namely outsourcing vendor selection, architecture, development process, project management, requirements engineering, configuration management, culture, communication, collaoration, control and distance), risk su-category, strength of the evidence for the risk (i.e., the numer of primary papers that suggested that the factor is a risk) and who should have the responsiility for controlling the risk. If an appropriate risk category or sucategory was not availale the researchers were asked to define extra categories as required. Different SLR authors use different terms for risk including challenge, arrier, threat, and prolem. The first author extracted the risks identified in each of the studies. To check her data extraction the other four authors each extracted the risks present in all papers relating to two studies. This gave us eight studies where we could compare the first author s data with that extracted y the other authors (thus giving a 22% data extraction check). As we are interested in identifying common risks ased on evidence, those risk items with support from fewer than three primary studies were disregarded. 2.6 Data synthesis To answer our research questions we sorted the risks on general risk category and su-category, then on supporting evidence. We applied the themes and su themes shown elow, which we had used earlier in [20] and had een developed y thematic analysis [12]. 1. GSD outsourcing rationale, including high level and detailed vendor selection strategy. 2. software development, including requirements engineering, architectural design, software development process, and configuration management. 3. human resources including culture and social integration, training, communication, and collaoration. 4. project management including planning, and coordination, and control. 3. RESULTS In this section, we report on the results related to our research questions. Tale 1 lists the GSD studies included in this research. Primary studies included in the SLR studies range from 9 [S20] to 122 [S13]; two studies, [S1, S14], did not make clear how many primary studies were included in their SLR. Overall we identified a total of 123 risks with support from at least three primary studies. Categorization of the risks was difficult as many items overlapped two or more categories e.g., In GSD differences among national cultures of the participants affect their collaoration, can e considered to e a cultural risk as well as a collaoration risk. Thus, we classified risks where we felt they est fitted; however, other researchers may have organized our risks differently. We discuss the risks, arranged y categories and sucategories, in more detail elow, and list the risks with the most empirical support in tales. For each risk we furnish evidence in terms of 1) the actual SLR studies that identified the item as a risk, and 2) the numer of primary studies in each SLR that provides support for the risk. If we found 10 or fewer risks for a su-category we list all the risks we identified in the tale; if we found more than ten risks for a su-category we include the 10 risks with the most empirical support.

4 Study No of Studies S1 unknown 1.5 a S Tale 1: GSD SLR studies and papers reporting those studies Quality Paper Reference Ågerfalk P. J., B., Holmström H., Lings B., Lundell B., Conchúir E.Ó., (2005) "A Framework for Considering Opportunities and Threats in Distriuted Software Development" Proceedings of the International Workshop on Distriuted Software Development, Austrian Computer Society, p.p Lings B., Lundell B., Agerfalk P., and Fitzgerald B., (2006) "Ten strategies for successful distriuted development", The Transfer and Diffusion of Information Technology for Organizational Resilience, Springer, pp Ali, N., Beecham, S., Mistrík, I., (2010) "Architectural knowledge management in gloal software development: A review" Proceedings of the 5th International Conference on Gloal Software Engineering, ICGSE, pp S S a S5 c S S S S S a S a c S a S a Alsudairi M, Dwivedi Y. K., (2010) "A multi-disciplinary profile of IS/IT outsourcing research" Journal of Enterprise Information Management, Vol. 23 Iss: 2, pp Costa C., Cunha C., Rocha R., Franca A., da Silva F., Prikladnicki R., (2011) "Models and tools for Managing Distriuted Software Development: A systematic literature review" Proceedings of EASE. da Silva F. Q. B., Prikladnicki R., França A. C. C., Monteiro C. V. F., Costa C., Rocha R., (2011) "Research and Practice of Distriuted Software Development Project Management: A Systematic Mapping Study" sumitted to Information and Software Technology. da Silva F.Q.B., Prikladnicki R., França A., Monteiro, Costa C., Rocha R.,(2011) "An evidence-ased model of distriuted software development project management: results from a systematic mapping study", Journal of Software Maintenance and Evolution: Research and Practice (on line). da Silva F.Q.B., Costa C., França A.C.C., Prikladnicki R., (2010) "Challenges and solutions in Distriuted Software Development Project Management: A systematic literature review" Proceedings of the 5th International Conference on Gloal Software Engineering, (ICGSE), pp Eling, T., Audy, J.L.N., Prikladnicki, R., (2009) "A systematic literature review of requirements engineering in distriuted software development environments" ICEIS 2009,Proceedings of the 11th International Conference on Enterprise Information Systems,, ISAS, pp Fauzi S. S. M., Bannerman P. L., and Staples M., (2010) "Software Configuration Management in Gloal Software Development; A Systematic map" APSEC '10 Proceedings of 2010 APSEC, pp Hossain E., Ali Baar M., Paik H.-Y., (2009) "Using scrum in gloal software development: A systematic literature review" Proceedings th IEEE International Conference on Gloal Software Engineering, ICGSE, pp Huang H., (2007) " Cultural Issues in Gloally Distriuted Information Systems Development; A Survey and Analysis" AMCIS Proceedings, paper 254. Jalali, S., Wohlin, C. (2011) "Gloal software engineering and agile practices: a systematic review" Journal of Software Maintenance and Evolution: Research and Practice (on line). Jalali, S., Wohlin, C., (2010) "Agile practices in gloal software engineering - A systematic map" Proceedings of the 5th International Conference on Gloal Software Engineering, ICGSE 2010, pp Jiménez M., Piattini M., and Vizcaino A., (2010) "A Systematic Review of Distriuted Software Development: Prolems and Solutions" chapter in Handook of Research on Software Engineering, IGI Gloal and Productivity Technologies: Implications of Gloalization, pp Jimenez M., Piattini M., (2009) "Prolems and solutions in Distriuted Software Development: A Systematic Review" Second International Conference on Software engineering approaches for offshore and outsourcing development, LNIBIP Vol. 16, pp Jiménez M., Piattini M., and Vizcaino A., (2009) "Challenges and improvements in distriuted software development: A systematic review" Advances in Software Engineering. Khan, S.U., Niazi, M., Ahmad, R., (2011) "Barriers in the selection of offshore software development outsourcing vendors: An exploratory study using a systematic literature review" Information and Software Technology, 53 (7), pp Khan, S.U., Niazi, M., Ahmad, R., (2009) "Critical arriers for offshore software development outsourcing vendors: A systematic literature review" Proceedings - Asia-Pacific Software Engineering Conference, APSEC, pp Khan, S.U., Niazi, M., Ahmad, R., (2011) "Factors influencing clients in the selection of offshore software outsourcing vendors: An exploratory study using a systematic literature review" Journal of Systems and Software, 84 (4), pp Khan, S.U., Niazi, M., Ahmad, R., (2009) " Critical success factors for offshore software development outsourcing vendors: a systematic literature review" Proceedings th IEEE International Conference on Gloal Software Engineering, ICGSE, pp

5 S14 Unknown ut at least Kroll, J., Luis J., Audy N., and Prikladnick R., (2010), Mapping the evolution of research on gloal software engineering: A Systematic Literature Review, ICEIS (3) 2011: S López, A., Nicolás, J., Toval, A., (2009) "Risks and safeguards for the requirements engineering process in gloal software development" Proceedings th IEEE International Conference on Gloal Software Engineering, ICGSE, pp S S S S a a c Noll J., Beecham S., Richardson I., (2010) "Gloal software development and collaoration: arriers and solutions" August, ACM Inroads, Volume 1 Issue 3. Nurdiani I., Jaangwe R., Smite D., Damian D., (2011) "Risk Identification and Risk Mitigation Instruments for Gloal Software Engineering: A systematic review and survey results" PARIS workshop ICGSE. Persson J S., Mathiassen L., (2011) "A process for managing risks in Distriuted teams" IEEE Software 27, 1. Jan- Fe., pp Persson J.S., Mathiassen L., Boeg J., Madsen T. S., Steinson F., (2009) "Managing Risks in Distriuted Software Projects An Integrative framework", IEEE TEM Vol 56 Iss 3 August, pp Prikladnicki R., Audy, J.L.N., (2010) "Process Models in the practice of distriuted software development: A systematic review of the literature", Information and Software Technology Vol 52 No 8, August, pp Prikladnicki, R., Audy, J.L.N., Shull, F., (2010) "Patterns in effective distriuted software development" IEEE Software, 27 (2), pp Prikladnicki R., Damien D., and Audy J. L. N., (2008) "Patterns of evolution in the practice of distriuted software development: quantitative results from a systematic review" Evaluation and Assessment in Software Engineering (EASE), Bari, Italy, pp S S a c S S S Rocha R. G. C., Costa, Rodrigues C., de Azevedo R., Junior I., Meira S., Prickladnicki R., (2011) "Collaoration models in distriuted software development a systematic review" CLEI Electronic Journal Vol 14 No 2, August, paper 1. Šmite, D., Wohlin, C., Gorschek, T., Feldt, R. (2010) "Empirical evidence in gloal software engineering: A systematic review" Empirical Software Engineering, 15 (1), pp Šmite, D., Wohlin, C., (2011) A whisper of evidence in gloal software engineering IEEE Software, 28 (4), pp Šmite D., Wohlin C., Gorschek T., Feldt R., (2008) "Reporting Empirical Research in Gloal Software Engineering: a classification scheme" International Conference on Software Engineering (ICGSE). Steinmacher I., Chaves A.P., Gerosa, M.A., (2010) "Awareness support in gloal software development: A systematic review ased on the 3C collaoration model" 16th International Workshop on Groupware (CRIWG), Lecture Notes in Computer Science (including suseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), LNCS, pp Treude C., Storey M-A., Weer J., (2009) " Empirical studies on collaoration in software development A systematic Literature Review" Review Literature And Arts Of The Americas and Technical Report DCS-331-IR, Department of Computer Science, University of Victoria, Decemer. Yalaho A., (2006) "A Conceptual Model of ICT-Supported Unified Process of International Outsourcing of Software Production,", 10th IEEE International Enterprise Distriuted Oject Computing Conference Workshops (EDOCW'06), pp GSD Outsourcing rationale Outsourcing rationale risks are found at two levels, 1) GSD high level vendor selection strategy (related to organizational issues), and 2) GSD detailed vendor strategy (related to a specific project). Four studies [S5, S12, S16, S17] contriute 21 GSD risks with support from at least three primary studies (17% of the risks we found overall). Almost all risks should e the concern of senior management in the client organization, although successful senior vendor management must e aware of the client s selection criteria as the criteria may well provide risks for the vendor. The vendor is actually the focus of [S12] where arriers to vendor selection are extracted; vendors are told to focus on the arriers if they wish to attract clients High GSD Vendor Strategy Selection Four studies identify high level GSD strategy risks [S5, S12, S16, S17]; almost all of the 12 risks identified have a reasonaly good level of empirical support. Tale 2 lists the 10 high level GSD vendor selection strategy risks with the most empirical support. This set of risks highlights factors that should e considered for vendor selection y a client organization whose senior management is contemplating GSD. Risk #3 Lack of protection for intellectual property rights is supported y two SLR studies. Tale 2: High level GSD vendor selection strategy risks level 1 Language and cultural arriers. [S12] 55 2 Vendor country instaility. [S12] 50 3 Lack of protection for intellectual property rights. [S12] Vendor s unethical ehaviour, e.g., [S12] 27 revealing knowledge of strategy of one client to another. 5 Poor vendor infrastructure. [S12] 32 6 Previous delays in delivery. [S12] 22 7 Time zone differences etween client [S12] 19 and vendor. 8 Vendor incompatiility with client. [S12] 10 9 Vendor s strategic inflexiility. [S12] Organizational challenges caused y GSD eyond distance and cultural differences. [S16] 6

6 3.1.2 Detailed GSD Vendor Selection Strategy Two studies discuss risks at the detailed level [S5, S12]. Tale 3 provides particulars of the risks and their empirical support. All nine of the risks we found are under the control of the client and are focussed on vendor selection for a specific project. Because of the relatively large numer of primary studies extracted y [S12] there is good empirical support from primary studies for this sutheme. Tale 3: Detailed GSD vendor selection strategy risks 1 Vendor s lack of project management. [S12] 48 2 Vendor s lack of technical capaility. [S12] 46 3 Vendor s poor relationship [S12] 43 management. 4 Communication gap etween client and [S12] 43 vendor. 5 Poor vendor contract management. [S12] 42 6 Poor vendor quality of service and [S12] 42 systems/processes. 7 Hidden vendor costs. [S12] 37 8 Vendor lack of control over project. [S12] 33 9 Many different client stakeholders Software Development We found 28 risks related to the software development lifecycle (23% of risks found overall). All the software development risks we found relate to 1) requirements engineering, 2) the software development process, 3) architectural design, and 4) software configuration management. Many risks relating to the development process are under the control of the vendor, mainly the vendor project manager, though some are appropriate for project leads, technical leads, quality assurance managers and software developers. However, it should not e forgotten that the client project manager must continuously monitor what is happening during the development process. Of course during requirements engineering oth client and vendor need to e involved Requirements engineering Most of the risks deal with communicating requirements, and collaoration, so that vendor practitioners may more easily understand exactly what the project needs to deliver. Tale 4 lists the risks with the most empirical support. Three studies [S5, S6, S15] furnished the 14 risks we found. There is a moderate level of support for the risks though none is supported y more than two SLRs and seven primary studies. Given the criticality of getting the requirements right, and the importance of client stakeholder involvement in requirements elicitation, it is surprising that there is not more evidence for risks related to stakeholder collaoration during requirements elicitation. Research on failed outsourced software development projects [2] highlights the importance of good requirements. Many projects fail ecause the client and client stakeholders too frequently do not grasp the importance of participation in requirements elicitation. The client project manager (with senior management support) can play an essential role in ensuring that stakeholders make themselves availale when required. Without serious high level management input, client stakeholders often feel disinclined to make time for systems analysts as they do not comprehend the importance of their participation; all too frequently there is nothing in it for them except the threat that the automated system may cause them to e made redundant [21]. Some of the risks in Tale 4 should e controlled y the vendor project manager; although anything associated with requirements elicitation should also involve the client project manager. Requirements engineering risks overlap with communication, configuration management, collaoration and cultural issues. Tale 4: Requirements engineering risks # Risks Study Support 1 National, organizational, and cultural differences of participants affect [S6] [S15] 7 4 their collaoration in requirements engineering. 2 Requirements change management [S6] 6 issues. 3 Lack of a common understanding of requirements. [S6] No suitale tools or methodologies [S15] 5 for requirements elicitation. 5 Requirements communication issues lead to misunderstandings. [S6] [S15] Lack of collaoration for [S6] 3 requirements engineering. 7 Lack of common goals. [S6] 3 9 Stakeholders located in different [S15] 3 time zones. 10 Lack of a well defined requirements engineering process. [S15] Software Development Process Two studies furnished risks related to the software development process; one was an SLR concerned specifically with agile methods [S8]. We identified seven risks and Tale 5 provides their details. Even though most of the risks were extracted from a SLR specifically focussing on agile methods all the risks, except for risk #2, could apply to any software development method. Most risks are related to communication in the vendor environment. We earlier extracted 64 risk mitigation advice items related to GSD and agile methods [20], although only 24 of the items had empirical support from three or more primary studies. As we only found seven evidence-ased agile method risks we are not convinced that agile practices are actually used in many (commercial) GSD projects. We note the comment in [10] that the role of on-site customer seems to e unsustainale for long periods and that it is difficult to introduce agile methods into large and complex projects. Tale 5: Software development process risks 1 Asymmetry in processes, policies and 14 standards. 2 Application of agile practices 10 3 Lack of synchronous communication in [S8] 9 agile development. 4 Collaoration difficulties in agile [S8] 6 development. 5 Poor communication andwidth for [S8] 6 agile development. 6 Lack of tool support for agile [S8] 6 development. 7 Prolems caused y large teams for agile development. [S8] 5

7 3.2.3 Architectural Design We found two overlapping risks from [S5, S18] related to poor architectural design (shown in Tale 6 elow), that result in increased communication, coordination and integration prolems. Both risks should e controlled y the vendor project manager. Tale 6: Architectural design risks 1 Not dividing the work into well-defined modules and carrying out progressive integration 8 2 High task coupling etween task segments increases the need for inter-site communication, coordination, and integration, and it can lead to lower level of performance as well as increase the numer of failures Configuration management Five configuration management risks were identified originating from three SLR studies [S5, S7, S18]. Tale 7 provides details of the risks and their empirical support. Risk #1 as it stands, is somewhat vague and would e difficult to manage without more detail. The configuration management risks listed have some overlaps with communication and project management. These risks should e controlled y the vendor project manager, and/or the vendor configuration manager. Tale 7: Configuration management risks 1 Group awareness prolems. [S7] 6 2 Lack of deployment of configuration 4 management system 3 Prolems concerning configuration 4 management tool differences, slow and unreliale sites, lacking awareness of product changes, and ug fixes etween sites. 4 Dependency, delay and increased time [S7] 4 required to complete maintenance requests. 5 Working in different SCM environments leads to maintenance requests eing handled at various levels in the project. [S7] Human resources We divided the human resources risks into three su-themes: culture and social integration, training, and communication and collaoration. In all, we identified 35 risks; 28% of all risks identified in this study. Each of the su-themes is discussed further elow Culture and social integration Five studies provided ten risks related to culture and social integration [S5, S6, S16, S17, S18]. Tale 8 lists details of the risks and their empirical evidence. Prolems caused y cultural diversity, difficulties with different languages, and mutual trust are the risks supported y the most evidence. Empirical support for the risks is reasonale in most cases. These risks are under the control of the vendor project manager in most cases, except for risks #8 and #10 and there are overlaps with communication and collaoration risks. Tale 8: Culture and social integration risks 1 Cultural diversity etween 36 development sites or teams. [S6] Language differences etween sites. 3 Mutual trust is important ut hard to otain. 4 Lack of mutual (shared) understanding among team memers in different development sites. 5 Differences in work culture may render cooperation difficulties, i.e., when sites are different in terms of team ehaviour, alancing of collectivism and individualism, perception of authority and hierarchy, planning, punctuality, and organizational culture. This may lead to decreased conflict handling capailities and lower efficiency or even paralyse the project. 6 Cultural ias occurring when project participants consider their norms and values as universal and neglect to reflect on to what extent values, norms, and iases are founded in their own cultural ackground. Cultural ias may lead to erroneous decisions and insecurity aout other participants qualifications and it can have a devastating impact on communication and collaoration efforts. 7 Fear aout the future, of jos and roles, erodes trust. 8 Stakeholders are less likely to commit to the project organization and its task when cultural differences and lack of face-to-face interaction makes it difficult to estalish a clear project identity. 9 Goal distriution can lead to conflicts related to task interpretation, process principles, and prolem resolution approaches and can result in site wars and low performance. Goal distriution is more likely in GSD ecause of faulty transfer of information and focus on own site performance. 10 Lack of collaoration etween distriuted stakeholders due to differences in culture, language, distance and processes. [S16] [S16] [S6] 3

8 3.3.2 Training Two SLRs [S5, S17] provided three training risks related to practitioner education. Tale 9 furnishes details. All three risks are the responsiility of the vendor project manager although at times risk #1 may refer to knowledge transfer prolems etween the client and vendor. It will then need to e dealt with y oth client and vendor project managers. Tale 9: Training risks 1 Different knowledge levels or 11 knowledge transfer prolems. 2 Ensure team memers share equal knowledge of the domain Lack of training in communication and collaoration tools Communication and collaoration It is very difficult to differentiate etween risks related to communication and collaoration so we consider the two together here. We identified 22 risks from five SLR studies [S5, S6, S16, S17, S18]. There is some agreement etween the SLRs with all four identifying communication and collaoration risks resulting from geography and time zone differences. Tale 10 provides details of the ten communication and collaoration risks with the most empirical support. Communication and collaoration risks potentially have two dimensions. Firstly vendor-client interaction risks in which case the risks need to e managed y oth vendor and client. However, if the vendor sucontracts to other countries or has sites in different countries, the vendor must manage all vendor-sucontractor interaction risks. While the client project manager may not e ale to control these risks he/she must ensure that risks in this area are monitored. 3.4 Project management We divided the risks into two su-themes: planning, and control. We extracted 39 project management risks, i.e., 32% of the total numer of risks found overall. We now discuss the risks provided for each su-theme in turn Planning Three studies identified ten planning risks [S5, S17, S18] with supporting evidence; all are under the control of the vendor project manager except for risk #4, no alignment of stakeholders goals and interests, which is of concern to the client project manager. Although most of the planning risks are not under the direct control of the client manager the client must e concerned with a numer of these risks including #7, effort estimation. Tale 11 provides details of all the risks we identified. Planning risks have overlaps with communication and collaoration Coordination Two SLR studies [S5, S17] provide five risks related to coordination; this su-category is closely tied to control as well as communication and collaoration. There is a reasonale amount of evidence supporting the risks. Tale 12 lists all five coordination risks which are of concern to oth client and vendor, and their empirical support. Risk #1 is again rather vague. Tale 10: Communication and collaoration risks 1 Lack of effective communication; 46 communication issues [S6] Temporal and physical distriution increases complexity of planning and coordination activities, makes multisite virtual meetings hard to plan, causes unproductive waits, delays feedack, and complicates simple things like time referencing and time settings. 3 Limited possiility for informal communication due to dispersion of sites (i.e. a lack of spontaneous communication through social relationships). 4 Lack of synchronous communication or interaction (i.e. inaility to communicate in real time). 5 Lack of team cohesiveness (i.e. lack of a sense of teamness). 6 It may e difficult to estalish effective coordination mechanisms in GSD projects and overcome challenges such as little face-to-face interaction, prolematic task coupling, different time zones, local holidays, weak social networks, and unclear lines of communication. 7 Poor collaoration and communication infrastructure. 8 Limited face-to face meetings can negatively impact trust, decision quality, creativity, and general management. [S16] [S16] [S16] 7 9 Overall visiility 5 10 Being separated, interaction media 5 ecomes primary communication link etween sites, ut their properties or use may cause prolems such as jumled sequences of messages; mix-ups etween past, present, and future messages; and loss of contextual information sharing. Such prolems, may lead to confusion and misunderstandings among participants and lower morale Control Four studies address GSD control [S5, S16, S17, S18]. Twenty four risks were identified (20% of the risks we identified overall). Most risks relate to 1) socio-cultural distance and hence have overlaps with culture and social integration, and 2) geographic distance with overlaps related communication and coordination. Much of the risk should e controlled y the vendor project manager, however, these risks are of interest to the client project manager as well (especially risk #9), and need close monitoring y oth project managers throughout the project. Tale 13 lists the ten risks with the most supporting evidence. 4. LIMITATIONS We may have missed some studies and hence underestimated the extent of SLRs in GSD. However, as noted earlier, we checked the papers we found against those identified in earlier work. No 7 6

9 authors discovered any papers descriing SLRs in GSD that we did not find. Tale 11: Planning risks 1 Selection of inappropriate 12 information and communication 13 technology. 2 Lack of project planning 8 3 Inappropriate activities distriution. 6 4 No alignment of stakeholders' goals 6 and interests. 5 Poor identification of roles and 8 responsiilities 6 Lack of detailed planning 6 7 Inaccurate effort estimation. 5 8 Not tailoring organizational structure 5 to reduce delays in prolem resolution. 9 Temporal distriution increases the complexity of planning and coordination activities, and makes multisite virtual meetings hard to plan, causes feedack, and complicates simple things like time referencing, leads to unproductive waits, delays and time settings. 10 No creation of communication protocols Tale 12: Coordination risks Lack of coordination Limited collaoration time due to 7 little or no overlapping work hours etween sites. 3 Failure to plan appropriate activities 6 distriution making coordination difficult. 4 Coordination prolems ecause of 4 limited experience in working together as a team. 5 Lack of appropriate information flow throughout the project. [S17 4 As we extracted only risks with support from at least three primary studies we did exclude some risks. We may also have inadvertently omitted some risks that should have een included although to ensure that the data extracted was complete, we compared a sample of the first author s data extraction with that from the other authors. We found good agreement etween authors. Because we only extracted risks with support from at least three primary studies from the SLRs we will have omitted some minor additional support for the risks we did find. Categorisation of the risks into themes was difficult ecause many risks overlapped two or more themes. Different researchers would no dout come up with different themes and su-themes and may well have categorised the risks differently, even if they had they used the same categories. Many of the SLR papers had titles that included terms such as map, review or profile. The aim of most of the SLR studies was not to provide risk advice to industry, ut rather to present a summary of the literature to researchers. However, this does not invalidate the risks we identified, it simply made them more difficult to find. A few papers and/or studies directly focussed on risk, e.g., [S5, S12, S15, S17, S18] and this made the data extraction easier. We could not of course, sum the evidence provided y the different studies as the primary studies referred to y the SLRs in many cases overlapped; hence we would have een counting the same evidence twice. In addition, many of the SLR studies did not provide details of the evidence supporting the risks they identified; this made it difficult to e sure aout the strength of the evidence for a numer of risk items that were excluded. Tale 13: Control risks 1 Poor or inconsistent project 13 infrastructure. 2 Asymmetry in processes, policies, 13 and standards. 3 Prolems with tracking and control Prolems with people management, 10 conflict resolution, and staff 3 turnover. 5 No transparency and progress visiility of project status (to all sites involved in project). 6 Disorganized task allocation. 7 7 No process alignment in terms of 6 traditions, development methods, and emphasis on user involvement will often differentiate etween sites, possily resulting in incompatiility and conflicts. 8 Lack of scope and change 7 management. 4 9 No maintenance of stakeholders 5 participation and commitment to the project. 10 Spatial distriution complicates the project manager s aility to monitor participants and progress, increases travel udgets, limits face-to-face 5 interaction, and weakens social relations. The evidence we have included for each risk varies in quality oth etween risks, and within the evidence for a single risk. Even the SLRs that defined the quality of all the primary studies they included mostly did not associate the quality of the evidence with a specific risk. Just one SLR in [S5a], carefully defined the quality of the evidence for each risk they identified. For example, while Tale 2, risk #3 ( Lack of protection for intellectual property rights ), has supporting evidence from one SLR [S12] with 46 primary studies of undefined quality, it also has support from of five primary studies, one study is of high quality, and four are of average quality. Even when the SLR authors defined the quality of the primary studies they included they use different quality assessment frameworks. Thus it is impossile to assign a single quality value for the evidence of a risk without going ack to all the SLRs and redefining quality for each primary study (using the same quality assessment for all the primary studies) and then associating these with each risk identified. We consider it possily more important for a risk to have supporting evidence from multiple (independent) SLRs rather than support from a larger numer of primary studies listed in a single SLR. However, given 7 4

10 the lack of research in many of the areas this is open to deate. 5. DISCUSSION AND CONCLUSIONS Five studies [S5, S11, S12, S15, S17, S18] specifically extracted risks (arriers or challenges) from their primary studies; this certainly made data extraction, particularly from the studies that provided supporting evidence, much easier. However, only three studies [S5, S12, S17] consistently made clear the numer of primary studies that supported the risks they identified. Overall most of the risks we found had moderate levels of primary study support. Most SLR studies did not make clear who should oversee and control the risks they identified. Whilst many of the risks should oviously e monitored y the vendor project manager, most authors did not specify the recipient. Client risk management is usually ignored except for requirements risks, and risks related to GSD outsourcing rationale. Smite et al [S21a], note that most of the empirical findings... for GSD studies are ased on intra-organizational industrial collaoration etween two geographically distriuted sites. There is a clear lack of studies aout inter-organizational collaoration. Because most GSD SLR studies do not make clear what kind of usiness models underpin their primary studies the authors do not appear to consider who should e responsile for monitoring and controlling most risks. In many cases the research focus appears to e on just the software development, with an assumption that the developers are of different cultures located in different time zones. The primary studies, from which the SLRs derived their data, may sometimes have made their usiness model clear, ut it is not at all ovious in the SLRs. We elieve that project context is important and that different risks will have a different likelihood depending on the usiness model used. If a client in country A, outsources to a vendor in Country B with a single site, the vendor company may need to employ a cultural liaison officer. However, after requirements have een specified, there will e fewer risks related to control, culture, coordination, and communication etc.; developers will not e located at multiple sites, we will have no prolems with developer time zones, and developer cultural differences will e minimized. While higher level communication etween the client and vendor project managers will of course e required, the vendor project manager will not e managing developers with multiple cultures, at different sites. We elieve that more research, related to the actual usiness models used for GSD, is necessary. We need to know more aout how: 1) the usiness model used for GSD, and 2) the project context, impact the development itself and their effect on risk patterns. Interestingly, when we consider the GSD risk mitigation advice identified in our previous research [20] we found 440 suggestions. However, only 96 of the 440 items had empirical support from at least three primary studies, i.e., much advice is suggested, ut little empirical support is provided. In comparison empirical support for the GSD risks is marginally etter as our 123 risks had empirical support ased on three or more primary studies (even though the quality of the evidence is unclear). When projects go awry there can e a disinclination to investigate the real reasons for this. It is less emarrassing for a company to ury the project and move on, particularly if the mistakes were overarching high level management errors, e.g., not providing sufficient high level support so that the stakeholders did not feel inclined to cooperate [21]. Few failed projects result in litigation; of those that do, most are settled out of court and gag orders imposed making it difficult to find out what actually happened [2]. Hence, we keep on making the same mistakes. The answers to our research questions are: RQ1 What are the most common GSD project risks identified in the SLRs? We classified the 123 risks we identified under four major themes, 1) GSD outsourcing rationale, 2) software development process, 3) human resources, and 4) project management. Thirty nine (34%) of the risks identified were related to project management and 34 (28%) were categorised as human resources. The su-theme with the most risks was control where we found 24 (20%) risks; this is followed y communication and collaoration with 22 risks (18%). The single risk with the most supporting evidence was a high level vendor selection strategy risk, language and cultural arriers which had support from 55 primary studies although the quality of the primary studies was not defined. RQ2 What empirical evidence is there for these risks? The risks supported y the most empirical evidence (if we consider the numer of SLRs identifying the risk), had support from four SLR studies. They were: 1) language differences etween sites a culture and social integration risk with evidence from 18, 14, 14, and 11 primary studies, 2) temporal and physical distriution increases the complexity of planning and coordination activities, makes multisite virtual meetings hard to plan, causes unproductive waits, delays feedack, and complicates simple things like time referencing and time settings, a coordination and communication risk with evidence from 20, 14, 13, and 5 primary studies. Overall, empirical support for most themes is moderate ut the quality of the evidence is undefined in most cases. Empirical support for risks related to GSD outsourcing rationale is highest in terms of primary study support ut, ecause this is not a common research area, we are lacking other SLR support for most of the risks identified. Just ecause we have not included a particular risk does not mean that it does not exist; it simply means that we were not ale to find enough empirical evidence to include it. RQ3 Who is responsile for controlling the risks? Most of the SLR studies did not make clear who should control the risks. In many of the SLRs there is an implicit focus on software development and the vendor and developers. Risks specific to the client are limited although those risks tend to have greater empirical support. It is interesting to note that [S12], which provided the client risks with the greatest empirical support, was actually written with a vendor focus and was intended to give vendors advice so that they could make themselves more attractive to clients. If the usiness model and project context is unclear, as it is in most of our studies, it can e difficult to determine who should e controlling the risks. In conclusion, most of the SLRs we investigated provided a summary of the literature and mappings for researchers. Although some studies identified industry risks, and evidence for those risks, most did not. We elieve that the evidence furnished, for the risks identified y most of our SLRs, though mostly undefined as regards quality, provides a good start, ut more research into GSD risks and the evidence supporting the risks is required. However,

11 the research must include a consideration of the project context and usiness model, as well as the quality of the supporting evidence. We intend to continue research in this area and are particularly concerned with risks from the client perspective. We are also interested in investigating real life variations of the client-vendor usiness model. We will then e ale to identify risks, risk mitigation practices, and the strength of the supporting evidence as it relates to the different usiness models employed. 6. ACKNOWLEDGEMENT Verner s work is supported y the European Union Seventh Research Framework. She is a Marie Curie Fellow at Keele University. 7. REFERENCES [1] Adullah L. M., Verner J.M., (2009) Outsourced Strategic IT Systems Development Risk IEEE RCIS, Fes, Morocco, April, pp [2] Adullah L. M., Verner J. M. Analysis and Application of an Outsourcing Risk Framework accepted y Journal of Systems and Software, in press [3] Beulen E., van Fenema P., Currie W., (2005) From Application Outsourcing to Infrastructure Management: Extending the Offshore Outsourcing Service Portfolio European Management Journal Vol. 23, No. 2, pp [4] Bondi A. B., Ross J. P., (2009) Experience with Training a Remotely Located Performance Test Team in a Quasi-agile Gloal Environment, ICGSE, pp [5] Casey V. (2009) Leveraging or Exploiting Cultural Differences, ICGSE, Limerick, Ireland, pp [6] Cruzes D. S., Dyå T, (2011), Research synthesis in software engineering: A tertiary study, Information and Software Technology 53, no. 5: [7] da Silva F. Q. B., Santos A. L. M, Soares S., Franca A. C. C., Monteiro C. V. F., and Maciel F. F., (2011) Six Years of systematic literature reviews in software engineering: An updated tertiary study, Information and Software Technology, availale online April [8] da Silva F. Q. B., Santos A. L. M, Soares S., Franca A. C. C., Monteiro C. V. F., (2010) A critical appraisal of systematic reviews in software engineering from the perspective of the research questions asked in the reviews ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. [9] da Silva FQB, Prikladnicki R., Franca ACC., Costa C., Rocha R (2011) Research and practice of distriuted software development project management: A systematic mapping study sumitted to Information and Software Technology. [10] Dyå T., and Dyngsør T., (2008) Empirical studies of agile software development: A systematic review Information and Software Technology, Vol., 50, pp [11] Fariek M., van den Brand M., Brinkkemper S., Harmsen, F., Helms, R. (2008) Reasons for Success and Failure in Offshore Software Development Projects, European Conference on Information Systems, Ireland, pp [12] Howitt D., Cramer Duncan (2011), Introduction to Research Methods, 3/E,Prentice Hall [13] Khan N., Fitzgerald G., (2004) Dimensions of Offshore Outsourcing Business Models Journal of Information Technology Case and Application, Vol 6, No 3 pp [14] Kitchenham B., Pretorius R., Budgen D., Brereton O. P., Turner M., Niazi M., Linkman S., (2010) Systematic literature reviews in software engineering - A tertiary study, Information and Software Technology, Vol. 52, no. 8, pp [15] Kitchenham B., Brereton O. P., Budgen D., Turner M., Bailey J., Linkman S., (2009) Systematic Literature reviews in software engineering a systematic literature review, Information and Software Technology, 51(1), pp [16] Kitchenham B., Charters S. (2007) Guidelines for performing Systematic Literature Reviews in Software Engineering, Keele University and Durham University Joint Report, EBSE [17] McCue A., (2005) Outsourcing flops lamed on tunnel vision, Silicon.com, Pulished on ZDNet News, June 22. [18] Verner J. M., and Adullah L. M., Exploratory Case Study Research: Outsourced Project Failure, accepted y Information and Software Technology, in press. [19] Verner J. M., Brereton O. P., Kitchenham B. A., Turner M., Niazi M., Systematic Literature Reviews in Gloal Software Development: A Tertiary Study sumitted to EASE [20] Verner J. M., Brereton O. P., Kitchenham B. A., Turner M., Niazi M., Risk Mitigation Advice for Gloal Software Development Extracted from Systematic Literature Reviews sumitted to ICGSE [21] Verner J. M., (2011) More Carrot and Less Stick in The dark side of software engineering: the ethics and realities of suversion, lying, espionage, and other nefarious activities, Rost Johann, and Glass Roert L., IEEE Press.