Comparative analysis of risk assessment methods in project IT

Jacek Winiarski * Comparative analysis of risk assessment methods in project IT Introduction IT projects implementation as described in software development methodologies is usually distributed into several stages. It begins with requirements specifications and ends with operation and maintenance of the implemented product [Chapman, Ward, 1997, p. 168]. Throughout the course of software development cycle the events affecting its course may take place. In practice, they are described by probability of their occurrence and the potential scope of damage they may create in the implemented project. The product of these quantities is defined as the risk measure [Frączkowski, 2003, p. 128]. It is used for forecasting negative impacts on the course of the implemented individual activities within the task or the entire project. Project management is the approach oriented at the accomplishment of the set objectives within the assumed time and budget [Kaczmarek, 2005, p. 65]. It is an art of maintaining the project failure risk at a possible lowest level throughout the entire project cycle. Risk management is one of many elements of the project management process. In principle, it is distributed into the following stages [Chong, Brown, 2001, p. 45]: identification and distribution of risk sources, identification of exposed project tasks, risk assessment, planning of response to the risk and risk and risk monitoring in the course of project implementation. 1. Failures in implementation of IT projects There are scores of institutions acting in the field of risk analysis in IT projects. These are mainly academic centres but there are also many organizations associating experts with know-how. One of the most often cited publications are reports called The CHAOS Chronicles, published regularly by The Standish Group International, an American institution dealing with monitoring of IT projects implemented in the USA. The * PhD, Department of Electronical Business, Faculty of Economic, University of Gdansk, email: ekojw@ug.gda.pl

180 Jacek Winiarski analysis of documents published on the Web fosters detailed specification of the results of IT projects monitoring in the form of statistics. According to the data publish on The Standish Group International web pages, approximately 2/3 of IT projects fail to end in a full success. Throughout their implementation (according to the statistics in every second project), there are deviations from time schedule or budget assumptions, failure to develop all the designed functions of the programme or abandonment of the entire project. The main objective of the risk assessment at the planning and implementation stages of IT projects is to reduce the number of projects which are likely to exceed the scheduled resources or be eventually terminated [Szyjewski, 2004, p. 125]. The risk assessment techniques presented in the paper compel the IT Project Managers to continuously update the information on potential hazards. One of the frequent mistakes made by IT Project Managers is to assume that the risk of successful accomplishment of the project remains the same throughout the entire project of its implementation [Pańkowska, 2001, p. 84]. It is to the contrary. The risk is incessantly variable, thus constant monitoring as per a carefully designed schedule is of essence. As shown in practice, the meticulousness and thoroughness of the risk assessment contribute considerably to the end success of the project. 2. Characteristics of the selected IT project The study was based on a project of system development aimed at servicing the Lending Library of Higher Education School. The undertaking in question was designed as a set of 349 tasks to be carried out by a group of 5 IT Specialists within 64 days. The accomplished and implemented software is to facilitate students access to the book catalogue by means of web browsers. The detailed comparative analyses were based on eight tasks from the entire project exposed to the risk of failure, taking into account the scheduled time, scope and budget (Table 1). Table 1. Selected risk-prone projects tasks Probability of noncompliance with the Implementation ID Task name: Duration costs [PLN] scheduled resources Requirements 40 14 days 0,4 4500 Specification

Comparative analysis of risk assessment methods in project IT 181 ID Task name: Duration Probability of noncompliance with the scheduled resources Implementation costs [PLN] 43 Definition of classes 3 days 0,3 800 52 Code development 30 days 0,5 10 300 67 Code testing 5 days 0,6 4 400 104 Code adjustment 10 days 0,5 7 300 109 Software installation 30 minutes 0,3 500 110 Preparation of user 10 days 0,3 5 500 documentation 111 User training 2 days 0,2 4 800 Source: Own elaboration. 3. Examples of application of risk assessment techniques in the selected IT Project case study The popular risk assessment methods applied to IT Projects comprise: 2x2 matrix method, probability and effects matrix method, Heeg s method, failure Analysis of failure effects. Less popular techniques encompass: sensitivity analysis, spot techniques, probability analysis (using e.g., Monte Carlo simulation), flow diagrams (e.g., critical path analysis) or decision tree analysis (e.g., PERT, VERT, GERT analyses). 3.1. 2x2 Matrix Method One of the primary tools fostering risk management process in project works is the so-called 2x2 matrix. It defines risk as the probability function for the occurrence of harmful event and the effect thereof [Pritchard, 2002, p. 122]. Table 2. Risk assessment using 2x2 Matrix method Impact Small Probability Quarter 1 Large 40 Quarter 3 Small 40 Source: Own elaboration. Large Quarter 2 Quarter 4 40

182 Jacek Winiarski 2x2 Matrix should be completed following the previous identification of potential hazards and preparation of the list thereof. Next, they are filled (depending on the probability of occurrence and the scope of potential loss they may cause) in their respective matrix quarters. The parts of the table specify [Chong, Brown, p. 65]: Quarter 1 represents the area of hazards of high probability of occurrence and inconsiderable negative effects for the project, Quarter 2 represents the area of hazards of high probability of occurrence and simultaneously substantial negative effects for the project implementation process, Quarter 3 represents the area of hazards of small probability of occurrence and inconsiderable negative effects for the project implementation process. This is the least risk-prone area. Quarter 4 represents the area of hazards of small probability of occurrence and considerable negative effects for the project, If the 2x2 table is not completed with hazards but with the tasks exposed to the risk of non-compliance with the scheduled resources, then after the analysis of the possibility of reallocation thereof along the directions recommended by the technique in question, precautions will be developed with the view of diminishing the risk for the accomplishment of particular tasks and thus the entire project. Task 40 requirements specification was placed in the 2nd quarter by the project risk manager. Next, the Project Manger suggested that Task 40 should be performed in compliance with the forms adopted in PRINCE2 method. The suggestion caused the task to be moved to the first quarter 40 (prim variant). Next, another Project Manager decided to carry out additional audit of the prepared requirements specification by an external expert. This decision caused the task 40 to be moved to the third quarter 40 (bis variant). All project tasks may be examined individually in the same manner. 3.2. Method of Matrix of probability and effects Another more complex tool is the so-called matrix of probability and effects. It is an elaboration of the 2x2 matrix concept. It is more detailed as far as the probability of estimates and the effects of hazard occurrence are concerned. Like previously, particular hazards are filled in the respective fields in the extended version of the table. After all hazards identified in the project have been filled, preventive measures are designed to eliminate the risk sources allocated in the second quarter [Chong, Brown,

Comparative analysis of risk assessment methods in project IT 183 2001, p. 64]. The ultimate objective of the risk management process is the reallocation of the most probable and most perilous hazards tops other areas of the matrix. The measures provide the basis for the risk management in the project planning process. The table used for the method in question may upon its modification serve for calculation of the measurable total scope of project risk. The aforesaid modification provides for allocation to particular cells of hazard weights representing the scope of probability of occurrence of a given hazard and the potential effects thereof (Table 3). Table 3. Risk Assessment Method of Matrix of Probability and Effects Effects Probability Minimal Minimal Minimal Minimal Minimal Extremely high 43 40 67 (0.8 1) (2,0) (3,5) (7,0) (8,0) (9,0) High (0.6 0.8) (1,5) Average 43 (0.4 0.6) (1,2) Low (0.2 0.4) 40, 104 (1,0) Extremely low 110,109, 111 (0 0.2) (0,5) Source: Own elaboration. 52 (2,0) 67 (1,8) 52 (5,0) (7,0) (8,0) 104 (4,0) (5,0) (7,0) 110, 109 (1,5) (3,0) (4,0) (5,0) 111 (1,0) (1,5) (3,0) (4,0) Each identified hazard, which may occur during performance of the project, shall be allocated to particular cells of the table. As a next step, the weight of a given cell should be multiplied by the number of hazards allocated thereto and sum up all the achieved numbers. The sum shall be divided by the total number of hazards in the analysed project. The end result is a measurable quantity of the total project risk. With respect to the project, the entire introductory risk of the project was 3.31. After the preventive measures have been recommended and applied, the total project-related risk was reduced to 1. The presented example, like 2x2 matrix method was based on the study of risk-prone tasks and not the tasks themselves.

184 Jacek Winiarski 3.3. Heeg s method The method recommended by Heeg in [Chong, Brown, 2001, p. 167] comprises three stages. These include: risk identification, risk assessment, selection. The presented method is based on the identification of projectspecific hazards. According to the author of the method, the risk sources may be the identified in several ways. One of the commonly used techniques is the analysis of task packages described by means of e.g., Work Breakdown Structure - WBS. It may be presented in the form of a Table 4. Table 4. Risk assessment using Heeg s method ID 40 43 52 Task name: Requirements Specification Definition of classes Code development 67 Code testing Potential risks Omission of required functiona lities Incomple te classes Syntactic errors Data transfer errors Semantic errors Incompat ibility Probability of occurrence Costs of neutralization [PLN] Probable costs [PLN] 0,4 8000 3200 0,2 1000 200 0,01 500 5 0,3 4000 1200 104 Code adjustment 0,04 300 12 109 Software installation 0,1 2000 200 Preparation of 110 user Deadline 0,01 3000 30 documentation 111 User training Deadline 0,01 1500 15 Source: Own elaboration.

Comparative analysis of risk assessment methods in project IT 185 Following identification of the risk-prone tasks and detailed specification of potential risk sources, which may affect the implementation process, it is necessary to determine the probability of occurrence of detailed hazards (Table 4, Column 4). Next, the planned costs related to elimination of potential losses are to be estimated (Table 4, Column 5). The last column of Table 4 comprises probable costs i.e. product of probability and foreseen costs of loss compensation (Table 4, Columns 4 and 5). Thus computed quantities of probable costs must be sorted in descending order and the group of tasks for which the sum of quantities in Column 4 Table 4 will be 75% of the total probable costs of the analysed project [Chong, Brown, 2001, p. 98] must be specified (starting from the highest values). In the example in question, these include tasks 40 and 67 (amounting to 90.5% of the total costs). Thus identified set of task groups shall be given a particular attention from the Project Managers. The possibility to undertake protective measures for these groups must be taken into consideration. The sum of total probable costs shall be 4862 PLN. 3.4. Failure Mode Effect Analysis Failure mode effect analysis was proposed by Maylor and described in [Chapman, Ward, 1997, p. 87]. This method analyses three parameters describing all tasks within the project. Each of these parameters must be expresses as a number on a scale from 0 to 10. The author adopts one point scale for all parameters. The requested quantities include: meaning of failure of implementation of a given task (failure), probability failure oversight, probability of failure occurrence during performance of a particular task. Each of the parameters must be examined individually. The objective of the presented analysis is to calculate a given total risk task constituting a function dependant on the aforementioned parameters. The risk is calculated on the basis of the following dependence: Risk = failure significance * probability of failure omission * probability of failure occurrence

186 Jacek Winiarski The higher the risk values, the more serious hazard is related to a particular task. With respect to activities exposed to the highest risk, additional measures alleviating potential losses should be proposed. Table 5. Risk Assessment Through Failure Effects Analysis ID Task name: Failure significance Probability of failure oversight Probability of failure occurrence 40 Requirements Specification 8 3 4 96 43 Definition of classes 3 2 3 18 52 Code development 7 1 5 35 67 Code testing 8 2 6 96 104 Code adjustment 7 2 5 70 109 Software installation 5 1 3 15 110 Preparation of user 4 7 3 83 documentation 111 User training 5 1 2 10 Source: Own elaboration. Risk For each task examined in the project by Failure Mode Effect Analysis, two additional parameters must be provided. These include: failure significance and failure oversight probability. Having performed the calculations illustrated in Table 5, one may discern that the highest risk pertains to tasks 40 and 67, while the least to the tasks no. 109 and 18. The total project risk is the sum of values in the last column, which is 423. 4. Comparative analysis of applications of risk measurement methods in IT projects Applications of the results obtained by means of 2x2 Matrix Method are not vast. This method is suitable for presentation of risk mitigation issues, since it clearly illustrates the required trends of preventive measures. IT project risk assessment based on this method may occur vague, general and eventually not yielding satisfactory results for the Managers. The concept of 2x2 matrix is focusing on risks, not the risk-

Comparative analysis of risk assessment methods in project IT 187 prone tasks, which definitely affects the profile of the analyses being carried out. 2x2 matrix does not allow risk quantification, with respect either to a part or the entire project. This technique is an easy-to-use tool for risk assessment in small projects. Its application supports the strategy of compensating potential effects of identified risks. If 2x2 matrix is completed with risk-prone tasks instead of identified risks, then after the main assumptions of the method have been applied, the effect of its use shall consist in development of preventive measures plan targeted at reduction of risks for individual tasks, hence for the entire project. Thus applied 2x2 matrix method will facilitate the assessment of the proposed preventive measures (after twice risk quantity measurement, before and after the preventive measures have been taken). Those who wish to use this technique shall be good experts in risk management, as this technique is based on the intuition, which determinant for the usability of the obtained results. Probability and effects matrix is the extended version of 2x2 matrix method. It has two advantages, which differentiate it significantly from the original pattern. One advantage is the fact that it fosters calculation of risk for individual tasks, task groups or the entire project, before and after the preventive measures have been taken. The other advantage is the clarity of results, not only for small but also for medium-sized IT projects. Heeg s method is the first from among the presented techniques, which assigns individual risks to the planned project tasks. Each of the tasks may be assigned more than one risk. This technique requires specification of the probability of occurrence of all identified risks. It is an interesting parameter, since within the framework of the risk definition another quantity is searched for, which is the probability of task nonperformance [Knight, 1934, p. 120]. In Heeg s method it is indispensable to specify potential costs of the reduction of effects of the identified risk occurrence. In practice, both quantities are identified on the basis of experience and intuition of the researchers. The method facilitates a detailed identification and analysis of risk sources, which may occur during the implementation of the project. By means of these method, the sources may be easily identified and assigned to particular tasks. This possibility is an essential advantage of the method in question. Unfortunately, this method also employs heuristic quantities in final risk assessment.

188 Jacek Winiarski In failure mode effects analysis it is necessary to specify further parameters. These parameters are not required in any other risk assessment techniques. It is indispensable to determine: failure significance, failure oversight probability as well as failure occurrence probability. The last parameter is identical to the scope of risk of a nonperformed task taking into consideration scheduled time resources, scope and budget. The person managing the risk must express all these values on a scale from 0 to 10. Failure mode effects analysis does not employ mathematical tools facilitating objectivity of data use for calculations. Like the aforementioned methods, the values used are based on the intuition. It will prove, however, in comparative analyses. From among the presented methods, this one allows the largest number of details to be used in the study. It occurs that Managers value the possibility of taking into account the probability of failure oversight. Failure mode effects analysis may be easily used for the risk assessment in large and middle-sized IT projects. Table 6. Comparative applications of risk assessment applications IT projects Required Results resources Method name Costs Duration of implementation Easiness of application Time involvement 2x2 Matrix Method l s e s l l Method of Matrix of Probability and Effects l s e m m m Heeg s Method l s m h m h Failure Mode Effect Analysis l s d m h h Key: low (l), medium (m), high (h), short (s), easy (e), difficult (d). Source: Own elaboration. Precision Usability Conclusion The paper depicted four methods of risk assessment methods applied in IT projects implementation. The same part of the IT project was examined through a comparative analysis. The obtained results foster concept that matrix techniques focus predominantly on the analysis of identified risks. They only indirectly examine the project tasks, for which it is probable, not comply with the scheduled time resources, scope or

Comparative analysis of risk assessment methods in project IT 189 budget. As a consequence, although these methods mark out the directions of preventive measures aimed at reduction of the risk, they seem not to be useful for large or complex IT projects. They are suitable for rough analyses in small projects. Further two methods foster a more detailed risk assessment. Based on the identified potential risks assigned to particular project tasks, Heeg s Method specifies probable costs, the company will have to incur in case of an anticipated risk. This project does not foster total risk calculation. This technique is suitable for comparative analyses for several variants of implementation of the same task, while it is not appropriate for the entire IT projects. The last presented technique Failure Mode Effect Analysis introduces two additional (crucial) parameters: failure significance and failure oversight probability. Thanks to these parameters, the method enables calculation of risk for implementation of all individual tasks as well as the entire project. This technique is commonly used in IT project risk management as the calculations involving two new parameters are very useful. The Project Manager s ultimate decision on the choice of the method for risk assessment in planning and implementation of an IT project will depends first and foremost on the specificity of project requirements (scope and innovativeness), funds and selected implementation methods. References 1. Chapman Ch., Ward S. (1997), Project risk management processes, techniques and insights, J. Wiley & Sons, Chichester. 2. Chong Y.Y., Brown M.E. (2001), Zarządzanie ryzykiem projektu, Oficyna Ekonomiczna, Dom wydawniczy ABC, Kraków. 3. Frączkowski K. (2003), Zarządzanie projektem informatycznym, Oficyna Wydawnicza Politechniki Wrocławskiej, Wrocław. 4. Kaczmarek T.T. (2005), Ryzyko i zarządzanie ryzykiem. Ujęcie interdyscyplinarne, Difin, Warszawa. 5. Knight F. (1933), Risk, uncertainty and profit, London. 6. Pańkowska M. (2001), Zarządzanie zasobami informatycznymi, Difin, Warszawa. 7. Pritchard C.L. (2002), Zarządzanie ryzykiem w projektach. Teoria i praktyka, WIG-PRESS, Warszawa.

190 Jacek Winiarski 8. Stabryła A. (2006), Zarządzanie projektami ekonomicznymi i organizacyjnymi, Wydawnictwo Naukowe PWN, Warszawa. 9. Szyjewski Z. (2004), Metodyki zarządzania projektami informatycznymi, Wydawnictwo PLACET, Warszawa. 10. Winiarski J. (2007), Analiza metod zarządzania ryzykiem w pracach projektowych z dziedziny informatyki, Pieniądze i Więź, Nr 2 (35), Gdańsk. Comparative analysis of risk assessment methods in project IT (Summary) The paper is targeted at the comparative analysis of practical applications of risk assessment methods of projects in IT industry. The study was based on an IT Project comprising 349 activities, wherein 8 were selected which were the most likely not to comply with the scheduled time, scope and budget resources. Next, four techniques were described and apply to provide the assessment of the same part of the project: 2x2 Matrix Method, Probability and Effects Matrix Method, Heeg s Method, Failure Mode Effects Analysis. The obtained results were discussed and on the basis thereof, conclusion were formulated to select the method depending on the specificity of an IT Project. Keywords IT projects, risk management, risk assessment techniques.