Assessment of IT Governance - A Prioritization of Cobit -

Paper #151 Assessment of IT Governance - A Prioritization of Cobit - Mårten Simonsson and Pontus Johnson KTH, Royal Institute of Technology Osquldas väg 12, 7 tr, S-100 44 Stockholm, Sweden ms101@ics.kth.se, pj101@ics.kth.se Abstract A shared view on the definition of IT governance is lacking and practitioners do not use present IT governance frameworks to support their decision-making. A commonly agreed upon definition of IT governance would be very useful and would serve the development and refinement of IT governance frameworks and assessment methodologies. This article presents an Architecture Theory Diagram, ATD, and a framework for defining IT governance based on an extensive literature study. IT governance is the preparation for, making of and implementation of IT-related decisions regarding goals, processes, people and technology on a tactical or strategic level. The framework for defining IT governance is eployed to compare how IT governance is defined in literature, and within a group of IT governance experts. Cobit is the most well-known framework for IT governance and it is frequently used by practitioners. While comparing Cobit s definition of IT governance to the previously identified concerns of literature and practitioners, it showed that Cobit does support most needs, but lacks in providing information on how decisionmaking structures should be implemented. Background to Research IT governance is a topic that has been increasingly discussed since the mid nineties. The topic has inherited much from the discipline of corporate governance, but has developed into a discipline of its own rights. However, a shared view on important concerns and how they should be handled is missing within the field. The definitions of IT governance are broad and ambiguous which in turn implicate difficult and inaccurate assessments. Most authors agree on IT governance as a top management concern of controlling IT s strategic impact, and the value delivered to the business c.f. (Weill 2004, ITGI 2005, De Haes 2005, Ribbers 2002). But whether the core of IT governance is a set of structures, processes and relational mechanisms (De Haes 2005), bundled performance metrics to aid IT process monitoring (ITGI 2005) or cascaded Balanced Scorecards (Kaplan 1996, Van Grembergen 2004) is not agreed upon. There is also a gap between what is stated in literature and the opinions of practitioners: The theories developed in literature are not frequently used by consultants or CIOs (Cumps 2006, Dahlberg 2006). Control Objectives for Information and related Technology, Cobit, is the most renowned framework for support of IT governance concerns (ITGI 2005, Guldentops 2004), but does it really address the concerns considered important in literature and by practitioners? Purpose. The purpose of this paper is to illustrate the differences in priority of IT governance concerns between literature, practitioners, and Cobit. The research is conducted within the -1-

Enterprise Architecture Research Program (EARP) at the Royal Institute of Technology (KTH) in Stockholm, Sweden. Within EARP, Architecture Theory Diagrams, ATD, are used as an approach to analyse various fields within the enterprise architecture domain (Johansson 2005). The Problem of Defining IT Governance The field of IT governance is defined differently in the numerous articles and books written on the topic. The lack of consensus is clear. Some of the prevalent definitions are: IT governance is the responsibility of executives and the board of directors, and consists of the leadership, organisational structures and processes that ensure that the enterprise s IT sustains and extends the organisation s strategies and objectives (ITGI 2005) IT governance: Specifying the decision rights and accountability framework to encourage desirable behaviour in the use of IT. (Weill & Ross2004) IT governance is the strategic alignment of IT with the business such that maximum business value is achieved though the development and maintenance of effective IT control and accountability, performance management, and risk management. (Webb et al 2006) The fact that the discipline lacks a uniform definition has previously been addressed by (Webb et al 2006), who also present a definition of their own, see last bullet above. Webb s definition is derived from literature, but is based on a fairly small amount of articles, and the methodology used to create the methodology remains unclear. During the past decades, several frameworks that support implementation of IT governance have been created. Cobit is a framework based on best practice, focusing on the processes of the IT organization and how their performance can be assessed and monitored (ITGI 2005). Although the problem has been partly addressed in the latest version of Cobit, little support is given on the arrangement of decision rights within the enterprise. The IT Infrastructure Library (Itil) provides useful best practice in the field of service management and service delivery, but does not cover the strategic impact of IT and the relation between IT and the business (OGC 2002). The information security standard ISO/IEC 17799 is often mentioned together with IT governance, see e.g. (Warland 2005, von Solms 2004). The common denominator here is IT risk management, separation of concerns and segregation of duties. Finally, (Weill & Ross 2004) has developed a framework for IT governance evaluation based on just a few questions. The framework has been used to map top-level assignment for IT responsibilities in 250 enterprises worldwide but cannot be used for in-depth assessments of IT governance. An attempt to overview IT governance frameworks, standards, and legislations can be found in (Holm Larsen 2006). As shown, there are several different frameworks and definitions of IT governance, but do practitioners within the field agree with them and strictly follow them in their quest for IT governance improvement? A survey conducted by Information Systems Audit and Control Association (ISACA) Sweden Chapter in late 2004 suggests that this might not be the case (ISACA Sweden Chapter 2004). Even though a grand part of the ISACA members responding the survey claimed knowing Cobit, Itil and ISO/IEC 17799 on a superficial level, few actually used the frameworks to support their work. This has been stated previously, c.f. (Cumps 2006, Dahlberg 2006), but the different priorities of IT governance concerns between literature, practitioners, and best practice frameworks have not been fully investigated. In order to detail -2-

distinct priorities within IT governance, a framework onto which both practitioners and theoreticians could map their concerns would be useful. Such framework should span the entire field of IT governance, and could be used to prioritize different concerns of e.g. literature and practitioners. A Framework for Defining IT Governance The first step towards creating a definition of IT governance was to gather information previously written on the topic. 102 sources of information on IT governance were identified when conducting an extensive literature search. The forums in which the articles have been published include the MIS Quarterly, Information Systems Control Journal, Information Systems Research, International Journal of Information Management, International Journal of Accounting Fig. 1 The Architecture Theory Diagram for IT governance. Information Systems, and the Hawaii International Conference on System Sciences, see e.g. (Hamaker 2004, Trites 2002, Ridley 2004, Sambamurthy 2000). 60 of the sources were selected randomly and analysed in order to find common denominators to base the definition upon. This resulted in the creation of a framework for defining IT governance, and is described more thoroughly in (Simonsson 2006a, Simonsson 2006b). An ATD was created in order to describe the content of different statements identified in literature. ATDs and their use are described in e.g. (Johnson 2004). A corresponding framework for defining IT governance was also developed, c.f. Fig. 1. and Fig. 2. Based on the analysis of 60 articles, it was concluded that IT governance is a matter of decision-making. Three Fig. 2. The framework for defining IT governance. dimensions are used for the framework for defining IT governance, namely the domain, phases and scope in which IT decisions are made and carried out. In the following subsections, each dimension is explained. Domain. The domain denotes what the decisions should consider. It comprises four dimensional units: Goals, processes, people and technology. Goals include strategy-related decisions, development and refinement of IT policies and guidelines, and control objectives used for performance assessments. Processes include the implementation and management of IT processes, e.g. acquisition, service level management, and incident management. People includes the relational architecture within the organization, and the roles and responsibilities of different stakeholders. Finally, IT governance is of course about managing the technology itself. The -3-

dimensional unit Technology represents the physical assets that the decisions consider, such as the actual hardware, software and facilities. The practitioners prioritized the dimensional units as they are presented below. Decisions on Goals. The development and refinement of an IT strategy, policies, guidelines, and control objectives to monitor whether the goals are achieved. Examples of issues to decide upon: Policies guiding IT use IT setting the direction of IT and its alignment with corporate strategy Control Objectives used to monitor the performance of IT processes Road maps describing how to reach the goals set in the IT strategy Decisions on Processes. The implementation and management of IT processes and related activities and procedures. Examples of issues to decide upon: Activities needed to perform IT related tasks Processes with standardized workflows for e.g. acquisition, service level management, and incident management Procedures describing how to accomplish IT related tasks Decisions on People. The relational structure within the organization, and the roles and responsibilities of different stakeholders. Examples of issues to decide upon: Roles defining who s doing what within IT Responsibilities describing the actions that each role is accountable for Stakeholder groups, such as committees for decision-making Corporate structure, the arrangement of roles and stakeholder groups Decisions on Technology. The physical IT-related assets. Examples of issues to decide upon: Infrastructure, such as servers, UPSs, firewalls and the corporate LAN Applications, such as the CRM system, ERP modules, operating systems, and desktop software Information storage, structure and use Facilities that host physical assets and personnel Decision-Making Phase. The decision-making phases denote different steps required to make decisions within the different domains. This dimension deals with the relation between IT, and the models of the reality used for decision-making. Before making any decision regarding e.g. the outsourcing of a helpdesk function, the organization must be clearly understood. Facts have to be thought over and investigated, and transformed into a model. The model might be a simple cognitive map, present nowhere else but in the head of the decision-maker, or a more formalized, abstract model put on print. This process of analysis and understanding is denoted the Understanding phase. Once the model is created, the actual decision can be made according to corporate IT principles, in a timely manner, by the right individuals, etc. In the IT governance definition, this is represented by the Decide phase, which also includes planning of how to make -4-

the decision. Finally, a decision is of little use unless its implementation is followed up and Monitored. This can be accomplished by implementing control objects for each process in order to assess real-world performance. The decision-makers compare the state of the reality with the should-be values obtained from the models. Note that these steps are not necessarily formal, but nevertheless exist in one way or another upon making decisions. The practitioners prioritized the dimensional units as they are presented below. Understand. The collection of information needed to make a correct decision. Examples of activities in the understand-phase: Understanding the organization and the implications of a certain decision Modelling complex problems to make them understandable for all stakeholders Stakeholder negotiations Decide. How and by whom the decision is made. Decisions are made according to corporate IT principles, at the correct level in an adequate forum, e.g. by a steering committee. Examples of activities in the decide-phase: Assigning decision-making authority Coordinating resources Aligning IT decision-making with external factors Monitor. How the implications of a decision are monitored. Examples of activities in the monitor-phase: Selecting control objectives Ensuring that the organization s performance is assessed Providing for audits Assigning accountability for IT monitoring Scope. The scope denotes different impacts implied by each decision. There is a long term aspect and a short time aspect of every decision that is made. Consequently, there is also a connection between the timeline of the decision and the level at which it is made. Top management make long time plans and set strategic goals, while lower management are authorized to make decisions affecting the near time. Further, strategically important decision requires more preparation than a tactic decision. The scope dimension is used to differentiate between different levels of decision-making. Firstly, there are detailed, rapidly carried out, IT-focused Tactic decisions. Examples of tactic decisions include whether to upgrade a certain workstation today or tomorrow, how to configure a user interface that is only used internally, or the manning of a single IT project. There also exists top management, low detailed, business oriented Strategic decisions with long timeline. A strategic decision might consider whether it is most appropriate to develop an application in-house or to purchase it off the shelf, or how the performance of IT processes should be reported to top management. The practitioners prioritized the dimensional units as they are presented below. Tactic decisions. Low-level management decisions, with many details and an impact primarily on IT. The decisions typically has an operations focus and a short timeline. Examples of tactical decisions: -5-

Whether to upgrade a server today or tomorrow How to configure a user interface How to man a single IT project Strategic decisions. Top-level management decisions, with few details and primarily a business impact. The decision features a business oriented focus with long timeline. Examples of strategic decisions: Whether to develop an application in-house or to purchase it off the shelf Whether to outsource IT operations The choice of decision-making structures Literature s and Practitioners Definitions of IT Governance It was the belief of the authors that IT governance would be defined differently in literature and by IT governance experts. Therefore, the framework for definition of IT governance was used to compare how literature and practitioners define the field. Literature s definition. All statements used to create the framework for IT governance definition were again analyzed in order to create a prioritization according to literature. The information was stored using a database. The statements were classified and the number of times that each dimensional unit (process, people, tactics, etc.) was mentioned explicitly or implicitly was counted. Fig 3. shows the results for this theoretical prioritization, i.e. literature s definition of IT governance. Results are normalized within each dimension, i.e. the total score for each dimension (e.g. Domain) is 100%. The theoretical prioritization shows that the dimensional units Strategic, Monitoring, and People were most Priority according to literature 100% 80% 60% 40% 20% 0% IT Governance Prioritization according to Literature Process Goal Technology People Understand Decide Monitor Domain Decision-making Phase Scope Fig. 3. 60 IT governance articles were classified using the framework for defining IT governance. frequently used within the 60 articles and within their dimensions respectively. As can be seen in the figure, IT governance mainly comprises strategic concerns according to literature. The daily use of IT, all the operational concerns for bread-and-butter IT are surely important, but they are not in the scope of IT governance. Regarding the decision-making phases, monitoring of ITrelated decisions is emphasized. In literature, IT control frameworks and legislations stipulating the need for internal control are often referred to, which is clearly reflected to in the figure. Technology issues are not the mayor concerns to decide upon, and literature rather stresses the importance of establishing roles and responsibilities, and an accountability framework that supports the organization s strive to achieve its business goals. Practitioner s definition. A survey with IT governance experts was conducted order to map their point of view onto the framework for defining IT governance. The study is just outlined here, but is described more thoroughly in (Simonsson 2006b). A web survey was sent out to 24 Swedish IT governance experts, asking them to prioritize the dimensional units of the IT -6-

governance definition. The survey was made using a commercial, web-based tool for online surveys. 1 18 participants responded to the survey. Among these, 72 % primarily had the role of consultants in IT governance change projects, but a few CIOs, security and risk managers, and internal auditors also participated. All respondents claimed previous involvement in at least one IT governance change project, 83 percent in two such projects or more. The practitioners were asked to prioritize the framework for IT governance definition. For each dimension, the respondents distributed 100 points between the dimensional units, to state what was most important to them in the achievement of good IT governance. The mean values for the practitioners priorities of the dimensional units, i.e. their definition of IT governance, can be found in Fig. 4. To test the credibility of the results, confidence intervals for (α=0.05) were calculated and are also displayed in the figure. The differences between dimensional units for Domain and Scope dimensions are statistically significant at that level, while the relative priorities for the Decision-Making Phase dimension remain a bit more uncertain. Priority according to Practitioners IT Governance Prioritization according to Practitioners 100% 80% 60% 40% 20% 0% Process Goal Technology People Domain Understand Decide Monitor Decision-Making Phase Scope Fig. 4. 18 IT governance experts prioritized the framework for defining IT governance. Diagram displays mean values with confidence intervals for (α=0.05). According to the 18 practitioners responding the survey, IT governance decision-making is mainly a strategy issue while tactical decisions are less important. Emphasis is put on understanding the situation at hand prior to making a decision, and solving practical issues regarding how each decision is carried out, such as assigning decision-making authority, coordinating resources, and aligning IT decision-making with external factors. Monitoring the implementation of decisions already made receives somewhat less attention from the practitioners, according to the survey. Practitioners do however agree that IT decisions are mainly about IT goal setting; strategy development, alignment of IT and business goals, etc. Another important topic is the establishment of a corporate decision-making structure with clear assignment of roles and responsibilities, while IT processes and technology issues are less stressed. Case Study: Cobit s IT Governance Definition Cobit is a well-known framework for IT governance improvement, risk mitigation and IT value delivery (Ridley 2004, Holm Larsen 2006, Debraceny 2006). It was first issued by the IT Governance Institute, ITGI, and Information Systems Audit and Control Association, ISACA, in 1998 and a fourth version became available in December 2005. Cobit describes the IT organization by means of 34 processes, divided into four different groups: Plan & Organize, Acquire & Implement, Delivery & Support, and Monitor & Evaluate. Each process contains a set of Control Objectives (statements of the desired results to be achieved by implementing control procedures for the processes), Key Performance Indicators, Critical Success Factors, and a CMM-style maturity model. The latest version of Cobit also contains RACI-charts to guide 1 Survey Monkey, http://www.surveymonkey.com -7-

which stakeholders should be Responsible, Accountable, Consulted, and Informed about certain activities. In order to evaluate Cobit s view of IT governance, each IT process was studied thoroughly, sentence for sentence thus mapping Cobit to the framework for defining IT governance. The Highand Low level control objectives of Cobit were included in the classification, and so were the RACIchart and the Goals and Metrics. The Maturity Model was excluded from the classification, since it just outlines and exemplifies what is said in the other sections of each process. The Inputs and Outputs were neither analysed, as they represent an alternative way of defining each process by the deliverables exchanged between the processes. The classification was carried out so that a single line of plain text featuring e.g. goals was given one point for Goals in the Domain dimension, etc. If Priority according to Cobit 4.0 IT Governance Prioritization according to Cobit 4.0 100% 80% 60% 40% 20% 0% Process Goal Technology People Domain Understand Decide Monitor Decision-Making Phase Scope Fig. 5. Cobit s prioritization of the framework for defining IT governance. the same line also featured monitoring aspects, Monitor of the Decision-making phase domain was also given one point, etc. Separate statements presented in tables, lists, etc, were given one point each. All in all, about 2500 lines of text or statements in Cobit were classified. Results, i.e. Cobit s definition of IT governance, are shown in Fig. 5., Monitoring and Processes were the dimensional units that received the highest marks. Once this classification was made, results were compared to prioritizations from literature and practitioners. Cobit compared to Literature. The results from Cobit s classification were compared to the prioritizations previously identified in literature, c.f. Fig. 6. The figure shows differences between Cobit and literature so that a perfect alignment would by equivalent to 0 %. The mean square difference between Cobit and Literature was 15 %, indicating that the prioritizations in general do align. In the Domain dimension, it is clearly visible that Cobit is focused on decisions regarding the Processes while People receive less attention. Further, Cobit spends more effort in discussing the Understand phase and less on the Decide phase. Strategic concerns are most often dealt with, while Tactical concerns are only briefly discussed. 50% IT Governance Prioritization: Cobit-Literature 50% IT Governance Prioritization: Cobit-Practitioners Difference beween Cobit and Literature 25% 0% -25% Process Goal Technology People Understand Decide Monitor Difference beween Cobit and Practitioners 25% 0% -25% Process Goal Technology People Understand Decide Monitor -50% Domain Decision-Making Phase Scope -50% Domain Decision-Making Phase Scope Fig. 6. IT governance is defined differently in literature and in Cobit. Fig. 7. IT governance is defined differently by practitioners and in Cobit. -8-

Cobit compared to Practitioners. Results from Cobit s classification were also compared to the practitioners prioritization, c.f. Fig. 7. The mean square difference was 8%, indicating good alignment. The figure shows that Cobit emphasizes Processes but lacks hands-on support for decisions regarding People and Goal settings. In the figure, it is also noticeable that Cobit focuses on decision Monitoring to a larger extent than what practitioners do, while the opposite is valid for Understand and Decide. Summary This article presented an ATD and a framework for definition for IT governance based on a study of 60 articles. IT governance is the preparation for, making of and implementation of ITrelated decisions regarding goals, processes, people and technology on a tactical or strategic level. Priorities in literature and of IT governance experts were mapped onto the framework for definition. A case study was carried out in order to prioritize Cobit. Results show that the major differences exist within the priorities of the decision-making phases: Cobit emphasises Monitoring of decisions while practitioners are trying to improve their Understanding of organizations and IT. Biography Mårten Simonsson is a Ph.D. Student in the field of IT governance at the Department of Industrial Information and Control Systems at KTH, Royal Institute of Technology in Stockholm, Sweden. Pontus Johnson, Ph.D, is a senior researcher at the same department. His research focus is Enterprise Architecture, IT value delivery and Enterprise Information Security. The authors would like to thank Mathias Ekstedt (Ph.D) for his valuable support upon creating the framework for IT governance definition. We are also deeply grateful to the IT governance experts that participated in the survey. References Cumps, B., Viaene, S., Dedene, G., and Vandenbulcke, J., An Empirical Study on Business/ICT Alignment in European Organizations. Proceedings of the 39th Hawaii International Conference on System Sciences, 2006 Dahlberg, T., and Kivijärvi, H., An Integrated Framework for IT Governance and the Development and Validation of an Assessment Instrument. Proceedings of the 39th Hawaii International Conference on System Sciences, 2006 Debraceny, R.S., Re-engineering IT Internal Controls: Applying capability Maturity Models to the Evaluation of IT Controls, Proceedings of the 39 th Hawaii International Conference on System Sciences, 2006 De Haes, S., and Van Grembergen, W., IT Governance Structures, Processes and Relational Mechanisms achieving IT/Business alignment in a major Belgian financial group. Proceedings of the 38 th Hawaii International Conference on system Sciences, 2005 Guldentops, E., Governing Information Technology through COBIT. In Van Grembergen, W. (Ed.): Strategies for Information Technology Governance. Idea Group Publishing, 2004-9-

Hamaker, S., and Hutton, A., Principles of IT Governance. Information Systems Control Journal, Volume 2, 2004 Holm Larsen, M., Kühn Pedersen, M., and Viborg Andersen, K., IT Governance Reviewing 17 IT Governance Tools and Analysing the Case of Novozymes A/S. Proceedings of the 39 th Hawaii International Conference on System Sciences, 2006 ISACA Sweden Chapter: FoU-kommitténs COBIT-undersökning. (In Swedish), 2004. Available online at www.isaca.se IT Governance Institute (ITGI), COBIT, 4 th Edition, December 2005. Available online at http://www.isaca.org Johansson, E., Assessment of Enterprise Information Security How to make it Credible and Efficient. Ph.D. Thesis at the Department of Industrial Information and Control Systems, Royal Institute of Technology, Stockholm, Sweden, 2005 Johnson, P., et al, Using Enterprise architecture for CIO Decicion-making: On the importance of Theory. Proceedings of 2 nd Annual Conference on Systems Engineering Research (CSER), 2004 Kaplan, R., and Norton, D., The Balanced Scorecard. Harvard Business School Press, 1996 Office of Government Commerce (OGC), IT Infrastructure Library Service Delivery. The Stationery Office, 2002 Ribbers, P.M.A., Peterson, R.R., and Parker, M.M., Designing information technology governance processes: Diagnosing contemporary practices and competing theories. Proceedings of the 35th Hawaii International Conference on System Sciences, 2002 Ridley, G., et al., COBIT and its utilization: A framework from the literature. Proceedings of the 37th Hawaii International Conference on System Sciences. 2004 Sambamurthy, V., and Zmud, R.W., Research Commentary: The Organizing Logic for an enterprise's IT Activities in the Digital Era - A Prognosis of Practice and a Call for research. Information Systems Research, Vol 11, No. 2, June 2000, pp 105-114 Simonsson, M., and Johnson, P., Defining IT Governance - A Consolidation of Literature. Working Paper of the Department of Industrial Information and Control Systems., 2006a. Availible online at www.ics.kth.se Simonsson, M., and Ekstedt, M., Getting the Priorities Right - Literature versus Practice on IT Governance. Accepted for publication at Portland International Conference on Management of Engineering and Technology, Istanbul, July 9-13, 2006b Trites, G., Director Responsibility for IT Governance. International Journal of Accounting Information Systems, vol. 5, Elsevier Inc., 2004, pp 89-99 Van Grembergen, W. Saull, R., and De Haes, S., Linking the IT Balanced Scorecard to the Business Objectives at a Major Canadian Financial Group. In (Ed. Van Grembergen, W., Strategies for Information Technology Governance. Idea Group Publishing, 2004 von Solms, B., and von Solms, R., The 10 Deadly Sins of Information Security Management. Computers & Security, vol 23, Elsevier Science, 2004, pp 371-376 Warland, C., and Ridley, G., Awareness of IT control frameworks in an Australian state government: A qualitative case study. Proceedings of the 38th Hawaii International Conference on System Sciences, 2005 Webb, P., Pollard, C., and Ridley, G. Attempting to define IT Governance: Wisdom or Folly Proceedings of the 39 th Hawaii International Conference on system Sciences, 2006 Weill, P., and Ross, J. W., IT governance How top performers manage IT decision rights for superior results. Harvard Business School Press, 2004-10-