A framework for the comparison of Maturity Models for Project-based Management

Transcription

1 Utrecht University A framework for the comparison of Maturity Models for Project-based Management Tjie-Jau Man, BSc. Student number Thesis number: INF/SCR Utrecht University supervisors: Lidwien v/d Wijngaert Sjaak Brinkkemper Capgemini supervisors: Chris ten Zweege Erwin Dunnink September 2, 2007 Capgemini

2 Preface Writing a thesis is unmistakably one of the hardest tasks that a graduating student has to deal with. Despite the enthusiasm at the beginning of the venture, you inevitably end up counting the days before the final deadline (that is, if you have decided upon one). As comical as it might sound, the whole process of doing the thesis assignment can be summarized with three consecutive thoughts (at least in my case). First, you will think that other students are exaggerating since there is no way you would take that much time to finish your thesis. After all, you are positive about the subject you have chosen. After several months, you will begin to wonder if you are on the right track because something just doesn t feel right. And then, finally, when you are nearing your final deadline, you would most probably wish you had taken the assignment more seriously from the beginning. Fortunately, support is provided to alleviate the recurring dips of motivation and selfdiscipline during this long and cumbersome journey towards graduation. In my case, I wish to thank my supervisors of the University of Utrecht, Lidwien van de Wijngaert and Sjaak Brinkkemper, for their guidance. I also want to thank the people at Capgemini Netherlands for providing me with a resourceful environment to conduct my thesis assignment, especially my supervisors Chris ten Zweege and Erwin Dunnink. Likewise to all members of the PMI project group, please accept my thanks for your cooperation and helpful feedback during and after the project meetings. And last but certainly not the least; I want to thank my parents for supporting me from the very beginning. Without them, I would never have come this far. Zevenaar, September 2007 Tjie-Jau Man i

3 Abstract To conduct project-based management (PM) as successful as possible, it is fundamental for organizations to invest time and effort to construct the necessary infrastructure, such as organizational structure, policies and competencies of people. Over time, more advanced organizations may wonder where they exactly stand in the whole process and what they should do to make further advancements. Maturity models for PM are developed to assist organizations that have these thoughts. By comparing their own practices against best practices described by these models, organizations can find out how mature or professionalized they are in performing project-based management and what they could do to realize desirable improvements in it. However, with more than 20 maturity models available in the field of PM, organizations have to consider carefully which one they can adopt. In order to do this, organizations need to know what aspects of these models are important to consider and how they should evaluate them. In this thesis, research is done on relevant dimensions to the evaluation of maturity models for PM. This set the stage for the selection of measures that are needed to evaluate similarities and differences between maturity models for PM. The research showed that maturity models for PM can be evaluated along three dimensions: structure, applicability and usage. And three measures were selected to operationalize these dimensions in the same respective order: Process-Data Diagrams, evaluation criteria and user interviews. These measures formed a framework that was applied to several maturity models for PM to determine its quality. The framework and its constituting measures proved useful in shedding light on the relevant similarities and differences between the models. It was able to show the strengths and weaknesses of each evaluated maturity model, which should be considered by organizations planning to adopt them. ii

4 Table of contents. INTRODUCTION RESEARCH QUESTION SCIENTIFIC & SOCIETAL RELEVANCE RESEARCH APPROACH MATURITY MODELS FOR PROJECT-BASED MANAGEMENT PROJECT-BASED MANAGEMENT MATURITY MODELS MATURITY MODELS FOR PM MATURITY MODEL SELECTION THE EVALUATION FRAMEWORK STRUCTURE DIMENSION Process-Data Diagram (PDD) Modeling PDD comparison method APPLICABILITY DIMENSION Evaluation criteria Criteria comparison method USAGE DIMENSION Interviews ANALYSIS & RESULTS STRUCTURE ANALYSIS Maturity reference model structure Assessment method structure Structure comparison summary APPLICABILITY ANALYSIS Maturity reference model Assessment method USAGE ANALYSIS DISCUSSION & CONCLUSIONS FRAMEWORK REQUIREMENTS SUGGESTIONS FOR SITUATIONAL SELECTION RECOMMENDATIONS FOR FURTHER RESEARCH...75 REFERENCES...76 APPENDIX...80 APPENDIX A- PMI PROJECT GROUP MEMBERS AND ROLES...80 APPENDIX A-2 CONSULTED EXPERTS...8 APPENDIX B PROCESS-DATA DIAGRAM MODELING...82 APPENDIX D- ASSESSOR & USER SAMPLE QUESTIONNAIRES...87 APPENDIX D-2 SUMMARY INTERVIEW FINDINGS...89 APPENDIX E PROCESS-DATA DIAGRAMS...92 APPENDIX F CONCEPT AND ACTIVITY TABLES...95 APPENDIX G CRITERIA RESULTS PER MODEL...04

5 . Introduction In general, there are two reasons why it is beneficial for organizations to adopt a maturity model for project-based management, which includes the management of projects, programs and portfolios. Ever since organizations began to adopt the projectbased way of conducting business, they have strived to deliver projects successfully. To do this, organizations require the necessary infrastructure, which includes processes (methods and techniques), governance structures, competences of people and tools []. Developing such an infrastructure may take several years, and because of this, more advanced organizations may start to wonder after a while where they exactly stand in the whole process and whether they are going the right way. This is when the adoption of a maturity model proves useful. A maturity model is able to assist organizations in verifying what they have achieved by describing activities and best practices and categorizing these descriptions into progressive levels of maturity. The second benefit for adopting a maturity model becomes apparent when an organization has finished assessing its current practices and aims for advancements to a desired level of maturity [2]. By comparing the results of a maturity assessment with the descriptions in a maturity model, an organization gains insight into their strengths and weaknesses and is able to prioritize its actions to make improvements. In addition to the above arguments, the execution of a maturity assessment in itself raises the awareness about what can be improved within an organization. In other words, members of an organization will focus more on the inefficiencies of their ways of working simply because they know they are being assessed. Besides deciding whether a maturity model should be adopted, an equivalently important process regards making the choice between the offered maturity models for project-based management. Many maturity models have emerged since the mid 90s [3] and one question that can be asked here is how organizations should evaluate them in order to select an appropriate maturity model. As an attempt to answer this question, a project group has been gathered by Project Management Institute Nederland (PMI-NL) and given the assignment to publish a book, in which different maturity models for project-based management are compared with each other. 2

6 Closely aligned to this assignment, research was done on maturity models for projectbased management. A framework was developed to support the evaluation and comparison between such models. This framework was applied to compare three maturity models for project-based management with each other. 3

7 .. Research question The purpose of this thesis is to develop a framework to evaluate and compare maturity models for project-based management. The central research question of this exploratory thesis research is formulated as follows: What measures are needed to evaluate the similarities and differences between maturity models for project-based management? The following sub research questions will set the stage for answering the main research question.. What is a maturity model for project-based management? Before analyzing a maturity model for project-based management, it is important to understand the reasons behind its existence and what this concept means. 2. What constitutes a maturity model for project-based management? Examining the components that constitute a maturity model sets the stage for determining important aspects or dimensions along which they can be evaluated. 3. What are relevant dimensions to the evaluation of maturity models for projectbased management? These dimensions provide guidance to the selection of the measures that facilitate the comparison between the maturity models. They will ultimately form the evaluation framework mentioned earlier. 4. What are the main similarities and differences between maturity models for project-based management? This sub question focuses on the main similarities and differences found after evaluating several maturity models for project-based management with the developed framework. 4

8 .2. Scientific & societal relevance This thesis research explores the aspects of maturity models for project-based management that are relevant to distinguishing them from one another. It is meant to shed light on the strengths and weaknesses of these models and how these affect their applicability in certain situations. However, it is not meant to indicate superiority or inferiority among them. The main reason behind this thesis lies in the assumption that organizations should take careful considerations of which maturity model to adopt. Maturity models in general are measurement tools used to assess and/or improve an organization s processes. Depending on the match between a maturity model and an organization s situation, the organization may end up assessing different capabilities than initially planned. This could affect the outcomes of the maturity assessment and may cause an organization to overlook some important weaknesses in its current processes. Supporting the above assumption is the contingency theory; a theory that takes many forms in the world of research. The earliest and extensively researched form of contingency theory was introduced by Fiedler in the 60s, which explains that group performance is a result of interaction of two factors: leadership style and situational favorableness [4][5]. Since then, the contingency approach has been applied and adopted many times [6][7][8][9]. A list of scientific articles that use the contingency theory is shown in [0]. Similar to what other researchers have done, this thesis applies the contingency theory on maturity models for project-based management. As mentioned before, the focus here is on finding measures to elicit similarities and differences among these maturity models. The results of this research could set the stage for further research on the possible contingency or fit between the models and organizational situations, and relate this to performance variables. Also, the differences found with the measures may prove useful in future research efforts on the categorization of maturity models for project-based management. And if the framework proves useful to the comparisons between them, it may provide a foundation or be used to evaluate and compare other maturity models of the same discipline. 5

9 .3. Research approach The thesis research starts off with the reviewing of literature and scientific papers to gather information about maturity models for project-based management. The next chapter of this report provides the necessary background information regarding this concept (Chapter 2). Chapter 2 also provides a long-list of existing maturity models for project-based management. From this long-list, several maturity models are selected for the purpose of testing the framework to be developed. The description of the selection process and the selected maturity models are described in the final paragraphs of Chapter 2. The examination of literature continues in combination with expert consults to find the dimensions in which maturity models may differ from each other. The experts here include all members of the project group and individuals who have had experience with maturity models for project-based management before. Both the selected dimensions and the evaluation framework that will comprise them are explained in Chapter 3. After the description of the framework, it is applied to several selected maturity models for project-based management to test the framework and the dimensions. The elaboration of the analysis and results of this application can be found in Chapter 4. In order to determine the quality of the evaluation framework, we have adopted the five requirements from the field of situational method engineering []. These requirements are used to assess the quality of a method assembled from method fragments to suit a situation specific for a project. As our framework describes a method to evaluate and compare maturity models for project-based management, these requirements can be used for its evaluation. The five requirements are defined as follows for this thesis: - Completeness: the framework describes all relevant dimensions for the evaluation of maturity models for project-based management; - Consistency: all activities and concepts are consistently defined and described throughout the framework; 6

10 - Applicability: the researchers are able to execute the method described by the framework; - Reliability: the method described by the framework is semantically correct and meaningful; - Efficiency: the method described by the framework can be performed at minimal cost and effort. This thesis concludes with the elaboration of discussion topics and conclusions based on the analysis results and the description of the framework s quality based on the above requirements (Chapter 5). 7

11 2. Maturity models for project-based management In order to understand the concept of maturity models for project-based management, it is necessary explain the two smaller concepts that constitutes it: project-based management and maturity model. These two concepts will be further elaborated in the following subsections. 2.. Project-based management As mentioned in the introduction, the term project-based management refers to all three domains of this discipline, which are project management, program management and portfolio management [2]. From here on, the abbreviation PM will be used from here on whilst referring to all three domains. In cases when the abbreviation is less appropriate, the term project-based management will be used. To understand the concept of PM, it is necessary to explain the three domains constituting it. This is done in the following paragraphs. A project is a temporary endeavor with a definite beginning and end. To complete a project successfully, it needs to meet the requirements predefined by all stakeholders and deliver a product or service different in some unique way from all similar products or services [3]. The Project Management Body of Knowledge (PMBOK) of the Project Management Institute (PMI) uses the following broad definition to define the management of projects: the application of knowledge, skills, tools and techniques to activities within a project in order to meet or exceed stakeholders needs and expectations [3]. Although this definition is similar to that of process management [4], it is different from process management in that project management is concerned with managing (collections of) temporary undertakings rather than ongoing activities. 8

12 Programs differ from projects in that they are carried out to achieve specific strategic business objectives or goals. Or as formulated in [2], a program s focus is on producing, in accordance with a vision of an end state consistent with organizational strategic objectives. An example of an end state is the realization of 5% cost reduction throughout the entire organization. To achieve this, a program will consist of a number of projects or functional activities including for example the implementation of a new logistics system and the development of a new IT system [2]. In addition, while the aforementioned project is successful when the logistics system is implemented conform specifications, the completion of the above program depends on the realization of the 5% cost reduction, and not with a new IT or logistics system. According to Wikipedia [5], the management of programs is: the process of managing multiple ongoing inter-dependent projects. (It) focuses on selecting the best group of programs, defining them in terms of their constituent projects and providing an infrastructure where projects can be run successfully but leaving projects management to the project management community. However, due to physical or financial constraints, organizations cannot undertake all projects or programs on their to-do-list at the same time. That is when the creation of a portfolio becomes relevant. A portfolio, in terms of PM, is a collection of projects and/or programs grouped together to meet strategic objectives [2]. Portfolio management is the centralized management of one or more portfolios. This domain is all about the prioritization, facilitation and management of projects/programs based on their alignment to the business strategy of an organization. According to [6] there has been an increase in the amount of organizations employing PM. The growing popularity is also reflected by the large amount of publications written about this concept. While some of them focus on defining the success of a project [7][8][9][20], others try to shed light on the various factors that might influence the effectiveness of PM [2][22][23]. The emergence of these literature points to the need to professionalize PM such that organizations can 9

13 undertake successful project continuously. This brings us to the discussion of maturity models, which are developed to facilitate the improvement of how PM is undertaken Maturity models The literature has paid a considerable amount of attention to the concept of maturity models [2][3][24][25][26][27]. This is because a maturity model allows an organization to assess and compare its own practices against best practices or those employed by competitors, with the intention to map out a structured path to improvement [3]. Basically, a maturity model is a framework describing the ideal progression toward desired improvement using several successive stages or levels. te that an organization in the context of maturity models for PM does not necessarily refer to an entire company. A maturity model can also be applied to a business unit, functional groups or departments. One well-known maturity model is the Capability Maturity Model (CMM), introduced by the Software Engineering Institute (SEI). This model was later replaced by its successor, the Capability Maturity Model Integration (CMMI) [28]. The development of Capability Maturity Models had inspired the emergence of other maturity models in the same field of Software Development. Examples of these are the Test Process Improvement (TPI) Model developed by Sogeti [29] and the Usability Maturity Model [30] Maturity Models for PM The existence of CMMI has also led to the development of maturity models for PM. Because of the role that PM plays in the software development process, many of the concepts of maturity incorporated in capability maturity models, such as the CMMI, were adopted by maturity models that emerged in the field of PM [3]. Building on what was explained about maturity models earlier, maturity models for PM are used to measure the degree to which an organization is executing PM by comparing its PM practices against practices in general or best practices. These models describe how mature or professionalized organizations are in conducting PM and what they could do to improve their way of working. 0

14 According to [3], there is no generally agreed definition of what a mature projectbased organization looks like. In spite of this, the current number of maturity models for PM is estimated at 30 [3]. During this research, an attempt was made to construct a long-list containing existing maturity models for PM. This list of maturity models is depicted in Table, along with their names and owners. Table Long-list of existing maturity models for PM Nr Acronym Name Owner OPM3 Organizational Project Management Project Management Institute (PMI) Maturity Model 2 P3M3 Portfolio, Programme, Project Office of Government Commerce (OGC) Management Maturity Model 3 P2M Project & Program Management for Enterprise Innovation (P2M) Project Management Association of Japan (PMAJ) 4 PMMM Project Management Maturity Model PM Solutions 5 PPMMM Project Portfolio Management Maturity PM Solutions Model 6 PMMM Programme Management Maturity Programme Management Group Model 7 PMMM Project Management Maturity Model KLR Consulting 8 (PM)2 The Berkeley Project Management Process Maturity Model Department of Civil Engineering University of California at Berkeley 9 ProMMM Project Management Maturity Model Project Management Professional Solutions Limited 0 MINCE2 Maturity Increments IN Controlled MINCE2 Foundation Environments PPMM Project and Portfolio Management PriceWaterhouseCoopers (PWC) Belgium Maturity 2 CMMI Capability Maturity Model Integration Software Engineering Institute (SEI) 3 SPICE Software Process Improvement and Capability determination 4 FAA-iCMM Federal Aviation Administration - Integrated Capability Maturity Model 5 Trillium Trillium Bell Canada Software Quality Institute Griffith University, Australia US Federal Aviation Administration 6 EFQM EFQM Excellence Model European Foundation for Quality Management (EFQM) 7 COBIT Control Objectives for Information and related Technology Information Systems Audit and Control Association (ISACA) 8 INK INK Managementmodel Instituut Nederlandse Kwaliteit (INK) 9 ProjectProof VA Volwassenheidsmodel Van Aetsveld 20 PAM Project Activity Model Artemis 2 Project The Project Excellence Model Berenschot Excellence Model 22 PMMM Project Management Maturity Model International Institute for Learning (IIL) H. Kerzner

15 Maturity models differ from one another in the concepts they embody and the suggestions they make as to how the path to maturity looks like [22]. Different maturity model for PM may define maturity differently and measure different things to determine maturity. Because of this, organizations should give careful consideration to the selection of a maturity model Maturity model selection In order to select maturity models to test the framework to be developed, each member of the project group were asked to rate each maturity model of the long-list based on several criteria. Information regarding the members of the project group is included in the Appendix section of this thesis (see Appendix A-). The maturity models were rated using the following criteria: - Method independency: the degree to which a maturity model is closely aligned to a PM methodology? - Public domain: the degree to which a maturity model and maturity assessment can be applied by anyone besides its owners. - Publication: the degree to which a maturity models is issued in publications. - Industry independency: the degree to which the application of a maturity model is limited to particular industry sectors. - Transparency: the traceability of the calculation of the maturity scores. - Toolset independency: the degree to which the usage a maturity model is bound to a toolset. - Years of existence: how many years a maturity model has existed. - Ease of use: the degree to which a maturity model is easy to use in practice. A candidate maturity model should be at least publicly available (or against moderate payment) through publication in a book (electronically or in print). This is to ensure that the needed information about a maturity model is accessible when it is evaluated using the framework. The table below shows the average scores given to the maturity models. 2

16 Table 2 Scoring table maturity model for PM Maturity model nr Selection Criteria Method independency Public domain Publication Industry independency Transparency Toolset independency Years of existence Ease of use Table 2 (continued) Maturity model nr Selection Criteria Method independency 0 0 Public domain 0 0 Publication 0 0 Industry independency 3 0 Transparency 7 0 Toolset independency 8 0 Years of existence 0 3 Ease of use 3 0 The maturity models were rated with scores ranging from 0 to 0 where 0 indicates Hardly and 0 indicates Completely. Grayed out maturity models are those excluded from the long-list based on their accessibility. The maturity models MINCE2 (nr. 4) and PM Solutions PMMM (nr. 8) have not been examined yet due to the limitations of time. The short-list that resulted from the selection process consisted of the following maturity models: - Organizational Project Management Maturity Model (OPM3) [2] - Capability Maturity Model Integration (CMMI-DEV) [28] - Kerzner Project Management Maturity Model (PMMM) [25] - Project, Program, Portfolio Management Maturity Model (P3M3) [32] - Maturity Increments IN Controlled Environments (MINCE2) [33] 3

17 The first three models of the list were selected to conduct this thesis, which are the OPM3, CMMI and PMMM. For this thesis, materials are used provided by the following institutions for the respective maturity models: Project Management Institute (PMI) for OPM3, International Institute for Learning (IIL) [34] for PMMM and Software Engineering Institute (SEI) for CMMI. A brief background of each model is provided in the following sections, including the reasons why they were or were not selected to test the framework OPM3 OPM3 is an acronym for Organizational Project Management Maturity Model. It is a standard developed under the stewardship of and introduced in December 2003 by the Project Management Institute (PMI). The development of this standard was inspired by the increasing interest in a maturity model that shows a step-by-step method of improving and maintaining an organization s ability to translate organizational strategy into the successful and consistent delivery of projects. In other words, OPM3 is meant to enable organizations to bridge the gap between organizational strategy and successful projects [35]. The purpose of OPM3 is not to prescribe what kind of improvements users should make or how they should make them. Rather, by providing a broad-based set of organizational project management (OPM) best practices, this standard allows an organization to use it as a basis for study and self-examination, and consequently to make its own informed decision regarding potential initiatives for changes [2]. The standard comprises three interrelated elements: - Knowledge. In this element, the user can become proficient with OPM3, be comfortable with the body of best practices knowledge it contains, with the idea of OPM and OPM maturity, and with the concepts and methodology of OPM3. - Assessment. The organization is compared to OPM3 in this element to determine its current location on a continuum of OPM maturity. 4

18 - Improvement. Here, organizations can decide to move ahead with change initiatives leading to increased maturity using the results of the assessment as a basis for planning. OPM3 is represented by two complementary parts: the Foundation and the ProductSuite. The Foundation is developed by PMI and the ProductSuite is developed by PMI in collaboration with Det rske Veritas (DNV) [36], an international consulting firm. The Foundation is a description of the OPM3 model itself, readily available to all organizations interested in knowing about the model. The ProductSuite, on the other hand, is a description of OPM3 as in how the model should be applied and what steps are taken during a maturity assessment. Access to the latter is limited to those who enroll for training sessions to become certified assessors, while the Foundation can be purchased in the form of a book. Both sources are consulted during the evaluation of OPM3. In cases where the distinction should be made between the two, the abbreviations OPM3-PS and OPM3-F will indicate information derived from the ProductSuite and Foundation respectively. OPM3 was selected for the evaluation because of its popularity in the field of PM. Information about this model was readily accessible since the chairman of the project group is a certified OPM3 assessor. This model is closely aligned to the PMBOK [2], which is a well-accepted standard approach for project management. Besides this, additional PMBOK guides have recently been developed to describe approaches to program and portfolio management. These additions are also embedded into the OPM CMMI CMMI stands for Capability Maturity Model Integration. Its first version (.) was introduced by the Software Engineering Institute (SEI) in 2002 as the successor of the Capability Maturity Model (CMM), which was developed from 987 until 997. The SEI (2007) defines CMMI as a process improvement approach that helps organizations integrate separate functions, set process improvement goals and priorities, provide guidance for quality processes, and provide a point of reference for appraising current processes. 5

19 The latest version of CMMI (2.), released in 2006, comprises a framework that allows the generation of multiple models, training courses and appraisal methods supporting specific areas of interest. CMMI for development is one of those models and provides guidance for managing, measuring, and monitoring software development processes. The development of CMM and CMMI was based on the premise, which states that the quality of a system is highly influenced by the quality of the process used to acquire, develop and maintain it [28]. These models comprise the essential elements of effective processes for one or more disciplines (e.g. Software Development) and describe an evolutionary improvement path from ad hoc, immature processes to disciplined, mature processes with improved quality and effectiveness. And the fundamental idea behind this is that even the finest people within an organization cannot perform their best if the process is not understood or operating at its best [28]. The purpose of the CMMI for Development model is to assist organizations in enhancing their software development processes for both products and services by describing characteristics of best practices. The reason why CMMI was included in this thesis is because of its rich history and worldwide acceptance. The model s long history sped up the search for information and experts who were willing to share their knowledge about it. In particular, the CMMI for Development is selected because of its availability. The model information was readily downloadable from the Word Wide Web. For the sake of brevity, the acronym CMMI will be used to refer to CMMI for Development for the rest of this thesis. During the evaluation, documents of SEI s CMMI and the Standard CMMI Assessment Method for Process Improvement (SCAMPI) [37] are consulted. It should be noted that SEI is not the only institution that provides CMMI maturity assessments. While SEI s assessment method is the only one being evaluated for CMMI, it is not the only that exists for CMMI. Thus, the findings here do not necessarily account for the procedures employed by other institutions that also provide CMMI assessments. 6

20 PMMM The Project Management Maturity Model (PMMM) was introduced by H. Kerzner in 998. The first edition of his book describing this model was published in 200. In 2005, he published the second edition. PMMM is a practical PMBOK-aligned standard [25]. The model sets out various levels or stages of development towards project management maturity; along with assessment instruments to validate how far along the maturity curve the organization has progressed. The original intent of the PMMM is to provide organizations with a framework that allows organizations to create an organization-specific maturity model. Each organization can have a different approach to maturity and that is why organizations are allowed to adapt the questions and answers of the PMMM questionnaire [25]. The physical part of the model consists of a book and an online assessment tool. Both components serve to provide individual assessment participants and their organizations with: - a breakdown on how they are doing in different categories in each maturity level; - a comparison on overall results against those of other companies and individuals who have taken the assessment; and - a high-level prescriptive action plan to follow for individual and organizational improvement. This model was chosen because of its simplicity and availability. It is interesting to examine the differences between PMMM and OPM3 since both of them are aligned to the PMBOK P3M3 P3M3 stands for Portfolio, Program and Project Management Maturity Model. It was formally published in February 2006 by the Office of Government Commerce (OGC) after some refinements were made [27]. The model describes the portfolio, program 7

21 and project-related activities within process areas that contribute to achieving desirable project outcomes. Although P3M3 was eligible to be included in this thesis, access to information about this model was only granted to people of accredited institutions. Several attempts were made to contact these accredited institutions, but most of them could not fulfill the request for information due to license agreements. Some of the contacted persons agreed to provide answers to questions only relevant to conducting the evaluation, but because this took place when most of the research was already done it was no longer possible to include P3M3 in the thesis MINCE2 The last candidate maturity model for PM was MINCE2, acronym for Maturity INcrements IN Controlled Environments 2. The MINCE2 Foundation (established in May 2007) developed this model in order to: - determine the project maturity level an organization is in; - report in a standardized way regarding the findings; and - indicate what to do in order to increase the maturity [33]. MINCE2 was not included in the thesis because at the moment of the maturity model selection, the owners of the model were still working on the publication of the model. All details about MINCE2 were going to be published in August So although one of the project group members had access to information about the model, it could not be used for the thesis research due to publishing rights. In this chapter, the concept of maturity models for PM was explained. The next chapter elaborates on the development of the framework used to evaluate the selected maturity models. 8

22 3. The evaluation framework The chairman of the PMI-NL Project Group organized monthly meetings to guide the construction of the framework. Whenever prompt feedback was needed, all members could contact each other by or phone. The description of the project group members and their roles is included in Appendix A-. After several brainstorm sessions and informal meetings, the project group members agreed upon a staged approach in the evaluation framework. It was decided that the framework should evaluate a maturity model s structure, applicability and usage. The following sections describe these three dimensions. After explaining what a dimension holds and how it can be operationalized, a measure is described to elicit the characteristics of a maturity model on that dimension. The description of each dimension concludes with an elaboration of how the results of the models, found with the selected measure, are compared with each other. 3.. Structure Dimension The first dimension along which the framework evaluates a maturity models for PM is structure. Perhaps it is important to mention here that a maturity model for PM is made up of two parts: a maturity reference model and an assessment method. From an assessor s point of view, the maturity reference model is considered a measuring staff; it elaborates on what an assessor should assess in order to determine the maturity of an organization. The description of the assessment method, on the other hand, describes how assessors should carry out the assessment to determine maturity. The characteristics of an assessment method are just as important as the maturity reference model because it affects the repeatability of an assessment and therefore, also the reliability of the results. Besides the fact that these two parts have different measurable characteristics, another reason why a distinction should be made is because a maturity model for PM does not necessarily have only one assessment method to apply its reference model. As mentioned earlier, the SEI is the owner of CMMI, and has described SCAMPI as its standard assessment method. 9

23 The structure of the maturity reference model comprises a collection of concepts and relationships between these concepts. Each concept says something about the concept of maturity as defined by a maturity model. And it is the relationship between these concepts that illustrates their importance and role in the definition. Shedding light on the structure of the model concepts makes it easier for organizations to understand the purpose and essence of a maturity model. The assessment method of a maturity model can be broken down into multiple process phases and activities. By knowing the structure of these process phases and activities, organizations will know what to anticipate when engaging in an assessment. The products resulting from the assessment activities are also important, especially their relationships with the concepts underlying the maturity reference model. After all, these products contain the data that assessors use to assess maturity. And they do that by comparing this data with the measures defined by maturity reference models. So the relationships between the products resulting from the assessment activities and the concepts of the reference model must not be ignored. To depict the structure of the maturity reference model and assessment method as good as possible, a meta-modeling technique was selected as a measure for the structure dimension Process-Data Diagram (PDD) Modeling Objective comparisons of the maturity models structures require them to be on a higher abstraction level where it is possible to describe them uniformly. A metamodel is a model constructed on a higher abstraction level, used to describe the features of an underlying model. The usage of meta-models would prove useful because maturity models for PM are described in different languages using various terminologies. They may use various descriptions even though they imply the same underlying concepts. Since the purpose of the framework is not to determine superiority or inferiority between maturity models, it is inappropriate to select one model (and its terminology) as a starting point and compare the other models with it. Another reason why the framework should use a meta-modeling technique as a measure for structure is because it enables a simplified illustration of the maturity 20

24 models. One quick glance at a meta-model allows an organization to understand the hierarchy of concepts or activities of a maturity model for PM. In [38], the authors were able to compare different object-oriented analysis and design techniques with each other using a meta-modeling technique. According to the authors, the construction of meta-models is a uniform and formal way to compare methodologies with each other as objective as possible, provided that the same constructs are used to model them. In this thesis, the framework will employ a type of meta-models called Process-Data Diagrams (PDD) described in [39]. A PDD is made up of two different meta-models, namely a meta-process model and a meta-data model. As mentioned earlier, a maturity model for PM incorporates a maturity reference model and an assessment method. The two types of meta-models are suitable for modeling each of the two parts. More specifically, the meta-process model will be used to depict the process phases and activities of an assessment method, while the meta-data model models the concepts underlying the maturity reference model. After creation, these two metamodels will be combined to create a PDD in which the relationships between the activities of the assessment process and the concepts of the model are shown. PDDs are able to answer the following three questions for each maturity model for PM: - What process phases and activities is the assessment method made up of? (using the meta-process model) - What products do the activities of the assessment method deliver? (meta-data model) - What concepts underlie a maturity model for PM, and how are they related to each other and the products of the assessment method? (process-data diagram) A condensed description of the PDD modeling technique and notations employed by the framework is provided in the Appendix section (see Appendix B). A more thorough explanation of this method can be found in [39]. 2

25 3..2. PDD comparison method The comparison between the PDDs follows the same approach described in [28]. This approach begins by defining all concepts and describing all activities depicted in the PDD of each maturity model. Each maturity model for PM can use a different terminology, so a thorough understanding of the definitions of the activities and concepts is needed before comparisons can be made with other models. The definitions are shown using activity tables and concept tables. They are derived from publications and literature about the maturity models. People who have had experience with a particular maturity model are also consulted to gain more and to verify the definitions. After all activities and concepts are defined, they are used to create two additional tables: an activity comparison table and a concept comparison table. The activity comparison table consists of a consolidated reference list of the activities of all maturity models that are selected to test the framework. This means that overlapping activities of the maturity models are combined and non-overlapping ones are added to the table. The activity comparison table also contains consolidated process phases of the assessment methods. Similarly, the concept comparison table contains a consolidated reference list of concepts of the selected maturity models. These tables are used for the actual comparison of the activities and concepts of the selected maturity models. This is done by filling in the fields in the Maturity model columns using the symbols explained below. For the activity comparison table, filling in: - = means that an activity of the reference list is present and also equivalent to the one in the PDD of a maturity model - > or < means that an activity of the reference list is present and comprises respectively less or more than the one in a PDD - a (2) after one of the previous symbols means that an activity is described in the second process phase of a PDD. 22

26 For the concept comparison table, filling in: - = means that a concept of the reference list is present and also equivalent to the one in the PDD of a maturity model - the name of a concept means that the concept in the reference list is present in a PDD but under a different name. - REPORT means that a concept of the reference list is present but not depicted in the PDD for the sake of space and structure. And if the concept of the reference list was depicted, it would be described within the REPORT concept. Finally, an empty field means that an activity or concept in the reference list is not present in the PDD of the respective maturity model for PM. Examples of comparison tables resulting from the analysis are depicted below: Table 3: Example Activity comparison table. Process phase Maturity Maturity model model 2. Activity. = <.2 Activity.2 < (2) >.3 Activity.3 = 2. Process phase 2 Maturity Maturity model model 2 2. Activity 2. < < () 2.2 Activity 2.2 > 2.3 Activity 2.3 > = Table 4: Example Concept comparison table. Process phase Maturity Maturity model model 2. Concept. CONCEPT =.2 Concept.2 = CONCEPT.3 Concept.3 = = 2. Process phase 2 Maturity Maturity model model 2 2. Concept 2. CONCEPT3 2.2 Concept 2.2 = 2.3 Concept 2.3 CONCEPT5 A comparison table allows a quick overview of what activities or concepts are present in the PDD of a maturity model for PM and what the main differences are with other models. It should be noted that only the concepts related to the assessment activities are compared using a concept comparison table. The core concepts underlying a maturity model are depicted as gray boxes in the respective PDDs and are compared using narrative text instead. This was decided because a concept comparison table 23

27 only compares the presence/absence of concepts in PDDs. However, whether one maturity model embodies more or fewer concepts than another one is not relevant. More important is to look how different maturity model are actually built-up. This is easier to describe using narrative text than a comparison table. 24

28 3.2. Applicability Dimension The second stage by the framework focuses on the applicability dimension of maturity models for PM. This dimension is selected because it sheds light on the properties of a maturity model for PM that affects the context in which the model can be applied. It is complementary to the previous dimension since it elicits information about a maturity model that cannot be depicted by a PDD. Besides the structure of a maturity model, it is important to know about properties that determine whether a maturity model for PM is suitable for an organization to adopt. For instance, a maturity model for PM may contain properties that limit its application in certain industries. A PDD is not capable of showing this piece of information. Eventually, the decision was made to use evaluation criteria to measure relevant properties of maturity models for PM. Criteria are appropriate measures to use for this dimension because of their flexibility. Each criterion can capture one property independently from each other, and this is useful especially when the list of criteria is not definitive. It allows the addition or removal of criteria without affecting other criteria on the list Evaluation criteria To decide which criteria were relevant to measure the applicability dimension, several sessions with the project group members were held. During these sessions, the list of criteria was changed and refined numerous times due to differing knowledge levels about maturity models for PM. Because of the limited time, the project group settled for a list of criteria shown in the tables below (see Table 5 and Table 6). The criteria are categorized into two main groups: criteria regarding the maturity reference model and those that focus on the assessment method. Table 5: Maturity reference model criteria Nr. Maturity reference Model (MM) criterion MM Openness MM2 Industry & size MM3 Scope MM4 Maturity level description MM5 Dimensions of maturity MM6 Process areas MM7 Process area dependencies 25

29 Table 6: Assessment method criteria Nr. AM AM2 AM3 AM4 AM5 AM6 AM7 Assessment Method (AM) criterion Assessment commitment Competence level Assessment method description Data gathering method Length of questionnaire Supportive assessment tools Benchmarking Because the purpose of the evaluation framework is descriptive rather than normative, no scores or ranks will result from the evaluation based on the criteria. For this reason, the findings per maturity model will be presented in the format shown in Table 7. The value indicates that a maturity model meets a criteria and indicates otherwise. The Reference column contains references to the information sources used and the Explanation column provides a brief explanation of the findings. Table 7: Example evaluation results format per maturity model Criteria Aspect Value Reference Explanation Maturity model criteria Criterion A Reference # <description of findings about criterion A in reference #> Criterion B Aspect ba Reference # <description of findings about aspect ba in Aspect bb Reference #2 Reference # and #2> Besides books and publications, assessors, experts and accredited associations are consulted to gather and verify information. Experts and assessors are people who have had experience with a maturity model for PM before during maturity assessments. Accreditation associations are organizations qualified to train and dispatch consultants to conduct maturity assessments for other organizations. These associations may also provide training sessions to those who wish to be certified assessors or those who want to know more about a particular maturity model for PM. Each criterion is defined and explained below to clarify the property it measures, the reasons why this property is relevant and how it is used to measure this property. 26

30 Maturity reference Model (MM) criteria MM - Openness Openness is the degree to which a maturity model for PM is available for public and whether its usage is limited to certain individuals or organizations. This criterion employs four possible situations that have significant implications for the openness of a maturity model. It measures whether a maturity model for PM and/or assessment materials: - can be accessed by without payment (free access) - can be accessed against payment (paid access) - is meant to be used by certified assessors (certified usage) - can only be used by specific organizations (proprietary access & usage) These situations are not independent of each other. A maturity model can, for instance, be made openly available while its assessment can only be done by certified assessors. The resulting schema will indicate whether the four situations hold for a maturity model and assessment. In de Explanation column, additional information will be provided for the values found. MM2 Industry & size Maturity models for PM can be specifically designed for a certain organizational context (e.g. type, structure, industry sector). Therefore, it is important for a model description to contain information about its applicable areas. This criterion measures whether a maturity model specifies limitations in its application to organizations operating in particular industry sectors or to organizations of specific sizes. MM3 - Scope As explained before, the discipline of PM includes the domains of project management, program management and portfolio management. Because of the differences between projects, programs and portfolios, the corresponding best practices and processes will differ as well. The scope of a maturity model for PM is the extent to which a maturity model embodies PM. Depending on the domains employed, a maturity model may be structured differently and describe different processes to improve in. One maturity model for PM may describe all three domains 27