Designing and validating an instrument to measure the degree of customization and cost of use and maintenance for a packaged ERP system

Transcription

1 Eindhoven, February 2010 Designing and validating an instrument to measure the degree of customization and cost of use and maintenance for a packaged ERP system By A.S. EHSARY, (Ali) BSc Industrial Engineering and Management Science TU/e 2007 Student identity number in partial fulfilment of the requirements for the degree of Master of Science in Operations Management and Logistics Supervisors: dr.ir. R.M. Dijkman, TU/e, IS prof.dr. R.J. Kusters, TU/e, IS drs. D. Zijlstra, Accenture, SI&T Consulting

2 TUE. Department Technology Management. Series Master Theses Operations Management and Logistics Subject headings: Information systems, Maintenance Costs, Measurement instrument ii

3 Abstract Standard Enterprise Resource Planning (ERP) packages are often customized by the adopting organizations in order to close the gap between the functionalities offered by the package and the specific needs of the organization. When customizing these packages, most organizations only take into account the extra costs during the implementation phase and neglect the ongoing use and maintenance costs. A reason for this is the lack of a good estimation tool which can be used to estimate the expected use and maintenance cost as a function of the degree of customization (DOC). To develop such a tool, data should be gathered from practice on degree of customization and cost of use and maintenance but in order to be able to do that first a validated measurement instrument is needed. This research focuses on developing and validating such an instrument which can be used to quantify the DOC and estimate the cost of use and maintenance. Two case studies are conducted: one to design the instrument and one to validate it. iii

4 Summary Standard Enterprise Resource Planning (ERP) packages developed by vendors such as SAP and Oracle are designed to support standard business processes. However, since every organization has some uniqueness in their processes at a certain level, there is always a gap between business and IT. This gap could either be closed by customizing the ERP package (IT) or changing the existing business processes. To avoid costly change management practices, most organizations choose to customize the ERP package and while doing that they only account for the extra costs during the implementation and neglect the costs of using and maintaining the package. This research is carried out at Accenture Consulting and focuses on the relation between the cost of use and maintenance for a specific ERP package, PeopleSoft HRMS, and the degree to which the package is customized. The following problem statement is formulated: There is a lack of insight into the relation between the degree of customization to the HRMS package and its use and maintenance costs after the implementation. Moreover, there are no tools which can be used to: (1) quantify the degree of customization (DOC) and (2) estimate the associated cost of using and maintaining the customized HRMS package based on the DOC. Taking the above stated problem into account the following research goal is formulated: Designing a measurement instrument which can be used to: 1. Quantify the extent to which the ERP package is customized and 2. To estimate the cost of using and maintaining such a package, with the objective of gathering data from practice to do research on the exact relation between the degree of customization and the associated costs of use and maintenance so that the research results can be used to develop a tool, similar to existing tools at Accenture, which can be used by Accenture consultants to estimate the cost of use and maintenance for a certain level of customization. To design the instrument, knowledge from the literature is combined with findings from a case study which is conducted at a Dutch Regional Education Centre (ROC) and validated by conducting an additional case study at an Academic Hospital in Leiden, Netherlands (LUMC). The measurement instrument consists of two frameworks: one for estimating the cost of use and maintenance and one for quantifying the degree of customization. Figure 1 represents the first framework. From the figure we can conclude that there are seven main use and maintenance activities which bring costs along three different dimensions (people, process and products&services) and that these costs can be estimated by using several sources of information within an organization. iv

5 Figure 1: Measuring cost of use and maintenance To quantify the degree of customization for a specific system, the amount of development time is taken as a measure. The total degree of customization is thus the total amount of development time spent on customizing the ERP package. The DOC can be estimated in two ways: either by simply summing up all the effort and calculating the total development time in hours or using the matrix of figure 2. Customization types Reports Interfaces Extensions Conversions Workflows v Complexity Simple Medium Complex Very complex 0 < dt 56 i = 31 0 < dt 44 i = 33 0 < dt 44 i = 21 0 < dt 90 i = 56 0 < dt 57 i = 49 Figure 2: Measuring the degree of customization 56 < dt 84 i = < dt 116 i = < dt 103 i = < dt 212 i = < dt 71 i = < dt 157 i = < dt i = < dt 227 i = < dt i = < dt i = < dt i = < dt i = 281

6 The matrix can be used to estimate the development time based on experts judgment on how complex a piece of customization is. First it should be evaluated to which of the five types the customization belongs (vertical axis of the matrix) and then how complex it is using the four categories: simple, medium, complex and very complex (horizontal axis of the matrix). To assess the complexity the assessor can also make use of the ranges in development time (dt) for each complexity level. Then, depending on which cell is pointed out in the matrix the complexity index (i) can be used to calculate the total degree of customization which is the sum of all the complexity indices: Where CI i is the complexity index of the i-th customization. Figure 3 presents the findings from the two case studies conducted at ROC and LUMC. As we can see the differences are large for people and products&services dimensions but there is almost no difference in costs along the process dimension ROC (DOC = 2120) LUMC (DOC = 3911) ROC (DOC = 2120) LUMC (DOC = 3911) Process People Products & Services Figure 3: Findings from the two case studies vi

7 Table of Contents Abstract... iii Summary... iv Table of Contents... vii 1 Introduction Background Standard ERP systems Misfit between ERP and business Customizing ERP to resolve misfits ERP lifecycle Definition of ERP maintenance Research context Problem statement Research goal Report overview Research approach Literature review Analysis of Accenture methodology Case study research Designing the measurement instrument Validation process Analysis of results Framework for measuring cost of use and maintenance Findings from literature Maintenance categories Organizing maintenance Cost dimensions Findings from ROC case study ERP maintenance at ROC Eindhoven vii

8 3.2.2 Collecting data on cost of maintenance activities Measurement instrument for cost of use and maintenance Cost of people Cost of products and services Cost of processes Conclusion Framework for measuring the degree of customization (DOC) Findings from literature Customizing ERP packages Customization implications Quantification of the degree of customization (DOC) Developing the DOC measurement framework Development time as a metric to quantify the DOC Findings from ROC case study Design process Measuring the DOC in practice Conclusion Validation of the measurement instrument Construct validity Content validity External validity Academic Hospital Leiden (LUMC) Measuring cost of maintenance at LUMC Measuring the DOC at LUMC Conclusion Findings General findings ROC Eindhoven Cost of maintenance viii

9 6.2.2 Degree of customization LUMC Cost of maintenance Degree of customization Conclusion Conclusions and future work Conclusions Cost of maintenance Degree of customization Findings Future work References Appendix A Customization objects in PeopleSoft Reports... Error! Bookmark not defined. Interfaces... Error! Bookmark not defined. Extensions... Error! Bookmark not defined. Conversions... Error! Bookmark not defined. Workflow... Error! Bookmark not defined. ix

10 1 Introduction Standard ERP systems are off-the-shelve software packages designed to support typical business processes within a whole field or industry. Unlike tailor-made information systems, ERP systems are based on best practice processes from the specific industry. Since these ERP systems are standard and organizations have some uniqueness in their processes on a certain level of detail, there is often a misfit between the ERP system and the adopting organization s business processes. Academic literature provides two main kinds of solutions to resolve these misfits. Either the organization should adopt the business processes on which the ERP package is based or the ERP package should be tailored to meet the organizations demand. Tailoring the ERP package can vary between simply configuring the system in order to choose between different executions of processes and functions to changing the software code. While configuring the system is seen as normal by the ERP vendors, all the other changes or customizations are discouraged due to limitations and increasing costs concerning maintenance, future updates and migrations. Despite this, the occurrence of customizations in ERP implementation projects is very common. However, a large number of companies which customize their ERP, fall back to a new version without any customization at all (also called the vanilla system) because of cost overrun during the usage period of the previous version. One of the hypothetical reasons for this is that the costs of using and maintaining a customized system turns out to be much higher than most organizations anticipated during the implementation phase. Taking the above in mind the conclusion can be drawn that there is a lack of insight into the use and maintenance cost of a customized ERP package and especially into the relation between the degree (or extent) of customization and the associated use and maintenance costs after the implementation. To explore this relation it is necessary to measure the cost of use and maintenance at one side and the degree of customization at the other side in practice and compare the cases with each other. But to be able to do that first it has to be defined what is exactly meant by customizations and the cost of use and maintenance and how these two can be measured. This research concentrates on developing a measurement instrument which can be used to measure the cost of use and maintenance and the degree of customization for an ERP package. The instrument is designed for a specific ERP package, namely PeopleSoft Human Resources Management System (HRMS) module of Oracle developed through a case study at a regional education centre and validated through another case study conducted at an Academic Hospital in Leiden. This introduction chapter is organized as follows. First some background information is given on the topic in section 1.1. Within the research context, which is described in sections 1.2, the problem statement is formulated in section 1.3 and taken the problem statement in mind the research goal is stated in section Background In this section some background information is given on ERP customization. First, it is explained what exactly a standard ERP system is and secondly what kinds of misfits could exist between 1

11 such a software system and the business processes of the organization adopting it. Finally, it is explained how a standard ERP software can be modified to resolve the misfits Standard ERP systems ERP systems are process aware information systems which support the operational processes of the whole enterprise by integrating and optimizing the processes and managing transactions throughout the whole organization (Moon 2007). According to Moon (2007), ERP finds its origin in the manufacturing which needed an information system that was capable of calculating more efficiently materials required through the whole manufacturing process as well as enabling efficient use of manufacturing capacity and optimizing scheduling (hence the terms Resource and Planning). Soon after these kinds of information systems proved their value for manufacturing, companies began to realize that their functionality could be extended beyond manufacturing and encompassing finance, sales and distribution and human resources (Klaus et al. 2000). To avoid major development costs, during the last decades, more and more companies choose to purchase packaged ERP software resulting into the rise of major ERP vendors such as SAP and Oracle. In order to be able to earn back the large development costs these vendors needed to sell as many software packages as possible and therefore standardize the software as much as possible. Although standard ERP packages are developed for specific processes (such as Human Resource and Supply Chain Management) and for specific industries (such as financial or manufacturing industry) and based on so called best practices within these fields, there still remains a certain misfit between these ERP packages and the business processes of the adopting organizations. This misfit is addresses in the next subsection Misfit between ERP and business Misfit between ERP packages and organizations is primarily caused by the fact that standard ERP systems are designed to support standard processes in a certain field whereas each organization has its own unique way of working, which is influenced by a combination of company specific, sector specific and country/culture specific factors (Soh et al. 2000). There are four types of misfits identified in the literature: Data, Functional, Output and Interface (Hong and Kim 2002; Soh et al. 2000): 1 Data misfit refers to incompatibilities between organizational requirements and ERP package in terms of data format and the relationships between entities as represented in the underlining data model. 2 Functional misfit can be divided into Access, Control and Operational misfit. Access misfits occur when the required access to perform certain tasks is not present, control misfits refer to missing validation procedures or checking routines and operational misfits refer to missing operational steps or the presence of certain steps which are inappropriate. 3 Output misfit refers to incompatibilities between requirements and ERP package in terms of the information content and presentation format of the output and. 2

12 4 Interface misfit occurs when there is a gap between the way the graphical user interface (look and feel) of the ERP package is designed and what the users within the adopting organization are accustomed to work with Customizing ERP to resolve misfits Although standard ERP packages allow the adopting organization to configure the software to meet most of their specific needs, there always remain some misfits which cannot be resolved by simply configuring the package. In order to resolve these misfits, there are basically two things that an organization can do: either changing their processes to better match the software package or modifying the software to meet their specific needs. The latter is referred to as customizing the ERP package ERP lifecycle There are four basic stages within an ERP systems lifecycle which can be distinguished from each other (Brehm and Markus 2000; Esteves and Pastor 1999; Markus and Tanis 1999): Acquisition: in this phase the decision whether or not to buy a packaged ERP system is analyzed as well as which package would be the most suitable one for the organization. Implementation: in this phase the package is actually configured and implemented. All the decisions regarding whether, what and how much to customize are made in this phase. Use and maintenance: during this stage the system is actually used and maintained. This phase is or at least ought to be the biggest stage in the ERP systems lifecycle since in this phase the organization actually benefits (or should benefit) from the ERP system and there is return on investment. Evolution: after using the ERP package for a while, internal and external factors might cause the system to evolve either to a complete new package or an upgrade of the existing one. Internal factors could be anything from changing the processes within the organization or changes in the IT environment of the ERP package whereas external factors could be the opportunity to upgrade to a better system with more functionalities or the expiration of support from the vendor for the current version Definition of ERP maintenance Maintenance of traditional software systems is defined by the IEEE as the modification of a software product after delivery to correct faults, to improve performance or other attributes, or to adapt the product to a modified environment (IEEE 1990). This definition points out that maintenance is a post-delivery activity and comprises all the tasks necessary to correct faults, improve performances and to adapt to changed environment. Correcting, improving and adapting could either be done to the software code, functionality or the hardware on which the software system is running. Since there are significant differences between standard ERP packages and traditional tailormade software systems (Klaus et al. 2000), this definition should be adjusted for ERP. Ng et al. (2002) state that ERP maintenance involves all post-implementation activities related to the 3

13 packaged application software undertaken by the client organization from the time the system goes live (i.e. successfully implemented) until it is retired from an organization s production system: - To keep the system running, - To adapt to a changed environment in order to operate well, - To provide help to the system users in using the system, - To realize benefits from the system, and - To keep the system a supported-version and meet the vendor s requirements for standard code. (Ng et al. 2002) 1.2 Research context This research is carried out at a company called Accenture. Accenture is a global management consulting, technology services and outsourcing company which among other things offers consulting services during ERP implementation projects. One of the various ERP modules for which Accenture offers consulting services is the PeopleSoft Human Resource Management System (HRMS). The research is carried out within the Service Integration and Technology (SI&T) department of Accenture Netherlands. The sub department Oracle within SI&T is responsible for implementation projects of Oracle modules and the focus group Oracle HR focuses specifically on the PeopleSoft HRMS package and has implemented this package with and without customizations in various companies in the Netherlands. The HR focus group is mainly interested in a tool which enables their consultants to predict the expected cost of using and maintaining the HRMS module for client companies who are considering customizing the package to a certain extent. PeopleSoft HRMS was originally developed by the company PeopleSoft Inc, which was specialized in human resource management, customer relationship management and student administration systems. In 2005 PeopleSoft was taken over by Oracle Corporation; the PeopleSoft name and product line are now marketed by Oracle, who also is responsible for developing maintenance patches and upgrades for the software. Table 1 provides an overview of HRMS versions released during the last decade along with the availability of upgrades and patches for each version. Since Oracle does not release any upgrades and patches for older versions than HRMS 8.9, it is likely that most companies currently use version 8.9 or higher. 4

14 Table 1: Overview of PeopleSoft HRMS versions Version Release Updates, Fixes, Security Alerts, and Upgrade Scripts Tax, Legal and Regulatory Extended Support Ends Sustaining Support Ends HRMS 8.0 SP1 Dec 2000 Mar 2007 Mar 2007 Not Available Indefinite HRMS 8.3 Nov 2001 Mar 2007 Mar 2008 Not Available Indefinite HRMS 8.8 Dec 2002 Dec 2007 Dec 2008 Dec 2011 Indefinite HRMS 8.9 Dec 2004 Dec 2009 Dec 2010 Dec 2012 Indefinite HRMS 8.9 ANZ Sep 2005 Sep 2010 Sep 2011 Sep 2013 Indefinite HRMS 9.0 Dec 2006 Dec 2011 Dec 2012 Dec 2014 Indefinite 1.3 Problem statement Since standard ERP systems, like the HRMS package, do not fully meet the requirements of all the companies using it, there is always a gap between the ERP functionality and the business processes of the companies. To fill this gap, the adopting organizations have the option to adjust their processes and/or to tailor the ERP system (Ehsary 2009). According to Accenture consultants, most organizations choose to customize the ERP system and this choice is often based on cost and benefit analysis concentrating on only the implementation phase. As long as the cost of customizations fit within the implementation budget, companies mostly choose to customize the ERP system in order to avoid difficult change management practices. But what often is neglected are the update and maintenance costs incurred after the implementation phase, during the rest of the ERP system s lifecycle, as a result of these customizations. Figure 4 shows hypothetically the difference in total lifecycle costs between a vanilla- and customized system as it is expected to be according to Accenture consultants. The figure demonstrates that although these costs may not seem much at the implementation phase, during the systems usage period the costs could dramatically increase for a customized system because of implications during update and maintenance activities. Moreover, the more complex the customizations are, the more dramatically the costs are expected to increase at a certain point, which results into exceeding the allocated budget for the ERP package before the accounted depreciation period is reached. 5

15 Figure 4: expected total cost of vanilla system versus customized system To be able to offer better services to the customers and to help them avoid major cost overrun during the lifecycle of the (customized) HRMS package, Accenture consultants need to gain more insights into the exact relation between the complexity of customization and the use and maintenance costs of the HRMS package. So to conclude, Accenture consultants face the following problem: There is a lack of insight into the relation between the degree of customization to the HRMS package and its use and maintenance costs after the implementation. Moreover, there are no tools which can be used to: (1) quantify the degree of customization (DOC) and (2) estimate the associated cost of using and maintaining the customized HRMS package based on the DOC. 1.4 Research goal Taking the problem statement in mind, the main goal of the project can be defined as: Designing a measurement instrument which can be used to: 1. Quantify the extent to which the ERP package is customized and 2. To estimate the cost of using and maintaining such a package, with the objective of gathering data from practice to do research on the exact relation between the degree of customization and the associated costs of use and maintenance so that the research results can be used to develop a tool, similar to existing tools at Accenture, which can be used by Accenture consultants to estimate the cost of use and maintenance for a certain level of customization. 6

16 The above stated research goal is further divided into the following sub goals: 1. Development of a framework which can be used to estimate the total cost of maintenance and use. 2. Validation of the framework for cost of maintenance and use through case study analysis 3. Development of a framework which can be used to quantify the degree of customization (DOC) for the PeopleSoft HRMS package. 4. Validation of the DOC framework through case study analysis 5. Analyzing the results of the case studies and drawing preliminary conclusions regarding the relation between DOC and cost of use and maintenance 1.5 Report overview This report is structured as follows. After this introduction chapter, in chapter 2 the research approach is explained in detail. Chapters 3 and 4 focus on the frameworks for estimating the cost of maintenance and quantifying the degree of customization respectively. The validation process for both frameworks is explained in chapter 5. In chapter 6 the findings from the case studies are discussed and chapter 6 concludes the report. 7

17 2 Research approach In this chapter the research approach is explained in more detail. Figure 5 represents the research process on a high level. As it can be noted from the figure, the starting point is the research goal as defined in section 1.4. Taking the research goal in mind, a literature review (2.1) is conducted to find out how literature can contribute when designing the instrument. Next to literature, also findings from a case study analysis and an analysis of currently used methodology at Accenture are used to design the measurement instrument. The analysis of Accenture methodology and the case study analysis are more thoroughly explained in sections 2.2 and 2.3 respectively. In section 2.4 it is explained how exactly the design process is carried out. The instrument is validated through an additional case study. This validation process is explained in more detail in Section 2.5. Finally, section 2.6 explains how the findings of the case studies are analyzed to draw preliminary conclusions on how the relation between the DOC and cost of maintenance looks like. Literature review (2.1) Research goal Analysis of Accenture methodology (2.2) Designing the measurement instrument (2.4) Validation through case study (2.5) Analysis of results (2.6) Case study analysis (2.3) Figure 5: Research approach 2.1 Literature review In order to avoid reinventing the wheel it is important to first study relevant literature on ERP packages to see what already has been researched and how it can contribute to the design of the measurement instrument. Therefore, the research goal is used to define the following set of research questions for the literature review: Research questions on ERP maintenance 1. What is ERP maintenance and which types of ERP maintenance exist? 2. How can maintenance activities be organized? 3. What kinds of costs are involved with ERP maintenance? Research questions on ERP customization 4. How can an ERP package be customized? 5. What are the implications of customizations on (cost of) maintenance? 6. How can the degree of customization be quantified? 8

18 To answer the above stated questions, academic journals and books as well as research papers published by large commercial research organizations were consulted. Table 2 presents an overview of the used resources and search terms. Table 2: Overview of literature sources and search terms Sources Academic research libraries ACM Digital Library Elsevier ScienceDirect Emerald IEEE IEE Electronic Library (IEL) IngentaConnect JSTOR SpringerLink Wiley Interscience Commercial research organizations Gartner Research Burton Group Forrester Research International Data Group (IDC) Search term(s) ERP maintenance ERP application management Information systems maintenance Information systems management ERP lifecycle cost ERP maintenance cost Cost of IS maintenance ERP (mis)fit ERP modification ERP customization Tailoring ERP systems It should be noted here that these search terms were only used to get started and to find the most important research and surveys on these topics. A substantial part of the references used in this report are found by looking into the references used in the surveys. 2.2 Analysis of Accenture methodology Accenture consultants already use a methodology, called Accenture Delivery Methods (ADM), to estimate the amount of effort needed for ERP implementation projects. To determine the estimated effort, ADM also take into account the amount of customization a client needs or asks for. ADM tools provide a good starting point for designing the instrument which can be used to quantify the DOC and therefore it is wise to study them to see how they can be used in the design process. Another reason for first analyzing ADM tools is because of the terminology used in it. Since the goal of the measurement instrument is to collect data which can be used to develop a tool for Accenture consultants, it would be convenient to use the same terminology as in the already existing Accenture tools. The following research questions are used when assessing the Accenture methodology: 9

19 7. How is ADM developed and how does it work? 8. How are ADM tools used by Accenture consultants? 9. Which parts of ADM is relevant for this research? 10. How can ADM be used when designing the measurement instrument? 2.3 Case study research Next to literature and Accenture methodology, a third important source of information is conducting field research in order to be able to find out what kind of information is available in practice and how and where exactly within an organization one could gather data on cost of maintenance and degree of customization. Therefore, the following research questions are addressed by the field research: 11. How do organizations organize and execute ERP maintenance activities? 12. How can one gather data on cost of maintenance activities in an organization? 13. How do organizations keep track of the number and type of customizations in their ERP package? 14. How can one gather data on the degree of customization of an ERP package used in an organization? There are several ways to conduct field research such as experiments, surveys, workshops, focus groups, case study analysis etcetera; see (Cooper and Schindler 2001) for an extensive list. However in this research, we have chosen to conduct a case study because of two specific characteristics of case study research: (1) high complexity of the unit of analysis and (2) possibility to do in-depth contextual analysis (Cooper and Schindler 2001; Yin 2008). Since ERP packages are enterprise wide systems and people from different departments and with different expertise are maintaining and using the package, the unit of analysis, which is the organization using the ERP package, is very complex. Also, to answer the above defined how-questions, an in-depth contextual analysis is necessary and therefore a case study approach is used here (Yin 2008). The case study analysis is carried out at a Dutch education institution ( ROC Eindhoven ) which has implemented the PeopleSoft HRMS package and has been using it for a few years. To conduct the case study, methodology of Yin (2008) is used. Within the organization first an intake interview with the head of Application Management was carried out. The intake interview was not structured and its goal was to present the problem statement and research goal in order to determine together with the respondent which people and roles within ROC Eindhoven are responsible for maintaining the PeopleSoft HRMS package. After the intake interview, three semi-structured interviews were conducted later on with (1) the change manager, (2) PeopleSoft developer and (3) PeopleSoft HRMS functional application manager to answer the research questions stated above. Next to the interviews, also some documentation and archival records were provided by the respondents in order to support their answers and increase construct validity (Yin 2008). 10

20 2.4 Designing the measurement instrument The measurement instrument has been developed by answering the research questions defined in sections 2.1, 2.2 and 7; thus combining theory form academic literature and commercial papers with Accenture methodology and findings from a field research. The instrument basically consists of two frameworks which are elaborated in more detail in chapters 3 and 4: 1. Framework for measuring cost of use and maintenance of ERP packages (Chapter 3) 2. Framework for measuring the degree of customization (Chapter 4) Figure 6 depicts how the three sources of information are combined to design the instrument. As it can be noted from the figure, the answers to research questions 1-3, 11 and 12 are used to design framework 1 and the answers to research questions 4-6, 13 and 14 to design framework 2. Research questions 7 through 10 focus on how Accenture methodology can contribute in the design of the whole instrument and thus the answers to those questions are used for both frameworks (this is illustrated by the dashed line in the centre column between framework 1 and 2. Literature review Accenture methodology Field research 1. What is ERP maintenance and how are maintenance activities organized 2. What kinds of costs are involved with ERP maintenance 3. Which types of ERP maintenance exist? 4. How can an ERP package be customized? 5. What are the implications of customizations on (cost of) maintenance? 6. How can the degree of customization be quantified? 7. How is ADM developed and how does it work? 8. How are ADM tools used by Accenture consultants? 9. Which parts of ADM is relevant for this research? 10. How can ADM be used when designing the measurement instrument? 11. How do organizations organize and execute ERP maintenance activities? 12. How can one gather data on cost of maintenance activities in an organization? 13. How do organizations keep track of the number and type of customizations in their ERP package? 14. How can one gather data on the degree of customization of an ERP package used in an organization? Figure 6: Combining literature, Accenture methodology and field research findings to design the measurement instrument 2.5 Validation process In order to ensure the quality of the measurement instrument, its validity should be tested first. Two types of instrument validity are taken into account here: external and internal validity (Carmines and Zeller 1994; Yin 2008). External validity refers to the extent to which the 11

21 instrument is applicable in other situations, or in other words the extent to which the instrument is generalizable and transferable whereas internal validity refers to how well the instrument does what it is designed to do. The latter can again be divided into construct and content validity (Figure 7). Figure 7: Validity Construct validity refers to the extent to which the instrument measures what it is supposed to measure. In this study the construct validity can be determined by evaluating after the case study whether that what has been measured matches with what was intended to measure. So the following questions are taken into consideration: 15. Did the measurement framework measure the costs of maintenance for the specific ERP package? 16. Did the measurement framework quantify the degree of customization of the specific ERP package? Content validity ensures that the instrument measures everything that is relevant and leaves nothing out. To evaluate the instruments content validity, the findings are compared to that of similar researches in academic literature. The following questions are addressed here: 17. Are all relevant costs of maintenance for the specific ERP package taken into account? 18. Are all kinds of customization options taken into account? External validity of the instrument is addressed by conducting an additional case study and evaluating for each of them whether the instrument consistently measures what it supposed to measure. Thus the research questions are again evaluated, but now at a different organization using the same measurement instrument. To ensure the external validity we need to ask the following: 19. Are there any significant differences between the answers to the research questions when we compare the ROC case study with the LUMC case study? The validity of the instrument is addressed in chapter 5. 12

22 2.6 Analysis of results Once the instrument is designed and validated, the data collected during the case studies can be analyzed to see whether we can draw any conclusions regarding the relation between the cost of maintenance and the degree of customization. Based on the data gathered we try in this step to answer the following questions: 20. Are there any differences in the magnitude and composition of the total cost of maintenance for the ERP package between the two cases? 21. Are there any differences in magnitude and composition of the degree of customization of the ERP package between the two cases? 22. Can we conclude on the basis of the findings that there is a positive relation between the degree of customization and the total cost of use and maintenance? The answers to the above stated questions can be found in chapter 6 where the findings from the case studies are discussed. 13

23 3 Framework for measuring cost of use and maintenance In this chapter the framework for measuring the cost of use and maintenance of a standard ERP package is presented. First the findings from the literature are discussed in section 3.1. Section 3.2 covers the findings from the case study analysis and in section 3.3 the framework for cost of maintenance is presented. Section 3.4 finally concludes this chapter. 3.1 Findings from literature In sections through findings from the literature review are presented which cover the following research questions: 1. What is ERP maintenance and which types of ERP maintenance exist? How can maintenance activities be organized according to literature? What kinds of costs are involved with ERP maintenance? Maintenance categories With ERP systems being relatively new compared to traditional information systems, there is little scientific literature available about its use and maintenance stage. Therefore, in this subsection research on software maintenance in general together with that on maintenance of ERP systems in particular is reviewed to understand what kinds of activities typically are carried out in this phase. From the literature there are seven maintenance categories identified: - Corrective: refers to fixing errors in the program, fixing performance issues and repair implementation failures (Abran and Nguyenkim 1991; Burch and Grupe 1993; Ng et al. 2002; Swanson 1976) - Adaptive: refers to adapting the system to meet changing environment such as changes in software and hardware or business processes and regulations (Abran and Nguyenkim 1991; Burch and Grupe 1993; Ng et al. 2002; Swanson 1976) - Perfective: eliminating the programs inefficiencies or other imperfections such as source code readability and poor use of computer operator time (Abran and Nguyenkim 1991; Burch and Grupe 1993; Ng et al. 2002; Swanson 1976) - Preventive: proactive maintenance which consists of a periodic inspection and review of the system to uncover and anticipate problems (Burch and Grupe 1993) - User-support and training: supporting and training the end users in using the system (Abran and Nguyenkim 1991; Ng et al. 2002) - Upgrade: upgrading the software to a newer version which has been developed by the vendor with major functional and technical improvements (Ng et al. 2002) - Patch maintenance: implementing small patches distributed by the vendor in order to fix, problems, adjust to changes in regulations or to realize small functional and/or technical improvements (Ng et al. 2002) Organizing maintenance In order to be able to measure the costs of ERP maintenance activities it is important to find out how these activities are typically managed within an organization. Since ERP is an information 14

24 system (IS) and maintaining such a system is something which is part of the IS management field, we approach ERP maintenance from this viewpoint. IS management involves all the activities and tasks needed to design, implement and manage the various information systems within an organization. IS management can be approached from several different perspectives such as the nature of the tasks and the object(s) on which the task is performed, the states in which an IS can find itself, the specific processes carried out and the way IS management tasks are organized (Looijen 2004). The latter is relevant for our framework. Looijen (2004) links the state in which an IS can find itself (use, change and exploit) to task areas and assigns them to one of the three types of management: functional management, application management and infrastructure management as depicted in Figure 8 (Looijen 2004). Figure 8: Linking IS states to management areas (Looijen 2004) Since information systems, like ERP systems, are software systems (application) which run on hardware (infrastructure) and used to support business processes (functional) and since the software, hardware as well as the functionality of it has to be maintained, it can be concluded that ERP maintenance activities are performed in all these IT management areas (Figure 9). 15

25 Business Internal IT organization Users and business processes Functional management Application management ERP Maintenance activities Infrastructure management Organization Figure 9: Organizing maintenance, adopted from (Meijer and Smalley 2006) Combining the three IS management areas with the maintenance categories of section 3.1.1, Table 3 presents all the maintenance activities which are performed within an organization. Table 3: Maintenance and use categories combined with IS management areas Functional management Application management Infrastructure management Corrective - All activities needed to keep the ERP application running when problems occur All activities needed to keep the infrastructure running when problems occur Adaptive All activities needed to adjust the application to the business when changes occur in the business processes (new users, new procedures etcetera) or regulations All activities needed to keep the application running when something has been changed All activities needed to keep the infrastructure running when something has been changed 16

26 Perfective Changing the way ERP is used to perform business operations in order to gain more benefits from it Changing the application parameters to improve performance Changing the infrastructure to improve performance Preventive General periodic maintenance to anticipate on changes in the business environment General periodic maintenance to keep the application in good condition and anticipate on problems General periodic maintenance to keep the infrastructure in good condition and anticipate on problems User support and training Providing user support on functional level Providing technical user support - Training the users to work with the ERP module Upgrade Activities needed to arrive at the functional design for the upgrade Technical design and implementation of the upgrade Adjusting the infrastructure to support the upgrade Patch maintenance All activities regarding the decision of implementing a vendor patch or parts of it Implementing a vendor patch or parts of it Adjusting the infrastructure to support the patches (if necessary) Cost dimensions The costs of the use and maintenance activities, identified in the previous section, manifest themselves in different forms. When designing an instrument to measure the total costs of use and maintenance it is important to first identify how these costs manifest themselves. Literature on cost of ERP and packaged application maintenance, as well as literature on total cost of ownership (TCO) for such systems, is consulted to see how costs were measured in earlier studies. It was found out that there are basically three different cost dimensions, namely: 1. Products & services: all costs for which an organization receives an invoice and has to pay in monetary units (Gable et al. 1998; Somers and Nelson 2004) 2. People: cost of effort spent by the internal IT personnel on maintenance activities (Abts et al. 2000; Boehm 1981; David et al. 2002; Somers and Nelson 2004) 17

27 3. Processes: indirect costs at the business side as a result of downtime due to planned or unplanned ERP maintenance activities; this also includes time spent on training by endusers. (David et al. 2002) 3.2 Findings from ROC case study The goal of the case study research was to find out: 11. How do organizations organize and execute ERP maintenance activities in practice? 12. How can one gather data on cost of maintenance activities in an organization? The case study was conducted at the Regional Education Centrum (ROC) Eindhoven. First an intake interview took place with the head of application management to understand how maintenance is organized within ROC and to identify which people and roles were involved in conducting maintenance activities specifically for PeopleSoft HRMS. Then, depending on the specific function and responsibility within the organization, for each identified respondent a different list of questions was prepared to conduct the semi structured interviews. The maintenance activities as listed in Table 3 were taken as a basis to formulate the questions. Next to the above stated two research questions also two research questions regarding the degree of customization were built into the interviews but those findings are discussed in the next chapter ERP maintenance at ROC Eindhoven HRMS maintenance activities are organized within ROC as depicted in Figure 10. All maintenance activities are divided over the three areas: functional management, technical application management and infrastructure management. The functional managers are dedicated to the HRMS package, within PeopleSoft technical management there is only one PeopleSoft developer which is responsible for changes in PeopleSoft HRMS as well as PeopleSoft Campus Solutions (CS) and the people within infrastructure management are not dedicated to any application at all. Since the latter makes it very difficult to measure the amount of effort spent on HRMS maintenance by Infrastructure management, it was decided to leave infrastructure (right column in Table 3) out of the framework. The following roles within ROC were identified as people responsible for maintaining the HRMS package and thus interviewed during the case study: - Change manager: responsible for managing all modifications to applications. - PeopleSoft developer: responsible for technical design of the modifications to PeopleSoft HRMS and CS. - PeopleSoft HRMS functional application mangers: responsible for providing functional support to HRMS users as well as performing functional maintenance activities such as training and functional design of the modifications to the HRMS package. Maintenance activities to HRMS package are, with the exception of user support and training, performed through change requests. A change request can either by initiated by end users 18

28 because of changes at the business side or by the application management department because of changes at the IT side (corrective, adaptive, preventive and perfective maintenance from Table 3). For each change request first the functional managers make a functional design which is then built by the PeopleSoft developer and again tested by the functional managers before it is implemented by the database administrator (DBA). Figure 10: HRMS maintenance at ROC Eindhoven Table 4 presents an overview of the maintenance activities which are performed for the HRMS package. The table includes almost all the maintenance activities except patches since ROC does not actively implement official Oracle patches. Whenever necessary, parts from the official patches are implemented into the system through a change request. 19

29 Table 4: Overview of maintenance activities at ROC Eindhoven PeopleSoft HRMS Functional application manager PeopleSoft developer Database administrator (DBA) Modifications through change requests (adaptive, corrective, perfective and preventive maintenance) Minor changes to the application Assessment of change requests Functional design of the modifications Testing the modifications built by the PeopleSoft developer Building the functional designs of the functional manager Testing the modifications Solving problems (bug fixes) Implementing the modifications Solving problems (bug fixes) User support Providing user support whenever necessary Training Training the users whenever new functionalities are added Upgrades and extension of used functionality Functional design Technical design and testing Collecting data on cost of maintenance activities The first goal of the case study was to identify (or verify) which maintenance activities are performed from the ones mentioned in the literature and how these activities are organized. The second goal of the case study was to see how we can gather data on cost of maintenance. 20

30 During the case study analysis, it became clear that maintenance activities involving infrastructure were impossible to ascribe to a single application, such as HRMS package, since the hardware is mostly shared with other applications. Also the people responsible for infrastructure maintenance are not committed to a single application. Besides, the respondents agreed that infrastructure maintenance activities are unlikely to be affected by the amount of customization in the application software. Therefore, taking this in mind, infrastructure maintenance is left out of the framework for cost of use and maintenance. For the remaining maintenance activities from Table 3 (the columns functional and application management), for each of the three cost dimensions products&services, people and processes it is identified which sources of information should be used. To gather data within an organization two kinds of sources can be distinguished: the knowledge available in the information systems (either or not automated) and the knowledge of people within the organization. Taking the maintenance activities and cost dimensions as a basis the following sources of information is identified in the case study analysis for measuring the cost of use and maintenance: Information systems: Incident and Problem management system, change management, Time reporting system, procurement system, financial system. People and roles within organization: Functional application manager, Technical application manager, application developer, Purchase/contract manger. 3.3 Measurement instrument for cost of use and maintenance Combining the findings from literature and ROC case study and the three cost dimensions, Table 5 was constructed. The total cost for maintaining the PeopleSoft HRMS package can be obtained by evaluating for each maintenance category the cost along the dimensions Products & services, People and Process. The overview is basically a matrix with the use and maintenance activities which are taken into account along the vertical axis and the cost dimensions along its horizontal axis. To obtain a complete view of the total cost of use and maintenance of the ERP system, all of the boxes in the matrix should be evaluated. 21