Working Papers Management Sciences Gr Behavioral factors influencing performance management systems use

Transcription

1 Working Papers Management Sciences Gr Behavioral factors influencing performance management systems use Taco Elzinga Shell International Exploration and Production Kesslerpark 1, 2288 GS Rijswijk, The Netherlands Tel ; t.elzinga@shell.com Bé Albronda (corresponding author) Faculty of Management Sciences (MW) Open University of the Netherlands (OUNL) P.O. Box 2960, 6401 DL Heerlen, The Netherlands Tel: ; be.albronda@ou.nl Frits Kluijtmans Faculty of Management Sciences (MW) Open University of the Netherlands (OUNL) P.O. Box 2960, 6401 DL Heerlen, The Netherlands Tel: ; frits.kluijtmans@ou.nl Abstract The purpose of this paper is to substantiate conclusions from De Waal s (2002) case study research about the role of behavioral factors in the use of performance management systems. De Waal s exploratory research is replicated in four organizations. Data were collected through questionnaires, interviews and document research. Data of the additional case studies were combined with De Waal s. Pattern matching was used to compare all case studies. De Waal s initial research design was extended through the inclusion of a number of sensitivity analyses in the pattern matching exercise. The behavioral factors in the use of performance management systems De Waal deemed significant appear rather arbitrary, due to methodological choices made during the pattern matching and data analyses. Sensitivity analyses shows that behavioral factors can be ranked according to their relative importance. Although the addition of sensitivity analyses has shed some light on the relative importance of behavioral factors, there still remain unanswered questions with regard to the greater validity and reliability of De Waal s original research design. Some assumptions made have to be studied in more detail to strenghten findings of the research. Keywords: Performance management systems, management behavior, behavioral factors

2 Behavioral factors influencing performance management systems use Introduction In recent years, it has become clear that behavioral factors play an important role in the successful use of a performance management system (PMS) in organizations. Nevertheless, little research on the subject has been done to date. De Waal (2002, 2003, 2004) is one of few authors who has studied the subject in detail. De Waal based his exploratory research, Quest for balance, on four case studies in three Dutch companies. He identified 18 behavioral factors that seem to be important for the regular use of a PMS by the management of a company. To substantiate his conclusions, additional studies are required. Our research, in which we replicate De Waal s study in four more organizations, is a first step to contribute to that goal. The results from our additional case studies were not particularly consistent, neither between these studies themselves, nor (even more importantly) with the original findings of De Waal. This poses the question how these rather confusing results can be explained and what impact the research set-up could have had on the results. In the next section we will give an overview of the literature on the subject of the influence of (behavioral) factors on the use of PMSs. In the third section we present an overview of De Waal s results. In this section we also describe how we have replicated his study, and in which organizations the additional studies were carried out. 1

3 In section 4 we compare the results of our case studies, and integrate these with De Waal s results. Finally, in section 5, we try to give some explanations for our puzzling results. De Waal s research emphasizes the important, but underexposed role of behavioral factors for the effective use of performance management systems. Our aim was to extend his study and to substantiate its conclusions. However, our results compel us to reconsider the methodology in use and to propose an alternative route to obtain better (read more valid and reliable) results. Literature review In recent years, an increasing number of companies have implemented performance management systems (PMS) that are based on critical success factors (CSF) and key performance indicators (KPI). A frequently used framework in this context is the balanced scorecard (BSC). Surveys among corporate executives conducted by the consulting firm Bain and Company revealed that by 2004, 64 percent of their respondents in North America and 57 percent of the respondents worldwide were using BSCs in their companies (Rigby and Bilodeau, 2005). According to Kaplan and Norton (1992, 1993, 1996) senior executives should not rely on one set of measures to the exclusion of others, but they want a balanced set of financial and operational measures. They therefore proposed to base the BSC on four perspectives (financial, customer, internal business, and innovation and learning perspective) and defined a number of goals and measures for each perspective. 2

4 The above is supported by Lingle and Schiemann (1996), who describe the results of a study that confirms that companies who balance financial and non-financial measurements, but also link strategic measures to operational ones, update their strategic scorecard regularly and clearly communicate measures and progress to all employees, are better performers. Since Kaplan and Norton introduced their concept of the BSC as a framework for translating a company s strategic objectives and CSFs to financial and non-financial performance measures (KPIs), there have been many articles on this subject. Initially research focused on how to design a BSC. In the late 1990s the attention shifted to the implementation process, and since around 2000 the literature also addresses the use of PMSs, and to what extent organizations actually take advantage of their PMS and use the performance data to initiate management actions. However, despite the increase in experience with these systems, there is still a lot to be learned about the factors that influence their effective use. The literature generally agrees that a good PMS consists of a balanced set of factors that are critical for the success of a company, and a limited number of indicators that are a measure for organizational performance. But what are the characteristics of a good KPI and how should this measure of performance be defined? According to Neely et al. (1995) it has long been recognized that inadequately designed performance measures can result in dysfunctional behavior. From the literature they derived a list of 22 recommendations with regard to the design of performance measures, based on which they have developed a record sheet describing how companies should define their KPIs. 3

5 A number of authors (e.g. Globerson, 1985; Maskell, 1989; Bauer, 2004) have provided guidelines to select the preferred set of performance criteria, but the development of PMSs doesn t stop with the selection and definition of measures. Bourne et al. (2000) emphasize that the development process is not a simple linear progression from system design to the use of performance measures. Instead, the PMS requires developing and reviewing at a number of different levels as the situation changes. According to them a PMS should include a process or mechanism for 1) reviewing and revising targets and standards, 2) developing individual measures as performance and circumstances change, 3) periodically reviewing and revising the complete set of measures in use, and 4) challenging the strategic assumptions. In their discussion and conclusions, Bourne et al. state that the task of implementing and using a PMS is far from complete at the end of the design stage and that specifically the development of the use of the measures is a problem which has hardly been researched, and for which few tools and techniques are currently available. In a review of the literature on performance measurement in manufacturing organizations (operations), Lohman et al. (2004) concluded that most papers appear to deal with a green field situation, in which PMSs are designed more or less in isolation, independent from existing systems in the organization. Olsen et al. (2007) point to this as a normative direction, focusing largely on how an integrated PMS should be designed, as opposed to addressing the more realistic problem of how to design, implement and improve a PMS in its organizational context. Both Lohman et al. and Olsen et al. propose a change in the perspective of an integrated PMS from a design approach to a coordination approach. 4

6 Analoguously, Franco and Bourne (2005) state that a PMS cannot be fully understood using a narrow analysis perspective that only focuses on the single process of designing and/or implementing the system. Following Pettigrew s framework (Pettigrew, 1985), they propose a contextualist approach, in which the development of a PMS involves the identification of the contextual factors that influence or are being influenced by the system. Pettigrew s framework shows the factors that should be taken into account during a change management process, and contains three basic components: context, process (here PSM design, implementation and use), and outcome. Based on a literature review and interviews with a number of key professionals, Franco and Bourne (2003, 2005) have investigated which critical factors play a role in enabling organizations to effectively use their PMSs. They hereby focus their attention on what is called the knowing-doing gap (i.e. organizations struggle to transform performance information into effective improvement actions) and try to understand what differentiates companies that are able to manage through measures (i.e. using their performance management system effectively) from those that are not. In the end, they investigated 11 process factors and five context factors that facilitate a more effective use of a PMS (see table I). Based on the literature, among these 16 factors, the implementation ones seem to be crucial. 5

7 Table I. Critical factors (Franco and Bourne, 2005) Categories Factors Sub-factors Process Design PMS framework (like BSC) and strategy map Measures and targets Alignment and integration Information infrastructure Implementation Use Top management agreement and commitment The three E s: empower, enable and encourage Communication Review and update measures Data analysis, interpretation, decion-making and action-taking Rewards Performance measurement helping tools and management processes Context Internal Firm strategy Culture Organizational structure and size External Industry Environment What can be said about the benefits to the company after it has implemented a PMS? Lawson et al. (2003) have investigated 150 organizations of which two-thirds agreed that significant benefits had been realized from using a scorecard system. Feedback from these two-thirds indicates that implementing a scorecard system improves communicating the company strategy to the employees, that employees see the benefits of using a scorecard, that employees have become more aware of business plan goals and that the system has helped to align operational improvements with the overall strategy of the organization. They emphasize the need for a formal tie between scorecard and strategy and recommend 6

8 to implement this by assigning accountability for actions and placing appropriate measures on the scorecards of responsible people or groups of people. Lawson s research seems to confirm the outcomes of earlier studies on PMS benefits (e.g. Lingle and Schiemann, 1996; Hoque and James, 2000; Davis and Albright, 2004), in that companies perform better if they are managed using formalised, balanced and integrated performance measures (Bititci et al., 2006). On the other hand, research conducted by Ittner et al. (2003), Neely et al. (2004) and others, suggests that the use of PMSs does not make any difference to business performance. Still other authors (e.g. Braam and Nijssen, 2004) suggest that the impact of performance management is contingent upon the way it is used. Despite the fact that research outcomes seem to point in different directions, one observation turns out to be rather common: providing information on performance is not sufficient to improve business performance results (Nudurupati and Bititci, 2005). The real success of a PMS lies in how people use this performance information (e.g. Eccles, 1991; Davenport, 1997; Prahalad and Krishnan, 2002; Nudurupati and Bititci, 2005). Many authors believe that the main reason performance measurement in organizations is often short-lived is because of people s behavior with the information (Marchand et al., 2000; Bititci et al., 2002). However, the influence of user s characteristics on the use of PMSs has been underexposed in scientific and professional literature (Vagneur and Peiperl, 2000; Krause, 2000; De Waal, 2002). 7

9 In other words, behavioral factors are important for the successful use of a PMS (e.g. Lipe and Salterio, 2000; Malina and Selto, 2001). This subject is studied in detail by De Waal (2002), who turned away from performance management research with a focus on the technicalities of designing and implementing a PMS. His focuses on behavioral issues linked with the use of PMS. In his research, De Waal identified 18 behavioral factors that are important for the regular use of PMS. In a later article, De Waal (2004) has further developed his work by grouping behavioral factors that are important for the regular use of PMS into five areas (manager s understanding, manager s attitude, PMS alignment, PMS focus, and organizational culture), to which a company should pay special attention. There is no fundamental difference with conclusions from the 2002 studies, but rather a different way of presenting. De Waal s work is based on four case studies in three companies. Additional studies are required to substantiate his conclusions. Our study is a first step to contribute to that goal. Methodology: Quest for balance revisited In many organizations performance management systems (PMS) are introduced to improve performance in today s dynamic and turbulent environments. However, results are often disappointing. As we have seen in the previous section, it may not be the system itself that is the problem, but the actual use of it by management. The main focus of De Waal s study Quest for balance was on the human element in PMSs: which behavioral factors influence managers use of a PMS? 8

10 Based on a literature review, De Waal deduced behavioral factors that could have a positive or negative impact on the use of a PMS. All in all, he identified some 45 factors. Then he selected criteria which denote when the use of a PMS and its components (like critical success factors, key performance indicators, etc.) is valuable for management. This led to a total of seven so-called criteria for regular use. In order to determine the relationship between the behavioral factors and the degree of the use of a PMS, De Waal conducted a multiple case study, in which four Dutch organizations, both profit and not-for-the profit, were researched in depth. The description of the data collection and analysis process below closely follows De Waal (2002). Data were collected through the use of questionnaires, interviews and document research. The questionnaire focused on the purposes managers used the PMS for, and on their attitude towards the PMS. Interviews were held with key persons in the design and implementation stages of the PMS, and with users. A structured review list was used for the research of various written sources, like management reports, project documentation, and minutes of management team meetings. For each organization, the gathered information was integrated in a case study description. To determine whether or not an organization satisfied a particular behavioral factor, scores were awarded as follows. If the results for a behavioral factor from all data were positive, a plus (+) was given; if the results were all negative, a minus (-) was given. If the results were not clearly in one direction, a zero (0) was given. 9

11 Next, a final score for each of the stages (start, development, and use) was determined. This was done by calculating the average of all behavioral factors in a particular stage. If the average was below -0.2, the end result was denoted a being (-). For an average above +0.2, the end result (+) was given. For an average between -0.2 and +0.2, the end result was (0). A similar procedure was followed to evaluate whether or not the criteria for regular use were satisfied. Subsequently, final analyis of the data took place by comparing the case studies with each other. Pattern matching was used to identify those behavioral factors with the greatest impact on the use of PMSs. Patterns between the case studies were identified through a detailed examination of matches between the scores on individual behavioral factors, the end scores for the three stages, and the scores for the criteria for regular use. In his analysis, De Waal distinguished behavioral factors that show a complete match (a match between all four case study scores), a partial match (a match between two or three scores), and a non-match (a match between none or one of the scores). According to De Waal, behavioral factors that show a complete match seem to have a general similarity with a successful implementation and use of a PMS. Consequently, these factors can be considered essential, whereas behavioral factors with a partial match may be important for successful PMS implementation and use. All in all, De Waal identified 18 behavioral factors that play an important role in the use of performance management systems. 10

12 In an attempt to substantiate De Waal s conclusions, we replicated his research in four more Dutch organizations. Like De Waal, we selected both profit and not-for-the-profit organizations. Generally speaking our research in each organization followed De Waal s design. First a questionnaire was distributed to all or most of the managers who used the PMS. The first part of the questionnaire focused on the behavioral factors that could have played a role in the different stages (design, implemetation and use) of the system s development. The second part addressed the degree of manager s use of the PMS. Secondly, interviews were held with a few managers and with those people who were responsible for the development project. And finally document research should shed light on the reasons to introduce a PMS and on the course of the implementation process. Table II. Main characteristics of case studies A B C D Kind of IT service BSC since Kind of BSC PMS since Performance Management 2002 since with 18 system in use system (ITIL) KPIs since since 2004 Number of respondents (managers) Number of interviews Document yes yes yes yes study 11

13 In table 2 we give an overview of the main characteristics of each case study. For confidentiality reasons, we have indicated the organizations as case A to D respectively. Organization A is the common management organization of ICT systems of a Dutch ministry. More than 100 people are employed. Clients are owners of computer systems and applications, and end-users. Organization B is a business enterprise in home appliances. The company is the result of a recent merger of three separate organizations, till then competitors, and gives employ to 350 people. Organization C is the common maintenance organization of a division of the Dutch defense organization. More than 2300 mostly technical civilians are employed in this part of the organization. Organization D is an energy company and the case study is carried out in the business area that is responsible for oil and gas exploration and production. The results of these additional case studies were compared with the original findings of De Waal. At first sight, the results did not seem to be very consistent. For example, in our studies the number of important behavioral factors varied between 13 and 28, whereas De Waal identified 18 factors. These varying results raised some serious questions. Were the additional case studies too different from the original ones? Was the research method used too unreliable to generate consistent results? Or where there other factors that played a role? 12

14 With these questions in mind, we decided to combine the data of our additional case studies with De Waal s original ones, thus extending the data set from four to eight case studies. We then re-analyzed all data, following De Waal s pattern matching technique. In this way we changed the scope of our research. Apart from gaining a deeper insight into the relationship between behavioral factors and manager s use of PMSs, we now also looked for an explanation of the variability of the earlier results. Before discussing the details of our findings, it is important to also take note of a number of assumptions De Waal has made in his research, because, as will become clear later on, this has given additional insight into reasons for different results in different case studies. During the review of De Waal s results, a number of shortcomings in his work have been identified and therefore the decision was made to first assess the impact of these. Firstly, due to errors in his pattern matching analysis, not 18, but 20 behavioral factors should have been identified as important for the regular use of PMSs. Secondly, De Waal has drawn a number of firm conclusions, based on the assumption that behavioral factors that show a partial match have a 50-75% matching score (two or three out of four scores). But one can argue whether a match of 50% should be considered as a partial match. Why not a minimum of 75% (three out of four scores)? In that case, the conclusion would have been that only four out of the initial 20 behavioral factors would have been identified as important. 13

15 Therefore, we decided to include a number of sensitivity analyses in the pattern matching exercise, in which we varied the matching scores of behavioral factors. By so doing we extended De Waal s initial research design. Our analysis will be further described in the next section. Results In the previous section it was described that the additional case studies did not show a consistent result at first glance. Although a direct one-to-one comparison between the results of these case studies and De Waal s work could not always be done (due to the fact that some of the studies also introduced a number of new behavioral factors), one recurrent finding was that the number of important behavioral factors varied between studies. The reason for this inconsistency becomes clear when a sensitivity analysis is carried out on the pattern matching exercise. In De Waal s research a behavioral factor is called important for the regular use of PMS if a match of at least 50% (two out of four scores) is obtained. Based on this definition he concluded that there are 18 (20) important factors. In our study we have carried out pattern matching analyses for a number of definitions of a match and looked at the impact on the number of behavioral factors that are identified as important. The new analyses were done based on the total of eight case studies (De Waal s original ones and the ones described in the previous section). Pattern matching (following De Waal s methodology) has been applied to identify the behavioral factors that seem to be important for the use of PMSs. The results are given in Appendix A. 14

16 It turns out that the definition of a match (e.g. 50% or 75%) is of key importance for the number of important behavioral factors. The more stringent the definition, the less behavioral factors are identified as being important. The relation between matching criteria and number of important behavioral factors appears to be linear. The results are shown in figure 1. Number of important factors vs matching criteria number of important factors pattern matching criteria (%) Figure 1. Number of important factors vs matching criteria The 18 (20) factors De Waal concluded to be important for the regular use of performance management systems are, therefore, rather arbitrary because it is the consequence of methodological choices made during pattern matching and data analysis. One could find 20, 10 or even only six important factors, depending on the criteria selected. The conclusion from the analysis is that one cannot identify a single firm number of factors that are important for the use of PMSs. 15

17 A spin-off of the results can be obtained if we use the analyses to determine the relative importance of the behavioral factors. The results from the analyses with different matching criteria have been presented with increasing number of corresponding scores, i.e. from 25% to 75%. In this way one can clearly see which factors are important in all analyses, and which ones only in analyses with less strict matching criteria. The behavioral factors have been ranked in order of satisfying more stringent matching criteria. Behavioral factors that satisfy even the most stringent matching definition are obviously the ones most important for the regular use of performance management systems. Based on the eight case studies, the results are listed in table 3 below, ranked from most important for the regular use of PMSs on top to least important at the bottom. 16

18 Table III. Behavioral factors ranked in order of importance Behavioral Factor Managers realize the importance of CSFs/KPIs/BSC to their performance Managers accept the need for performance management Managers have earlier (positive) experiences with performance management 25% match 37.5% match 50% match 62.5% match 75% match Managers' frames of reference contain similar KPIs Managers are involved in making analyses Managers do not experience CSFs/KPIs/BSC as threatening Managers clearly see the promoter using the PMS Managers agree on the starting time Managers understand the meaning of KPIs Managers can influence the KPIs assigned to them Managers find the PMS relevant because it has a clear internal control purpose. Managers trust good quality analyses Managers and their controlling systems have mutual trust Managers agree on changes in the CSF/KPI set Managers are stimulated to improve their performance Managers are informed about the status of the PMS project Managers have insight into the relationship between business processes and CSFs/KPIs Managers' KPI sets are aligned with their responsibility area Managers understand the CSF/KPI/BSC reporting Managers find the PMS relevant because only those stakeholders' interests are incorporated that are important to the organisation's success. Managers trust the performance information Managers' results on CSFs/KPIs/BSC are openly communicated Managers have an active role during the development stage of the PMS project Managers are involved in setting KPI targets Managers accept the promoter Managers do not get discouraged by the collection of performance data Managers have insight into the relationship between cause and effect Managers' activities are supported by KPIs Managers' information processing capabilities are not exceeded by the number of CSFs/KPIs Managers can use their CSFs/KPIs/BSC for managing their employees Managers use the PMS regularly during the planning and control cycle Managers have been involved in decision making about the project start time Managers work in a stable, relatively tranquil environment Managers are actively communicating about the PMS project Managers are involved in defining KPIs Managers do have insight into the relationship between strategy and CSFs and KPIs 17 Managers are involved in making the CSF/KPI/BSC report layout Managers use the CSFs/KPIs/BSC that match their responsibility areas

19 The above results allow companies, who want to improve the use of their performance management system, to pay attention to the most important factors first before addressing the others. Discussion Despite the promising results presented in the previous section, there are still questions unanswered. There is, for example, a number of factors in the top 10 that one would not expect to be amongst the most important ones. In other words, some results are surprising and there is no clear explanation why certain factors are more important than others. Further analysis of the results in table 3 does not give any lead here. The top 10 most important factors come from all three stages (design, implementation and use) identified by Franco and Bourne (2005), and thus there does not seem to be a reason to assume that one phase is more important than another. Instead of accepting the results at face value, it is important to carefully consider De Waal s methodology. He has used a number of assumptions which have not been justified or verified. For example, De Waal assumes that all behavioral factors in the list are independent, which can be easily challenged. Another important assumption relates to the criteria for regular use. De Waal lists seven criteria, without describing the basis where they originate from, or justifying their selection. 18

20 One can argue the relevance of some of these criteria, e.g. what does the criterion Are their plans for follow up projects tell you about the regular use of the PMS? Another point of attention is that there appears to be an overlap between some of the criteria for regular use and the behavioral factors. Take for example the criterion for regular use Regular communication about KPI results. This is very similar to the behavioral factor Managers results on CSFs/KPIs/BSC are openly communicated. One can even argue whether some of the behavioral factors shouldn t be more appropriate as a criteria for regular use, e.g. Managers use the PMS regularly during the planning and control cycle. Concluding one can say that De Waal has set an important first step in raising the awareness that behavioral factors are important for the regular use of performance management systems. Based on his work a number of additional studies have been carried out and described in this article, which give interesting new insights in the relative importance of these factors. However a number of questions have been raised with respect to De Waal s methodology. 19