Kongkiti Phusavat, Ph.D. Department of Industrial Engineering Kasetsart University Bangkok, Thailand

Discussion on the Term Key Performance Indicators: Issues for Philosophies, Interpretations, and Demonstrations By Kongkiti Phusavat, Ph.D. Department of Industrial Engineering Kasetsart University Bangkok, Thailand Richard Dwight, Ph.D. Faculty of Engineering University of Wollongong Wollongong, Australia Abstract The paper explicitly focuses on describing the term Key Performance Indicator (KPI) based on the authors perspectives and experience. Alternate views are presented. There are several issues that have been applied to address and to gain basic understanding on this term. They include the methods of identification, data, information, and analysis concepts. After having examined the KPI from these four issues, it can be concluded that there are two approaches for designing, developing, and implementing the KPI. The first approach can be referred to as the quantitative approach (implying the use of quantitative data unit dimensions and specific frequency). In this approach, there are two types of the KPI, i.e., KPI with trend-specific and with variation (control or monitoring)-specific. On the other hand, the second approach may be called the qualitative approach (implying that the data stems from the combination of rating, judgment, and expectation or standard). By demonstrating the results from these two approaches, it is expected that the audiences are able to better understand and utilize KPI for their management processes. The materials contained in this paper are applicable primarily at the organizational, functional, process levels. Key terms: KPI, Quantitative Approach, and Qualitative Approach The organizational performance measurement system has long been regarded as an important management tool for continuous improvement. The system provides feedback to the following three questions. How well an organization is performing? Is the organization achieving its objectives? How much has the organization improved from a last period? In addition, performance measurement helps create feedback to managers with respect to the effectiveness of improvement interventions (implying corrective and preventive decisions). To simplify the scope and the roles of performance measurement, Sink (1990) provides the Kurstedt s management system framework. This framework depicts the components and interfaces in the management system. The Kurstedt s management system consists of three components. They are who manages, what is managed, and what is used to manage. The three interfaces are decision/action, measurement/data, and information portrayal/information perception. Sink (1990) further elaborates the what is managed -1-

component or the domain of responsibility for a manager should be divided into five elements. These elements are in the sequential order of upstream system, inputs, value-added processes, outputs, and downstream system. The interfaces help link the three components together. The first interface of decision/action represents improvement interventions made by the manager. The second interface, measurement/data, is designated as the assessment of effects and impacts from past improvement interventions. These results then forward to the third interface, known as information portrayal/information perception. Data on performance measurement needs to be organized and portrayed to the manager in the form and the format of information that can assist him/her in making quality decisions. V Figure 1 demonstrates the Management Systems Model. Who Manages? Perception Portrayal Data-to-Information Conversion Decisions Measurement Actions Data Upstream Systems Organizational System (in the context of Input/ Output Analysis) Inputs Value-added Processes Outputs Downstream Systems Figure 1: Roles of Performance Measurement in the Management Systems Understanding the Blanchard s Systems Engineering, Sink s Performance Measurement Methodology, and the Kurstedth s Management Systems Model therefore becomes a fundamental basis for KPI understanding. Their approaches, referred to as the system view, extensively focus on many aspects during KPI design, development, and implementation. These aspects include the alignment between KPI with organizational policies and objectives, explicit integration with an organization s planning process, strong interface with database, capability for information usage, and deriving the horizontal view of an organization (sometimes refereed to the Input/output Analysis or the Upstream-input-process-output-downstream Chain). As a result of this system view, a proposed set of KPI should be part of a performance measurement system structure. In this structure, there are 3 important components, i.e., key mechanisms (such as data collection and reporting), decision-making process, and performance criteria. Within the performance criteria, there exists the KPI. Figure 2 exhibits the performance measurement system structure. -2-

Performance Measurement Structure Mechanism Performance Criteria Decision-making Process Data collection Data-to-information Reporting KPI Performance analysis Performance improvement Figure 2: Performance Measurement Structure The next step needs to focus on what to be measured. Sink and Tuttle (1989) specifically state that there are seven criteria that reflect the term performance. These seven criteria are profitability, productivity, quality, quality of work life, innovation, effectiveness, and efficiency. It should be noted that there are other criteria to be measured as suggested by others. For examples, Kaplan and Norton (1992) propose that the performance consists of 4 perspectives, i.e., financial, customer, internal business, and innovation and learning. Harper (1984) recommends 7 performance aspects. They include productivity, unit cost, price, factor proportion, cost proportion, product mix, and input allocation. Figure 3 illustrates Sink and Tuttle s 7 performance criteria with respect to an organizational system. Organizational System Upstream Systems Inputs Value-added Processes Outputs Downstream Systems Productivity Efficiency Effectiveness Innovation and Quality of Work Life Quality Profitability Figure 3: Input/output Analysis with Key Performance Criteria -3-

Given the understanding of what to be measured, the next step is to be familiar with the KPI formation. One of the primary reason for identify the KPI in terms of a ratio format is due to the information analysis approach. There is no need for the KPI to be further normalized for external comparison such as comparison with industrial averages or benchmarking. In addition, in cases of changes in operations such as acquisition of new plants or expansion of production lines, the quality of information is not compromised. For example, with a new plant expansion, the use of revenue may not be appropriate since the increase is likely due to this expansion. It should be noted that, with qualitative data, the portrayal of information could be in either the tabular or the graphical formats. To better utilize a ratio format for the KPI identification, Sink and Tuttle (1989) advocate the extensive use of quantitative data with clear definitions and specific frequency for review. This is quite helpful for data collection and reporting effort. For example, different individuals may interpret the terms revenue, delay, damage differently. In order to clarify data to be collected, the dimensional units must also be stated. The term labor has at least 3 dimensions, i.e., time (hours), headcounts (persons), and financial value (dollars). The term maintenance may be collected in various dimensional units such as frequency (times), cost (dollars), and headcounts (persons), and time (hours). The specific frequency of review (daily, weekly, monthly, quarterly, and so on) is helpful for reporting. Table 1 demonstrates some of the KPI example with a ratio format according to the 7 performance criteria can be demonstrated as follows. Given a ratio format application, many lessons have been learned. There may be two types of the KPI in a ratio format. The first one may be referred to as having a desirable trend. The second type may be described as having a desirable result on stability. In other words, the result should exhibit less variation. Another lesson includes the rules that can be applied as a framework for the KPI identification. The first rule is called the absolute rule. This rule deals with an attempt to measure for an average value for assessment. The second rule can be labeled as a frequency rule. This rule is suitable when there is a standard or expectation value available. Some of the examples associated with the KPI types and the identification rule are as follows. (1) Trend-specific KPI: Revenue labor cost, Return delivered products, Compensation cost due to workplace injuries operating cost, and so on (2) Variation-specific (control or monitoring) KPI: Employees with the 35 years old or above employees, Labor cost material costs, Service revenue product revenue, and so on (3) Absolute rule framework: Total time taken to repair number of repairs, Total time taken to respond to incoming calls number of incoming calls, and so on (4) Frequency rule framework: Number of repairs completed within 30 minutes number of repairs, Number of respond to incoming calls made within 45 seconds number of incoming calls, and so on -4-

Performance Criteria Profitability Productivity Quality Quality of work life Innovation Effectiveness Efficiency Table 1: Demonstration of the KPI Definitions Ratio Formulas (According to the Input/output Analysis) Relationships between revenue and cost Relationships between outputs and resources consumed Assurance of quality at 5 checkpoints (upstream, inputs, processes, outputs, and downstream) Feeling of workforces on key factors in an organization such as pay, compensation, responsibility, safety, autonomy, and so on Applied creativity for managing changes over time Degree to which an organization accomplishes what it set out to accomplish Degree to which an organization utilizes the right things Downstream or outcomes inputs Outputs inputs Downstream downstream Downstream outputs Outputs Outputs Process Inputs Inputs Inputs upstream Upstream Upstream Process Outcomes Outcomes (Product innovation) Process (Process innovation) Actual outputs planned outputs Inputs expected to be consumed inputs actually consumed Examples Revenue cost Profit cost Products labor Products (labor + materials) Revenue from rework revenue Return products Rework products Average amount of time to respond to a customer complaint Rejects incoming materials Rejects suppliers with ISO 9001 certificates Audited suppliers suppliers Absent hours from unsafe and unsuitable working environment working hours Employees who have undergone training in the past 12 months employees Revenue from new products revenue Suggestions implemented suggestions Actual products produced in a specified timeframe targets Allocated machine hours actual machine hours used -5-

To summarize the previous discussion subjects, the system view on the KPI is quite beneficial for successful identification and implementation. The quality of information for managerial evaluation, analysis, and synthesis of current and potential problems is cited as a great contribution from performance measurement. Figure 4 provides a brief overview of the integration of the KPI into a performance-based management process. Organizational System Analysis (Input/output Analysis) Consideration into policies, objectives, and planning assumptions Performance Criteria Definition of terms KPI Identification (Ratio Format) Performance analysis (Internal analysis: trend and variation analyses) (External analysis: standard comparison and benchmarking) Issues of data availability, accuracy, and reliability, of rule applications (absolute or frequency terms), of KPI types, and information analysis Performance improvement thru root-cause analysis Planning (Target setting) Project interventions Process improvement Figure 4: Identification of the KPI and Its Integration into Performance-based Management Up until now the discussion has centred on finding quantified KPIs. The advantages for this are considerable and have been identified. These advantages appear to include the ability to undertake external comparisons and some robustness against changes in operations. Quantification also simplifies the task of data collection and unambiguous reporting. A number of issues arise from this desire for quantification. -6-

The objective of KPI setting should not be quantification for its own sake. While the results of the process so far reported here might be the identification of quantified KPIs this may not always be the right outcome. Where the goal of KPI setting is not specifically set at identifying quantified measures then the resulting approach could be referred to as qualitative. It has been established that the purpose of performance measurement might be to either determine how well the organization is going, whether it is achieving its objectives, whether it is improving, or, whether an initiative is having the desired effect. These organizational objectives are all problematic and the task can be simplified if achievement is transformed to mean the meeting or exceeding of particular targets for specific KPIs. Once KPIs have been set then the effective manager can set about meeting them. The problem is to establish that the KPIs so set actually relate to organization performance particularly in an overall or long-term sense. Another issue relates to the need for a standard against which performance is judged. This is of particular importance if the question being asked is How good are we. In a general view of performance measurement, the issue of determining how well an organization is doing requires a standard (Dwight, 1999). Such a standard should consider the means available to the organization to allow a particular level of attainment. Somehow a concept of what might be possible to achieve in terms of increasing the value of the organization needs to be determined. This will at some point become arbitrary depending on the level of decision considered possible. In other words what is considered to be within the system rather than the environment. There is no particular boundary on the types of decision that could be made. The business strategy chosen is presumably a key determinant of performance. Indeed the business that the organization is in, the particular needs that it decides to seek to fulfill will be a key cause of a particular performance. The resulting performance will depend on the nature of the resources that the organization either owns, procures, or, develops. In such circumstances the setting of a standard of performance will be subjective. Motivation towards the organization s goals is yet another key determinant of what might be appropriate KPIs. From a qualitative perspective, a KPI must first and foremost motivate people towards those issues that determine performance for the organization. If they are indeed key performance indicators then they must point in the direction for improvement. To achieve this the measures must somehow represent in a useful manageable and actionable format the products, as Pritchard (1990) defined them, of the organization. Having identified these products, the method of determining the quality of those products may not be feasible in a quantitative sense without loosing the essence of what is to be measured. In this case KPIs may fall short of being quantified while meeting the main objective of being reasonable performance indicators. Managing the issues that determine whether an organization will survive and grow is a common aim for organizations. Possibly for some government departments, a definition of organization may need careful consideration. The survival and growth of a department may not be a valid objective. Often the objective may be to minimize the size of the organization by in some cases acting to reduce demand. -7-

Normally an important part of survival is satisfaction of customers. Other determinants would be successfully dealing with what Porter(1985) termed the 5 forces of a competitive environment. Such an analysis leads to quite specific requirements that the organization would need to control and influence. This guidance to the organization s resource structure must also be the key determinant in the measures of performance, KPIs. To a large extent the concept of qualitative setting of performance measures is encapsulated in the so called ProMES approach proposed by Pritchard(1990). In brief, the process requires the work group to develop, through a series of meetings and feedback sessions, an understanding of the products they are responsible for, a method of measuring them and an understanding of the contingencies. Here contingency' refers to the scaling of the measure of performance relative to good and bad performance. The basic steps are listed as: 1. Determine products (tasks), missions related to those products, and indicators of performance. 2. Operationalise the indicators identified in step 1, i.e. determine suitable KPIs. 3. Determine Contingencies or standards of performance for each specific indicator and its relationship to overall performance. Standards are set by those who will be measured against them. Maximums, minimums and zeros are set this way. Visibility of the method of performance measurement setting allows management to contribute where appropriate. The resulting measures may be either quantitative or qualitative depending on the nature of the product identified and the ease with which a reasonable quantitative measure can be identified. Prichard identifies the importance of the positive feedback loop from performance to motivation and again to performance. Such relationships are also implied by Sink and Tuttle s 7 performance criteria as illustrated in Figure 3 previously. In terms of the discussion here, Pritchard s contribution was to identify the need to both establish a full set of products produced by an organizational entity and to think laterally about possible measures of performance. As an example of lateral thinking on performance measures, Kleingeld and van Tuijl (1992) report such measures against products of a photocopier repair group. 4 of these are identified below. Some could be classed as quantitative KPIs and others are qualitative. A sample of 4 attributes, their associated controllable attribute and KPI identified were: 1. Attribute: Quality; Controllable Attribute: Effective; KPI operationalized: Mean time between repairs, % repeat calls 2. Attribute: Cost; Controllable Attribute: Efficiency; KPI operationalized: Labour time per job 3. Attribute: Administration; Controllable Attribute: Accurate records; % of information recorded correctly 4. Attribute: Ambassadorship : Controllable Attribute: Company image promoted; KPI operationalized: Observed quality of personal presentation (say) -8-

The argument is that the KPIs derived are central to motivating improved performance consistent with organizational goals. They are not necessarily easily quantified. To have some confidence that the KPIs do actually reflect an organisation s performance a general approach is proposed. It should be noted that the actual process used may depend on the situation of the organization and its business. In some cases the focus may easily be determined to be a specific goal and associated limited set of attributes. A simple process can be suggested as follows: 1. Determine what the goals are, in particular the products of the organization (This requires a strategic analysis to determine factors affecting competitive advantage.) e.g. to become indispensable to clients. 2. Establish the connection between these products and the organisation s resources (cost and value drivers). e.g. indispensability may be contributed by contract supervision. It may be contributed by improvement process coordination and what latitude is provided to those dealing directly with the client. 3. Given the connections, establish measures of the quality of those connections and measures for the outcomes. e.g. Indispensibility could be measured in terms of the extent to which key client activities are undertaken by the organization; improvement process coordination by the number of suggestions logged and the number implemented or the absolute monetary value of the customer savings from suggestions. Whether these measures are easily quantifiable or easily rolled up into an overall ratio may not be key to their use in motivating improved performance consistent with the business goals. This process can be expanded into a more detailed set of steps in the form of what could be termed an audit approach to performance measurement. The process for audit approach applied to a maintenance activity of an organization is as follows: 1. Establish and rate the relevant organization goals as they relate to the technical systems it employs and their relative importance or contribution. 2. Determine the attributes that the organization of interest controls relative to these organizational goals and establish the interrelationships. For example the maintenance organization partly control attributes of production disruption, maintenance direct cost, equipment performance and equipment life. 3. Identify the important business activities. 4. Relate the business activities to the desired attributes. In the case of maintenance the functions that control the disruption to production, for example. 5. If required, the business activities could be rated in terms of importance to each desired attribute, for example scheduling could be considered as a key activity to achieve low direct maintenance cost, along with other activities. 6. Consider what might be the possible options for the configuration of each business activity. This would constitute the standard against which performance would be judged. 7. Confirm the standard as being the achievable and most desirable configuration and state of the activity relative to the business requirements. -9-

8. Establish the actual performance, possibly by using measures of performance, KPIs if you will. For example, the maintenance execution function influences disruptions, in part, by responding quickly. The KPI is this case might be response time and time to repair. 9. Combine KPI values into an overall KPI by relation back through the business attributes and so to the business goals. The result of this process will be a set of KPIs. Whether they are quantitative or qualitative depends on the particular nature of the product or business outcome, attribute and activity for which the KPI is sought. KPI (Organizational and Functional Levels) Quantitative Approach Qualitative Approach Trend-specific Variation-specific (monitoring and control) Process- specific Outcome - specific Key Consideration 1. Rule applications: absolute or frequency rules 2. Data: clear dimensional units with specific frequency 3. Analysis: internal (trend and variation analysis with own standard or expectation values) and external (industrial standard values and benchmarking) Key Consideration 1. Product definition 2. Standard setting 3. Determination of underlying causes of product quality. 4. Fix on business performance 5. Quantification a possible outcome Figure 5: Overall perspective on KPI definition -10-

In conclusion, Figure 5 is offered as an overall summary of the different perspectives as well as the definition of KPIs. The qualitative and quantitative approaches identified here share many key aspects. These include the identification of business goals, recognition of the basic dimensions of the attributes of specific organizations, and, the quest for measures that adequately represent those dimensions. Some divergence occurs in the final stages of establishment of actual measures. In both cases quantification of performance is sought. With rigid adherence to this objective comes clarity of measures, unambiguous representation of performance and some robustness in the face of variation in the actual business undertakings. With rigid adherence to the need for relevance to the business objectives comes the promise of more appropriate indicators of actual performance possibly at the expense of clarity and ability to compare across different businesses. It seems that both approaches have something to offer to organizations in their quest for practical measures that can be used to manage the organization s activities and to support their quest to set up measures that both reflect the organizations required direction for improvement and motivate people to make improvements coherently. -11-

References 1. Blanchard, B. S. and Fabrycky, W. 1990 System Engineering and Analysis Prentice-Hall: Englewood Cliffs, NJ 2. Blanchard, B. S. 1998 System Engineering and Management Wiley-Interscience: Singapore 3. Deming, W. E. 1986 Out of Crisis MIT Center for Advanced Engineering Study: MA. 4. Dixon, J. R.; Nanni, A. J.; and Vollmann, T. E. 1990 The New Performance Challenge: Measuring Operations for World-Class Competition Dow Jones- Irwin: New York 5. Dwight, R A, 1999 Searching for Real Maintenance Performance, Journal of Quality in Maintenance Engineering 6. Kaplan, R. S. and Norton, D. P. 1996 Translating Strategy into Action: the Balanced Scorecard Harvard Business Review: Boston, MA. 7. Kaplan, R. S. 1991 New System for Measurement and Control the Engineering Economist: Spring 8. Kleingeld P and van Tuijl H, Individual and Group productivity enhancement in a service department, TUE June 1992 rev July 1992 9. Hodgetts, R. M. 1998 Measures of Quality and High Performance AMACOM: New York 10. Kurstedt, H. A. 1992 Management System Theory and Practices Course Lectures for ISE 4015, 4016, and 5004, Department of Industrial and Systems Engineering at Virginia Tech 11. Kurstedt, H. A. 1985 Management System Model Helps Your Tool Work for You Responsive Article Published by the Management Systems Laboratory, Department of Industrial and Systems Engineering at Virginia Tech. 12. Neely, A. 1998 Measuring Business Performance: Why, What, and How the Economist Boos: London, Great Britain 13. Porter, M E, 1985, Competitive Advantage: Creating and Sustaining Superior Performance, New York, The Free Press 14. Pritchard, R.D.(1990), Measuring and Improving Organizational Productivity: A Practical Guide, Praeger, New York 15. Sink, D. S. and Tuttle, T. C. 1989 Planning and Measurement in Your Organization of the Future IE Press: Norcross, GA. 16. Sink, D. S. 1990 Theory and Practice of Measurement for Improvement in the Organization of the Future Productivity Management Frontier II -12-