How To Measure Performance Of A Redesign

Transcription

1 Performance Measures to evaluate the impact of Best Practices M.H. Jansen-Vullers, M.W.N.C. Loosschilder, P.A.M. Kleingeld, and H.A. Reijers Department of Technology Management, Eindhoven University of Technology P.O. Box 513, NL-5600 MB, Eindhoven, The Netherlands. {m.h.jansen-vullers, p.a.m.kleingeld, Abstract. Business process redesign best practices help to generate alternative process redesigns. Best practices are often used by practitioners to repeat design techniques that have proven effective in practice, but it is not clear why they are the best, and in what respect. More importantly, it is unclear why a best practice might be best for one process and not for another. In this paper, we propose a framework for performance measures that can be used to evaluate the effect of a redesign with the dimensions of, cost, quality and flexibility. For each dimension we propose a set of measures, which have been operationalized as much as possible. We conclude that a simulation approach is very well applicable to estimate the impact of redesign best practices, especially for the, cost and flexibility measures. For quality measures additional methods are necessary to complement the estimation. Keywords: Business Process Redesign, Best Practices, Performance Measurement. 1 Introduction The domain of business process redesign can roughly be divided in two different approaches: revolutionary and evolutionary. In the revolutionary approach, a redesign starts from a clean sheet. In the evolutionary approach, the existing business process is taken as a starting point. Application of redesign best practices fits within this approach. Reijers provides an overview of all heuristics currently encountered in literature [21]. Further, a rough qualitative estimation of the expected effect is given [22]. However, quantitative research is necessary to determine a more concrete impact of a best practice on the performance of a workflow. Although not much is known about the impact of redesign heuristics on the performance of a workflow, some papers have been found that are based on a simulation study. These studies include several heuristics: knockout heuristic [1], extra resources heuristic [9], specialist-generalist heuristic [9, 19], flexible assignment heuristic [19], task composition, triage and case types heuristic [26].

2 The main shortcoming of the above-mentioned literature is that none of the authors provide guidelines for the redesign of workflows: what heuristic should be applied in what situation, process or setting? Other deficiencies are a lack of a general approach to quantify the impact of heuristics, the limited number of different dimensions of performance, and the limited number of aspects per measured dimension. Further, none of them, with the exception of [19], quantifies the impact of the implementation of more than one heuristic at the same. In our research, we aim to quantify redesign best practices within as many dimensions as possible. This paper provides an overview of available performance frameworks and related performance measures. The resulting performance measures can be used in simulation studies to quantify the impact of a redesign best practice and to evaluate the adequateness of a business process design. The paper is organized as follows. In Section 2 the dimensions of performance are investigated. Next, each performance dimension is discussed separately (Section 3 Section 6). The performance measures have been specified for every dimension based on the findings in literature and metrics are proposed. The paper concludes with a reflection on and discussion of the results. 2 Performance measurement This study focuses on the quantification of a redesign heuristic s impact on the performance of a business process. Subject of study is the business process that is being redesigned, in contrast to, for example, the performance of individual employees or entire organizations. In the last twenty years a variety of performance measurement systems has been developed. We assessed this literature to find out what dimensions of performance they discern and which are suitable for measuring business process performance. Six different systems have been considered. Here we summarize the dimensions (in some approaches also called categories or perspectives) they discern. Performance pyramid: four dimensions on the operational level, i.e. quality, delivery, process and cost [6]. Performance measurement matrix: four different dimensions, i.e. internal and external, cost and non-cost measures [12]. Results/determinants matrix: six dimensions, i.e. competitiveness, financial performance, quality of service, flexibility, resource utilization and innovation [5]. Balanced scorecard: financial and non-financial measures are combined, resulting in four dimensions, i.e. the customer, internal business process, innovation and financial perspective [11]. Devil s quadrangle: four dimensions, i.e., cost, quality, and flexibility. The quality dimension further distinguishes internal quality (pertaining to the workforce) and external quality (of the process and the output) [4]. Performance prism: this approach indicates what areas an organization must focus on, i.e. stakeholders, strategies, processes and capabilities [3].

3 We assessed the dimensions of the above-mentioned performance measurement approaches for possible evaluation of the impact of best practices. In the performance pyramid quality, process and costs can be considered as performance dimensions of business processes. Delivery is a measure that is concerned with the ly delivery of the required quantities, which overlaps with the process from the viewpoint of (administrative) business processes. In the performance measurement matrix the three relevant performance pyramid dimensions can be mapped onto three dimensions of the matrix; the external cost measures dimension is less suitable, because it focuses on customers instead of business processes. In the results/determinants matrix quality of service and flexibility are suitable performance dimensions. Resource utilization is more a measure of the cost or dimension than a performance dimension on its own. The remaining dimensions refer to organizational performance. From the balanced scorecard and the performance prism only the internal business process is relevant, though it does not add any specific performance dimension. When considering the devil s quadrangle, we conclude that, cost and flexibility already have been identified as suitable dimensions. The (internal and external) quality dimension are also relevant when depicting trade-offs and effects of a business process redesign effort. It can be concluded that the dimensions of the devil s quadrangle are suitable for measuring the performance of a workflow (see Figure 1). The dimensions cover the concept of performance and the other performance measurement systems do not provide additional relevant dimensions. The next four sections give the performance measures for each of the discerned dimensions. Quality Cost Time Flexibility Fig. 1. The devil s quadrangle [4]

4 3 The Time dimension Time has been described as both a source of competitive advantage and the fundamental measure of performance [18]. Based on the information on measurements found in the literature [4, 15, 18, 21, 25], a set of performance measures for the dimension, specifically for workflows, has been derived. Figure 2 shows the relation among these measures. Lead Throughput Move Queue Setup Service Wait Client Task 1 Task 2 Client Fig. 2. Performance measures - dimension Lead is the it takes to handle an entire case. Throughput is the between the moment a task is completed and the moment the next task is completed. This is composed of: Service : the that resources spend on actually handling the case Queue : the a case spends waiting in a queue Wait : all other delays for a case, e.g. the a case has to wait in a parallel branch for completion of all other branches, in order to be able synchronize Move : the it takes to move a case between tasks Setup : the it takes to setup a task for a case, for example the to get acquainted with the case Based on standard event logs with information, the above-mentioned measures can easily be calculated (see e.g., [2]). 4 The Cost dimension The most commonly used performance targets are of a financial nature. The cost dimension is closely related to the other dimensions. For example, long lead

5 s can result in a more costly process; low quality can lead to expensive rework and low flexibility can also result in a more costly process execution. We focus on the direct costs of running the process. Most of the literature [4, 14, 15, 18, 25] is about cost measurement in production processes, here viewed from a workflow perspective. This resulted in: Running costs: costs for executing the workflow Labor costs: costs of the workforce (salary worked hours) Machinery costs: an investment in machinery per new automated task Training costs: costs for training employees (based on the number of employees and the number of new tasks) Inventory costs: costs of keeping records and products (e.g. file cabinets) Transport costs: costs for moving (intermediate) products and sharing information (distance price per km) Administrative costs: costs for keeping the entire workflow intact Resource utilization: the ratio of in use and available of resources These measures can be categorized as either constant costs or variable costs. Constant costs are costs that are independent of production volume, e.g. investments or labor (size of the workforce is constant) [13]. Variable costs vary with the level of the output, e.g. for part employees. 5 The Quality dimension The quality of a workflow can be judged from at least two angles [21]. External quality is defined from the customer s side, the person or organization that initiates the workflow and will receive the output. Internal quality is defined from the worker s side. External quality can be measured as client satisfaction with either the product or the process. Satisfaction with the product is the degree to which the customer feels that the product is according to specification or feels satisfaction with the delivered product. With the advancement of total quality management (TQM) the emphasis has shifted from conformance to specification towards customer satisfaction [18]. The satisfaction of a customer with the process concerns how a workflow is executed [21]. Literature has been found on both the quality of a product and the quality of a process [7, 14, 15, 25]. This resulted in the following set of external quality performance measures: Quality of the output Performance: the degree to which a product s primary operating characteristics meet customer s requirements Conformance: the degree to which a product s design and operating characteristics meet established standards Serviceability: the speed, courtesy, competence and ease of correcting mistakes Quality of the process

6 Information on application status: the degree to which information on the application status is provided Bureaucratic language simplification: the clearness in the presentation Information availability: the required to get updated on the status Measuring external quality in a workflow model is less straightforward than measuring or costs. Many different factors influence and determine the external quality. Performance and serviceability may be influenced by: the number of specialists that work on a case, the number of control tasks per case, and the assignment of a case manager, to name a few. The quality of the process may be influenced by two aspects: the assignment of a case manager and the number of employees that work on a case. The more employees work on a case, the lower the expected quality of the process, as the commitment of the employee increases. Whether a specific aspect influences the external quality of a process and the degree to which it affects the quality is highly dependent on the type of process. It is therefore not possible to capture the measures of external quality in a metric. The same problem came up during the operationalization of internal quality and process modification flexibility (see below). To settle this, we have decided to consider such aspects of external quality as proxies: a change in one or more of the aspects will have some impact on the external quality dimension. However, the exact size of impact cannot be determined in a simulation model. Internal quality can be seen as the quality of a workflow from an operator s perspective. In this context, internal quality involves the conditions of working [21]. Psychological and social factors are very important. High internal quality can result in high motivation, high job satisfaction, high external quality and low absenteeism. Internal quality is commonly measured with a survey among employees, although some measures might be obtained by simulation as well. The literature contains slightly different, but comparable internal quality measures [10, 16, 17, 20, 21]. This resulted in the following set of performance measures: Skill variety: the degree to which a job requires a variety of different activities in carrying out the work, which involves the use of a number of different skills and talents of a person (number of different tasks and case types). Task identity: the degree to which the job requires completion of a whole and identifiable piece of work; that is, doing a job from beginning to end with a visible outcome (ratio of number of executed tasks and total number of tasks per workflow). Task significance: the degree to which the job has a substantial impact on the lives or work of other people, either in the immediate organization or in the external environment. Autonomy: the degree to which the job provides substantial freedom, independence, and discretion to the individual in scheduling the work and in determining the procedures to be used in carrying it out the work (ratio of number of authorized decisions and total number of decisions).

7 Feedback: the degree to which carrying out the work activities required by the job results in the individual obtaining direct and clear information about the effectiveness of his or her performance. Co-worker relations: the quality of the relations between an employee and his/her co-workers. The operationalization of internal quality is comparable to external quality in the sense that it is not possible to capture the measures in metrics. Differences among people moderate how they react to the complexity and challenge of their work [10]. The aspects that can be used as a proxy for internal quality are: the number of executed tasks and case types per resource, the number of contact moments, the number of teams a resource is participating in, the number of authorized decisions and the number of new tasks and case types per resource. 6 The Flexibility dimension Flexibility is the least mentioned criterion to measure the effect of a redesign effort. Flexibility can be defined as the ability to react to changes [21]. This section describes the performance measures for the flexibility dimension found in literature [8, 18, 21, 23 25, 27]. It appears that flexibility can be identified for individual resources, for individual tasks, and for the workflow (process) as a whole. The following set of flexibility performance measures is proposed: Mix flexibility: the ability to process different kinds of cases for resources: number of case types a resource can handle for tasks: number of case types a task can handle for the workflow: number of case types that can be handled Labor flexibility: the ability to perform different tasks for resources: number of executable tasks for the workflow: available resources per task per case Routing flexibility: the ability to process a case using multiple routes (number of different sequences in the workflow) Volume flexibility: the ability to handle changing volumes of input (available per employee) Process modification flexibility: the ability to modify the process (number of subflows in the workflow, complexity, number of outsourced tasks, etc.) For each of these types, different aspects of flexibility can be considered, such as range (the range of variations that can be handled), (the amount of required to adapt to change) and cost (the amount of money required to adapt to change). Another way of looking at flexibility mentioned in literature is to discern run flexibility and build flexibility. Run flexibility concerns the possibilities to handle changes and variations while executing a specific workflow. Build flexibility concerns the possibility to change the workflow structure [21]. This distinction is already present in the set of measures. Process modification flexibility is comparable to build flexibility and the other measures are all run flexibility types.

8 7 Discussion The application of heuristics in a redesign effort is a fruitful approach. However, little quantitative data is available to predict the impact of the application of a best practice. A few studies provide quantitative data, but these have a limited scope. This paper provides an overview of different dimensions for performance measurement and for each dimension an overview of possible performance measures. The dimensions are taken from literature on performance measurement systems, and especially from the devil s quadrangle [4]. For the dimensions of and costs, the computation of the measures in a simulation model is quite straightforward. Determining the external quality of the output and the process with a simulation model is much more difficult, or even impossible. We identified a large number of different proxies, of which a few were mentioned in this paper. Although this set of proxies can be computed in a simulation study, this does not do complete justice to the real complexity of external quality in practice. Something similar holds for the internal quality measures; these can be implemented, but also in this case this will provide a limited view on what will happen in reality. Furthermore, internal quality is highly dependent on the perception of the person involved. With respect to the flexibility measures, all measures can be implemented (except process modification flexibility). Here we expect that actual changes in our metrics values accurately reflect what will happen in practice. The research is suitable both for quantifying the impact of redesign best practices, as well as for measuring the adequateness of a process design in general. The discussion above provides a twofold conclusion when applying simulation in this context. For a large part the approach is very well applicable and several measures have been identified for this purpose. For a smaller, but very significant part, we conclude that a simulation approach does not provide the desirable insight. Since the respective identified performance dimensions are all relevant, different methods are necessary to complement the analysis of the impact based on the devil s quadrangle. References 1. W.M.P. van der Aalst. Reengineering Knock-out Processes. Decision Support Systems, 30(4): , W.M.P. van der Aalst, B.F. van Dongen, J. Herbst, L. Maruster, G.Schimm, and A.J.M.M. Weijters. Workflow mining: a survey of issues and approaches. Data and Knowledge Engineering, 47(2): , C. Adams and A. Neely. Prism Reform. Financial Management, 5:28 31, N. Brand and H. van der Kolk. Workflow Analysis and Design. Kluwer Bedrijfswetenschappen (In Dutch), T.J. Brignall, L. Fitzgerald, R. Johnston, and R. Silvestro. Performance Measurement in Service Businesses. Management Accounting, 69(10):34 36, K.F. Cross and R.L. Lynch. The SMART Way to Define and Sustain Success. National Productivity Review, 8:23 33, 1988/1989.

9 7. D.A. Garvin. Competing on the Eight Dimensions of Quality. Harvard Business Review, D. Gerwin. An agenda for research on the flexibility of manufacturing processes. International Journal of Operations & Production Management, 7(1):38 49, R.H.J.J.M. Goverde and H.A. Reijers. Resource Management: a Clear-Headed Approach to Ensure Efficiency. Workflow Magazine, 4(6):26 28 (In Dutch), J.R. Hackman and G.R. Oldham. Motivation through the design of work: Test of a theory. Organizational Behavior and Human Performance, 16(2): , R.S. Kaplan and D.P. Norton. The Balanced Scorecard: Measures that Drive Performance. Harvard Business Review, 70:71 79, D.P. Keegan, R.G. Eiler, and C.R. Jones. Are your Performance Measures obsolete? Management Accounting, 70(12):45 50, E. Lekanne. Bedrijfseconomie Gids: Begrippen uit de Bedrijfseconomie. Uitgeverij NIB, Zeist (In Dutch), G.K. Leong, D.L. Sneyder, and P.T. Ward. Research in the process and content of manufacturing strategy. Omega, The International Journal of Management Science, 18(2): , A. Longo and G. Motta. Design Processes for Sustainable Performances: a Model and a Method. In Proceedings of the Workshop on Business Process Design, F. van de Looij and J. Benders. Not just money: Quality of working life as employment strategy. Health Manpower Management, 21(3):27 33, V. Mehta and H. Shah. Characteristics of a work organization from a lean perspective. Engineering Management Journal, 17(2):14 20, A. Neely, M. Gregory, and K. Platts. Performance measurement system design: A literature review and research agenda. International Journal of Operations & Production Management, 15(4):80 116, M. Netjes, W.M.P. van der Aalst, and H.A. Reijers. Analysis of Resource Constrained Processes with Colored Petri Nets. In K. Jensen, editor, Proceedings of the Sixth Workshop and Tutorial on Practical Use of Colored Petri Nets and the CPN Tools, C.H.P. Pava. Old and new sociotechnical approaches. National Productivity Review, 4(1):87 89, H.A. Reijers. Design and Control of Workflow Processes: Business Process Management for the Service Industry. Springer Verlag, Berlin, H.A. Reijers and S. Limam Mansar. Best Practices in Business Process Redesign: an Overview and Qualitative Evaluation of Successful Redesign Heuristics. Omega, The International Journal of Management Science, 33: , A.K. Sethi and S.P. Sethi. Flexibility in manufacturing: A survey. International Journal of Flexible Manufacturing Systems, 2(4): , N. Slack. The flexibility of manufacturing systems. International Journal of Operations & Production Management, 7(4):35 45, A. De Toni and S. Tonchia. Performance measurement systems: Models, characteristics and measures. International Journal of Operations & Production Management, 21(1):46 70, M. Zapf and A. Heinzl. Evaluation of Generic Process Design Patterns: An Experimental Study. In W.M.P. van der Aalst, J. Desel, and A. Oberweis, editors, Business Process Management: Models, Techniques, and Empirical Studies, volume 1806 of LNCS, pages Springer-Verlag, Berlin, Q. Zhang, M.A. Vonderembse, and J. Lim. Manufacturing flexibility: Defining and analyzing relationships among competence, capability, and customer satisfaction. Journal of Operations Management, 21: , 2003.