SEAN J. GALLAGHER. A paper submitted to the faculty of. The University of North Carolina at Chapel Hill

Transcription

1 Linking Individual Performance Appraisals with Departmental Outcomes: Does the Type of Individual Performance Appraisal System Influence Departmental Performance? By SEAN J. GALLAGHER A paper submitted to the faculty of The University of North Carolina at Chapel Hill In partial fulfillment of the requirements for the degree Master of Public Administration March 2010 This paper represents work done by a UNC-Chapel Hill Master of Public Administration student. It is not a formal report of the School of Government, nor is it the work of School of Government faculty. Executive Summary A number of professional organizations promote the use of performance measures by local governments to quantify progress toward organizational goals. Strategic human resource management proponents advocate aligning performance appraisal systems with organizational goals. Yet, little research exists that links performance appraisal systems to performance results or performance measures for local governments. This study seeks to determine if there is a link between a department s performance appraisal system and its results, as measured by North Carolina Benchmarking Project data.

2 Introduction The performance appraisal system 1 serves as the primary method of feedback to the individual employee. Performance appraisal systems touch every employee in an organization. Managers and supervisors rate their employees, commonly, on an annual basis to evaluate individual performance. Many organizations, including local governments, invest significant human and financial resources into their performance appraisal system. Little is known about the extent to which departments align their individual performance appraisals with desired departmental performance outcomes. Kevin R. Murphy and Jeanette N. Cleveland noted, The activation and guiding processes (i.e. goals) within an organization and the role of the appraisal in the process needs to be addressed more directly through research. 2 This study seeks to determine if certain individual performance appraisal characteristics are associated with better performance results. 3 Specifically, does the type of performance appraisal system a department utilizes impact that department s performance? The answer to this question should interest managers and human resource professionals in local government because performance appraisal systems are costly and challenging to change, as most municipalities codify the appraisal system in local ordinances. Performance Appraisal Systems The centerpiece of a performance management system is usually the performance appraisal, a formal systematic process for reviewing performance and providing oral and written feedback to employees at least annually. 4 The performance appraisal is key to the performance management process, but little research exists connecting performance appraisals and performance measures. 5 Dennis Daley states: Performance appraisal systems require a specific appraisal instrument; there are no generic appraisal processes. 6 While there is no generic appraisal process, several common characteristics are shared among the performance appraisal systems of the 17 participating cities in the North Carolina Benchmarking Project, the focus of this study. Composition, communication, level of form customization, and ties to compensation decisions were characteristics of the performance appraisal systems that were examined. Most of the analysis focused on the composition characteristic. Composition refers to the content of the performance appraisal form. Three key variables used by this study to assess composition are: Number of quantitative performance targets. These targets are measures of individual performance tied to an objective, measurable standard for the rated performance dimension. Number of qualitative performance targets. These targets are subjective measures of individual performance in a rated dimension that can be influenced by training such as job knowledge, quality of work, time management, and customer service. Number of behavior performance targets. These targets are subjective measures of individual performance in a rated dimension that is a personality trait of behavior and not easily corrected through formal training such as initiative, cooperation/teamwork, and dependability. The other three characteristics were a secondary focus of the study. The communication component examines the various methods of communicating goals, standards, and performance expectations for the performance appraisal system, such as goal setting at the start of the appraisal process and defining expected performance dimensions on the appraisal form. The level of form customization component refers to the level of specificity of the performance appraisal form for a given job title. Compensation refers to the degree to which pay incentives were tied to the performance appraisal system in each municipality. The content analysis revealed considerable variance among the municipalities 1

3 regarding the tying of pay incentive policies to the performance appraisal process, but little variation among in other examined characteristics. Those municipalities that tied their pay incentives to performance appraisals had similar methods for determining pay incentives and size of incentives. For example, the city of Wilmington, NC, distributes merit pay awards to its employees based on performance appraisal rating. The prescribed pay increase is 2 percent (increase in base pay) for standard performance, 3.5 percent for above standard performance, and 5 percent for outstanding performance. Methodology This exploratory, cross-sectional study used a content analysis of performance appraisal systems of the 17 participants in the North Carolina Benchmarking Project, a cooperative project providing comparative performance measures for several municipal services. Pearson s correlation was used to assess association between the dependent and independent variables. The content analysis results were then analyzed against the results of the higher-order performance measures (efficiency and effectiveness) for selected services to identify any relationship between performance appraisal system characteristics and performance results. The content analysis covered three documents provided by the 17 participating municipalities: performance appraisal forms, performance appraisal policies, and pay incentive policies. The forms and policies were analyzed to determine if there was a correlation between the form s content, characteristics, relationship to pay incentives, and the efficiency and effectiveness measures published by the benchmarking project. The North Carolina Benchmarking Project data were selected as the dependent variables for three reasons. First, the project is voluntary. Municipalities that elect to join are inclined to use performance measures in their management systems. Second, the project s data are recognized as reliable and valid performance measurements by the participants. Third, the comparative data allow cross-municipality comparison. The areas of and Building Inspections were selected as they were thought more likely to have performance targets on individual performance appraisal forms. 7 This study used data published by UNC School of Government in Final Report on City Services for Fiscal Year : Performance and Cost Data. The independent variables were compiled following content analysis of documents provided by all 17 participating municipalities. The content analysis focused on four elements of performance 8 appraisals: composition, communication, level of form customization, and ties to compensation. Performance appraisal system characteristics were translated into 18 independent variables based on mention of these characteristics in the human resource management literature. These 18 dichotomous variables indicate the presence or absence of key appraisal attributes in a given city s forms or policies. For example, a variable used for the level of form customization characteristic was: Is a single form used citywide? The answers were coded 0 for no and 1 for yes. See Appendix A (Content Analysis Methodology) for complete list of independent variables. Limitations Scope. The scope of the sample is limited to two service areas in North Carolina municipalities that participate in the benchmarking project. This limited the N of the study to 11 municipalities in building inspections (several municipalities consolidate their building inspections with county governments) and 17 municipalities in fleet maintenance. 9 The limited data set does not allow conclusions to be generalized beyond these cities, but can identify areas for more extensive research with a larger sample size. Findings cannot be interpreted to apply outside the service areas used in the study. Cross Sectional Study Method. This study examined the forms, policies, and performance results from fiscal year This one time look at data does not take into account changes in performance or 2

4 performance appraisal processes that happened before or after the studied fiscal year. In particular, all municipalities that link pay incentive to performance appraisals were fully funded in fiscal year Content analysis method. The content analysis of published forms and policies focuses on the intent of the performance appraisal system. By looking at the espoused framework of the performance appraisal system, it is assumed that the process works as written in policy. However, there could be a disconnect between what the process should be and actual application. Ideally, semi-structured interviews would be used to explore this possibility. However due to time constraints and resource availability, this additional step was not taken in this study. Purpose of Appraisals and Performance Measures. The research explored the individual performance appraisal system as a management tool. The author concedes there are many purposes for a performance appraisal system other than measuring performance, such as training and development, validating recruitment and selection processes, and identifying employees for promotion. Findings The study used a p-value of.05 or less as the criterion for determining statistical significance. The Pearson s r-value explains the level of association. Pearson s r-values range from -1.0 to 1.0. An r- value closer to -1.0 or 1.0 indicates a stronger measure of association. A negative correlation means that as one variable increases the other decreases. A positive correlation means that both variables move in the same direction. Using a p-value threshold of <.05 means that there is less than 5 percent chance any measure of association deemed statistically significant occurred by chance. Statistically significant correlations in this study are shown in Table 1. Findings. The fleet maintenance service provided the greatest number of statistically significant relationships between performance appraisal characteristics and departmental performance results. The most notable finding among seven significant results is the strong relationship between pay and percentage of work orders requiring repeat repair within 30 days. There was a strong relationship between pay incentives and work quality. The favorable (negative) correlation between pay incentives and the percentage of work orders requiring repeat repair within 30 days reinforces the concept of pay for performance. This is the result a manager is looking for with this effectiveness performance measure. A decrease in the number of repeat repairs indicates an increase in the quality of work. Pay incentives are based on rewarding results; an increase in quality of work is a result worthy of reward. Anecdotally, three municipalities specifically use the performance measure (percentage of work orders requiring repeat repair within 30 days) as an evaluated performance target on their performance appraisal. All three perform above the mean and median of the 17 cities. Two municipalities are among the top four performers. Performance appraisal forms or policies that encourage the use of quantifiable objectives were found to be favorably (negatively) correlated with the percentage of work orders requiring repeat repair within 30 days. While some forms incorporate quantifiable targets on the form, many of the policies and forms encourage supervisors to use quantifiable objectives in performance appraisals. The negative correlation suggests that the percentage of repeat repairs will decrease when the policies encourage the use of quantifiable objectives. These findings were isolated to the performance measure of percentage of work orders requiring repeat repair within 30 days. 3

5 Table 1. Statistically Significant Findings Between Appraisal Characteristics and Performance Results Service Area N Independent Variables Building Inspections 11 Building Inspections 8 *Statistically significant at p<.05 Form identifies core competencies Use of goal setting at start of process Rating scale/definitions incorporate quantifiable objectives Number of behavior targets Performance dimension defined on forms Policy/form encourages use of quantifiable objectives Pay incentives Periodic review required by form and policy Number of qualitative performance targets Dependent Variables maintenance cost per work order maintenance cost per vehicle equivalent unit maintenance cost per vehicle equivalent unit % of preventive maintenances completed as scheduled % of preventive maintenances completed as scheduled % of work orders requiring repeat repair within 30 days % of work orders requiring repeat repair within 30 days % of inspection responses within one working day of request % of inspections that are reinspections Dependent Variable Desired Result Pearson s r Costs 0.663* Costs * Costs 0.519* Higher * Higher 0.762* * * Higher * 0.727* There is an unfavorable (positive) correlation between identifying core competencies expected of fleet maintenance employees on the performance appraisal and cost per work order. This finding suggests that focusing on core competencies in performance appraisal does not lower costs per work order in fleet maintenance. The efficiency measure of fleet maintenance costs per VEU is favorably (negatively) correlated with goal setting and is unfavorably (positively) correlated with a performance appraisal form using a rating scale or performance dimensions that incorporate quantifiable objectives. 10 This finding suggests that communicating objectives to an employee at the start of the evaluation period can lower cost per VEU. Surprisingly, the positive relationship between cost per VEU and using a rating scale or performance dimensions that incorporate quantifiable objectives means that using a rating with quantifiable objectives does not lower costs. The appraisal forms that defined their performance dimensions to communicate the standards for evaluation to employees were strongly associated with percentage of preventative maintenance completed on schedule. This favorable relationship is the highest level of association at the.76 level. This finding suggests that stating the definitions of performance dimensions on the appraisal form may lead to a higher 4

6 percentage of preventative maintenance completed on schedule. Also, the percentage of preventative maintenance completed as scheduled was found to be unfavorably (negatively) correlated with the number of behavior targets. The analysis shows that more behavior targets on a performance appraisal tends to result in a lower percentage of preventative maintenance completed on schedule. Conversely, a performance appraisal form that uses fewer behavior targets tends to lead to more preventative maintenance completed on schedule. This finding supports the use of fewer behavior targets on an appraisal form. Building Inspections Findings. Communication through periodic face-to-face engagement during an individual s rating period is favorably (negatively) correlated with the percentage of inspection responses completed within one day. This finding indicates that fewer face-to-face meetings with employees during the rating period tend to lead to a higher percentage of inspection requests responded to within one working day. Second, the number of qualitative performance targets in the appraisal is unfavorably (positively) correlated with percentage of inspection that are reinspections. This suggests that loading building inspector s performance appraisals with qualitative targets will not reduce reinspections. Conclusions This exploratory study set out to determine whether the type of performance appraisal system a department uses impacts that department s performance. The small number of statistically significant findings identifies few tendencies linking individual performance appraisal characteristics to department performance results in the studied services areas. However, the small number of statistically significant correlations yielded promising findings that will be encouraging to performance management proponents. The statistically significant findings hint that performance appraisals possessing the characteristics of communicating goals and performance dimensions, encouraging the use of quantifiable objectives, and linking performance to pay incentives can impact certain performance results in the service areas studied. The findings suggest that the concept of linking performance appraisals and performance results may have some validity. However, these relationships refer primarily to documents declaring appraisal systems intent. The impact of actual implementation deserves further scrutiny. The statistically significant findings for the composition characteristic indicate no correlation at a statistically significant level between performance appraisals using quantitative targets and department performance results. However, performance appraisal forms using behavior targets and qualitative targets were found to have one statistically significant finding each that was an unfavorable association with performance results. These findings suggest that while using quantitative targets may not impact results, using behavior and qualitative performance targets may lead to unfavorable results. The small number of statistically significant findings merits more research using a larger sample, incorporating qualitative research methodologies, and utilizing a longitudinal research design to see the impacts over time on performance. A larger sample can possibly determine if more statistically significant findings emerge when studying other service areas besides fleet maintenance and building inspections. A study applying qualitative methods to examine implementation of performance appraisal policies can possibly look at the impact of other variables linking performance appraisal systems and performance results. 11 A longitudinal test of a larger sample can possibly determine if the statistically significant findings in this study hold up over time in a greater number of municipalities and if other relationships are also statistically significant. The finding that more quantitative performance targets do not significantly impact performance is puzzling. Broadly, the small number of significant findings should stimulate conversation among human resource professionals and managers about the importance of aligning performance appraisal systems and organizational goals. 5

7 Bibliography Ammons, David (editor), Leading Performance Management in Local Government. ICMA. Washington, DC Daley, Dennis M. Strategic Human Resource Management: People and Performance Management in the Public Sector. Upper Saddle River, NJ. Prentice Hall Murphy, Kevin R. and Cleveland, Jeanette N. Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspective. Thousand Oaks, CA: SAGE Publications Mohrman Jr, Allan M., Resnick-West, Susan M., and Lawler III, Edward E. Designing Performance Appraisal Systems: Aligning Appraisals and Organizational Realities. San Francisco, CA. Josey-Bass Publishers North Carolina Local Government Performance Measurement Project. Final Report on City Services for Fiscal Year : Performance and Cost Data. School of Government. Chapel Hill, NC Selden, Sally Coleman. Human Capital: Tools and Strategies for the Public Sector. Washington, DC: CQ Press

8 ENDNOTES 1 For the purposes of this study, performance appraisal system refers to the complete process of appraising an individual employee s performance in accordance with the stated policy, appraisal form and pay incentive policy. 2 Murphy, Kevin R. and Cleveland, Jeanette N. (1995) Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspective Thousand Oaks, CA: SAGE Publications, For the purposes of this study, performance results are the results of the units of analysis (performance measures) the UNC School of Government publishes for each fiscal year. This study used data for fiscal year Selden, Sally Coleman (2008) Human Capital: Tools and Strategies for the Public Sector. Washington, DC: CQ Press Seldon, p For the purposes of this study, performance measures will refer to the set of higher order performance measures (measures of efficiency and effectiveness rather than workloads) published by the NC Benchmarking Project. These are units of analysis for performance. 6 Daley, Dennis M. (2002) Strategic Human Resource Management: People and Performance Management in the Public Sector. Upper Saddle River, NJ. Prentice Hall, There are 11 service areas of data the NC Benchmarking Project collects annually. Some areas are incident driven (such as police, fire and emergency communication) which would limit the impact of individual performance on performance measures. 8 See Appendix A for an example of the content analysis methodology. 9 Select performance measures had a lower N due to certain municipalities not reporting data for that performance measure. With no performance results, those municipalities variables were removed from correlation analysis for that particular performance measure. 10 An example of a performance appraisal form using a rating scale or performance dimensions that incorporate quantifiable objectives is found on Charlotte s performance appraisal form. Under the performance dimension of job knowledge and skills, technicians are evaluated on their ability to obtain and maintain certification for ASE Master Technician. Achieve Expectation= Certification 6 for heavy vehicles and Certification 8 for light vehicles. Exceed Expectations= obtaining 1-3 certifications above the Achieve Expectations level. Exceptional= obtain 4 or more certifications in ASE, EVT or Factory (Ford, GM, Cummins, Allison, etc ) certifications. 11 The author recognizes that correlations cannot be used to determine causality of the co-movement of two variables. Nevertheless, the human resource management literature assumes that performance appraisals are not ends themselves but means to ends. Performance appraisals are expected to influence various desired ends. The focus of this study was to examine the impact of performance appraisal systems on performance results. Based on the content analysis and resulting Pearson s correlations, it appears that performance appraisals characteristics might influence some results. Although the author concedes the possibility that the apparent relationship could be spurious (that is, the product of a third factor s influence on both the appraisal system and the performance results), the existence of at least a modest causal relationship between appraisal systems and performance results seems likely. 7

9 Appendix A- Content Analysis Methodology Characteristic Independent Variables Type of Answers Variable Composition Does the appraisal form or policy provide instructions to link Nominal Yes/No (0/1) performance accomplishments to city/department goals? Communication Does the appraisal form/policy identify core competencies for Nominal Yes/No (0/1) individual performance? Composition Number of Qualitative Performance Targets: (e.g.- subjective Interval Range (0-40) measure of performance in a dimension that can be influenced by training such as : Job Knowledge, Quality of Work, Time Management, Customer Service) Composition Number of Quantitative Performance Targets: (e.g.- any measure Interval Range (0-6) of performance that is tied to an objective, measurable standard for the rated dimension) Composition Number of Behavior Performance Targets: (e.g.- subjective Interval Range (0-13) measure of performance in a dimension that is personality traits of behavior and is not easily corrected through formal training such as Initiative, Cooperation/Teamwork, Dependability) Composition Number of open ended evaluation blocks to be filled in by Interval Range (1-25) supervisor Composition Does the appraisal form attempt to quantify scores for qualitative Nominal Yes/No (0/1) performance? Communication Are performance dimensions defined on the appraisal form? Nominal Yes/No (0/1) Communication Level of Form Customization Level of Form Customization Level of Form Customization Does the performance appraisal policy or form require periodic Nominal Yes/No (0/1) performance reviews during rating period? Is the appraisal form used city wide? Nominal Yes/No (0/1) Is the appraisal form tailored to the department? Nominal Yes/No (0/1) Is the appraisal form tailored to the job title? Nominal Yes/No (0/1) Communication Does the performance appraisal process use goal/ objective Nominal Yes/No (0/1) setting at the start of the process? Composition Do the instructions/policies for supervisors filling out the Nominal Yes/No (0/1) performance appraisal encourage the use of quantifiable objectives? Communication Do the performance rating scale/performance definitions Nominal Yes/No (0/1) incorporate quantifiable objectives? Compensation Is the level of rated performance directly related to the size of a Nominal Yes/No (0/1) pay incentive? Compensation Is the appraisal form used to determine pay incentives? Nominal Yes/No (0/1) Compensation Can an individual receive a pay incentive for standard performance? Nominal Yes/No (0/1) 9