Improving Public-Sector Performance Management: One Step Forward, Two Steps Back?

Transcription

1 Improving Public-Sector Performance Management: One Step Forward, Two Steps Back? Carolyn J. Heinrich University of Wisconsin-Madison LaFollette School of Public Affairs and Institute for Research on Poverty August, 2003 Please do not quote, cite or distribute without permission. This research was funded by a grant from the IBM Endowment for the Business of Government, and the author thanks Mark Abramson for his support and guidance throughout the project. Stephen Wandner and Jonathan Simonetta of the U.S. Department of Labor provided data, technical assistance and feedback that were also vital to this work. Will DuPont and Lynette Mooring were excellent research assistants.

2 Abstract The U.S. Department of Labor introduced performance standards and outcome measures to its public employment and training programs more than two decades ago, and a strong emphasis on performance accountability continues as a key feature of the current Workforce Investment Act (WIA) programs. This study uses the WIA performance management system to identify challenges and prospects in implementing performance management systems effectively in public agencies. The process for setting performance standards is studied, and empirical analyses investigate relationships among these standards, states attained performance levels, and differentials between states performance and the standards. The study findings show that setting performance targets is a key task that determines the nature of incentives in the performance management system. In the absence of regular adjustments to these standards for changing local conditions, however, the WIA system appears to have promoted increased risk for program managers rather than shared accountability. Program managers appeared to make undesirable post-hoc accommodations to improve measured performance. This study produces both general lessons about the implementation of performance management systems and more specific feedback and strategies for improving the effectiveness of the WIA system.

3 Introduction Although performance measurement as a management tool has a long history dating back to the 1800s, it is primarily in the last two decades that public-sector performance management has shifted to an explicit focus on measuring outcomes and rewarding results (Heinrich, 2003; Radin, 2000). The Government Performance and Results Act (GPRA) of 1993 mandated the development of outcomes-based performance measurement systems in federal agencies, including annual performance plans specifying quantitatively measurable goals and levels of performance to be achieved and annual reports comparing actual performance with goals. This study is one in a growing body of work that aims to describe and draw lessons from public agencies experiences in implementing these systems and to identify ways to increase their effectiveness, in addition to improving agency performance (Hatry et al., 2003; Heckman, Heinrich and Smith, 2002). 1 Among federal government agencies, the Department of Labor (DOL) has been a pioneer in the development of performance management systems (Barnow, 2000). Before GPRA, the Job Training Partnership Act (JTPA) of 1982 introduced performance standards for public employment and training program outcomes (e.g., job placement rates and trainee earnings) and the use of budgetary incentives based on performance to motivate agency staff. In addition, two randomized experimental evaluations, of the JTPA program in the 1980s and the Job Corps program in the 1990s, provided important information for assessing the performance of these performance standards systems in measuring program impacts. Policymakers and public managers have since drawn from the results of these studies to inform the design and operation of performance standards systems in government programs. 1

4 In the Workforce Investment Act (WIA) of 1998 that recently replaced the JTPA program, a greater emphasis on performance accountability has been described as a hallmark of the legislation (Sheets, 2002; U.S. DOL-ETA, 2001). Some of the broader principles guiding the evolution of this performance management system include those originating in total quality management and reinventing government reforms--the measurement and analysis of results, continuous performance improvement, shared accountability, and a customer and market focus. The DOL is also actively supporting the use of the Malcolm Baldridge Criteria for Performance Excellence as a tool for improving organizational effectiveness. 2 Two new features of the WIA performance management system that were intended to strengthen these principles in implementation are: (1) a new approach to setting performance standards that involves the negotiation of performance targets with states, and (2) new performance measures of customer (participant and employer) satisfaction. This study uses the WIA performance management system as a case study to elucidate some of the challenges and prospects for making basic principles and components of performance management systems work effectively in public agencies. Early studies of the WIA performance management system have suggested that the system is working poorly and is in need of important reforms (U.S. GAO, 2002). Through the analysis of data from states five-year WIA implementation plans, DOL records on state negotiated standards and performance, and other sources of data on participant and local area characteristics, this study produces both general lessons about the implementation of performance management systems and more specific feedback and strategies for improving the effectiveness of the WIA system. The information 2

5 generated in this study should also contribute to ongoing debate and discussions associated with the reauthorization of WIA. The paper proceeds as follows. An overview of the WIA program, the specific goals of the WIA performance management system, and notable changes compared to the JTPA system are presented first. The data and methods for the study are briefly described next. A qualitative analysis of how states determined performance goals, the levels of performance standards, and adjustments to standards under WIA is followed by empirical analyses of variation in and relationships among negotiated standards, states attained performance levels, and differentials between states performance and their negotiated standards. The larger question these analyses address is: How effective is the WIA performance management system in gauging program performance and creating the right incentives to guide program administration and operations in improving outcomes for workers and employers? The paper concludes with recommendations for how the WIA performance management system and similar systems in other government programs might be improved. The WIA performance management system: background information, key features and issues Since the inception of the JTPA program, federal workforce development programs have sought to actively engage the private sector and to promote strong local governance so that employment and training services can be tailored to meet local employer and worker needs. Although the WIA program retains the basic structure and operational components of the JTPA program, important changes were made in the eligibility criteria for workforce development 3

6 services, the types of services made available, and the processes for performance accountability under WIA. In brief, WIA makes available a broader range of core services to the general public (e.g., labor market information and job search assistance), not solely to those who qualify based on lowincome criteria. Individuals access to more intensive levels of service (e.g., comprehensive assessment and case management, vocational or on-the-job training) proceeds sequentially if they fail to achieve success in the labor market following receipt of basic services. These services are typically provided through one-stop centers that include programs of the DOL, the Department of Education, the Department of Health and Human Services, and the Department of Housing and Urban Development. The DOL does not require monitoring and tracking of participants using self-directed, core services or non-wia services at the one-stop centers, but rather only those participants who receive substantial staff assistance in the WIA programs. WIA also established new performance measures and requirements for using specific types of data to evaluate performance. Table A-1 in Appendix A shows the current WIA performance measures and indicates which of these are new to WIA. The addition of the participant and employer satisfaction performance measures was intended to make the workforce development programs broadly accountable to their primary customers: participants and employers. Other new measures are the credential rates for adults, dislocated workers and older youth, which indicate the attainment of a degree, the certification of skills or training completed. The DOL directed states to develop management information systems (MIS) for tracking performance and to use unemployment insurance (UI) records to compute the employment and earnings outcomes of participants. Although some states were able to modify their existing JTPA 4

7 MIS systems, a number of states had to develop new procedures and systems to collect these data. As the GAO (2002) reports, states have struggled to meet DOL requirements for these systems, including the need to maintain lists of participants (i.e., a sampling frame) to use in supplemental data collection through follow-up surveys, the collection of performance data at different time points for different measures, and the use of different participant subgroups (e.g., employed at registration, type of program or level of service received) in calculating performance outcomes. A key feature of the new WIA performance management system (and a primary focus of this study) is the negotiation of performance standards that states are required to meet. Under JTPA, the DOL established expected performance levels using a regression-based model with national departure points. States could use the optional DOL adjustment model or develop their own adjustment procedures, although the state-developed procedures and any adjustments made by the governor had to conform to the DOL s parameters (see Social Policy Research Associates, 1999). A majority of states adopted these models and used the DOL-provided performance standards worksheets (see Appendix A, Table A-2) to determine performance targets, although some with modifications. Under WIA, states negotiate with the DOL and local service delivery areas to establish performance targets, using estimates based on historical data that are similarly intended to take into account differences in economic conditions, participant characteristics and services delivered. The pretext for making this change to a system of negotiated standards was to promote shared accountability, described as one of the guiding principles of the Workforce Investment Act (U.S. DOL-ETA, 2001, p. 8). States own reports of procedures used to determine WIA 5

8 performance standards suggest that there is substantially greater discretion and variation in both the processes and types of information used to establish the state-level standards. Because there are strong incentives (rewards and sanctions) for performance outcomes in WIA, it is important that the data collected and measures used are comparable across states and localities. The level of the negotiated standard is also critical in the determination of performance bonuses and sanctions. In order to be eligible for an incentive grant (up to $3 million), states are required to achieve at least 80 percent of the negotiated performance level for all 17 measures. States that do not meet their performance goals for two consecutive years may be penalized with up to a 5-percent reduction in their WIA grant. Reflecting the increased emphasis on continuous performance improvement in the WIA system, the targeted levels of performance negotiated by states increase over each of the first three years of WIA (PY ) for most states. And although the law allows states to renegotiate standards in cases of unanticipated circumstances, the data show that few states exercised this option in the first three years of the program. The history of the JTPA performance management system suggests important reasons for concern about the determination of performance standards and the incentives they create for program managers and staff. In the mid to late 1980s, reports emerged describing how JTPA program administrators and case workers limited access to program services for more disadvantaged applicants in the effort to improve measured performance, a practice more widely known as cream-skimming. (Dickinson et al., 1988; Orfield and Slessarev, 1986; Anderson, et al., 1993). In addition, Courty and Marschke (1997) showed how program managers strategically organized their trainee inventories and timed participant program exits to maximize end of the year performance levels. Other studies associated a shift to shorter-term, less intensive service 6

9 provision under JTPA with the pressure to produce more immediate, low-cost job placements (Zornitsky and Rubin, 1988; Barnow, 1992). A recent U.S. General Accounting Office ( GAO) report (2002) suggests that history may be repeating itself. The GAO interviewed WIA program administrators in 50 states and visited five sites to assess the effectiveness of the WIA performance management system. The report notes that many states have indicated that the need to meet performance levels may be the driving factor in deciding who receives WIA-funded services at the local level (p. 14). It also describes how some local areas are limiting access to services for individuals who they perceive are less likely to get and retain a job. Observing the serious challenges that states and localities have faced in implementing the system, the GAO suggests that even when fully implemented, WIA performance measures may still not provide a true picture of WIA-funded program performance (U.S. GAO, 2002, p. 3). In a summary report to the U.S. Department of Labor on the implementation of WIA, Barnow and Gubits (forthcoming, fn. 12) also found, based on meetings with officials from about 20 states, that the greatest dissatisfaction in every instance has been with the way the performance management system has been implemented. Study data and methods As described in the preceding section, three elements are key to the WIA performance management system and to similar systems in other government programs: (1) performance measures to evaluate progress toward performance goals, (2) a method for setting standards and measuring performance against the standards, and (3) rewards and sanctions that generate incentives for the achievement of performance goals. This analysis begins with an investigation, 7

10 primarily qualitative, of how performance goals and performance standard levels were established under WIA s new system. Data for this first part of the study come from: 1. Five-year plans, mandated by the DOL and developed by states, describing how states would implement the WIA program and the performance management system. 2. Guidelines issued by the DOL for performance standards negotiations and parameters recommended for use as baseline values in negotiations. The DOL also established national goals for the WIA performance measures. 3. Data from the DOL on the final levels of negotiated performance standards set by the states. 4. Data from the DOL s Standardized Program Information Reports (SPIR) on local participant characteristics and services delivered by JTPA agencies in program year 1998, the baseline year used by a majority of states in determining performance standards. In the WIA five-year plans, states had to indicate the performance standards established for each of the core indicators (see Appendix A, Table A-1) for program years and to explain how they determined the levels of performance goals. They were also required to describe the management information systems and reporting processes used to track performance, and how these data would be disseminated and used to improve services and customer satisfaction. States were given the option to submit the plans for early transition by July 1, 1999 or to submit them later by April 2000, before the July 1, 2000 WIA start date. 8

11 Although the WIA state plans are stored in the DOL electronic archives, less than half of the electronic links were functional in early Contact with state WIA officials and website searches produced a total of 50 (out of 52) of these plans. 4 Information in these plans about states negotiated performance targets, the process by which these performance levels or standards were established, and how they compared to national goals and projected national averages of the standards was extracted for analysis. Forty-four states had complete information about their specific performance targets. The second part of the study applies correlation and regression analysis to investigate relationships among negotiated performance standards, states attained performance, and differentials between states performance and their negotiated standards. The data for these analyses include: 1. Information from the DOL on states reported (actual) performance levels in 8 quarters under WIA (2 nd quarter PY st quarter PY 2002) for each of 17 performance standards. 2. DOL data on the final levels of negotiated performance standards set by the states. 3. Bureau of Labor Statistics data on state economic conditions by year, in addition to the SPIR data on other state and local characteristics. Four sets of analyses are conducted to compare states workforce development performance to negotiated standards and other relevant variables. The first set of analyses computes the differential between states performance and their negotiated standards and examines how these differentials vary across states and by program year. A second set investigates the relationship between states attained performance levels and baseline participant 9

12 and area characteristics to determine if there are associations between performance and these variables for which adjustments to standards were intended to be made. A third set of analyses examines associations between the performance differentials and states baseline participant and area characteristics. If, in fact, the process of negotiating standards effectively adjusts states standards to account for local participant and area characteristics, then the relationships among these variables should be weaker than those in the second set of analyses described above. Finally, the last set of analyses focuses on the new participant and employer satisfaction performance measures and investigates whether there are significant associations between these measures and the more objective measures of employment, earnings, retention, education and skill attainment of WIA participants. Determination of Performance Standards under WIA As this research and some of the studies discussed above suggest, the determination of performance standards (or minimum levels of performance to be achieved) is a key task in the design and implementation of performance management systems that significantly influences the incentives for public managers and staff operating programs. An important concern for those involved in setting standards is to use a process that creates a level playing field (Social Policy Research Associates, 1999). Public managers do not want to be held accountable for factors outside their control or to be unfairly compared to other agencies with different client populations, economic environments, and other extenuating factors. At the same time, policymakers want to use the system to motivate performance improvements, and in the case of WIA, to promote shared accountability for results. This is 10

13 likely to require an approach that engages public managers in the process of setting performance standards and makes an attempt to balance risks (e.g., for unanticipated conditions or changes in external factors that affect outcomes) among the parties involved. Procedures for Setting State Performance Standards in WIA One important source of data for setting performance standards is historical (or baseline) information on past levels of performance achievement, to the extent that these data are available. Since performance data were collected in the JTPA program, more than half of the states used some baseline performance measures to determine appropriate levels for the WIA negotiated performance standards. The baseline data typically came from several different sources: projected national averages for the negotiated standards provided by the DOL (based on the experiences of seven early implementation states), federal baseline numbers (available in the federal performance tracking system, i.e., SPIR data), 5 unemployment insurance data, and states own performance baselines from previous program years. Georgia, for example, used program year (PY) 1998 state performance records combined with the projected national averages in negotiations with regional office representatives and local-level officials to determine the performance targets for the first three years of WIA. Indiana reported that it used PY 1999 performance data to determine the performance standards, but it did not have time for consultations with local workforce development officials in setting the goals; only first-year (PY 2000) goals were presented in Indiana s five-year plan. Some states, such as New Hampshire and Ohio, used UI data from earlier periods (PY ) combined with DOL performance data available in the SPIR to set performance levels. 11

14 About one-half of the states also explicitly indicated that negotiations with local workforce development officials were important in determining performance standards, and many of these also used some type of baseline data to inform the discussions. States were instructed to take into account differences in economic conditions, participant characteristics, and services provided. For a majority, these adjustments to standards were made informally during the review of baseline information and negotiations. For example, Wisconsin reported using PY 1997 data and the projected averages in negotiations with local officials to set the standards. A comparison of these data in Wisconsin s five-year plan shows that when Wisconsin s PY 1997 baseline was above the projected national averages, the projected averages were established as the targets. When Wisconsin s baseline numbers were below the projected national averages, the baseline values were typically set as the targets. The states of Washington, Nebraska, South Carolina and others followed a similar process. It was rare, as in the case of the state of New York, that all of the state s performance baseline measures were above the national targets and were set as the standards for PY Only the states of Texas, Maryland and the District of Columbia reported using statistical models to determine the performance standards. 6 Table 1 presents descriptive statistics on the levels of performance standards set by the states using data from the DOL on the final negotiated standards. These statistics confirm that there is considerable variation across the states in the levels of performance standards established through the negotiation process. Interestingly, among the new WIA performance standards (for which no historical performance information was available), the participant and employer satisfaction standards vary the least across the states (standard deviations 2.8 and 3.2, 12

15 respectively), while the credential rate measures (for all groups) have comparatively large standard deviations ( ). Comparison of State Performance Targets with National Goals The DOL also established national goals for the first three years of WIA performance measures (see Appendix A, Table A-3) that reflect the articulated objective of continuous performance improvement. As reported by the Employment and Training Administration (DOL, 2002: 2), WIA envisions a high-performance workforce system that is continuously improving and delivering high quality services to its customers. States negotiated performance standards were compared to these national goals. For about one-third of the states for which information on the specific levels of negotiated performance is included in their five-year plans, some of the state targets are above the national goals, and some are below, likely reflecting risk-balancing, standard-setting strategies such as those used by Wisconsin. The negotiated standards are mostly or all above the national goals for another third of these states, although only four had standards set higher than all of the national targets. Arkansas was unique in setting each of its standards (with the exception of the earnings change measures) exactly 1 percentage point above the national goals in the first year. Among the others, just three states had established performance standards that were all below the national goals. North Carolina, for example, used PY 1997 baseline data in its determination of performance standards, and all of the standards were set significantly below both the state baseline measures and national goals. A few states, such as Alabama, also adopted a more risk-adverse approach, setting some performance standards lower than baseline values to allow time for adjustment to the new system. 13

16 In general, the performance targets established in the state 5-year plans for WIA implementation reflected the continuous performance improvement objective. These planned targets and the final negotiated standards for the states 7 for the first three years of WIA show that the negotiated standards, on average, increased about 1 to 2 ½ percentage points between PY 2000 and PY 2001 and between PY 2001 and PY 2002 (see Table 2). In addition, the mean expected increase in performance levels is larger between PY 2001 and PY 2002 than that going from PY 2000 to PY 2001 for most standards. The states, in effect, set target levels that not only required that they improve over time, but also that the magnitude of the improvements increase from year to year. Adjustments to Performance Standards In addition to accounting for factors (demographic, economic or others) known at the time that performance standards are established, it is important to allow for adjustments to standards that will offset future or unknown risks of poor performance due to conditions or circumstances beyond the control of public managers. As described above, many states used baseline performance data from program years 1999, 1998, 1997 or earlier to establish performance standards for the first year of WIA and then also built in anticipated performance improvements for the two subsequent years. Economic conditions changed significantly, however, between the pre-wia period and first three years of WIA implementation. Between 1998 and 1999, unemployment rates were declining on average, with a median decline of 0.2 percent and 75 percent of all states experiencing a decline. This pattern continued in the year before WIA (1999 to 2000). Between 2000 and 2001, however, this trend reversed. More than 75 percent of the states experienced an 14

17 increase in unemployment rates over the course of this year, with a median increase of 0.7 percent. Increases in unemployment rates were even greater between 2001 and 2002, with all states experiencing an increase in unemployment except one that was unchanged. Thus, at the same time that unemployment rates were increasing and creating adverse labor market conditions for trainees in the first three years of WIA, the standards for performance achievement in the program were increasing. (See Figure 1 below.) Figure 1. Program Program Program Performance goal and local year 2000 year 2001 year 2002 labor market conditions Mean entered employment rate standard for adults Mean unemployment rate Despite these dramatic changes in economic conditions, less than a third of the states final negotiated standards were changed from those proposed in their 5-year plans. A few states final negotiated standards, such as those in North Carolina and Delaware, were higher than originally presented in their plan. Where changes were made, however, it was more common to lower the negotiated standards. The District of Columbia, Georgia, Idaho, Missouri, New York, Oregon and Washington, for example, adjusted one or more of their performance standards downward over these program years. Among the small number of states that made changes, they were most likely to lower their older youth or displaced worker standards. One Texas official expressed her concern in a phone conversation that even with Texas relatively sophisticated statistical model for setting performance standards, adequate adjustments had not been made for economic conditions. She noted that older youth were most likely to experience poor labor market outcomes in a recession, as adults would take any job and thereby displace the older youth. 15

18 Under JTPA, performance standards were adjusted annually, as shown in Appendix A, Table A-2. The WIA guidelines directed that the negotiated performance targets take into account local economic conditions, participant characteristics, and services delivered in the states. Renegotiation appears to be more of an exception, however, than a routine procedure. The relationship of the final negotiated (or re-negotiated) standards to these local variables was examined empirically in correlation and regression analyses using DOL SPIR data that was only available through program year 1998, the baseline year used by a majority of states in determining performance standards. 8 The question of interest in this analysis is whether the negotiated or re-negotiated performance standards appear to account (or adjust) for differences across states. The simple correlation analyses showed only two consistent associations among negotiated performance standards and participant characteristics. States with higher percentages of Hispanic and limited English proficiency populations had significantly lower performance targets for all adult, dislocated worker, and youth performance measures (correlation coefficients ranging from r = to , p< for all). The correlation between percentage Hispanic and limited English proficiency, not surprisingly, was very high at r=0.819; the percentage of the participant population that was Hispanic was also significantly and positively correlated (p<0.0001) with the percentage who were single heads of households, had less than a high school degree, lacked work experience, had a skills deficiency, and were not in the labor force. Among the three states with performance targets in their 5-year plan that were all below the national goals, California had the largest proportion of Hispanics among its PY 1998 participant population (34.4%), nearly onefourth of Rhode Island s participant population was Hispanic, and North Carolina had the fastest 16

19 growing Hispanic population (over 400% increase) between the 1990 and 2000 U.S. Censuses. In addition, correlations with state unemployment rates (in 1998) showed that states with higher rates of unemployment had significantly lower standards for adult, dislocated worker and older youth entered employment rates and younger youth employment retention rates. Ordinary least squares regressions indicated a few more statistically significant relationships among baseline participant and economic characteristics and the performance standards negotiated by the states, although these relationships tended to vary across the different standards. 9 For example, a more highly educated participant population in 1998 was significantly and positively associated with higher standards for entered employment rates, although not for earnings change or employment retention standards. In addition, the most important factors affecting entered employment rate standards for adults and dislocated workers were unemployment rates in 1998 and the change in unemployment rates between 1998 and The regression models also included measures of the employment and earnings outcomes of PY1998 participants to account for past performance, and these variables were statistically significant and positively associated with performance standard levels in most models. However, it was still the case that the most consistent relationships across standards were the statistically significant and negative associations between higher percentages of Hispanics or participants with limited English proficiency and performance standard levels. Although documentation is not available to confirm that adjustments were being made deliberately in negotiations to account for these specific baseline characteristics, the empirical findings above suggest this may be occurring. Interestingly, Table A-2 in Appendix A shows that the PY 1998 JTPA performance standards adjustment worksheet did not explicitly allow for 17

20 adjustments for the proportion of Hispanics, 10 although it did adjust for lack of work history, less than a high school education and not in the labor force, all of which are positively correlated with the percentage of Hispanics. The JTPA adjustment model did take into account the percentage of the local area that was black, however, where being black was significantly correlated with having less than a high school degree and a skills deficiency. The analyses presented in the next section provide some indication of how effectively the new system of negotiated performance standards works in adjusting for local characteristics and economic conditions. States Performance under WIA: Is It Up to Standards? States performance relative to negotiated targets The difference between a state s attained performance level in a given quarter and the performance target for that particular program year was computed for each of the 17 performance standards over the 8 quarters for which performance data were available. Table 3 presents some descriptive statistics on the magnitude of these differentials by program year. A positive differential indicates that, on average, states were exceeding their negotiated targets. The relatively large standard deviations associated with each of these measures suggests that there is considerable variation among the states in their performance achievements (relative to standards). Table 4 shows the proportion of states that met or exceeded their performance targets in the 2nd- 4th quarters of PY 2000, PY 2001, and the first quarter of PY Simple correlations among the computed performance differentials for the 17 different measures showed nearly all positive correlations, some weaker and some stronger and significant relationships. This is a result program managers should like to see, as it suggests that there are 18