Sheena McConnell, Irma Perez-Johnson, and Paul Decker. October 2005

Transcription

1 USING VOUCHERS IN GOVERNMENT TRAINING PROGRAMS: FINDINGS FROM THE INDIVIDUAL TRAINING ACCOUNT EXPERIMENT Sheena McConnell, Irma Perez-Johnson, and Paul Decker October 2005 This research was funded under a contract from the US. Department of Labor (contract number N ). The views expressed herein do not necessarily reflect the policies or opinions of the U.S. Department of Labor.

2 Vouchers are increasingly being used to deliver government-funded goods and services. Vouchers meet the government s desire to target consumption by limiting the consumers choices to a specified set of goods or services; at the same time vouchers allow consumers to choose the type of goods or services and their provider that is best for them. Examples of government-funded goods and services delivered by vouchers include food (such as food stamps and WIC), childcare, housing to low-income families (such as Section 8 housing), K-12 schooling, job retention services (such as clothing, gas, and computer training), and prescription drugs (Posner et al. 2000). Since implementation of the Workforce Investment Act of 1998 (WIA), vouchers have also been used for government-funded occupation training. The effectiveness of any voucher program depends on how the vouchers are administered. Vouchers vary in how they are administered in three main ways. First, they differ in the amount of information and assistance consumers are provided to help them make informed choices. In some programs, little information is provided; while in others, extensive information is provided. In childcare programs, for example, the government usually provides families a guide to local childcare providers. The second way that the administration of vouchers varies is the extent to which the program ensures a minimum level of the quality of goods and services purchased with the vouchers. Typically, the choice of goods and services are restricted to those that have met certain minimum quality standards. The U.S. Department of Housing and Urban Development, for example, requires that housing units that can be rented using Section 8 vouchers meet certain standards for health and safety. The third source of variation is how the vouchers are used to ration goods or services among consumers. Thus vouchers vary in their value, whether the value is the same for all consumers or varies by consumers needs or other factors, and the extent to which the consumers are required to contribute payment for the goods or services. 2

3 This paper explores the impacts of different approaches to administering vouchers or individual training accounts (ITAs)--for occupation training under WIA. As WIA requires that local workforce agencies determine their own approach to implementing ITAs, this study is of particular policy interest. It is also interesting given the Bush Administration s interest in a new but similar voucher initiative known as Personal Reemployment Accounts (PRAs). PRAs are vouchers that can pay for a wide range of goods and services for unemployed workers or to pay a cash bonus for rapid reemployment. Using a classic experiment implemented in eight sites, we examine the impact of three different approaches to administering ITAs. These approaches vary in how they ensure that the consumer, or customer in the language of WIA, has made an informed choice, whether the customer s choice can be rejected, and how the value of each ITA is determined. We examine the impact of each approach on the amount of counseling customers receive about their choices, the likelihood the customers will use the training voucher, the price of selected training programs and the cost of the ITA to the workforce agency, and the training program and provider selected. The rest of this paper is organized into seven sections. Section A provides the policy context for the use of vouchers in occupation training. Section B describes issues that arise in the administration of ITAs. Section C describes the three tested approaches. The experimental design and data collection are described in Section D. Section E describes the characteristics of the customers in the sample. The main study findings are described in Section F. The paper ends, in Section G, with a discussion of the implications of the findings and a description of further research. 3

4 A. POLICY CONTEXT The mandate to use vouchers under WIA reflects a trend toward promoting customer choice and provider competition in government training programs that has been ongoing for several decades. The first large-scale government-funded training program, established in 1962 under the Manpower Development and Training Act (MDTA), contracted directly with service providers for classroom training. The program directed trainees to one of these contracted providers. Under the Comprehensive Employment and Training Act (CETA), which replaced MDTA in 1973, local service providers competed to receive CETA funding disbursed by local workforce agencies or prime sponsors. Under the Job Training Partnership Act (JTPA), which superseded CETA in 1982, local workforce agencies still contracted with providers to provide blocks of classroom training. However, local workforce agencies could also refer individual customers to courses already offered by service providers if preferred by the customer (Barnow 2000). And indeed many local workforce agencies chose to move away from the use of contracted training and toward individual referrals (D Amico et al. 2001). Some local workforce agencies even began to use vouchers for training (Barnow and Trutko 1999). One of the key objectives of WIA, which replaced JTPA in 1998, was to provide customers more choice in employment and training services. WIA mandated that ITAs be given to nearly all persons eligible for WIA training. This included persons funded under both the dislocated worker program and those funded under the adult program (for persons 18 years of age or older who are not dislocated workers). An ITA has a dollar value and can be used to purchase training services up to that dollar value. Customers can purchase more expensive training by supplementing the ITA with their own funds or funds from other sources. However, any balance remaining in the ITA after the training program is paid for is kept by the workforce agency. 4

5 WIA placed two restrictions on how ITAs can be used. First, the customer must choose a program that will provide training for an occupation in demand in the local area. However, WIA left up to local discretion how occupations in demand were to be identified. Second, the training program that the customer chooses must be on the eligible training provider (ETP) list. Training providers need to apply for each of their training programs to be on the ETP list by submitting information about their program and student outcomes to the local workforce agencies. To promote informed choice, WIA also required states to make available to customers a Consumer Report System which provides information on training programs including student outcomes, program cost, program length, and certifications available on completion of the program. Another key objective of WIA was to provide flexibility for local workforce agencies to design their own policies. Hence, local workforce agencies decide how to implement their ITA program--the amount of each ITA, counseling requirements, the criteria under which a training program is approved, and the criteria for whether an occupation is in demand. B. DESIGNING VOUCHERS FOR TRAINING The theoretical advantage of using vouchers for delivering training is that they promote economic efficiency. Vouchers promote efficiency in two ways. First, informed customers choose training programs that maximize their individual well-being and hence social welfare (as defined by the government and its goals). Second, customer choice increases competition among training providers, which may lead to better quality or less costly training programs. 1. Providing Information Underlying the theoretical advantage of vouchers is the assumption that customers are informed. However, the efficiency benefit of vouchers may not be realized if customers are 5

6 inadequately informed about their own abilities, the quality of the training provider, or the expected wages and employment prospects in the occupation for which they are training (Barnow 2000). Customers may be uninformed because it is too costly to become informed or they may not have the ability to become informed (Steuerle 2000). To the extent that customers are not informed, local workforce agencies may need to provide information to customers or require that customers show they are informed. Concerns about uninformed customers have led many workforce agencies to require customers to receive counseling and conduct research on the selected occupation and training program before they are issued a voucher. Studies of voucher programs under JTPA and WIA found that customers are typically required to undertake an assessment of their skills and abilities, to show that they have engaged in extensive research on training programs and the labor market, and to have extensive discussions with the counselors before a voucher is issued (Barnow and Trutko 1999; D Amico and Salzman 2004). In some sites, customers are required to submit a formal application in which they justify their choice of training program. 2. Constraining Choices The efficiency benefit of vouchers may also not be realized if individual customers choosing training programs that maximize their own well-being do not also maximize social welfare (Barnow 2000). For example, it may be that the government places a higher emphasis on the financial return to training than the customer. As public assistance removes some of the cost of unemployment, customers may place a lower emphasis on the probability of job placement after training than is socially desirable. Some customers may evaluate training programs based on factors that the government views as less important, such as whether they have friends who will also attend the program. 6

7 If individuals maximizing their own well-being do not maximize social welfare, it may be optimal for workforce agencies to place more restrictions on customers choices. WIA already places some constraints on customers choices: they can only choose providers that are on the ETP list (and so meet some minimum quality standards) and choose occupations that are in demand. Local workforce agencies vary in how they determine whether an occupation is in demand (D Amico and Salzman 2004). Some draw on lists of occupations with job vacancies and relatively high wages developed locally or by the state s labor market research unit. Others just rely on employment counselors to make the determination. Local workforce agencies can place further constraints on customers choice of training. In a study of training voucher programs under JTPA, Barnow and Trutko (1999) found that local workforce agency staff felt that the authority to reject a customer s training choice was important to prevent customers making poor training choices and wasting resources. In practice, however, few agencies constrained customers choices any further. D Amico and Salzman (2004) found that employment and training counselors in the workforce agencies in their study rarely rejected a customer s choice of training program. 3. Rationing Resources The Voucher s Value Funds for training are limited and hence need to be rationed among customers. One approach is to provide each customer the same value voucher. This approach has the advantage of being equitable and simple. However, if the return to training differs by customer, this equity and simplicity may be gained at the cost of a loss of social welfare. If the local workforce agency can identify those customers who receive the largest return to training, it may be better for the agency to provide a larger voucher to those customers who will receive a higher return on their training. 7

8 In practice, most local workforce agencies provide an ITA with the same fixed value to all its customers (D Amico and Salzman 2000). Some local workforce agencies, such as the one in Jacksonville, Florida, however, set the ITA value at three different levels, providing a higher ITA for training for occupations with higher wages. C. THE APPROACHES TESTED IN THE ITA EXPERIMENT While some workforce agencies had experience with administering vouchers for training after WIA was passed, the vast majority of workforce agencies did not. But, as described above, they are faced with key decisions about how to administer ITAa how much information to provide, the constraints placed on customer choices, and how to set the value of the ITA. To help workforce agencies make informed decisions about how to administer the ITAs, the U.S. Department of Labor sponsored Mathematica Policy Research, Inc. to conduct the ITA Experiment. The ITA Experiment tested three approaches to administering ITAs (Table 1). The three approaches we tested vary along the three key implementation dimensions: (1) the intensity of mandatory counseling about the training choice; (2) whether the counselor can reject the customer s choice; and (3) whether the ITA amount is the same for all customers or is set by the counselor depending on the expected return to training. Table 1: Summary of Approaches Approach 1: Structured Choice Approach 2: Guided Choice Approach 3: Maximum Choice Counseling Mandatory, Mandatory, Voluntary Intensive Less intensive Counselors Can Yes No No Reject Choices? Award Amount Customized Fixed cap Fixed cap 8

9 The approaches tested reflect a spectrum of different approaches to administering ITAs. At one end of the spectrum, we tested an approach in which counselors play a major role in the choice of training program (the structured choice approach). At the other end of the spectrum, we tested an approach with few constraints or requirements (the maximum choice approach). The approach in the middle of the spectrum, the guided choice approach, approximates the approach that many workforce agencies currently use. Under this approach, counselors have some limited role in the training decisions. 1. Approach 1: Structured Choice Approach 1 places considerable emphasis on the counselors role in guiding customers to make wise investments in training. The assumption underlying this approach is that without counseling customers would not always choose the training program with the highest expected return to the government s investment. Hence, the counselors provide extensive counseling and require that customers show that they are informed. They can also reject a training choice that is not expected to have a high economic return. The counselors also tailor the amount of the ITA to each customer, giving a larger amount for training with higher returns. The counseling activities required before an ITA is issued under Approach 1 are highly structured. Customers are required to identify an occupation to train for from a list of high-wage, high-demand occupations developed by the site. Customers are also required to show that they have thoroughly researched the various training programs available for their chosen occupation. For example, customers are required to complete a form for each of two or three training programs with detailed information about the program, such as typical class size, beginning date of the course, and the time of the classes. They are also required to list all the costs of the training including books, supplies, and any necessary childcare. Counselors are required to discuss the customers family budgets during the time they will be in training to ensure that they 9

10 have sufficient resources to complete training. Finally, counselors are required to walk through with the customer a benefit-cost analysis of the customer s training choice. This involves considering all the costs of the training (even those incurred by the government), including the opportunity cost of the customers time, and comparing these costs with the expected benefits from increased earnings from receiving the training. Approach 1 is the only approach in which the counselor has the ability to reject a training choice made by the customer. Potential reasons for rejecting a choice includes that the expected benefits from the training do not exceed its costs, the net benefits of another training program are higher than the one chosen by the customer, or the counselor thinks that the customer has insufficient financial resources to live on while participating in the training program. If the counselor rejects a choice, the customer can make another choice, but the counselor is not required to accept any choice. Under Approach 1 the counselor tailors the value of the ITA to each customer. Counselors are directed to provide an ITA of greater value to people choosing training with higher net benefits and to provide an ITA of lower value or to deny an ITA to people choosing training with lower net benefits. The goal was that the workforce agency should spend on average the same amount on customers under Approach 1 as it does on customers under Approaches 2 and 3. In response to requests from the sites, we placed a cap on the amount of the ITA a counselor could award under Approach 1. However, the cap was set high enough that in most cases it was not expected to be binding. The cap on the ITA award was set at $8,000 in six sites, $7,000 in one site, and $6,000 in one site. 2. Approach 2: Guided Choice 10

11 Approach 2 approximates the approach currently used by many workforce agencies. Like Approach 1, it assumes that without counseling some customers would not be sufficiently informed to make good decisions. Hence, like customers under Approach 1, Approach 2 customers are required to conduct some counseling activities. The counseling requirements are similar to those under Approach 1, except that the customers do not need to consider the benefits and costs of the training (although the counselors would discuss these if requested to) nor do they need to choose an occupation from the list of high-wage/high-demand occupations. Unlike Approach 1, however, this approach assumes that well-informed customers will make decisions that are in line with maximizing social welfare. Hence, under Approach 2, counselors cannot reject the customer s choice as long as the program is on the ETP list. Following the usual practice of workforce agencies, each Approach 2 customer receives an ITA of the same value. The value of the ITA was set by each site at a level that was sufficient to pay for many occupation training programs in the area, but low enough that they did not expect to run out of training funds before the end of the year. The ITA value was $3,000 in five sites, $4,000 in two sites, and $5,000 in one site. 3. Approach 3: Maximum Choice Approach 3 is the least restrictive voucher approach. This approach assumes that customers will voluntarily acquire the information they need to make and informed training decision and by maximizing their own well-being, they also maximize social welfare. Hence, customers under Approach 3 are not required to participate in any particular counseling activities, but can voluntarily choose to participate in any counseling activities available to customers under Approaches 1 and 2. As under Approach 2, counselors cannot reject the customer s choice as long as the program is on the ETP list and each customer is provided an ITA of the same value. 11

12 The ITA value was set at the same value for Approaches 2 and 3 customers. Hence, the only difference between Approaches 2 and 3 is the mandatory counseling required under Approach 2. D. EVALUATION DESIGN The three ITA approaches were implemented side by side in each of eight study sites: (1) Phoenix, Arizona; (2) Maricopa County, Arizona; (3) Bridgeport, Connecticut; (4) Jacksonville, Florida; (5) Atlanta, Georgia; (6) Northeast Region, Georgia; (7) North Cook County, Illinois; and (8) Charlotte, North Carolina. These study sites were chosen purposively by the U.S. Department of Labor from among the workforce agencies that applied to participate in the study. The enrollment period for the study differed slightly by site. It began on a rolling basis between December 2001 (in North Cook County) and August 2002 (in Bridgeport) and continued for about 18 months in each site. The first month of enrollment was considered a pilot; data from that time period were not included in the analysis. All customers who were found eligible for training during the enrollment period were randomly assigned to one of the three approaches. They were assigned with a probability of onethird to each approach. Before being randomly assigned, customers gave their written consent for participation in the study and completed a baseline information form that requested demographic and socio-economic information. In total, about 8,000 people were randomly assigned to the three ITA approaches. After random assignment, all customers were sent a letter notifying them of their assigned approach. The letter provided a brief description of their assigned approach, including the value of the ITA under Approaches 2 and 3 (but not the cap on the value of an ITA under Approach 1). Customers were then required to attend an approach-specific orientation, which provided details about the ITA approach to which they were assigned. The site assigned the study 12

13 participant to a counselor in the usual way and counselors were notified of the customer s assignment. The use of random assignment ensures that customers assigned to each of the three approaches will have similar baseline characteristics on average. As a result, any observed differences in participant outcomes across the approaches can be directly attributed to differences in the ITA approaches with a known degree of statistical precision. Hence, most of the findings in this paper involve comparing average outcomes across the three approaches. All counselors in the study worked with customers under all three approaches. Hence, none of the differences in outcomes across approaches can be attributed to differences in the counselors across approaches. Our monitoring of counseling suggested that counselors were successful in administrating a different approach to different customers. One potential limitation of testing all three approaches in each site, however, is that we are not able to test whether the three approaches differ in their effects on the training providers. It is possible that the way the vouchers are implemented could affect the pricing and quality decisions of the training providers. A computerized information system was designed specifically for the study. This study information system was used (by both site staff and researchers) to track study participants, including their participation in counseling, whether they received training, the training program they chose, and the value of the ITA. Researchers also conducted three rounds of in-depth site visits to each site (in 2002, 2003, and 2004). E. CHARACTERISTICS OF THE STUDY SAMPLE As expected, the sample of nearly 8,000 customers was evenly distributed across the three ITA approaches. Table 2 shows the characteristics of these customers using data collected on the baseline information form just prior to random assignment. For the most part, the random assignment process yielded statistically equivalent groups of participants under each approach. 13

14 There was one exception. Relative to Approach 2 customers, Approach 1 customers were slightly more likely to be employed at the time of enrollment in the study. These customers are more likely to be funded under the adult WIA program rather than the dislocated worker program. This difference does not affect the overall findings of the paper as we found the impacts similar for customers funded by the adult and dislocated worker programs. 14

15 Table 2. Characteristics of ITA Study Participants, by Approach (Percentages Unless Otherwise Noted) Characteristic All Approach 1 Approach 2 Approach 3 Gender Female Race One race White Black or African American American Indian or Alaskan Native Asian or Pacific Islander Other Two or more races Hispanic or Latino Average Age at Enrollment (Years) Years of Regular Schooling Less than to More than Employment Status at Enrollment Unemployed Employed 9 10** 8 10 Had never worked Years Worked at Current/Most Recent Job Less than 1 year to 5 years to 9 years or more years Average Funding Source Dislocated worker 69 67*** Adult 31 33*** Received Public Assistance at Enrollment Sample Size 7,922 2,646 2,649 2,627 Source: Baseline information form. */**/*** = Difference relative to Approach 2 is statistically significant at the.10/.05/.01 confidence level. 15

16 The customers in the ITA study are quite diverse in their characteristics (Table 2). While nearly all study participants had more than 12 years of education, 10 percent had fewer than 12 years of education, and 9 percent had more than 16 years of education. While their average age was 40, some were as young as 18 and others as old as 81. Just over half the customers were female; just under half were white. Most customers were unemployed when they were randomly assigned; only about nine percent were employed. Most customers had less than five years of experience in their current or most recent job. About two-thirds of the sample members were funded through the dislocated worker program. The study sample is not evenly distributed across sites (Table 3). One site, North Cook County, accounts for 23 percent of the study sample, while Northeast Region, Georgia accounts for only 2 percent of the sample. These differences reflect the differences in enrollment periods (the enrollment period was about four months longer in North Cook County), differences in the size of the populations served, and differences in the funds allocated to training in each site. The distribution across sites makes little difference to our findings, as the estimated impacts of each approach are remarkably similar across sites. Table 3. Participation in the ITA Experiment (Percentages) Overall Approach 1 Approach 2 Approach 3 Total Enrollment Phoenix, AZ Maricopa County, AZ Bridgeport, CT Jacksonville, FL Atlanta, GA Northeast Region, GA North Cook County, IL Charlotte, NC Sample Sizes 7,922 2,646 2,649 2,627 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). 16

17 F. FINDINGS The impact of mandatory counseling and the ability of the counselors to reject customers choices depends on the customers prior knowledge of occupations and training programs, the time and burden cost to the customers of counseling, the knowledge and expertise of the counselors, and how additional information changes the customers decisions. The impacts of tailoring the value of the ITA to the needs of the customers depends on the extent to which the value of the ITA is a binding constraint to the choice of training program and the ability of the counselors to tailor the ITA appropriately. This paper examines the impacts of the administration of ITAs on four outcomes: (1) receipt of counseling, (2) receipt of an ITA, (3) the price of training programs chosen by ITA recipients and the expenditures on ITAs, and (4) the customers selections of training programs and providers. Before describing our estimated impacts, however, we describe how the planned approaches were actually implemented by the sites. 1. Implementation of the Approaches by the Sites Our monitoring of counseling activities and discussions with workforce agency staff suggests that, in general, the sites implemented Approaches 2 and 3 as planned. Discussions with counselors in the study sites revealed that they were most comfortable with Approach 2--the approach most similar to the approach used by the sites prior to the study. Counselors were less comfortable with Approach 3. They were concerned that without counseling, customers would make poor decisions. Despite this concern, counselors implemented Approach 3 as planned and did not require customers in Approach 3 to participate in counseling. Counselors did not, however, implement Approach 1 as planned. While they enforced the counseling requirement and tailored the ITA amount to each customer, they found it very difficult to steer customers to high-return training, and rarely, if ever, denied training to anyone. 17

18 They also found it difficult to control the expenditures on ITAs. As shown later, expenditures on ITA under Approach 1 were much higher than under Approaches 2 and 3. Instead of taking the more directive approach required under Approach 1, counselors tended to defer to the customer. Many customers already had chosen a training program when they were found eligible for training and counselors rarely attempted to change the customer s choice. Counselors gave several reasons for the lack of fidelity to Approach 1. Many counselors felt that counseling should be a collaborative process and it was important to respect customers wishes. They felt that if they steered customers to training programs that were not their first choice, the customers would be less likely to complete the training. Many felt the information that was available to calculate the net benefits of a training program was too inaccurate to use as a basis for decisions. And for some specialized occupations, such as information technology, counselors felt unqualified to prescribe a specific training strategy. This deviation of actual from planned implementation of Approach 1 implies that, in practice, the three approaches differed mainly along only two dimensions: the amount of required counseling and whether the ITA cap was fixed or customized. 2. Receipt of Counseling One of the key ways in which the approaches differed was the extent of mandatory counseling. Counseling was mandatory under Approaches 1 and 2, with the counseling requirements being more extensive under Approach 1. Counseling was voluntary under Approach 3. While this paper discusses counseling received after random assignment, most of the study sites also required counseling prior to determining eligibility for training. As this occurred prior to random assignment, the amount of counseling received by customers would on average be the same under each approach. 18

19 The counseling requirements under Approaches 1 and 2 seem to discourage customers from pursuing ITA-funded training. To receive an ITA, all customers had to attend a mandatory approach-specific orientation after random assignment. The proportion of invited customers who attended this orientation varied by approach (Table 4), even though all that had differed between approaches at this point was the content of the letters notifying the customers of their assignment. Approach 3 customers were significantly more likely to attend the orientation than Approach 2 customers (74 percent compared with 67 percent). Although the overall rate at which study participants attended an orientation varied considerably across sites, being assigned to Approach 3 increased the probability of the participant attending an orientation in every site and the impact was statistically significant in five sites (Table 4). This suggests that just the knowledge of the counseling requirements discouraged customers from pursuing training. On the other hand, there was no significant difference between the likelihood of customers in Approaches 1 and 2 attending an orientation overall. The proportion of Approach 1 participants who attended to an orientation was comparable to the proportion of Approach 2 participants who attended (68 percent compared with 67 percent). Approach 1 customers were slightly more likely to attend an orientation in three sites, although the difference was significant at a 5 percent confidence level only in Maricopa County. 19

20 Table 4. Participation in ITA Counseling (Percentages Unless Otherwise Noted) Overall Approach 1 Approach 2 Approach 3 Attended Approach-Specific ITA Orientation Overall *** Phoenix, AZ *** Maricopa County, AZ ** *** Bridgeport, CT *** Jacksonville, FL * Atlanta, GA Northeast Region, GA North Cook County, IL * ** Charlotte, NC * ** Attended Counseling Beyond the ITA Orientation *** *** Average Number of Counseling Sessions All ITA study participants *** *** Participants who attended an ITA orientation *** *** Sample Size 7,922 2,646 2,649 2,627 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). Notes: Percentages may not sum to 100 due to rounding. */**/*** = Difference relative to Approach 2 is statistically significant at the.10/.05/.01 confidence level. After the orientation, any further counseling for Approach 3 customers was voluntary and Approach 3 customers rarely requested it. Less than 4 percent of all customers assigned to Approach 3 participated in any counseling beyond orientation, compared with more than half of customers under Approaches 1 and 2 (Table 4). Even when Approach 3 customers did request further counseling after the orientation, they attended just over one additional counseling session on average (Table 4). This result was found consistently across our sites. After attending the ITA orientation, Approach 1 customers were significantly more likely to attend some counseling than Approach 2 customers even though some counseling was mandatory for customers under both Approaches 1 and 2. Of all customers randomly assigned, 65 percent of Approach 1 customers attended some counseling beyond the ITA orientation 20

21 compared with 58 percent of Approach 2 customers (Table 4). This suggests that information provided at the orientation was more likely to encourage Approach 1 customers than Approach 2 customers to continue the process toward receiving an ITA. While the letter to Approach 1 customers did not tell them they could receive a higher ITA than under the other approaches, anecdotal evidence suggests that many customers were aware of this difference. We suspect that the perception that they could receive a higher ITA offset the burden of any additional counseling under Approach 1. Approach 1 customers were required to complete more counseling to receive an ITA, and they in fact did. On average, Approach 1 customers attended 1.7 counseling sessions compared with 1.5 for Approach 2 customers (Table 4). Those Approach 1 customers who attended the ITA orientation participated in an average of 2.5 counseling sessions; in comparison Approach 2 customers who attended an ITA orientation participated in an average of 2.2 counseling sessions. 3. Receipt of ITAs Although all customers found eligible for training are eligible to receive an ITA, some may decide not to receive one. This may be because they decide after being found eligible for training that they do not want to pursue training, perhaps because they find a job. Alternatively, they may decide to pursue training but finance it from their own funds or other sources. The likelihood of a customer pursuing ITA-funded training depends on its expected costs and benefits, both of which may be affected by the approach to administering ITAs. The costs of ITA-funded training include the burden of completing the activities necessary to receive an ITA. It also includes earnings forgone when training or waiting to receive an ITA. The benefits of ITA-funded training depend on the dollar value of the ITA received and the restrictions placed on how the ITA can be used. 21

22 Overall, only about two-thirds of all customers who were randomly assigned received an ITA (Table 5). This proportion varied by site. It was particularly low in two sites Atlanta, Georgia and Charlotte, North Carolina. In Atlanta, the relatively low ITA training rates may reflect the availability of state-funded alternatives to ITA. We have no explanation for the low rate in Charlotte. Most of the attrition occurred prior to the mandatory orientation 87 percent of those who attended the orientation received an ITA. Approach 3 customers were significantly more likely to receive an ITA than customers under Approaches 1 and 2 (Table 5). About 65 percent of all Approach 3 customers who were randomly assigned received an ITA compared with 59 percent of Approach 1 customers and 58 percent of Approach 2 customers. Compared to Approach 2 customers, Approach 3 customers were more likely to receive an ITA in every site, and the difference was statistically significant in four of the eight sites. The difference between the rate of ITA receipt under Approaches 2 and 3 was mainly due to differential attrition between the receipt of the letter notifying the customer of his/her assignment and the orientation. Of those customers who attended an orientation, the proportion of Approach 3 customers who received an ITA (88.5 percent) was not significantly different than the proportion of Approach 2 customers who received an ITA (86.6 percent). This suggests that it is the expected, rather than actual, burden of the counseling that explains the lower rate of ITA receipt under Approach 2. 22

23 Table 5. Receipt of ITAs (Percentages) Overall Approach 1 Approach 2 Approach 3 All Study Sites *** Phoenix, AZ *** Maricopa County, AZ ** *** Bridgeport, CT ** Jacksonville, FL Atlanta, GA Northeast Region, GA North Cook County, IL Charlotte, NC * *** All Study Sites, Among those who Attended an Orientation Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). */**/*** = Difference relative to Approach 2 is statistically significant at the.10/.05/.01 confidence level Overall, Approach 1 and 2 customers had comparable rates of receipt of ITAs. This does, however, mask some difference across sites. For example, the proportion of customers who received an ITA was significantly higher under Approach 1 in Maricopa County, Arizona and Charlotte, North Carolina. The similar overall rate of ITA receipt between Approach 1 and 2 customers suggests either that the differences between the two approaches had little effect on ITA receipt or that opposing factors were offsetting. We expect that the possibility of a higher value ITA, to the extent that the customers were aware of it, may have encouraged ITA receipt. However, the additional counseling requirements may have discouraged ITA receipt. After random assignment, it took ITA recipients under Approach 1 about one week longer than Approach 2 ITA recipients to receive an ITA (Table 6). The additional time involved also increases the likelihood that the customers will receive a job offer and decide against pursuing training. As the ability of the counselors to reject the customer s choice was rarely exercised, we doubt that this ability had much impact on ITA receipt. 23

24 Table 6. Average Number of Weeks Between Random Assignment and ITA Receipt OVERALL APPROACH 1 APPROACH 2 APPROACH *** Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). */**/*** = Difference relative to Approach 2 is statistically significant at the.10/.05/.01 confidence level. 4. Price of Selected Training Programs and ITA Expenditures We examined two approaches to determining the value of each ITA. The first approach was to tailor the amount to each customer providing a larger amount to customers who pursue training with a higher expected economic return and a lower amount to other customers (Approach 1). The second approach was to provide the same fixed ITA to all customers (Approaches 2 and 3). We found that customers assigned to Approach 1 chose significantly higher-priced training programs on average than those assigned to Approaches 2 and 3 (Table 7). Approach 1 customers chose training programs that were on average $4,800 about $1,600, or over 50 percent, more expensive than those chosen by Approaches 2 and 3 customers. On average, Approach 1 customers chose more expensive programs than the other customers in all sites and the difference was statistically significant in six of the eight sites. Although the fixed value of the ITA was the same under Approaches 2 and 3, the required counseling under Approach 2 may have affected the program chosen. However, we found that the receipt of counseling under Approach 2 had little impact on the price of the program chosen. Within each site, Approach 2 and 3 customers chose programs of similar prices, on average $3,116 and $3,133 respectively. 24

25 Table 7. Average Price of Selected Training Programs (Dollars) Study Site Approach 1 Cap Approach 1 Price Approaches 2 and 3 Fixed ITA Value Approach 2 Price Approach 3 Price Phoenix, AZ 8,000 4,791*** 3,000 2,489 2,518 Maricopa County, AZ 8,000 5,659*** 3,000 3,204 3,089 Bridgeport, CT 7,000 3,810*** 3,000 2,784 3,034 Jacksonville, FL 6,000 4,834*** 3,000 3,388 3,300 Atlanta, GA 8,000 4,429 5,000 4,210 4,220 Northeast Region, GA 8,000 3,931 4,000 3,556 3,547 North Cook County, IL 8,000 5,210*** 3,000 2,536 2,578 Charlotte, NC 8,000 5,055*** 4,000 3,245 3,229 Overall 7,626 4,764*** 3,430 3,116 3,133 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). */**/*** = Difference relative to Approach 2 is statistically significant at the.10/.05/.01 confidence As customers can contribute other sources of funding to pay for the training program, the workforce agencies expenditures on ITAs may be less than the price of the programs. On average, for those who received an ITA, ITA expenditures were about $1,800 to $1,900 (66 percent) higher for customers under Approach 1 than under Approaches 2 and 3 (Table 8). The average expenditure per ITA recipient under Approach 1 was significantly higher in all sites except Northeast Region, Georgia (not shown). In contrast, there was no statistically significant difference in average expenditures per ITA recipient between Approaches 2 and 3 overall, or in any site. 25

26 Table 8. Average Expenditures of Workforce Agencies Approach 1 Approach 2 Approach 3 Average Price of Selected Programs $4,764*** $3,116 $3,133 Average Expenditures per ITA Recipient $4,731*** $2,849 $2,857 Average Expenditure per ITA Recipient as a Percent of Average Price 99.3% 91.4% 91.2% Percentage of Customers who Receive ITAs 59.0% 57.6% 65.1%*** Average Expenditures per Customer $2,791*** $1,641 $1,853*** Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). Notes: Estimates do not include WIA administrative costs or the costs of ITA-related counseling. */**/*** = Difference relative to Approach 2 is statistically significant at the.10/.05/.01 confidence level. Customers assigned to Approaches 2 or 3 were more likely to use non-ita funds to help pay for training than were customers assigned to Approach 1. In total, customers financed about 9 percent of the cost of their training under Approaches 2 and 3 from some other sources (Table 8). In contrast, under Approach 1, the cap on the ITA value was set so high that few customers chose a training program that cost more than the cap. Customers financed less than 1 percent of the cost of the training programs under Approach 1 from other sources (Table 8). This suggests that the more generous ITA funding available under Approach 1 tended to discourage customers from finding other sources of training financing. Our findings in Table 8 also suggest that the ITA approach has a substantial effect on the workforce agencies expenditures on ITAs averaged over all their customers (including those who do not receive an ITA). The average training cost per customer is 70 percent higher under Approach 1 than under Approach 2 ($2,791 under Approach 1 compared with $1,641 under Approach 2). This higher cost is mainly because Approach 1 customers choose more expensive programs and in most cases this additional expense was entirely paid for by an ITA. 26

27 The average training cost per customer under Approach 3 is also significantly higher than under Approach 2 ($1,853 under Approach 3 compared with $1,641 under Approach 2). This is because, although the ITA expenditures per ITA recipient is similar under Approaches 2 and 3, Approach 3 customers are significantly more likely to receive an ITA than Approach 2 customers. While the expenditures on ITAs are a large part of the cost of the training to the government, the provision of counseling is also costly. As the costs of counseling are likely to be slightly higher under Approach 1 than under Approach 2, if counseling costs are taken into account, the difference in cost between Approaches 1 and 2 would increase. However, as the counseling costs are relatively small for Approach 3, once the counseling costs are included, the difference in total costs between Approaches 2 and 3 would decrease and perhaps disappear or become negative. 5. Training Program Selections Ultimately, workforce agencies are concerned about the training choices made by customers and the employment they obtain after training. The way ITAs are administered could affect the choices made by the customers. Whether counseling is mandatory, whether the counselor can require the occupation to be on the high-wage/high demand occupation list and can reject the customers choices, and the value of the ITA may all affect the selection of training programs. The only difference between Approaches 2 and 3 is the counseling requirement. If counseling provided customers more information or made them think more about their choices, we would expect to see some differences in the training programs selected by Approach 2 and 3 ITA recipients. The occupations chosen by ITA recipients when counseling was mandatory were remarkably similar to the occupations chosen by ITA recipients when counseling was voluntary 27

28 and few received counseling. Table 9 shows the top 20 occupations corresponding to the training programs chosen by the ITA recipients in our sample. 1 Approach 3 ITA recipients were slightly less likely than Approach 2 ITA recipients to choose accounting or bookkeeping as an occupation, but the difference between the proportion of ITA recipients in Approach 2 and 3 who chose this occupation was only statistically significant at the 10 percent level. 1 Table 9 shows only the occupations chosen by those sample members who received ITAs. As the approach affected the likelihood of receiving an ITA, differences between the occupations chosen by the recipients under each approach may be due to underlying differences in the characteristics of ITA recipients rather than the effect of counseling on occupation choice. We found few differences in the observed characteristics of ITA recipients under each approach. 28

29 Table 9. Occupation Choices by Recipients of ITAs (Percentages) Occupation Overall Approach 1 Approach 2 Approach 3 Top 20 Occupation Choices: 1. Computer networking and information technology Commercial Driver s License Medical assistant/secretary Administrative assistant * Certified Nurse Aide ** Medical coding/billing Computer programmer Licensed practical nurse or registered nurse Database administration Web design Accounting/ bookkeeping ** * 12. Health technician/technologist ** Heating, ventilation, and air conditioning Dental assistant *** Paralegal Teaching Business/project administration Computer installation, repair, and support Computer-aided design and drafting Data entry/processing ITA Vouchers with Occupation Information 4,759 1,525 1,505 1,729 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). */**/*** = Difference relative to Approach 2 is statistically significant at the.10/.05/.01 confidence level. We were surprised to find, however, that mandatory counseling did seem to affect the type of training provider selected (Table 10). Approach 3 customers were significantly more likely to choose training at a community or technical college and somewhat less likely to choose training at private schools or four-year universities than Approach 1 or 2 customers. 29

30 Table 10. Training Providers and Average Duration of Training Program Selected by ITA Recipients Type of Training Provider Overall Approach 1 Approach 2 Approach 3 Private, For-Profit (Proprietary) School 62.7% 63.5% 63.4% 61.3% Community or Public Technical College *** Nonprofit Organizations State or Private (Four Year) University ** Other * * Average Duration of Training Program (Months) *** Sample Size 4,756 1,517 1,520 1,719 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). Notes: Percentages may not sum to 100 due to rounding. Table excludes data on Charlotte, North Carolina, due to missing training provider information. Mandatory counseling did not, however, seem to affect the length of the program chosen. Approach 2 and 3 customers chose programs of comparable lengths (Table 10). On average, the programs chosen by Approach 2 and 3 ITA recipients were about 7 months long. Approaches 1 and 2 differed in the intensity of counseling, the value of the ITA, and whether the counselors could reject a customer s choice. As counselors rarely rejected a customer s request, we suspect that their ability to do so had little impact on the selection of training programs by Approach 1 customers. Approach 1 and 2 recipients of ITAs chose similar types of training providers (Table 10). Hence, the additional counseling and the higher value ITA did not seem to affect the choice of the type of provider. Approach 1 customers signed up for longer programs on average probably because they were able to purchase more expensive programs (Table 10). Approach 1 customers chose programs with an average duration of 8.2 months, about one month longer than the programs chosen by Approach 2 and 3 customers. Interviews with program staff suggest that the 30