Labor Market Impact on Youth: A meta analysis of the Youth Employment Inventory 1

Transcription

1 Labor Marke Impac on Youh: A mea analysis of he Youh Employmen Invenory 1 Olga Susana Puero 2 A. Inroducion Inervenions o suppor young workers have been broadly applied in boh developed and developing economies alike, as concerns for unemploymen and inaciviy raes drive public and privae resources o inves in youh. Despie comprehensive experiences around he world, very lile is known abou he deerminans of success of youh employmen inervenions. In paricular, wha ype of inervenions work bes and wha are he key feaures in implemenaion design and argeing ha promp posiive impacs on employmen and earnings under differen economic and insiuional condiions. To respond o hese quesions, his paper uses a meaanalyical framework ha examines he evaluaion evidence colleced by he Youh Employmen Invenory, a World Bank iniiaive ha compiles world-wide inervenions designed o inegrae youh ino he labor marke. This mea-analyical framework combines informaion on program impac, program characerisics, and counry conex. I employs a probabiliy model o measure he likelihood of obaining posiive impacs on employmen and earnings, synhesizing he resuls from individual sudies in a common field. Empirical resuls from a sample of 172 evaluaed sudies including ne impac evaluaions and evaluaions wih gross oucomes - indicae ha program success is no deermined by he ype of inervenion bu raher by he program s argeing sraegies oward disadvanaged youh, he counry level of developmen, and he flexibiliy of he labor marke regulaions. Sensiiviy ess show hese resuls are sable under differen specificaions, paricularly when he sample size is consrained o sudies wih ne impac evaluaions (i.e. wih reamen and conrol groups). The paper has five secions including his Inroducion. Secion B sars wih a lieraure review of cross-counry sudies and mea-analyses of labor marke programs. Whenever available, esimaes on employmen and earnings of young people are discussed. Secion C provides a summary of he Youh Employmen Invenory, including design, coverage, and main feaures. I describes he main ype of inervenions and findings based on a descripive evaluaion of he invenory s evidence. Secion D displays he daase of programs included in he meaanalysis, and describes he key variables ino play. I also inroduces he model, empirical resuls and sensiiviy ess. Conclusions and implicaions are drawn in Secion E. 1 This is a background paper for he World Bank s 2007 Youh Employmen Invenory, conduced by he Social Proecion and Labor Uni of he Human Developmen Nework. Main resuls from his paper can be found in he invenory s synhesis repor (Becherman e al. 2007). 2 Correspondence: Olga Susana Puero, World Bank, opuero@worldbank.org. 1

2 B. Lieraure review of overview sudies of labor marke programs Focus on Youh There are few overview and cross-counry sudies on labor marke programs around he world, and he evidence is even more scan when he focus is on specific arge groups such as youh. Becherman e al. (2004) and Dar and Tzannaos (1999) already exposed he subsanial lack of solid informaion and evaluaion evidence on acive labor marke programs (ALMP). In heir review, Becherman e al. (2004) looked a he evidence from 159 ne impac evaluaions of ALMP, of which solely 12 percen (19 programs) were youh-oriened. Their assessmen on youh employmen programs, paricularly relaed o raining sysems, indicaes ofen negaive impacs on labor marke oucomes (a finding more accenuaed in developed economies), sressing he imporance of early and susained inervenions o reduce school dropous raes and improve educaional aainmen. Developing economies, on he conrary, repored posiive impacs when raining is offered as par of a comprehensive package. Evaluaion evidence is considerably richer in OECD counries, where here is a longsanding radiion in he design and implemenaion of impac evaluaions and where he susainabiliy of publicly-funded programs relies grealy on he evaluaion oucomes. 3 Several evaluaions have measured he effeciveness of ALMP in developed counries, and whenever possible, impacs on employabiliy and earnings of young people have been esimaed. Despie his amoun of evidence, very few sudies have pu ogeher cross-counry analyses o compare he effeciveness of acive measures and draw sysemaic lessons on wha can poenially deermine he success of employmen inervenions in OECD counries. Sysemaic cross-counry evidence from economerical sudies sems from he works of Heckman e al. (1999), Kluve and Schmid (2002), and Kluve (2006). Descripive cross-counry analyses for he region can be also found in Marin and Grubb (2001). Based on a sample of evaluaion sudies of ALMP implemened in Europe and he U.S. before 1994, Heckman e al. (1999) observed he impacs of inervenions such as job raining, job search assisance, and wage subsidies on employabiliy. The findings sugges a non-clear paern of success across caegories of inervenion and very moderae and raher disappoining oucomes, especially for youh. The paper also examines he large heerogeneiy in evaluaion mehods across counries, and paricularly beween he much less developed evaluaion evidence in Europe and he large experience of he U.S. When drawing mehodological lessons, he auhors sugges here is no an opimal mehod of choice for conducing program evaluaions, i.e. experimenal and non-experimenal mehods as well as oher economeric echniques may be in general- equally convenien o measure labor marke impacs so long as he qualiy of he underlying daa is ensured. Kluve and Schmid (2002) colleced a sample of 53 evaluaions of European programs implemened beween 1983 and 1999, and compared heir resuls o he U.S. programs previously sudied by Heckman e al. (1999). Their overview analysis suggess mixed program effecs across caegories of inervenion and arge populaion: while raining and job search assisance may be effecive o improve he paricipans labor marke prospecs, direc job creaion programs in he public secor may lead o negaive gains and hinder employabiliy of some arge groups. Young workers were he mos difficul group o assis among he unemployed. Following he previous sudies, Kluve (2006) se a mea-analyical framework o esimae he probabiliy of success, in erms of posiive reamen effecs on employmen, of ALMP in 3 As is paricularly he case in he Unied Saes. 2

3 Europe, wih special aenion drawn o programs implemened in he lae nineies and in he 2000s. The probabiliy of success is modeled by (i) he caegory of inervenion, (ii) he sudy design 4, (iii) he insiuional labor marke conex, and (iv) he economic counry-conex a he ime a paricular program was implemened. The mea-analysis used microeconomic evaluaions sudies of ALMP from 17 European counries, yielding a comprehensive daase wih 137 observaions or assessmens on he ne impacs of he inervenions by ype of program or arge group. 5 Abou 25 percen of hese observaions referred o youh employmen programs. Resuls indicae ha caegory of inervenion is he only clear deerminan of success of acive labor marke measures in Europe, and here is lile if any evidence ha sudy design or counry-conex facors explain he programs effeciveness. Kluve (2006) defines he se of caegories or program ypes as follows: labor marke raining, privae secor incenive programs (e.g. wage subsidies), direc employmen programs in he public secor (e.g. public works programs), and services and sancions (e.g. job search assisance and compulsory programs o mainain unemploymen benefis). Besides hese ypologies, programs were furher disaggregaed by arge group: youhs and disabled workers. Specific findings across caegories of inervenion sugges raher modes posiive impacs from raining programs on employabiliy. Relaive o raining programs, he probabiliy model indicaes significanly higher reurns from privae secor incenive programs and services and sancions programs: hey increase he likelihood of posiive labor marke impacs in 40 o 50 percenage poins more han raining programs do. On he oher hand, public secor employmen programs are 30 o 40 percen less likely o yield posiive impacs han raining programs. On specific arge groups, he model indicaed ha youhs are sill he hardes populaion o assis, reducing he probabiliy of posiive employmen impacs in 40 o 60 percenage poins. Addiional evidence of mea-analyical frameworks of labor marke programs is available in he Unied Saes. Greenberg e al. (2003) synhesized findings from 31 evaluaions of 15 publicly-funded raining programs o measure he programs effecs on paricipans earnings 6. Their model regresses he repored earning effecs agains (i) ype of raining, (ii) demographic characerisics of he arge populaion (e.g. gender and race), (iii) economic condiions of he area where he program was implemened, (iv) evaluaion mehod, (v) number of years since raining was received, and (vi) year in which he program was implemened. Of a oal of 315 observaions on earnings oucomes, 99 (31 percen) were relaed o raining programs for disadvanaged youh. Resuls sugges highly heerogeneous earning effecs among assised groups, i.e. men, women and youhs. Findings for youh were paricularly discouraging. Across program componens, classroom raining yielded consisenly posiive effecs relaive o on-hejob raining. Training appeared o be less effecive for whies and female beneficiaries han for all oher paricipans. 7 The sudy found a saisically significan posiive correlaion beween raining 4 Sudy design idenifies wheher he evaluaion followed a randomized experimen and also classifies programs by he experimen s sample size. 5 These 137 observaions originaed from 95 differen evaluaion sudies. 6 Programs included in he analysis are: Manpower Developmen and Training Ac (MDTA), Neighborhood Youh Corps (NYC), Job Corps, Comprehensive Employmen and Training Ac (CETA), he Naional Suppored Work Demonsraion, he JOBSTART Demonsraion, he New Chance Demonsraion, and he Job Training Parnership Ac (JTPA). They were all implemened in he U.S. beween 1962 and These findings are consisen wih an early paper by Gay and Borus (1980). In heir review, Gay and Borus (1980) use adminisraive daa on paricipans earnings from a longiudinal sudy of four federally sponsored employmen and raining programs o validae he usefulness of performance indicaors in assessing he effeciveness of he programs. The sudy idenified ne posiive impacs on earnings of ou- 3

4 effecs and program cos (e.g. a $1,000-increase in program cos raised he raining effec by abou $108). This is, o he bes of our knowledge, he exen o which overview analyses and more concreely mea-analyical frameworks have been applied o he sudy of labor marke programs. In fac, mea-analyses have been only recenly inroduced o economic discussions 8 o quaniaively combine and synhesize resuls from individual sudies. Previous applicaions of mea-analyses include fields such as educaion, medicine, and psychology (Hun, 1997). This paper builds on he se of sudies and mehodologies described above, and offers a cross-counry, cross-program analysis of employmen inervenions for youh, wih he aim of providing sysemaic and quaniaive lessons wih far-reaching implicaions for boh developed and developing counries alike. The paper relies on he informaion gahered by he World Bank s Youh Employmen Invenory, a comprehensive iniiaive ha compiles policies and programs designed o inegrae young people ino he labor marke. C. Evidence on Youh: The Youh Employmen Invenory 1. Designing he Youh Employmen Invenory The Youh Employmen Invenory is a World Bank iniiaive ha offers a wealh of informaion on employmen inervenions o suppor young workers around he world. I is based on available documenaion of curren and pas programs and includes evidence from 289 inervenions from 84 counries in all regions of he world. The inervenions included in he invenory have been analyzed in order o (i) documen he ypes of programs ha have been implemened o suppor young workers o find a job; and (ii) o idenify wha appears o work in erms of improving employmen oucomes for youh. The invenory includes compleed and ongoing inervenions aiming o faciliae he ransiion of young people ino he labor marke, wih a paricular focus on disadvanaged youh. The invenory is limied o pos-formal-schooling inervenions and allows he inclusion of boh labor marke policies and programs wih he following caveas: (i) Policy inervenions are eligible for inclusion if hey specifically arge young people; while (ii) labor marke programs are included even if hey did no explicily arge youh bu he documenaion indicaes ha young people were he primary paricipans. These limiaions and furher deails on he selecion crieria are discussed o greaer exend in he invenory s synhesis repor (Becherman e al. 2007). The invenory groups inervenions ino nine caegories, displayed in Table 1. These caegories se a clear framework for he inclusion and classificaion of inervenions. They are based on a wo-fold approach o youh employmen problems, consising of: (1) increasing labor of-school and black NYC beneficiaries; while here appeared o be significan negaive effecs for nonblack NYC paricipans and all Job Corps beneficiaries. 8 For insance, Jarrell and Sanley (1990) use a mea-analysis o measure wage differenials across union and non-union workers. Also, Sirmans e al. (2006) apply a mea regression analysis o deermine he value of housing. 4

5 demand in general in relaion o supply; and (2) increasing 'inegrabiliy' of disadvanaged youh, so ha hey can ake advanage of opporuniies ha arise when he labor demand increases 9. The caegories of inervenion are largely self-explanaory bu a few commens may be useful. Caegory 1, making he labor marke work beer for young people, includes inervenions ha improve informaion (counseling, job search skills), increase labor demand for youh (wage subsidies and public works), and remove discriminaion. Caegory 2, improving chances for young enrepreneurs, covers inervenions ha provide assisance (financial, echnical, and raining) o youh who are saring heir own business. Caegories 3 and 4 boh deal wih raining: he former includes he full range of pos-formal schooling raining programs while he laer includes inervenions inended o address raining marke failures by providing informaion, credi, and oher financial incenives. Table 1: Caegories used o classify inervenions in he Youh Employmen Invenory 1. Making he labor marke work beer for young people 1a. Counseling, job search skills 1b. Wage subsidies 1c. Public works programs 1d. Ani-discriminaion legislaion 1e. Oher 2. Improving chances for young enrepreneurs 3. Skills raining for young people 3a. Vocaional raining including appreniceship sysems 3b. Lieracy & numeracy young adul lieracy programs 3c. 2nd chance & equivalency programs 3d. Oher 4. Making raining sysems work beer for young people 4a. Informaion 4b. Credi (o individuals or enerprises) 4c. Financial incenives (subsidies, vouchers) 4d. Oher 5. Programs o counerac residenial segregaion of disadvanaged young people 5a. Transporaion 5b. Ohers 6. Improving labor marke regulaions o he benefi of young people 7. Programs for overseas employmen of young people 8. Comprehensive inervenions 9. Oher (e.g. volunary naional service programs) Locaion can also be a barrier for young people if where hey reside isolaes hem from learning or employmen opporuniies, or even a secure living environmen. Caegory 5 is mean o include inervenions (e.g., ransporaion services or residenial mobiliy) ha can help young 9 Inegrabiliy can be increased by (a) remedying or couneracing marke failure, (b) improving labor marke regulaions, and (c) improving he skills of disadvanaged youh (Godfrey 2003). 5

6 people overcome his form of barrier. Caegory 6 covers regulaory reforms (e.g., changes in labor law, minimum wage, ec.) ha are designed o enhance employmen opporuniies for young people. Caegory 7 includes programs o provide job opporuniies ouside he counry. Inervenions ha provide muliple ypes of services, and hus canno be included in one of he oher groups, are included in Caegory 8. Finally, Caegory 9 is a residual grouping. Several ools were used in he process of collecing and compiling inervenions. These ools include a quesionnaire o sysemaize and sandardize he informaion, and he creaion of a daabase o faciliae analysis and inerpreaion of he evidence (Box 1). Box 1: Compiling informaion for he Youh Employmen Invenory Templae A quesionnaire emplae was designed o ensure consisency and uniformiy in he collecion and recording of informaion for he invenory. Informaion colleced on each program includes inervenion caegory, counry, ime period in which i was implemened, curren saus, he specific labor marke problems i sough o address, main objecives, a deailed descripion of he program (scale, financing, ec.), as well as several performance indicaors o undersand he program s impac, summary measures on he qualiy of he evaluaion evidence and he qualiy of he inervenion (described below), and sources for furher informaion on he inervenion. To allow for quaniaive analysis of he daa, variables included in he emplae were coded on he basis of muliple choice measures wherever feasible. Daabase In he projec design sage, a decision was made o organize he daa collecion so ha he invenory could be in an elecronic forma in order o faciliae search capabiliies, updaing, and o do quaniaive analysis. The emplae was buil ino an Excel workshee and an independen machine-readable file was creaed for each inervenion included in he invenory. Afer he daa-collecion phase ended, an Excel macro was designed on Microsof Visual Basic o read every file and consruc a searchable daabase where he number of observaions (rows) mached he number of inervenions (files or workshees). Daa colleced in he quesionnaire boh plain ex and codes -- are displayed in he columns, creaing a daabase of program-specific informaion (Daabase 1). Simulaneously, a daabase of counry-specific informaion (Daabase 2) was creaed o conexualize he economic condiions of he counry. This informaion includes level of developmen, level of income, and a characerizaion of he labor marke regulaory/insiuional siuaion. Sources of informaion for he counry daabase are he World Developmen Indicaors and he Doing Business Daabase. The Excel macro links daabases 1 and 2 hrough a common key-variable, namely counry name, creaing a comprehensive daabase for he analysis of he invenory. The daabase can be accessed a hp://go.worldbank.org/48z06gmd70 Two criical variables figure prominenly in he analysis of lessons learned from he invenory. These are he qualiy of inervenion (QOI) and he qualiy of evaluaion (QOE). The QOI is he measure of program effeciveness, whose possible values are described in Table 2. The primary performance indicaors ha are considered in esablishing a QOI raing are he effecs of he program on he employmen and earnings of paricipans. A one level, he QOI value can be used o idenify impac i.e., o disinguish hose programs ha acually help paricipans in he labor marke (QOI = 1, 2, or 3) from hose ha appear o have no effec, or even a negaive effec (QOI = 0). This disincion is similar, bu no idenical, o deermining program success. To be specific, inervenions can have a posiive employmen impac bu are no cos-effecive (i.e., QOI = 1). These programs canno be considered successful. 6

7 Assessing he qualiy of he inervenion relies grealy on he informaion qualiy, which varies widely from basic descripive informaion o solid evaluaion evidence. The QOE variable idenifies he evaluaive basis for assessing program qualiy. The QOE measure is described in Table 3. Wih his variable, hen, assessmens of he effeciveness of inervenions can be judged wih knowledge of he underlying evidence. For example, one can consider only hose programs ha mee he mos exacing burden of proof (i.e., QOE = 3), wih he radeoff ha sample size will be reduced. On he oher hand, acceping a less demanding basis of evidence will increase he pool of programs under consideraion, bu a he expense of rigor. As we will see laer in his paper, assessmens of program impac are correlaed wih he qualiy of he evaluaion evidence. Table 2: Measuring he Qualiy of he Inervenion (QOI) QOI value Descripion 0 Inervenion had negaive or zero impac on labor marke oucomes. 1 Inervenion had posiive impac on labor marke oucomes bu is no cos-effecive. 2 Inervenion had posiive impac on labor marke oucomes and here is no evidence on coss. 3 Inervenion had posiive impac on labor marke oucomes and is cos-effecive. 99 Missing value. No enough evidence o make an assessmen. QOE value Descripion Table 3: Measuring he Qualiy of Evaluaion (QOE) 0 Inervenion has no evaluaion informaion available on oucomes or impac. 1 2 Evaluaion includes basic informaion on he gross oucomes of he inervenion (e.g. number of paricipans/ young people who found a job afer he inervenion, improvemen in earnings of paricipans) wihou considering ne effecs (i.e., here is no conrol group). Evaluaion includes esimae of ne impac on, e.g., employmen and earnings in he labor marke (using conrol groups o measure impac) bu no cos-benefi analysis. 3 Evaluaion includes ne impac plus cos-benefi analysis. 2. Coverage of he Youh Employmen Invenory Wha are he mos popular employmen inervenions for youh and where have been implemened? Is here any paricular orienaion of he inervenions owards disadvanaged youh? How are youh employmen inervenions being financed? This secion approaches hese quesions based on he evidence from he invenory. Firs, regarding coverage of he invenory by caegory of inervenion, skills raining programs paricularly vocaional raining and appreniceships sysems and comprehensive programs 10 are he mos popular inervenions supporing he enry of young people ino he labor marke. They accoun for 38 and 33 percen of all inervenions covered by he invenory, respecively (Table 4). Oher prevalen caegories are inervenions o make he labor marke 10 Comprehensive inervenions have in general very srong raining componens. They encompass job and life skills raining (in classroom and/or on-he-job), appreniceship sysems, enrepreneurship schemes, informaion, counseling, placemen, financial incenives, and oher suppor services. 7

8 work beer for young people and enrepreneurship schemes. All oher inervenion caegories have very small samples. 11 Across regions, mos programs covered by he invenory were implemened in OECD counries (42 percen) and Lain America and he Caribbean (24 percen). Europe and Cenral Asia, Sub-Saharan Africa, Souh and Eas Asia and he Pacific, and Middle Eas and Norh Africa had relaively low paricipaion in he oal number of documened inervenions. Advanced and middle income counries dominae he picure wih nearly 90 percen of colleced inervenions carried ou in hese counries. The disribuion of inervenions varies by he counries income level. While OECD counries end o have a more diverse and comprehensive se of policies and programs o approach youh, middle income counries have focused grealy on raining sysems, and low income counries in enrepreneurship schemes. Second, regarding argeing and he orienaion of he programs oward disadvanaged groups, he invenory shows ha mos inervenions are addressed o youh only while nearly 20 percen are open o people of all ages. Rural youh are argeed bu received relaively less services han urban youh: only 10 percen of inervenions served exclusively rural populaions, 28 percen were implemened in he ciies and he remaining 62 percen worked in boh, rural and urban areas. To measure he exen o which inervenions are oriened owards he disadvanaged, he invenory idenified he following arge groups: women, he disabled, ehnic groups, and individuals wih cerain income and educaion levels. The incidence of gender argeing is relaively low: only 16 percen of all programs are oriened owards young women and 2 percen are argeed explicily a young men. There are 32 programs (11 percen) for disabled young people, mos of which provide comprehensive services. Regarding ehniciy, here is only a small number of inervenions (20 in oal, wih 9 of hese providing raining) argeed owards paricular ehnic groups. Jus over half (52 percen) of all programs in he invenory are oriened owards lowincome youh. Training programs and comprehensive programs are especially likely o be argeed on income. Regarding educaion, 49 percen of all programs were argeed o youh wih low educaional aainmen. Comprehensive programs are again mos likely o have educaionrelaed argeing. Nine programs in OECD counries were found o assis educaed youh. Finally, regarding financing, he major sponsor of youh employmen programs is he governmen, bu join public-privae venures wih inernaional organizaions and bilaeral donors play an imporan role in he delivery of he programs neverheless. Abou 56 percen of programs are primarily governmen-funded, and 33 percen are financed by a mix of insiuions, such as cenral and local governmens, inernaional organizaions, bilaeral donors, civil sociey and he privae secor. Governmen-funded programs provide mos comprehensive inervenions as well as wage subsidies, financial incenives for raining and public works programs. Public programs are more prevalen in advanced economies and more likely o have rigorous evaluaion informaion and ne impac sudies han programs sponsored by oher insiuions. On he oher hand, he provision of raining programs and enrepreneurship schemes in developing counries relies grealy on donors resources and he paricipaion of non-governmen organizaions. 11 There is no evidence on programs relaed o ani-discriminaion legislaion, young adul lieracy, couneracing locaion barriers, or programs o promoe overseas employmen of young people. 8

9 Table 4: Coverage of he Invenory by Caegory of Inervenion and Region Caegory of inervenion Europe & Cenral Asia Lain America & Caribbean Middle Eas & Norh Africa OECD Souh and Eas Asia and he Pacific Sub- Saharan Africa Toal % 1. Making he labor marke work beer for young people % 1a. counseling, job search skills b. wage subsidies c. public works programs d. ani-discriminaion legislaion 0 1e. oher Improving chances for young enrepreneurs % 3. Skills raining for young people % 3a. vocaional raining including appreniceship sysems b. lieracy & numeracy young adul lieracy programs 0 3c. 2nd chance & equivalency programs d. oher Making raining sysems work beer for young people % 4a. informaion b. credi (o individuals or enerprises) 1 1 4c. financial incenives (subsidies, vouchers) d. oher Programs o counerac residenial segregaion of % disadvanaged young people 5a. ransporaion 0 5b. ohers 0 6. Improving labor marke regulaions o he benefi of young % people 7. Programs for overseas employmen of young people 0 0% 8. Comprehensive inervenions % 9. Oher (e.g. volunary naional service programs) % Toal % 9

10 3. Main Findings from he Youh Employmen Invenory: A qualiaive analysis This secion summarizes he main findings from he invenory based on descripive saisics and an overview analysis of inervenions across regions. 12 Imporan evidence-based lessons are also exraced from inervenions wih impac evaluaions i.e. hose wih reamen and conrol groups (Puero, 2007b). Training is he dominan form of inervenion used o inegrae young people ino he labor marke. The invenory shows ha inervenions wih srong raining componens, specifically skills raining and comprehensive programs are he mos common approach o youh. These programs are highly popular in OECD and middle income counries (paricularly in Lain America) and accoun for more han wo-hirds of inervenions around he world. This suggess a clear ineres in building up skills developmen sraegies o counerac un/under-employmen among he young. Inervenions are ofen argeed a low-income or poorly-educaed young people, paricularly in non-developed counries. A vas majoriy of inervenions in Lain America focus on young people from low income families, while educaed youh are mos argeed in Europe and Cenral Asia. Raings on he qualiy of he inervenions indicae ha programs argeing low-income and low-educaed youh have higher possibiliies of yielding posiive labor marke impac han programs wihou any income or educaion orienaion. The overall evaluaion evidence on youh employmen inervenions is weak. Only one-fourh of inervenions in he invenory have esimaes of ne impac, and jus one in en has evidence on cos-effeciveness (Table 5). Available informaion on youh employmen inervenions is acually sronger in developed counries. Indusrialized economies have a longsanding experience in he implemenaion of ALMP, which has grown over ime wih he need for documening and analyzing program effecs. Nearly 60 percen of inervenions wih ne impac evaluaions where carried ou in OECD economies. Table 5: Coverage of he Invenory by Qualiy of Inervenion and Qualiy of Evaluaion Qualiy of Evaluaion Qualiy of Inervenion (QOI) (QOE) Toal Toal Noe: See Tables 2 and 3 for definiions on Qualiy of Inervenion and Evaluaion. 12 Furher informaion on regional analyses can be found in he invenory s regional repors: Roher and Puero (2007) for OECD counries, Savreska (2006a) for Easern Europe and Cenral Asia, Puero (2007a) for Lain America and he Caribbean, Roher (2006) for Sub-Saharan Africa, and Savreska (2006b) for Asia, including boh Eas Asia and he Pacific and Souh Asia. 10

11 The assessed impac of an inervenion is affeced by he qualiy of he underlying evaluaion evidence. As displayed in Table 5, when informaion on gross oucomes only is available (QOE = 1), 90 of he 99 programs where a QOI assessmen was made were judged o have posiive impac 13. However, when a ne impac evaluaion has been carried ou (QOE = 2 or 3), he possibiliy of finding a posiive employmen impac decreases significanly, o 60 percen (44 ou of 73 programs). This has wo imporan implicaions: Firs, wihin he conex of his sudy, an overall assessmen of wha inervenions can do for he employmen and earnings of young workers is much more favorable when he sandard of accepable evidence is relaively ligh han when a higher sandard is se (i.e., ne impac evaluaion). Second, because of he lack of serious evaluaions especially in developing counries, policy-makers who end o focus on gross oucome measures are generally overesimaing how useful heir inervenions are in helping young people find employmen or increasing heir earnings. When cos-effeciveness is aken ino accoun, only abou one-hird of inervenions are successful (10 ou of 28 programs where QOE = 3). 14 This finding holds even afer inerpolaing programs for which here is no evidence available on cos. This simulaion exercise assumes ha he programs wihou cos informaion have he same probabiliy of being cos effecive as programs wih cos informaion. The final oucome indicaes ha 33.2 percen of inervenions wih ne impac evaluaion are successful (i.e. have posiive impac and are cos-effecive). Across caegories of inervenion, a descripive analysis of he inervenions wih impac evaluaions (i.e. wih reamen and conrol groups) shows relaively beer resuls from (i) inervenions o make he labor marke work beer for youh, (ii) comprehensive programs, and (iii) enrepreneurship schemes. Training programs were less successful han average, wih relaively beer oucomes in developing and ransiion economies han in developed counries. Sample sizes in hese caegories become a consideraion especially when we impose he resricion ha programs need o have an impac evaluaion. 15 Applying he simulaion exercise menioned before, he resuls show relaively lile variaion across programs. Inervenion qualiy raings indicae ha he labor marke impac ends o be more favorable in developing and ransiion counries han in indusrialized counries. While only 60 percen of programs in he OECD region had posiive impac, he corresponding raes in Europe and Cenral Asia and Lain America and he Caribbean he wo oher regions wih significan samples were 90 percen and 92 percen, respecively. Alhough he sample sizes are oo small in Souh and Eas Asia and he Pacific and Sub-Saharan Africa o draw firm conclusions, he limied evidence in hese regions offers addiional suppor for he conclusion ha youh programs have been more successful in developing counries. This is a surprising finding, given he more exensive experience of OECD counries wih employmen programs, heir greaer capaciy and resources, more available informaion and analysis, and generally beer funcioning labor markes. 13 Where impac is defined as he effec of he programs on he fuure employmen prospecs of paricipans, measured by pos-program employmen and/or earnings. 14 Where success implies having a posiive impac on he labor marke prospecs of paricipans and being cos-effecive (i.e. coss do no exceed benefis). 15 The number of inervenions wih impac evaluaions ranges from 34 for comprehensive programs o jus 3 for enrepreneurship. 11

12 Three hypoheses have been drawn in he analysis o explain why indusrialized economies have less effecive inervenions. The firs hypohesis suggess ha his finding is due o measuremen problems semming from he fac ha programs in indusrialized counries end o be more rigorously evaluaed. The daa show ha he availabiliy of ne impac evaluaion reduces he frequency of posiive employmen effecs, so i may be ha he differences are due o he fac ha he posiive impac of inervenions in non-indusrialized counries is oversaed and ha, if hey were evaluaed as rigorously as OECD programs, hese differences would disappear. This hypohesis will be challenge by he mea-analysis model in secion D. A second hypohesis implies real differences beween youhs in developed and nondeveloped counries. For example, disadvanaged young people he dominan clienele of youh programs everywhere may be a such a defici in OECD counries given he high levels of human capial and he skills inensiy of heir labor demand ha employmen inervenions are simply no enough o compensae. In developing counries, on he oher hand, he boos ha hese programs can give young people may be enough o give hem he advanage hey need o improve heir siuaion. Unforunaely, his hypohesis canno be esed wih he informaion colleced by he invenory. A las, he finding migh be explained by differences in labor marke insiuions or policies, such as employmen proecion laws (EPL). In general, employmen proecion rules could limi he effeciveness of youh programs since i is well documened ha, where such rules are sric, young people are likely o experience difficuly in enering he labor marke (e.g., OECD 2004). Wihin he OECD region, he evidence is a leas consisen wih his hypohesis: youh programs have a higher posiive impac rae in Anglo-Saxon counries (74 percen) where EPL is more flexible han in he res of he OECD, largely coninenal Europe (38 percen) where rules are more proecive. 16 However, observing he World Bank s Doing Business employmen rigidiy index here is no clear paern beween flexibiliy and program effeciveness. In fac, developed counries have some of he highes levels of labor marke flexibiliy around he world. D. Mea-analysis of he Youh Employmen Invenory: A quaniaive analysis 1. Daase and variables The invenory offers a considerable collecion of sudies and programs wih esimaes on labor marke oucomes on youh. In order o draw daa-suppored lessons on wha can poenially work for his arge populaion, he sample of inervenions colleced by he invenory has been filered o examine only hose programs wih relaively good qualiy of evaluaion, i.e. QOE = 1, 2, and 3. This se of programs ensures enough evidence o assess he qualiy of he inervenions and deermine wheher he programs yield a posiive impac on employabiliy and earnings of paricipans. As a resul, he daase for he mea-analysis sars wih a sub-sample of 172 observaions of inervenions where an assessmen was made, including boh programs wih gross oucomes and wih ne impac evaluaions, as shown in Table 6. A subsequen specificaion consrains he sample o inervenions wih only ne impac evaluaions, i.e. QOE = 2 and 3. Based on he measures of inervenion qualiy (or QOI, described in deail in Secion B), a binomial variable has been consruced o idenify he occurrence of posiive labor marke impacs. This is he dependen variable of he model, which will measure he probabiliy of 16 The differences beween he Anglo-Saxon and coninenal Europe (and oher) counries in he OECD are discussed in some deail in he OECD regional paper (Roher and Puero, 2007). 12

13 program success. The variable akes value 1 in 78 percen of he sample (134 cases), where he evaluaion repored posiive effecs on employabiliy and/or earnings of beneficiaries (i.e. QOI = 1, 2, or 3), and value 0 in he remaining 22 percen of observaions (38 cases) where negaive or zero oucomes were repored (QOI = 0). As can be seen, he consrucion of he dependen variable offers no consideraion on cos-effeciveness, given he subsanial lack of evidence on cos benefi analyses. Table 6: Sample of inervenions for he Mea-analysis (QOE = 1, 2, 3 and QOI 99) Qualiy of Evaluaion Qualiy of Inervenion (QOI) (QOE) Toal Toal Noe: See Tables 2 and 3 for definiions on Qualiy of Inervenion and Evaluaion. Explanaory variables are par of four groups: (i) caegory of inervenion, (ii) evaluaion qualiy, (iii) economic and insiuional counry conex, and (iv) specific characerisics of he program. Caegory of Inervenion. Given he evaluaion evidence available in he se of 172 programs, caegories of inervenion have been clusered in five ypes, as displayed in Table 7. Program ype 1 comprises inervenions o make he labor marke work beer; his corresponds o 15 percen of he sample. Program ype 2 includes all enrepreneurship schemes, prevalen in 9 percen of he cases. Training-relaed inervenions (i.e. caegories 3 and 4) are clusered under program ype 3, wih 36 percen of he cases. The highes paricipaion is ha of comprehensive programs, classified under program ype 4. The las ype clusers he remaining caegories (caegories 6 and 9) wih fairly low evidence on oucomes. Program ypes are inroduced in he model as five independen dummy variables, where raining-relaed programs represened he omied caegory. Type Table 7: Classificaion of caegories of inervenion by labor marke impac Caegory of inervenion Negaive or Zero impac Posiive impac 1 1. Making he labor marke work beer for young people Improving chances for young enrepreneurs Skills raining for young people Making raining sysems work beer for young people Comprehensive approach Improving labor marke regulaions Toal 9. Oher Toal

14 Evaluaion Qualiy. Wihin he sample of inervenions wih evaluaion evidence a furher disincion has been drawn beween evaluaions wih only gross oucomes (i.e. QOE = 1) and ne impac evaluaions (i.e. QOE = 2 and 3). This classificaion seeks o es wheher he ype of evaluaion affecs he repored labor marke oucomes. Economic and insiuional counry conex. Counry characerisics have been also considered in oher analyses (e.g. Kluve (2006) and Greenberg e al. (2003)) o capure he effec of he macroeconomic condiions and labor marke regulaions on labor marke oucomes. We disinguish beween developed and non-developed economies, in order o es wheher youh employmen programs are affeced by he counry s income level. Abou 58 percen of evaluaed inervenions ook place in non-developed counries (Annex, Table A.1). In addiion, we use he rigidiy of employmen index (as repored by he Doing Business Repor) o measure he effec of employmen regulaions flexibiliy on youh s labor marke prospecs. The rigidiy of employmen index is a composie measure of difficuly of hiring and firing, and rigidiy of hours. The higher he index, he more sringen he labor marke regulaions are. I varies from 26.2 in Eas Asia and he Pacific o 53.1 in Sub-Saharan Africa. The OECD area repored he second lowes level wih a raing of Specific characerisics of he program refer mainly o he feaures of he arge populaion, in paricular he programs specific focus on women, he disabled, specific ehnic groups, as well as low-income and low-educaed youh. Dummy variables were creaed for each arge group o es wheher focalizaion sraegies on he mos disadvanaged allow beer oucomes (Annex, Table A.2). The model also conrols by paricipan s age, given ha some programs (30 percen in his sample) were also open o workers of all ages. Addiional program conrols include: he decade when he inervenion was firs implemened and he curren saus of he program. Mos inervenions, nearly 72 percen, have been implemened during he nineies and 2000s, and over 60 percen are already compleed. The locaion of he program in rural and urban areas has also been considered in he model. The daa shows ha jus abou one hird of he inervenions have specifically argeed a cerain area, while wo hirds of he inervenions are implemened in boh rural and urban seings. The las variable under consideraion is he programs primary source of financing, which akes value 1 for governmen-sponsored inervenions (wo hirds of observaions) and 0 for ohers. 2. The model and resuls This analysis uses a probi model o esimae he probabiliy ha a cerain youh employmen program yields posiive impacs on he labor marke oucomes of is beneficiaries. Probi is a binary choice model ha esimaes he probabiliy of an even as a funcion of a se of aribues, assuming a normal disribuion in he daa. Following Hayashi (2000), in he probi model, a scalar dependen variable y is a binary variable, y { 0, 1}. In our case y = 1 indicaes ha a cerain program repored posiive labor marke impacs on youh, while y = 0 indicaes negaive or zero impacs. This even is deermined by a vecor of regressors x, namely caegory of inervenion, evaluaion qualiy, counry characerisics and program characerisics. As a resul, he condiional probabiliy of y given x is given by 14

15 f f ( y = 1 x ; β 0 ) ( y = 0 x ; β ) 0 = Φ = 1 Φ ' ( x β 0 ), ' ( x β ), where Φ (. ) is he cumulaive densiy funcion of he sandard normal disribuion. Given he binary feaures of y, his can be wrien compacly as f ' y ' 1 y ( x ; β ) = Φ( x β ) [ Φ( x β )] y Table 8 repors he resuls of wo model specificaions. Specificaion 1 includes he larger sample where QOE = 1, 2, 3, and Specificaion 2 resrics he sample o ne impac evaluaions, i.e., QOE =2, 3. The explanaory variables are he same, wih he excepion of he qualiy of evaluaion variable which is no needed in he second specificaion. Marginal effecs are displayed for each variable. These marginal effecs repor he change in he probabiliy of a posiive program impac for an infiniesimal change in each independen coninuous variable or for a discree change in he case of dummy variables. The compuaion of marginal effecs fis maximum-likelihood probi models, where he maximum-likelihood esimaor of β 0 is given by ' ' he funcion m( w ; β) = log f ( y x ; β) = y logφ( x β) + ( 1 y ) log[ 1 Φ( x β) ], where w is he -h observaion in he daase. The models esimaed coefficiens on which he marginal effecs are based are presened in he Annex, Table A On he firs se of variables regarding caegory of inervenion, he esimaes sugges here are no saisically significan differences among program ypes labor marke work beer and comprehensive programs in erms of he likelihood ha hey deliver posiive impac on he labor marke, relaive o raining programs. This resul holds for boh specificaions. The unclear paern of success across caegories of inervenion was also repored by Heckman e al. (1999) for a sample of OECD programs. In he esimaion process, wo caegories (enrepreneurship and ohers) were dropped due o collineariy effecs of heir small sample size on he prediced variable 18. On he oher hand, he mea-analysis confirms ha evaluaion qualiy maers. This is shown in Specificaion 1, where he saisically significan negaive coefficien for qualiy of evaluaion variable indicaes ha assessmens of program impac are likely more negaive when proper ne impac sudies have been carried ou. Having a ne impac evaluaion reduces he likelihood of success by 35 percenage poins (compared o he even of no having a ne impac evaluaion). This reflecs an over opimisic reading of resuls from evaluaions wih gross oucomes, and emphasizes he imporance of conducing rigorous evaluaions o capure he real effecs of programs. This resul suppors he qualiaive finding discussed in Secion C according o which he assessed impac of an inervenion is affeced by he qualiy of he underlying evaluaion evidence. 17 A logi model was also esimaed o es wheher a logisic disribuion beer fied he daa han a normal disribuion. The logi regression repored very similar esimaes han he probi. Resuls are shown in he Annex, Table A I is worh noicing he changes in sample size once all variables are considered ino play. Specificaion 1 goes from 172 observaions o 95, while Specificaion 2 goes from 73 o 59. This is mainly due o missing values in he programs characerisics, paricularly argeing on gender, disabled youh, and specific ehnic groups. 15

16 Table 8: Probi model reporing marginal effecs of youh employmen programs Specificaion 1 Specificaion 2 QOE=1, 2, 3 QOE = 2, 3 z-sa 6 Marginal effec z-sa 6 Marginal effec Caegory of inervenion 1 Labor marke work beer Comprehensive Qualiy of he evaluaion 2 Ne impac evaluaion * Economic and insiuional counry conex 3 Non-developed counries ** ** Rigidiy of employmen index ** * Specific characerisics of he program Time period and saus 4 Program implemened before he nineies * Compleed programs ** ** Targeing 5 Programs arge only youhs Programs locaed in specific areas Programs focus on women Programs focus on specific ehnic groups Programs focus on poor youh * * Programs focus on low-educaed youh Financing Governmen-sponsored Observaions = 95 Observaions =59 Pseudo R2 = 0.46 Pseudo R2 = 0.42 Noes: 1. Training-relaed programs (including skills raining and programs o make he raining sysems work beer) are he omied caegory. 2. Programs wih evaluaions reporing only gross oucomes are he omied caegory. 3. Developed counries are he omied caegory. The rigidiy of employmen index is a coninuous variable. 4. On decade of implemenaion, programs implemened during he nineies and 2000s are he omied caegory. On curren saus of he inervenions, ongoing programs are he omied caegory. 5. Omied caegories on argeing reflec none specific orienaion oward disadvanage people wihin hose groups. 6. The z-saisics es he null hypohesis of a zero coefficien following a sandard normal disribuion. The values of he z-saisics are repored in he hird column: * significan a 5%; ** significan a 1%. Economic and insiuional counry conex variables have highly significan effecs on program impac. The regressions show ha youh employmen programs are more effecive in developing counries and ransiion counries han in developed economies. The likelihood of success is beween percenage poins (depending on specificaion) higher when he program is implemened in a developing or ransiional seing han when i is carried ou in a developed counry. Given ha he qualiy of evaluaion is conrolled for, his resul 16

17 canno be explained by he fac ha impac evidence is more rigorous in developed counries. Anoher possible explanaion, which canno be esed here, is ha he skills disadvanage of paricipans in developed counries may ofen be oo large o be ackled by hese programs; while in developing counries where skills are scarcer, programs may provide enough of a boos o make a measurable difference. Likewise, he models sugges ha inervenions for youh end o be more successful in counries wih higher labor marke flexibiliy. In paricular, he higher he rigidiy of employmen index he lower he probabiliy of obaining posiive labor marke impacs. Noe ha he size of he coefficien in boh specificaions is very small and, while he effec may be saisically significan, he imporance seems relaively minor. Among program characerisics, he period of implemenaion and he curren saus of he program have significan effecs on he probabiliy of success. Firs, alhough saisical significance is borderline, he models sugges here is a learning process, where programs developed during he 1990s and afer end o yield beer oucomes han older programs. This is he case in Lain America where here has been a move owards demand-oriened programs ha mach effecively he needs of he producive secor, as well as open paricipaion of he privae secor and oher agens in he provision and financing of programs. Second, boh specificaions indicae ha ongoing programs perform beer han compleed programs. In erms of he programs orienaion oward paricular groups, of all characerisics of he arge populaion included in he model, only income level explains wih saisical significance program success. Programs argeing low-income youh perform significanly beer han programs wihou his orienaion, suggesing ha inervenions do have promise for improving he labor marke siuaion of low-income young people. Oher consideraions oward a paricular gender, he disabled, specific ehnic groups, and youh wih low educaion levels do no affec he oucomes. Similarly, he model esed wheher publicly-funded programs perform beer han oherwise, bu he marginal effec of source of financing lacked saisical significance. Resuls of Specificaion 1 were esed o ensure he bes fi of he model and o rule ou he possibiliy of ouliers. The firs es checked he sabiliy of he explanaory power in a smaller sample. Afer spliing he sample randomly in wo, he R-squared increases slighly from.46 o.51, suggesing a seady fi in he model. Marginal effecs of his model are repored in Table A.5, Annex. An addiional es was performed o ensure he sabiliy of he explanaory power by ruling ou he possibiliy of ouliers. Specificaion 1 is run ieraively by sequenially and randomly dropping one observaion wih replacemen. Niney-five models resuled from his exercise, and he R-squared repored ranged from.45 o.51 (Figure A.1, Annex) verifying he sabiliy of he specificaion s bes fi, and eliminaing he possibiliy of ouliers. E. Conclusions The Youh Employmen Invenory (YEI) is based on available documenaion of curren and pas programs and includes evidence from 289 sudies of inervenions from 84 counries in all regions of he world. These sudies have been analyzed based on he evaluaion evidence available in order o (i) documen he ypes of programs ha have been implemened o suppor young workers o find work, and (ii) idenify wha appears o succeed in erms of improving employmen oucomes for youh. 17

18 This paper uses a mea-analyical framework o examine simulaneously all inervenions colleced by he invenory wih evaluaion evidence on labor marke oucomes. I employs a probi model o esimae he probabiliy of obaining posiive program impacs based on he ype of inervenion, qualiy of he evaluaion evidence, characerisics of he programs, and counryspecific characerisics. Empirical resuls from a sample of 172 evaluaed sudies including ne impac evaluaions and evaluaions wih gross oucomes - indicae ha program success is no deermined by he ype of inervenion bu raher by he program s argeing sraegies oward disadvanaged youh, he counry level of developmen and he flexibiliy of he labor marke regulaions. Daa from he youh employmen invenory shows here are no major differences across ypes of inervenions in erms of impac, suggesing ha policy-makers should consider which ype of inervenion bes addresses he problem of concern. This unclear paern was also found by Heckman e al. (1999) when analyzing acive labor marke measures in he OECD area. However, i differs from Kluve s (2006) findings according o which some inervenions work in fac beer han ohers. In addiion, he mea-analysis of he invenory shows ha counry conex maers when assessing he impac of youh employmen programs. An employmen program implemened in a developing or ransiional counry has a leas a 50 percen higher probabiliy of yielding posiive impac for youh han a developed-counry program. The analysis proved his is no a measuremen problem, since he esimaes hold even when he sample is consrained o sudies wih ne impac evaluaion. Oher explanaions may come ino play, such as he human capial gap beween hese wo groups of counries. Labor marke insiuions appear o have small bu significan effecs on program impac. The model shows ha less flexible employmen proecion rules, measured by he rigidiy of employmen, slighly lower he probabiliy of obaining posiive oucomes from youh employmen programs. Cerain characerisics of he programs show ineresing effecs. Ongoing programs and hose carried ou since he 1990s have significanly beer performance han earlier inervenions, suggesing a poenial learning process. Moreover, argeing inervenions on economically disadvanaged youh appears o have subsanial posiive impac on paricipans labor marke prospecs. Sensiiviy ess show hese resuls are sable under differen specificaions, paricularly when he sample size is consrained o sudies wih ne impac evaluaions. 18

19 References. Becherman, Gordon, Karina Olivas, and Ami Dar Impac of Acive Labor Marke Programs: New Evidence from Evaluaions wih Paricular Aenion o Developing and ransiion Counries. Washingon, D.C.: World Bank, Social Proecion Discussion Paper Series Becherman, Gordon; Marin Godfrey; Olga Susana Puero; Friederike Roher; and Anonea Savreska Global Invenory of Inervenions o Suppor Young Workers: Synhesis Repor. Preliminary draf. Washingon, D.C.: World Bank. Dar, Ami and P. Zafiris Tzannaos Acive Labor Marke Programs: A Review of he Evidence from Evaluaions, Social Proecion Discussion Paper no. 9901, January. The World Bank. Washingon, D.C. Gay, Rober, and Michael Borus Validaing performance indicaors for employmen and raining programs, Journal of Human Resources. Winer 1980, 15, Greenberg, David H.; Charles Michalopoulos; Philip K. Robins A Mea-Analysis of Governmen-Sponsored Training Programs. Indusrial & Labor Relaions Review. Volume 57, Issue Aricle 2. Hayashi, Fumio Economerics, Princeon Universiy Press. Heckman, J.J., R.J. LaLonde and J.A. Smih (1999), "The economics and economerics of acive labour marke programs", in O. Ashenfeler and D. Card (eds.), Handbook of Labor Economics 3, Elsevier, Amserdam. Hun, Moron How Science Takes Sock: The Sory of Mea-Analysis. New York: Russell Sage Foundaion. Jarrell, S. B., and T. D. Sanley A Mea-Analysis of he Union-Nonunion Wage Gap. Indusrial and Labor Relaions Review, Vol. 40, No. 2 (January), pp Kluve, J The Effeciveness of European Acive Labor Marke Policy, IZA Discussion Paper, No. 2018, Bonn. Kluve, J. and C.M. Schmid (2002), "Can raining and employmen subsidies comba European unemploymen?", Economic Policy 35, Marin, J.P. and D. Grubb (2001), "Wha works and for whom: a review of OECD counries experiences wih acive labour marke policies", IFAU Working Paper 2001:14. Puero, Olga Susana. 2007a. Inervenions o Suppor Young Workers in Lain America and he Caribbean. World Bank: Washingon D.C. Puero, Olga Susana. 2007b.Learning from Inernaional Experiences, The Youh Employmen Invenory; Background paper for he Sierra Leone Youh and Employmen ESW, World Bank, Washingon D.C. 19

20 Roher, Friederike and Olga Susana Puero Inervenions o Suppor Young Workers in OECD counries. World Bank: Washingon D.C. Roher, Friederike Inervenions o Suppor Young Workers in Sub-Saharan Africa. World Bank: Washingon D.C. Sirmans, G. Sacy; Lynn MacDonald; David A. Macpherson; Emily Norman Ziez The Value of Housing Characerisics: A Mea Analysis. The Journal of Real Esae Finance and Economics. Volume 33, Number 3 / November, 2006 Savreska, Anonea. 2006a. Europe and Cenral Asia, Youh Employmen Invenory Summary Repor. World Bank: Washingon D.C. Savreska, Anonea. 2006b. Inervenions o Suppor Young Workers in Souh and Eas Asia and he Pacific. World Bank: Washingon D.C. 20