S P D I S C U S S I O N P A P E R NO. 0715 Authorized 41412 Disclosure Public Authorized A Review of Interventions to Support Young Workers: Findings of the Youth Disclosure Employment Inventory Public Gordon Betcherman, Martin Godfrey, Susana Puerto, Friederike Rother and Antoneta Stavreska Authorized October 2007 Disclosure Public Authorized Disclosure Public 41412 A Review of Interventions to Support Young Workers: Findings of the Youth Employment Inventory Gordon Betcherman, Martin Godfrey, Susana Puerto, Friederike Rother, and Antoneta Stavreska October 2007 JEL Classification: J13, J18, J24 Keywords: youth employment, program effectiveness, impact evaluation 41412 41412 TABLE OF CONTENTS Executive Summary.................................................................................................i A. Introduction: the objective of the inventory and of the report...................................1 B. Background: the nature of youth employment problems and policies to address them......................................................................................................2 1. Youth employment issues................................................................................2 2. Policies to address youth employment issues.......................................................4 C. The methodology for designing and compiling the inventory....................................8 1. Framework for classifying interventions............................................................8 2. The inventory ­ how it was compiled...............................................................11 D. Coverage of the inventory ­ what interventions have been implemented?..........................16 E. Quality of evaluations of programs in the inventory...............................................27 F. Quality of interventions in the inventory: Descriptive analysis.................................32 G. Quality of interventions in the inventory: Meta analysis.........................................55 1. Methodology..........................................................................................56 2. Results..................................................................................................58 H. Conclusions............................................................................................62 References.......................................................................................................70 1 41412 TABLES Table 1: Youth unemployment rate, ratio of youth to adult unemployment rate, youth labor force participation rate, and youth employment-to-population ratio, by region, 1995 and 2005 ..............................................................................................3 Table 2: Categories used to classify programs in the YEI.................................................10 Table 3: Measuring the quality of intervention (QOI)......................................................13 Table 4: Measuring the quality of evaluation (QOE).......................................................14 Table 5: Possible choices for Quality of Intervention Given Quality of Evaluation....................15 Table 6: Coverage of inventory by category of intervention and region..................................17 Table 7: Coverage of inventory by category of intervention, location, and age group served........22 Table 8: Orientation towards disadvantaged groups by broad category of intervention................24 Table 9: Coverage of inventory by category of intervention and source of finance....................26 Table 10: Quality of Evaluation (QOE) evidence by category of intervention...........................30 Table 11: Percentage distribution of Quality of Evaluation (QOE) by region............................31 Table 12: Quality of Intervention (QOI) disaggregated by Quality of Evaluation (QOE) .............36 Table 13: Simulation of overall program success rate (positive impact, cost effective) for all programs with evaluation evidence of any kind..................................................38 Table 14: Simulation of overall program success rate (positive impact, cost effective) for programs with evidence on net impact ............................................................38 Table 15: Summary rating of Quality of Intervention by category of intervention......................39 Table 16: Simulation of program success rate (positive impact, cost effective), by category of intervention ...........................................................................................41 Table 17: Assessment of labor market impact of selected training programs in transition and developing countries..................................................................................47 Table 18: Comprehensive programs with net impact evaluations..........................................49 Table 19: Summary rating of Quality of Intervention by region ...........................................51 Table 20: Relative impact and cost-effectiveness of interventions oriented towards the disadvantaged.........................................................................................55 Table 21: Probit model reporting marginal effects of youth employment programs....................59 Table 22: Summary of program targets, design and risks, and impacts, Youth Employment Inventory...............................................................................................67 Table 23: What program for which problem? ................................................................69 2 41412 FIGURES Figure 1: Categories of interventions by OECD country sub-groups.....................................19 Figure 2: Summary of Quality of Evaluations (QOE).......................................................28 Figure 3: Percentage of programs with net impact evaluations by category of intervention..........29 Figure 4: Percentage distribution of programs by Quality of Evaluation (QOE) for OECD country sub-groups..................................................................................32 Figure 5: Summary of Quality of Interventions (QOI) for all programs with evaluation evidence of any kind.................................................................................35 Figure 6: Summary of Quality of Interventions (QOI) for programs with evidence on net impact..................................................................................................36 Figure 7: Percentage of programs with positive labor market impact, regardless of cost, by category of intervention..........................................................................40 Figure 8: Percentage of interventions with positive labor market impact and with cost- effectiveness, by region.............................................................................51 Figure 9: Percentage of interventions with positive labor market impact and with cost- effectiveness, by income level (Panel A) and country type (Panel B)........................52 Figure 10: Percentage of interventions with positive employment impact and with cost-effectiveness, by country's rigidity of employment..............................54 BOXES Box 1: ECA wage subsidies for young people ­ An example of programs designed to make the labor market work better for young people (Category 1)............................21 Box 2: The Commonwealth Youth Credit Initiative -- An example of programs designed to improve chances for young entrepreneurs (Category 2)......................................25 Box 3: Entra 21 ­ An example of programs designed to provide skills training for young people (Category 3)...................................................................................27 Box 4: Kenya's Jua Kali voucher program ­ An example of programs designed to make training systems work better for young people (Category 4)....................................30 Box 5: Evaluation within a cost/ outcome framework ­ a mini-manual................................34 Box 6: LAC's Jovenes programs ­ An example of comprehensive multiple-service approaches (Category 8)...............................................................................50 3 41412 ANNEXES Annex A: Inventory template and coding system...........................................................74 Annex B: Template Reader Manual..........................................................................75 Annex C: Unit costs of a sample of Youth Employment Programs......................................77 Table C.1. Unit costs of Wage Subsidies Programs..............................................77 Table C.2. Unit costs of Public Works Programs.................................................77 Table C.3. Unit costs of Entrepreneurship Schemes.............................................77 Table C.4. Unit costs of Training Programs.......................................................77 Table C.5. Unit costs of Second-chance Programs in the U.S. .................................78 Table C.6. Unit cost estimates across Teenage Parent Demonstration Programs in the U.S......................................................................................................78 Table C.7. Unit cost estimates for Jóvenes Programs............................................78 Table C.8. Unit cost estimates for Comprehensive Programs in the OECD area............79 Annex D: Further results from the Meta-analysis..........................................................79 Table D.1.: Sample of interventions for the meta-analysis.....................................79 Table D.2: Classification of categories of intervention by labor market impact (for a sample of programs with QOE=1, 2, 3)......................................79 Table D.3: Classification of countries' level of development by labor market impact (for a sample of programs with QOE=1, 2, 3)......................................79 Table D.4: Number of interventions targeting disadvantaged youths by labor market impact (for a sample of programs with QOE=1, 2, 3)..............................79 Box D.1. The probit model.........................................................................80 Table D.5: Probit model: simple coefficients (Table 19, synthesis report)....................81 Table D.6: Probit model: Specification 1 dropping randomly 50 per cent of the sample....................................................................................82 Figure D.1: R-squares of 95 models featuring Specification 1 (Repeatedly dropping observations with replacement).......................................................82 4 41412 A Review of Interventions to Support Young Workers: Findings of the Youth Employment Inventory Executive Summary The Youth Employment Inventory has been compiled to improve the evidence base for making decisions about how to address the problem of youth employment. As policy- makers consider measures to help young people make the transition into the labor market and obtain decent work, they are hampered by a lack of information on what their options are, what works in different situations, and what has been tried and failed. To respond to this situation, the World Bank has compiled a world-wide inventory of the interventions that are designed to integrate young people into the labor market. This Youth Employment Inventory (YEI) is based on available documentation of current and past programs and includes evidence from 289 studies of interventions from 84 countries in all regions of the world. The interventions included in the YEI have been analyzed in order to (i) document the types of programs that have been implemented to support young workers to find work; and (ii) identify what appears to work in terms of improving employment outcomes for youth. This synthesis report pulls together the information from this inventory and a set of background reports to document the global experience with youth employment programs. The YEI includes programs designed to facilitate the transition of young people into the labor market, with a focus on disadvantaged young people. The inventory is meant to be as exhaustive as possible and is not confined to success stories, on the principle that there is a great deal to be learned from mistakes and failures. The YEI does not include new project information but, rather, is based exclusively on existing documentation gathered from a wide range of published and electronic sources. For practical reasons, the inventory is largely limited to post-formal schooling interventions. It covers ongoing and completed interventions specifically targeted at young people or that had young people as primary participants. The most common type of intervention for youth is skills training. This category accounts for 39 per cent of all interventions and is significant in all regions, but is especially popular in Latin America and the Caribbean where it represents 56 per cent of the programs included in the inventory. Comprehensive multiple-service interventions -- for instance, combining vocational and on-the-job training with wage subsidies and public works, or classroom and on-the-job training with paid work experience and job search assistance ­ account for 32 per cent of the total. One-half of these multiple-service programs are in OECD countries. Making the labor market work better for young people (especially through wage subsidies), and improving chances for young entrepreneurs each account for 12 per cent of the total. i 41412 The largest number of interventions is in the OECD area but Latin America and the Caribbean also has good coverage. The methodology for assembling the inventory emphasized the search for programs in developing countries. However, 42 per cent of the interventions in the inventory are from OECD countries; this reflects both the level of activity as well as the availability of documentation in industrialized countries. Among developing regions, youth programs have been most widely implemented in Latin America, which accounts for 24 per cent of the interventions included in the inventory. The shares in the other regions are 14 per cent in Eastern Europe and Central Asia, 10 per cent in Sub- Saharan Africa, 7 per cent in South and East Asia and the Pacific, and 3 per cent in the Middle East and North Africa. To the extent that programs are targeted, they most frequently target young people with low income and/or education. Just over half (51 per cent) of all programs in the inventory are oriented towards young people with low incomes, or in low-income families. The incidence of gender targeting is relatively low, with only 16 per cent of all programs oriented towards young women and 2 per cent towards young men. The inventory includes 32 programs (11 per cent) targeted at disabled young people, while only a small number of interventions (20 in total) are targeted at particular ethnic groups. The analysis of the patterns of program success concludes that interventions oriented towards disadvantaged youth are as good, if not better, than programs with no particular orientation. One of the major observations from the research is that the level of program evaluation has been weak, especially in developing countries. A strong conclusion is the need for major improvements in the quality of evidence available for youth employment interventions. For almost 40 per cent of programs included in the inventory, no evaluation information at all on outcomes or impact could be found. An additional 35 per cent have studies which cover only gross outcomes, and do not use a methodology (e.g., based on a control group) to estimate net impact. In other words, only about one-quarter of all programs included have some evidence on the net impact. And, of the programs that meet this evaluation standard, most (45 of 73) do not include any cost-benefit analysis. Overall, only one in 10 programs included in the inventory has an evaluation which measures both net impact and cost. Moreover, these figures likely overestimate the true of incidence of scientific evaluations of youth programs since interventions with extensive analysis and documentation were more likely to be captured for the inventory. The current reality is that, outside the OECD area (especially the Anglo-Saxon countries) and other than studies sponsored by international organizations, rigorous evaluations are quite rare. The absence of rigorous evaluations almost certainly leads to an overestimation of program impacts by policy-makers. Properly evaluated programs are less likely to lead to positive assessments of impact and effectiveness than judgments based on "non-scientific" methodologies. In the absence of such evaluations, then, policy-makers are likely to overestimate the benefit of their interventions and, as a result, allocate resources inefficiently. This is a particular concern in developing countries where resources are scarce and evaluations are uncommon. ii 41412 The majority of interventions included in the inventory appear to have positive labor market impacts for participants. Two specific performance indicators ­ post-program employment and earnings ­ are considered in assessing program "impact". An assessment of impact could be made for 172 interventions where either gross outcomes or net impacts were available. Of these 172 programs, 132 (78 per cent) were rated as having had a positive impact in terms of the employment and/or earnings of participants. When only programs with net impact evaluations were considered, the share with demonstrably positive labor market impacts for participants was 60 per cent. But once cost-effectiveness is taken into account along with labor market impacts, less than half of the programs in the inventory could be judged as successful. Although a full picture of the overall success of youth programs should also consider the cost dimension, it is not possible to determine cost-effectiveness in the vast majority of cases. Of the 134 programs assessed to have positive employment impact, only 25 have a cost-benefit analysis. Of these, 14 were cost-effective (56 per cent) while 11 (44 per cent) were not. Assuming that this observed ratio of cost-effectiveness applies to programs without cost information, we can estimate the overall success rate of interventions, where "success" is defined as having a positive labor market impact and cost-effectiveness. Our estimate, using all programs with outcome indicators, is that about 44 per cent of interventions are successful according to this definition. When we restrict our calculations to programs with net impact evaluations, the estimated success rate is 33 per cent. The factors associated with program impact were examined through both descriptive analysis and a more rigorous meta analysis. The statistical analysis of program outcomes looked at the role of various factors specific to the intervention and to the context in which it was implemented. To identify the determinants of positive program impacts systematically, a meta-analysis of the interventions in the inventory was carried out, using econometric methods to combine and synthesize results from the individual studies to get an overall picture. The results suggest that there are no major differences across categories of interventions in terms of impact or cost-effectiveness. Three categories of interventions ­ making the labor market work better for young people (primarily wage subsidies, public works, and job search assistance), skills training, and comprehensive programs ­ each had similar percentages of programs with positive impacts. Although entrepreneurship programs had the highest positive impact rating, the number of these interventions in the inventory is too small to draw firm conclusions. The meta analysis found no statistically significant differences in the impact of the different program types. The policy implication of this finding is that particular types of programs should not be favored but, rather, that interventions should be chosen based on the specific obstacles to employment that need to be overcome. iii 41412 The employment impact of youth interventions tends to be more favorable in developing and transition countries than in industrialized countries. The probability that programs will help young people in the labor market is greater in developing and transition countries than in industrialized ones. This is not due to the more rigorous evaluations in developed countries. The meta-analysis confirmed that the difference in program impact by level of development remained even after the quality of the evaluation evidence was taken into account. The study could not adequately explain this result, but it would be interesting to test two hypotheses. First, are disadvantaged youth so "disadvantaged" in developed countries that employment interventions are simply not enough to compensate? Second, are there institutions and policies that systematically differ by level of development that might explain the variation in program outcomes? Youth programs have a lower likelihood of having a positive impact in countries where labor markets are not flexible although the magnitude of the effect is not large. Research has shown that protective employment rules create barriers for new entrants and our results suggest that employment programs do not significantly overcome these barriers. The meta-analysis finds that the rigidity of employment protection rules is associated with a lower probability of positive employment benefits to participants, although the magnitude of the effect is very small. In any event, policy-makers need to take a comprehensive approach to improving youth employment, implementing well-designed interventions and also, ensuring that labor market policies and institutions do not block access for young people. iv 41412 A Review of Interventions to Support Young Workers: Findings of the Youth Employment Inventory1 A. Introduction: Objective of the Inventory and of the Report Youth employment has become a major concern in many countries around the world. As policy-makers consider measures to help young people make the transition into the labor market and obtain decent work, they are hampered by a lack of information on what their options are, what works in different situations, and what has been tried and failed. To respond to this situation, the World Bank has compiled a world-wide inventory of interventions that are designed to integrate young people into the labor market. This Youth Employment Inventory (YEI) is based on available documentation of current and past programs and includes evidence from 289 interventions from 84 countries in all regions of the world. The interventions included in the YEI have been analyzed in order to (i) document the types of programs that have been implemented to support young workers to find work; and (ii) identify what appears to work in terms of improving employment outcomes for youth. The inventory of programs itself is available as an on-line database.2 This report synthesizes the information from this inventory and a set of background reports to document the global experience with youth employment programs.3 As background, Section B provides a brief summary of the situation of young people in labor markets world-wide, and also reviews the existing literature on policies to address youth employment problems. Following this, we turn to the underlying framework and methodology used to assemble the youth employment inventory in Section C. In Section D, we consider the coverage of the YEI, which represents the sample of youth programs 1 The Youth Employment Network (YEN) and the German Technical Cooperation (GTZ) assisted in the collection of program information used for the inventory. The final version of this synthesis report has benefited from comments by Wendy Cunningham, Linda McGinnis, Jean Fares, and Sudharshan Canagarajah as well as comments received during review meetings involving the World Bank, YEN, and ILO staff, and at seminars in the Europe and Central Asia and Latin America and Caribbean regions of the World Bank. 2 The database can be accessed at http://go.worldbank.org/48Z06GMD70 3The background reports include regional reports for the OECD (Rother and Puerto, 2007), Eastern Europe and Central Asia (Stavreska, 2006), Latin America and the Caribbean (Puerto, 2006), Sub-Saharan Africa (Rother, 2006), and Asia (Stavreska, 2006). The Asia report includes both East and South Asia because of small samples in these two regions. No report has been prepared for the Middle East and North Africa because of an insufficient number of programs. In addition to these regional reports, two analytical background reports have been prepared ­ one covers lessons learned from the impact evaluations and the other presents the results of a meta-analysis of the evaluations (Puerto 2007a,b). 1 41412 identified in our global search of the available documentation. In effect, this sample provides a description of what types of programs have been implemented to support young workers. In addressing the question of "what works", it is critical to pay close attention to the quality of the evaluation evidence. This is discussed in Section E. We then turn to the analysis of the effectiveness of the interventions included in the inventory. The descriptive evidence is presented in Section F. In addition, we have undertaken an econometric meta-analysis to more systematically identify the determinants of program success and the results of this analysis are presented in Section G. Finally, conclusions and implications are drawn in Section H. B. Background: The Nature of Youth Employment Problems and Policies to Address them At the outset, it should be recognized that many young people make the transition into employment successfully, without requiring any special assistance. However, many others experience difficulty, either taking a very long time to gain a foothold or finding themselves outside the labor market completely. Certain groups have particular problems, especially those with poor education and without basic skills. Young women in many countries, youth with disabilities, those affected by HIV/ AIDS, ethnic minorities, demobilized soldiers, and migrants are often at a special disadvantage. 1. Youth Employment Issues Youth employment problems have various dimensions and can be manifested in different ways.4 The most familiar is unemployment; in fact, the unemployment rate of 15-24 year-olds is one of only two employment indicators in the Millennium Development Goals. Although this age group represents only 25 per cent of the world's working age population, it accounts for almost one-half of global unemployment (World Bank 2006). As Table 1 shows, youth unemployment rates have generally been increasing over the past decade, as has the ratio between the youth and adult unemployment rate. The extent of the youth unemployment problem does vary by region; however, it is a serious concern everywhere. Except in the developed economies, the Central and Eastern European and Commonwealth of Independent States (CEE/CIS) countries, and East Asia, unemployment rates are higher for young women than for young men ­ particularly in Latin America and the Caribbean and the Middle East 4The age range of 15-24 is generally used to define youth in this report. 2 41412 and North Africa (ILO 2006). In South Asia, Latin America, OECD, and CEE/CIS countries, youth unemployment rates tend to be highest among the less educated. However, in several developing countries including those in the Middle East and North Africa and in Africa, the highest youth unemployment rates have been observed among the more educated (ILO 2006). Where this is the case, it is often due to high reservation wages and/or selectivity in job search (e.g., queuing for public sector jobs), supported by relatively prosperous families. However, unemployment is not the only indicator of youth labor market difficulties. In fact, it often understates the magnitude of the problem for two reasons: first, many jobless young people who would like to work are "discouraged" and are not counted as unemployed since they are not actively searching for work, and second, many have little choice but to work even in very bad jobs. So other categories of young people ­ inactive "discouraged workers", unpaid family workers, self-employed earning very little income, badly-paid wage earners, etc. ­ are also at a disadvantage in the labor market. The ILO (2006) estimates the number of "young working poor" (earning less than $1 a day) in the world in 2005 at around 125 million, 23 per cent of the total workforce in this age group. The rate is particularly high in South Asia and Sub-Saharan Africa. Table 1: Youth Unemployment Rate, Ratio of Youth to Adult Unemployment Rate, Youth Labor Force Participation Rate, and Youth Employment-to-population Ratio, by Region, 1995 and 20051 Youth Ratio of youth Youth labor Youth unemployment to adult force employment-to rate (per cent) unemployment participation population rate rate (per cent) ratio 1995 2005 1995 2005 1995 2005 1995 2005 World 12.3 13.5 2.8 3.0 58.9 54.7 51.6 47.3 Developed economies & other EU 15.2 13.1 2.3 2.3 53.6 51.8 45.4 45.0 CEE (non-EU)/ CIS 19.6 19.9 2.6 2.6 47.2 41.8 38.0 33.5 East Asia 7.5 7.8 2.9 2.8 75.2 67.3 69.5 62.1 South-east Asia & Pacific 9.2 15.8 4.7 5.1 58.1 56.5 52.8 47.5 South Asia 9.9 10.0 3.6 2.8 50.6 47.2 45.6 42.5 Latin America & Caribbean 14.4 16.6 2.7 2.8 56.4 54.2 48.3 45.2 Middle East & North Africa 28.7 25.7 3.0 3.1 40.0 40.0 28.5 29.7 Sub-Saharan Africa 17.5 18.1 3.3 3.0 68.2 65.5 56.2 53.7 1. "Youth" defined as 15-24 year olds. Source: ILO (2006: Figures 2.3 and 2.4; Tables 2.3, 2.4 and 2.6). Table 1 also shows that both the youth labor force participation rate and the employment-to-population ratio have fallen over the past decade, both globally and in all 3 41412 regions except the Middle East and North Africa. Rates are particularly low in that region and in the transition economies. Declining participation and employment rates are partly due to an increase in educational enrollment ­ between 1990 and 2000 the number of students in the world's secondary schools increased by 15 per cent and in higher education institutions by 8 per cent (ILO 2005). However, these figures also reflect withdrawal from or failure to enter the labor force by an increasing proportion of school leavers. Indeed, cross-country household data assembled for the recent World Development Report show that many young people are neither studying nor are in the labor market (World Bank 2006).5 Many young people unable to find formal wage employment end up in the informal economy. The incidence of unpaid work is also high. In economies where informality is widespread (in itself often a symptom of policy failure), informal and unpaid work can be a stepping stone to better jobs in the future, but for many young people this is not the case (World Bank 2006). While informal employment is less prevalent in high-income countries, young workers disproportionately hold precarious jobs, such as temporary employment. In Latin America, the recent increase in temporary contracts has particularly affected young people, especially those from the poorest households (ILO 2004). Another indicator of difficulties youth experience in the labor market is the estimated 59 million 15-18 year olds worldwide who are in hazardous forms of work (ILO 2005). Many also earn extremely little, but unfortunately reliable earnings data are not widespread. 2. Policies to Address Youth Employment Issues This brief review illustrates the difficulties many young people face in the labor market, including the special challenges confronting certain categories of youth. Longer- term analysis in some countries has shown that part of the issue is a "transition" problem, with young people needing time to accumulate the experience and skills required to find good jobs. However, this is certainly not the case for all youth and, in any case, waiting out the transition period is not an option for policy-makers or for young people themselves. Moreover, the social and economic hardship young workers experience because of employment problems is intensified when longer-term "scarring" also occurs. Accordingly, there is great interest in how interventions can smooth the transition of young people ­ 5The major reason among young men is discouragement about finding work while young women are more often engaged in "non-market" activities (e.g., household responsibilities, raising children, etc.). 4 41412 especially vulnerable youth -- into the labor market, helping them to find their first jobs, to become economically self-sufficient, and to lay the groundwork for productive careers. The 2007 World Development Report reviews the key policy areas for broadening employment opportunities for young people. In fact, some of the most important policies lie outside the labor market. Since young people suffer disproportionately from weak labor demand, the overall health of the economy is critical. This underscores the importance of sound macroeconomic conditions and a positive investment climate. Without these preconditions, young people will have scarce employment opportunities. Of course, investment in human capital through formal education is essential for taking advantage of these opportunities. Finally, and closer to the concerns of this report, labor market policies, institutions, and programs can make a significant difference in terms of creating opportunities for young people, enhancing their capabilities to take advantage of these opportunities, as well as offering second chances to those who need them.6 Policy-makers have taken a range of measures to reform labor markets that are intended to improve employment opportunities for youth and others. For example, in 1990, Colombia substantially reduced the cost of dismissing workers, which increased turnover for formal-sector workers but also reduced the length of unemployment spells, particularly for young and more educated workers (World Bank 2006). Setting wages for apprentices below the minimum level, thus subsidizing on-the-job training has significantly increased the job opportunities for young workers in Chile. In middle-income European countries, youth- specific wage subsidies, if well targeted and of limited duration, have been found to have a large beneficial effect on employment (World Bank 2006). Reform of training systems can also be important. In industrialized countries, formal apprenticeship schemes, combined to a varying extent with part-time schooling, have had a positive impact on employment for young men and on earnings for young women. However, questions have been raised about how replicable these successes can be in developing countries where the formal wage sector is small and institutions are weak. Moreover, traditional apprenticeship systems are now running into trouble adjusting to the demands of a rapidly changing global economy, even in countries with a long tradition of dual education 6These elements of opportunity, capability, and second chance form the basis of the conceptual framework used in the World Development Report. See World Bank (2006) for more details on this framework. 5 41412 (Quintini and Martin 2006). The role of employers as a provider of skills for young people is limited. Surveys in many countries have shown that larger enterprises and those that export and use new technologies are likely to provide training; however, the bulk of enterprises do not invest in their young workers through formal training. Efforts are being made in many countries to reform rigid, low-quality training programs, disconnected from labor markets, by changing the role of government from that of provider to manager and policy developer, with more competition between public and private providers, and by moving away from a narrow focus on inputs to a focus on outcomes. Active labor market programs (ALMPs) have been widely used to enhance labor supply, stimulate labor demand, and improve the functioning of the labor market. These programs are often targeted at specific groups, including young people. The existing knowledge on what works in the area of ALMPs is hampered by the lack of solid information and evaluation evidence, especially on programs outside the OECD area (Betcherman, Olivas, and Dar 2004).7 Over the past decade, there have been various cross-country reviews of evaluations of ALMPs, including those targeted at young people. For example, Heckman et al. (1999) observed the impact of job training, job search assistance, and wage subsidies on employability, finding only very moderate and rather disappointing outcomes, especially for youth. Based on a sample of evaluation studies of ALMPs implemented in Europe and the U.S., Kluve and Schmidt (2002) found mixed program effects across different types of interventions and target populations: while training and job search assistance were effective in improving participants' labor market prospects, direct job creation programs in the public sector led to negative outcomes. Young workers were the most difficult group to assist among the unemployed. Reviews of the evaluation evidence by the World Bank and the OECD have come to similar conclusions (e.g., Betcherman, Olivas, and Dar 2004; Dar and Tzannatos 1999; Martin and Grubb 2001). On balance, these programs have not been a panacea for unemployment but when they are carefully designed, targeted, and implemented, they can improve the employment prospects for some workers. In their review, Betcherman, Olivas, 7In OECD countries, especially Anglo-Saxon ones, there is a tradition of impact evaluations. In some countries, such as the U.S., the availability of public funds relies greatly on evaluation outcomes. 6 41412 and Dar (2004) looked at the evidence on training for young people (usually targeted at those with low levels of education) and concluded that the impact of these programs had not been very favorable. Their results supported other studies (e.g., Godfrey 2003) indicating that it was difficult to reverse education failures through training. According to impact evaluations, the relatively few examples of positive outcomes appear to be limited to comprehensive programs that integrated training with other services such as remedial education, job search assistance, and social services. A background paper for the 2006 OECD Employment Outlook includes a useful summary of what features of ALMPs appear to work for youth in member countries (Quintini and Martin 2006: 28). Programs should come into play early ­ after a period of unemployment of at most six months (as in Australia, Belgium, Denmark, New Zealand, Norway, and the UK). Sweden activates such programs after 90 days, Finland immediately for those without a vocational qualification. Job-search assistance programs are found to be the most cost-effective for youth, with wage and employment subsidy programs having a positive short-term impact but a less positive net impact on the longer-term employment prospects of participants. In order to connect training programs to local or national labor market needs, the private sector and local communities need to be mobilized and involved in project design. Targeting of programs is crucial, distinguishing between teenagers (who should be helped to remain in school and acquire qualifications) and young adults (who need help in acquiring work experience), and focusing on school drop-outs. Programs should insist on tight work-search requirements, in the interests of an early exit from unemployment. Integration of services into a combined, comprehensive package seems to be more successful than separate provision. Effectiveness of programs is increased by greater involvement of social partners and of public authorities at all levels. Quintini and Martin (2006) emphasize two weaknesses of active labor market programs for youth in OECD countries. They can be rather expensive and it is extremely difficult to tackle the problem of very disadvantaged youth. The high cost of programs, they 7 41412 suggest, makes it important to ensure that the exit from unemployment is into real jobs rather than into excessively lengthy education and training or expensive job-creation schemes. And evaluation of several programs points to the need to identify the most vulnerable young people as early as possible during their transition and to provide them with specific assistance. Systematic information on these issues is very limited in the context of developing and transition countries. C. The Methodology for Designing and Compiling the Inventory The youth employment inventory (YEI) includes programs designed to facilitate the transition of young people into the labor market. In particular, the focus is on disadvantaged young people. The inventory is meant to be as exhaustive as possible and is not confined to success stories, on the principle that there is a great deal to be learned from mistakes and failures. The YEI itself does not include new project information but, rather, is based exclusively on existing documentation. This information has been gathered from databases, research papers, publications and web-sites of international organizations (the World Bank, the United Nations and its regional commissions, the International Labor Office, the Asian Development Bank, the African Development Bank, the Inter-American Development Bank, the Organization for Economic Cooperation and Development, the European Union and its institutions, other regional organizations, etc.), bilateral donor agencies, non-government organizations, national labor market programs, national research institutions, as well as academic publications, both books and journals, and conference reports. This section presents the methodology implemented to compile the inventory. It includes a description of the framework used to categorize interventions, and then summarizes the data-collection effort, focusing on the key methodological questions that define the scope and content of the inventory. 1. Framework for Classifying Interventions A basic issue to be resolved was setting the boundaries on what to include in the inventory. Most important was how far back into the education system the inventory should cover. Analytical considerations alone would suggest that it should go back a long way. Many studies have concluded that the impact of interventions on future employment outcomes of disadvantaged young people diminish with age ­ in other words, addressing 8 41412 potential problems early has a greater return than when young people have left formal education. For example, in reviewing the evidence, the OECD (2002) has concluded that "the biggest pay-off for disadvantaged youths comes from early and sustained interventions."8 In other words, any policy advice on addressing youth employment problems should emphasize that prevention is more effective than curing. However, while there is no denying the strength of this analytical point, there are practical grounds for limiting the inventory to post-formal-schooling interventions.9 One was the need to set boundaries to limit the inventory to a feasible size. The second was to give it a clear identity that differentiates the study from the enormous body of literature on formal education. By limiting the scope in this way, we do not intend to detract from the importance of formal schooling and early interventions in improving subsequent labor market outcomes. The template used to categorize programs in the inventory builds on an earlier framework developed by Godfrey (2003). That framework embodied a two-fold approach to policy to address the employment problems of disadvantaged youth: (1) increasing the demand for labor in general in relation to supply, and (2) increasing the 'integrability' of the disadvantaged young, so that they can take advantage of opportunities that arise when the demand for labor increases. Integrability can be increased by (a) remedying or counteracting market failure (e.g., in the labor market, credit market, or training market), (b) improving labor market regulations, and (c) improving the skills of disadvantaged youth. Based on these two premises, the inventory classifies youth employment interventions into 9 categories, displayed in Table 2. These groupings are largely self- explanatory but a few comments may be useful. Category 1, "making the labor market work better for young people", includes interventions that improve information (counseling, job search skills), increase labor demand for youth (wage subsidies and public works), and remove discrimination. Category 2, "improving chances for young entrepreneurs", covers interventions that provide assistance (financial, technical, and training) to youth who are 8The OECD review goes on to note that "...[S]uch interventions should begin even before children enter the compulsory schooling system, and they should be followed by intensive efforts to boost their performance in primary and secondary schooling and reduce drop-out rates." Pre-school and school programs that attempt to improve the relative access and learning outcomes of children from disadvantaged backgrounds (variously defined) are particularly interesting. 9There were five cases of programs (all in OECD countries) included in the inventory where participants could either be unemployed youth who participated as a "second chance program" or young graduates who continued in a vocational training program as part of formal schooling. 9 41412 starting their own business. Categories 3 and 4 both deal with training: the former includes the full range of post-formal schooling training programs while the latter includes interventions intended to address training market failures by providing information, credit, and other financial incentives. Location can also be a barrier for young people if where they reside isolates them from learning or employment opportunities, or even a secure living environment. Category 5 is meant to include interventions (e.g., transportation services or residential mobility) that can help young people overcome this form of barrier. Category 6 covers regulatory reforms (e.g., changes in labor law, minimum wage, etc.) that are designed to enhance employment opportunities for young people. Category 7 includes programs to provide job opportunities outside the country. Interventions that provide multiple types of services, and thus cannot be included in one of the other groups, are included in Category 8. Finally, Category 9 is a residual grouping. Examples of programs under each main category are included throughout this paper. Table 2: Categories used to Classify Programs in the YEI 1. Making the labor market work better for young people 1a. counseling, job search skills 1b. wage subsidies 1c. public works programs 1d. anti-discrimination legislation 1e. other 2. Improving chances for young entrepreneurs 3. Skills training for young people 3a. vocational training including apprenticeship systems 3b. literacy & numeracy ­ young adult literacy programs 3c. 2nd chance & equivalency programs 3d. other 4. Making training systems work better for young people 4a. information 4b. credit (to individuals or enterprises) 4c. financial incentives (subsidies, vouchers) 4d. other 5. Programs to counteract residential segregation of disadvantaged young people 5a. transportation 5b. others 6. Improving labor market regulations to the benefit of young people 7. Programs for overseas employment of young people 8. Comprehensive approach 9. Other (e.g. voluntary national service programs) 10 41412 2. The Inventory ­ how it was Compiled The inventory provides a wealth of information on each intervention and, as noted above, eligible interventions were not confined to success stories. The research team identified programs and gathered documentation from the range of sources described at the beginning of this section. With a view to maximizing synergies with other related initiatives, the compilation of the inventory was carried out in cooperation with other activities of the Youth Employment Network as well as youth employment initiatives at the World Bank. The screening and documentation process was based on a standardized screening and data-collection methodology developed by the research team. Since different researchers were undertaking the primary research, which included determining program eligibility, reviewing documentation, entering information into the inventory database, and assessing the quality of the intervention, it was essential that a standardized methodology was followed. Criteria for inclusion. A major methodological issue concerned the determination of what kinds of interventions would be included in the YEI. The question of how far back into the education system the inventory should go has already been discussed. Two other considerations relevant to defining scope were (i) whether the inventory should be restricted to programmatic interventions or also include policies like labor market regulations and minimum wages that affect labor market outcomes for young people; and (ii) whether it should include interventions that, while not targeted at youth specifically, could have a big impact on young people. With respect to policy, the determination was to base eligibility on the stated purpose and to include only those policy interventions that specifically targeted young people (e.g., a special youth minimum wage or contracting rules that only applied to young people). As we will see in the next section, though, there were very few policies included in the inventory; almost all interventions covered are programs. As for programs, they were eligible for inclusion even if they did not explicitly target youth if the documentation indicated that young people were the primary participants. As we will see in the next section, about 20 per cent of the programs included in the inventory did not have age restrictions. Also, both completed and ongoing interventions were eligible for inclusion. Further restrictions were imposed based on the quality of the information. Ideally, given the inventory's objective of providing information on what works, sound impact evaluations should have been a condition of inclusion. However, most interventions simply 11 41412 do not meet this condition, especially in developing economies, so imposing this restriction would have excluded the majority of the interventions identified. This would have severely limited the project's value in documenting what has been tried to support young workers, which was one of the objectives of the study. Nonetheless, a minimum amount of information was required for inclusion -- sound information on the intervention's objectives, implementation design, and targeting criteria. Also, the data collection placed priority on including evaluated interventions ­ i.e., those with net impact evaluations and cost-benefit analysis. To some extent, then, interventions with evaluations are overrepresented in the inventory. Template. A questionnaire template was designed to ensure consistency and uniformity in the collection and recording of information for the inventory. The template and the coding system used are shown in Annex A. Information collected on each program includes intervention category (as described in Table 2), country, time period in which it was implemented, current status, the specific labor market problems it sought to address, main objectives, a detailed description of the program (scale, financing, etc.), as well as several performance indicators to understand the program's impact, summary measures on the quality of the evaluation evidence and the quality of the intervention (described below), and sources for further information on the intervention. To allow for quantitative analysis of the data, variables included in the template were coded on the basis of multiple choice measures wherever feasible. The template and coding system are shown in Annex A. Inventory database. In the project design stage, a decision was made to use an electronic format for the database in order to facilitate search capabilities, updating, and quantitative analysis.10 The template was built into an Excel worksheet and an independent machine-readable file was created for each intervention included in the inventory. After the data-collection phase ended, an Excel macro was designed on Microsoft Visual Basic to read every file and construct a searchable database where the number of observations (rows) matched the number of interventions (files or worksheets). Data collected in the questionnaire ­ both plain text and codes -- are displayed in the columns, creating a database of program-specific information (Database 1), which includes, for each intervention, all the 10In fact, the inventory was conceived as a "live database" that could be regularly updated. This was another reason for investing in the creation of an electronic format. 12 41412 information shown in the template and coding system (summarized in the previous paragraph and shown in detail in Annex A). Simultaneously, a database of country-specific information (Database 2) was created to contextualize the economic conditions of the country. This information includes level of development, level of income, and a characterization of the labor market regulatory/institutional situation. Sources of information for the country database are the World Development Indicators and the Doing Business Database (2006). The Excel macro links databases 1 and 2 through a common key-variable, namely country name, creating a comprehensive database for the analysis of the inventory.. For details on how to create databases and informative tables from the inventory, see Annex B. Quality of intervention and quality of evaluation variables. Two critical variables in the inventory database are the "quality of intervention" (QOI) and "quality of evaluation" (QOE) (template, sections I and J, respectively). These figure prominently in the analysis of what we have learned from the inventory in terms of what works for supporting young workers. Both QOI and QOE values for each intervention have been determined by the research team according to standardized criteria described below. The "quality of intervention" is the measure of program effectiveness. The possible values for QOI are described in Table 3. The primary performance indicators that are considered in establishing a QOI rating are the effects of the program on the employment and earnings of participants. At one level, the QOI value can be used to identify impact ­ i.e., to distinguish those programs that actually help participants in the labor market (QOI=1,2, or 3) from those that appear to have no effect, or even a negative effect (QOI=0). A rating of 1 or 2 means that a program is judged to have had a positive impact, but this does not necessarily mean that it was successful. To be specific, interventions can have a positive employment impact but not be cost-effective (i.e., QOI=1).11 These programs cannot be considered successful. 11A program is considered cost-effective if the evaluation results indicate that the benefits (e.g., reduced use of social assistance, increased tax gains through participants who found a job, increased earnings, etc.) exceed program costs (income support, training material, cost of training, etc.). Since we are relying on available project documentation, specific methodologies used for the cost-benefit analysis can vary. 13 41412 Table 3: Measuring the Quality of Intervention (QOI) QOI value Description 0 Program had negative or zero impact on labor market outcomes. 1 Program had positive impact on labor market outcomes but is not cost- effective. 2 Program had positive impact on labor market outcomes and there is no evidence on costs. 3 Program had positive impact on labor market outcomes and is cost- effective. 99 Missing value. Not enough evidence to make an assessment. Determining a value for the quality of the intervention is complicated by the fact that the evidence on which to base the assessment varies widely. In some cases, solid evaluation results are available while in others, only basic descriptive information exists. The "quality of evaluation" variable is important for identifying the evaluative basis for assessing program quality. The QOE measure is described in Table 4. With this variable, then, assessments of the effectiveness of interventions can be judged with knowledge of the quality of the underlying evidence. For example, one could consider only those programs that meet the most exacting burden of proof (i.e., QOE=3), with the tradeoff that sample size will be vastly reduced. On the other hand, accepting a less demanding basis of evidence will increase the pool of programs under consideration, but at the expense of rigor. Table 4: Measuring the Quality of Evaluation (QOE) QOE value Description 0 Program has no evaluation information available on outcomes or impact. Evaluation includes basic information on the gross outcomes of the intervention 1 (e.g. number of participants/ young people who found a job after the intervention, improvement in earnings of participants) without considering net effects (i.e., there is no control group). Evaluation includes estimate of net impact on, e.g., employment and earnings in 2 the labor market (using control groups to measure impact) but no cost-benefit analysis. 3 Evaluation includes net impact plus cost-benefit analysis Table 5 identifies the possible choices for QOI, given QOE. Where cells are empty, the QOI-QOE combination is possible. However, there are three types of cases (identified by letters A,B, and C) where a particular QOI value cannot be assigned based on the available evaluation evidence: (A) Where there is no evaluation information whatsoever (QOE=0), impact must be unknown (QOI=99). (B) Where a net impact evaluation exists (QOE=2 or 3), 14 41412 the impact cannot be assessed as unknown (QOE cannot equal 99). (C) Where there is a net impact evaluation but no evidence on costs (QOE=1 or 2), the impact rating cannot indicate whether program is cost-effective or not (QOI cannot equal 1 or 3). Table 5: Possible Choices for Quality of Intervention Given Quality of Evaluation Quality of Intervention 0 1 2 3 99 Quality of Evaluation Negative Positive Positive impact Positive or zero impact but but unknown impact and Unknown impact cost- cost- cost- impact ineffective effectiveness effective 0 No evaluation information A 1 Basic information without net effects C C 2 Net impact evaluation 3 Net impact and cost-benefit B analyses Areas marking out impossible combinations: A ­ No evaluation information; therefore no assessment of QOI; B ­ Net impact evaluation; therefore some assessment can be made of impact C ­ Information on outcomes or impacts but no cost information; therefore, assessment can be made of impact but cost-effectiveness must be considered unknown. The most difficult situation to address in assigning a value for the quality of intervention arises where QOE=1. Where no evidence exists, we have already noted that the QOI score is 99 (unknown impact), by default. And where there is a net impact evaluation (QOE=2 or 3), it is generally possible to assess impact, although not always with cost- effectiveness. However, when QOE=1, there is some performance information on the program, but only in terms of gross outcomes. With no rigorous assessment of net impacts, one option would have been to assign all of these programs with a missing QOI value. However, 35 per cent of the cases in the inventory have only gross outcomes and this strategy would have seriously diminished the sample for addressing the question of what works. So, in order to capture information on effectiveness for programs where QOE=1, the research team used the following indicators, where available: Before and after measures of employment variables; Post-program comparisons of labor market outcomes for participants relative to others in the same sector; How well the program met explicit goals and targets in terms of job placement, activity rates, earnings, or enrollment rates in secondary school/college after the program; Whether program reached the objective population; and 15 41412 Qualitative results from interviews to participants and employers. In the absence of cost-benefit analysis, interventions with QOE=1 generally are assessed a QOI rating of 0, 2 or 99. Thus, a standardized methodology was followed to address the question of how to evaluate the quality of an intervention with only limited evidence on performance, and taking into account the tendency towards bias in self- reporting.12 However, some readers may still question the QOI ratings for programs without net impact evaluations and, as a result, when the evidence on the quality of interventions is presented, the quality of the underlying evaluation evidence can be taken into account. D. Coverage of the Inventory ­ What Interventions have been Implemented? Table 6 shows the coverage of the inventory in terms of the number of interventions in each category, by region. In addition to describing the sample for the subsequent analysis of the inventory, Table 6 can also be seen as a portrayal of the types of interventions that have been implemented globally and by region to support the entry of young people into the labor market. While programs can have more than one purpose and offer more than one type of service, we have tried wherever possible to identify the primary nature of the intervention and classify it accordingly (template, section A). Where this has really not been possible, the program has been grouped under Category 8. The regions included in Table 6 conform to the standard World Bank categories, plus the industrialized-country members of the OECD.13 Table 6 includes all categories of interventions included in the classification system. However, no programs were found under the headings 1d (anti-discrimination legislation), 3b (young adult literacy programs), 5 (counteracting the isolation of young people), and 7 (programs to promote overseas employment of young people). These headings are excluded from the remaining tables in this paper, but should be retained in the framework for what is intended to be a regularly updated inventory. Of all the interventions included in the YEI, 38 per cent are known to have been completed, 42 per cent are ongoing (of which more than half are judged to be self- sustainable) and the status of 20 per cent is unknown. 12 Moreover, to assure that the researchers applied uniform standards to assigning QOI and QOE values in this situation (as well as in general), the team discussed cases where ratings were not obvious and a sub-sample of programs were rated independently by all researchers. 13 Where OECD member countries are also in World Bank regions, they are classified here according to the World Bank grouping. 16 41412 Table 6: Coverage of Inventory by Category of Intervention and Region Category of intervention Europe Latin Middle OECD South Sub- Total & America & East & & East Saharan Central Caribbean North Asia & Africa Asia Africa Pacific 1. Making the labor market work better for young people 1a. counseling, job search skills 2 1 3 1 6 1b. wage subsidies 8 9 17 1c. public works programs 3 1 3 1 8 1d. anti-discrimination 0 legislation 1e. other 2 2 4 Sub-total 13 3 1 17 0 2 35 2. Improving chances for 3 5 1 11 6 7 33 young entrepreneurs 3. Skills training for young people 3a. vocational training including 13 36 2 33 8 6 98 apprenticeship systems 3b. literacy & numeracy ­ young 0 adult literacy programs 3c. 2nd chance & equivalency 3 1 3 1 8 programs 3d. other 2 1 2 5 Sub-total 18 38 2 38 9 6 111 4. Making training systems work better for young people 4a. information 1 2 3 4b. credit (to individuals or 1 1 enterprises) 4c. financial incentives 2 1 1 4 (subsidies, vouchers) 4d. other 2 1 3 Sub-total 0 0 0 6 1 4 11 5. Programs to counteract isolation of disadvantaged young people 5a. transportation 0 5b. others 0 Sub-total 0 0 0 0 0 0 0 6. Improving labor market 1 1 2 regulations to the benefit of young people 7. Programs to promote 0 overseas employment of young people 8. Comprehensive, multiple- 6 22 4 47 4 9 94 service approach 9. Other (e.g. voluntary 1 2 1 3 national service programs) Unclassified 0 Total 41 68 8 122 21 29 289 17 41412 Types of interventions. Overall, 289 interventions are included in the inventory.14 By main category, the most popular intervention is skills training for young people. This category accounts for 39 per cent of all interventions and is significant in all regions, but is especially popular in LAC where it represents 56 per cent of the programs included. Comprehensive multiple-service interventions -- for instance, combining vocational and on- the-job training with wage subsidies and public works, or classroom and on-the-job training with paid work experience and job search assistance ­ account for 32 per cent of the total. One-half of these multiple-service programs are in OECD countries. Making the labor market work better for young people (especially through wage subsidies), and improving chances for young entrepreneurs each accounts for 12 per cent of the total. All of the other intervention categories are very small in number. Interventions by region. The OECD area and LAC account for the largest shares of interventions included in the inventory. Of the 289 interventions, 122, or 42 per cent, are in OECD countries while 68 (24 per cent) are in LAC. The shares in the other regions are 14 per cent in ECA, 10 per cent in Sub-Saharan Africa, 7 per cent in South and East Asia and the Pacific (SEAP), and 3 per cent in the Middle East and North Africa (MENA). Closer analysis reveals interesting patterns of programming within the OECD and LAC regions, the two major contributing regions to the inventory. While these patterns are summarized here, more detail is available in the regional reports. Within the OECD group, a distinction can be made between Anglo-Saxon countries and the rest of the OECD, primarily countries in Continental Europe. In general, Anglo- Saxon economies are less interventionist in the labor market and use social policy, including active labor market programs, in a less activist way than the group of continental European countries.15 Although continental Europe in fact comprises several social systems (the Nordics, Mediterranean, and continental Europe itself, as described in Boeri (2002)), generally there is a strong reliance on social insurance instruments (pensions, health and unemployment insurance), unions are relatively involved in the labor market, and there is a 14The actual number of programs included in the inventory is slightly less because some programs have been evaluated more than once. 15The average (non-weighted) ALMP spending in Anglo-Saxon economies in 2004 was 0.41 per cent of GDP, compared to 0.86 per cent in continental European economies and others (Japan and Korea). 18 41412 significant investment in active labor market programs to support unemployed or otherwise vulnerable workers. Not surprisingly, given these institutional differences, these two groups of OECD countries use different types of interventions to support young workers. With a large sample of programs in the region to draw on (79 interventions in the Anglo-Saxon countries and 43 in the continental European and other countries), it is possible to identify these differences.16 These are summarized in Figure 1. In the Anglo-Saxon countries, there is a heavy reliance on comprehensive approaches, with this category accounting for 44 per cent of the total. While these types of programs are used in continental Europe, they are less important, at least quantitatively (28 per cent). The other major difference involves the group of interventions to counteract labor market failure, through counseling, job search assistance, temporary employment subsidies, and public works. In continental Europe, this group of ALMPs accounts for 23 per cent of all interventions, but just 9 per cent in the Anglo-Saxon countries. In both sub-regions, skills training programs account for roughly one-third of all interventions. Figure 1: Categories of Interventions by OECD Country sub-groups 1. Making the labor market 9% work better for young people 23% 2. Improving chances for 10% young entrepreneurs 7% 3. Skills training for young 30% people 33% 4. Making training systems 4% work better for young people 7% 6. Improving labor market 0% regulations 2% 44% 8. Comprehensive approach 28% 9. Other 3% 0% Anglo-Saxon countries Continental Europe and other OECDs 16 The Anglo-Saxon countries with interventions included in the inventory are Canada, the United States, Ireland, the United Kingdom, Australia, and New Zealand. The group of continental Europe and other countries consists of Austria, Belgium, Denmark, France, Germany, Italy, the Netherlands, Norway, Portugal, Finland, Spain, Sweden, and Japan. Since the single non-continental European country, Japan, has only four interventions, we can basically refer to this group as continental European. 19 41412 What is interesting about the LAC region is how the approaches to supporting young workers have changed over time. Three dominant models have been used in the region in the past few decades. First, a state-managed training model prevailed during the 1970s. This traditional supply-driven model offered specialized training and retraining to workers through centralized public providers. These training institutions tended to be financed by payroll taxes. The emphasis on this model was reduced in the 1980s as part of a broader realignment of economic policy toward market-driven principles (de Moura Castro et al., 1998). Nonetheless, some public institutions have survived and continue to provide vocational training services.17 The second dominant LAC model emerged in the early 1990s with the Jóvenes Programs (see Box 6 for details). This is a demand-driven model that targets economically disadvantaged youth, fosters private-sector participation, and promotes competition among training providers. The model was first applied in Chile and soon after replicated in Argentina, Uruguay, Paraguay, Peru, Colombia, Dominican Republic, and Venezuela. The programs are financed and coordinated by the government. Training has a comprehensive scope ­ from technical to life skills and from lectures to internships ­ accompanied by sound support services and financial incentives. The Jóvenes model has been successful in improving job placement and earnings, but became particularly expensive for some countries where it has been replaced by smaller and more focused interventions. The third and most recent model inherits the demand-driven orientation of the Jóvenes. It is a vocational training approach with on-the-job training and placement services. The Entra 21 Program (see Box 3 for details) is the most characteristic example of this model. This program started in 2002, and aims to provide business with skilled information- and-communication-technology workers while improving the employability of disadvantaged youths (age 16-29). 17This inventory includes an impact evaluation for SENA (Servicio Nacional de Aprendizaje), the largest public-training institution in Colombia. 20 41412 Box 1: ECA Wage Subsidies for Young People ­ an Example of Programs Designed to Make the Labor Market Work Better for Young People (Category 1) Programs designed to make the labor market work better for young people are relatively important in ECA. Several countries in the region have operated wage subsidy programs targeted at young people. In Poland, the Intervention Works Program was initiated in 1995 for people up to the age of 30. The program is estimated to have increased reemployment by 15.6 per cent in non-subsidized jobs, and by 13.1 per cent in any kind of job (including subsidized) but with lower monthly earnings. Based on a scientific evaluation with cost analysis (QOE=3), this program was assessed to generate positive employment impacts in a cost- effective manner (QOI=3). In the Czech Republic, a wage subsidy program has been in operation since 1996, for the benefit of young people. It achieved a statistically significant increase in employment of 12 per cent for participants (QOI=2; QOE=3). Women and less-educated participants (a considerable proportion of all participants) gained most from the program. Again, however, monthly earnings were lower than pre-program levels. In Bulgaria, a program for subsidized Employment in Public Administration of Young School Leavers has been in place since 2002. At the end of July 2004, 909 people were employed through the program, out of 1,090 young people so far included in the program. Monitoring and assessment of the program's implementation are carried out on a regular basis. This provides the possibility of correcting the scope and mechanisms of the program and adapting it to suit the conditions and needs of the labor market (QOE = 1, QOI = 99). In Slovakia, employers can receive a monthly contribution from government to cover the costs of employing unemployed school leavers (who also receive a grant to cover personal expenses) in a `graduate practice' scheme. In 2004, 14,462 job seekers participated in the scheme, of whom 68 per cent were women and 83 per cent were from disadvantaged groups. No analysis of the net impact of the intervention is available (QOE = 1, QOI = 2). In Latvia, a pilot project of subsidized work experience during the summer holidays for students from secondary and secondary vocational schools and vocational training students was organized in 2004. The pilot project can be evaluated as successful, but there is also room for organizational improvements, with better targeting ­ for instance to students from large families. Contracts were signed with 448 employers (enterprises and organizations) nationwide and 3,191 subsidized jobs were offered. The employers who offered the majority of the jobs were retailers, food factories and farm enterprises, while some positions were also offered by a children's hospital. Some employers were highly satisfied with the employed students and asked them to continue the cooperation after the pilot project was finished (QOE = 1, QOI = 2). In Kyrgyzstan a Youth Job Vouchers scheme has operated since 1996. A survey found both employers and young people to be highly satisfied with the scheme, although employers claim that it would be possible to create even more jobs if they only had to commit themselves to the young people for one year. Vouchers opened the door to a career start for 180 young women and 80 young men; 75 per cent of the jobs were assessed to be genuine new jobs (QOE = 1, QOI = 2). Sources: O'Leary (1998), Fretwell et al. (1999), EU (2005 and 2006) and BMZ (2006). Program targeting. As described in the methodology section, interventions were included in the inventory if they targeted youth or if young people were the principal beneficiaries. Table 7 shows that most of the interventions are aimed exclusively at young people, but 59 of the 289 total (20 per cent) were open to people of all ages. Over 80 per cent of programs included in the entrepreneurship, training, and multiple-service categories were youth-only interventions. 21 41412 Table 7: Coverage of Inventory by Category of Intervention, Location, and Age Group Served Category of Urban Rural Both Unknown Total intervention Only All Only All Only All Only All Only All Total young ages young ages young ages young ages young ages people people people people people 1. Making the labor market work better for young people 1a. counseling, job search 2 3 1 3 3 6 skills 1b. wage 1 11 5 12 5 17 subsidies 1c. public works programs 1 2 5 3 5 8 1e. other 1 2 1 3 1 4 Sub-total 2 2 1 0 18 12 0 0 21 14 35 2. Improving 5 7 1 16 4 28 5 33 chances for young entrepreneurs 3. Skills training for young people 3a. vocational 38 2 7 1 36 14 1 81 17 98 training including apprenticeship systems 3c. 2nd chance & 5 2 1 8 0 8 equivalency programs 3d. other 1 1 3 4 1 5 Sub-total 44 3 7 1 41 14 2 0 93 18 111 4. Making training systems work better for young people 4a. information 2 1 2 1 3 4b. credit (to individuals or 1 0 1 1 enterprises) 4c. financial incentives 1 1 2 2 2 4 (subsidies, vouchers) 4d. other 1 2 3 0 3 Sub-total 1 2 0 1 6 1 0 0 7 4 11 6. Improving labor market regulations to the 1 1 1 1 2 benefit of young people 8. 17 3 8 3 53 10 2 1 78 16 94 Comprehensive, multiple-service approach 9. Other (e.g. 1 1 1 2 1 3 voluntary national service programs) Total 70 10 23 6 136 43 4 1 230 59 289 22 41412 The inventory also distinguishes between programs by their location (Table 7). Few interventions (only 10 per cent of the total) are confined to rural areas. Somewhat more are targeted at urban areas (28 per cent). However, the majority (62 per cent) operate in both urban and rural areas. How far are interventions oriented towards disadvantaged young people? In order to answer this question, the inventory collected data on whether the program was oriented to a particular gender, to the disabled, to specific ethnic groups, and to certain income and education levels. The results are presented in Table 8. The only characteristics that actually are frequently targeted are income and education. Just over half (51 per cent) of all programs in the inventory are oriented towards young people with low incomes, or in low-income families; when we exclude programs where information on income targeting was not included in the documentation, this figure rises to 62 per cent. Training programs (category 3) and multi-service programs (category 8) are especially likely to be targeted at people with low incomes. Regarding education, 49 per cent of all programs were targeted at youth with low educational attainment (53 per cent if we exclude programs where we do not have the information). Here, also, multi-service programs are most likely to have education-related targeting. It is notable that 9 programs we have identified are aimed at the better-educated. A number of these are in OECD countries (i.e., Australia, Canada, Germany and Japan). 23 41412 Table 8: Orientation towards Disadvantaged Groups by Broad Category of Intervention Intervention category Gender 1 2 3 4 6 8 9 Total Women 7 6 17 6 9 0 45 Neutral 25 26 91 4 2 79 3 230 Men 3 2 5 Not known 1 1 1 6 9 Total 35 33 111 11 2 94 3 289 Disability 1 2 3 4 6 8 9 Total Disabled 5 3 4 1 18 1 32 Neutral 21 18 82 8 1 57 2 189 Non-disabled 1 1 1 3 Not known 8 12 24 2 1 18 65 Total 35 33 111 11 2 94 3 289 Ethnicity 1 2 3 4 6 8 9 Total Particular group(s) 3 2 9 5 1 20 Neutral 22 21 79 8 1 72 2 205 Negative 1 1 Not known 9 10 23 3 1 17 63 Total 35 33 111 11 2 94 3 289 Income 1 2 3 4 6 8 9 Total Low-income 12 14 64 3 51 3 147 Neutral 12 14 28 6 1 29 90 Non-poor 0 Not known 11 5 19 2 1 14 52 Total 35 33 111 11 2 94 3 289 Education 1 2 3 4 6 8 9 Total Low-education 18 10 50 4 1 56 3 142 Neutral 11 17 50 4 25 107 Non-low-education 1 1 4 1 2 9 Not known 5 5 7 2 1 11 31 Total 35 33 111 11 2 94 3 289 The incidence of gender targeting is relatively low. Only 16 per cent of all programs are oriented towards young women; 2 per cent are targeted explicitly at young men. The inventory includes 32 programs targeted at disabled young people, which represents 11 per cent of the total. The majority of these (18 programs) provide multiple services. Finally, we have found only a small number of interventions (20 in total, with 9 of these providing training) targeted at particular ethnic groups. 24 41412 Box 2: The Commonwealth Youth Credit Initiative -- an Example of Programs Designed to Improve Chances for Young Entrepreneurs (Category 2) The Commonwealth Youth Credit Initiative (CYCI) in India is a small enterprise scheme for unemployed young people involving "micro-credit" (small-scale lending), training, and enterprise development. The scheme aims to create employment opportunities by providing low-cost, easily accessible credit to establish successful businesses and training in financial and enterprise management. The program also increases the knowledge of young people and youth-related organizations in the operation and management of credit programs. Services for youth are provided through low interest rates, low training costs, partnership with NGOs, and ongoing training and monitoring of enterprises. The focus is on developing capacity for enterprise management, a pre-requisite for the self-employed. The program has three stages: Stage I: Pre Credit (community outreach support system and identification and selection of youth). Stage II: Training for Capacity Building (capacity building, group formation, basic credit management, and entrepreneurship training). Stage III: Credit Delivery & Support (credit dispersion, credit management system, post-training support for growth and expansion, and re-lending). The CYCI was designed by the Commonwealth Secretariat, an intergovernmental agency of the British Commonwealth. CYCI completed a three-year pilot cycle at Ahmedabad, Gujarat, India in 1999, which was conducted in collaboration with the International Centre for Entrepreneurship and Career Development (ICECD). It became self-sustaining after three years, with operational self-sufficiency of 98 per cent. After the three­year pilot program, 82 per cent of participants were successfully operating micro-enterprises on a self-sustainable basis. Female participation reached over 75 per cent of the assisted population. Over 2,500 young people in India were trained and provided with small loans. Similar schemes have been transferred to other Commonwealth member states in Africa, South Asia, and the Caribbean. This intervention was rated as QOI=2, based on QOE=1. Sources: www.thecommonwealth.org and www.icecd.org Financing. The interventions included in the inventory were financed from a variety of sources, including government (different levels), beneficiaries, employers, non- governmental organizations, and "other sources". This latter category includes two sub- categories: international organizations and other donor agencies or combinations of different types of funding sources, where a primary funder is not evident (e.g., donor agencies and government; government and employers; donor agencies and NGOs, etc.). The classification of the intervention categories by funding source is summarized in Table 9. The majority of interventions (56 per cent) were solely government-funded. Another third were classified under the "other" category. All of the other financing sources were infrequently cited. Somewhat different funding arrangements seem to characterize different types of programs. The ALMP interventions that are categorized under category 1 (making 25 41412 labor markets work better for youth) are typically funded by government (27 of 33 in the inventory). This is also the case for interventions to make skills training work better for young people and for multi-service interventions, where 8 of the 11 programs and 66 of 94 programs, respectively, are government-financed. On the other hand, self-employment assistance and skills training programs are much more often funded in ways other than straight government financing. Table 9: Coverage of Inventory by Category of Intervention and Source of Finance Category of intervention Source of finance Government Beneficiaries Employers NGOs Other NA Total 1. Making the labor market work better for young people 1a. counseling, job search skills 4 2 6 1b. wage subsidies 14 1 2 17 1c. public works programs 7 1 8 1e. other 2 1 1 4 Sub-total 27 0 1 1 6 0 35 2. Improving chances for 15 1 5 7 5 33 young entrepreneurs 3. Skills training for young people 3a. vocational training 40 5 1 48 4 98 including apprenticeship systems 3c. 2nd chance & equivalency 4 4 8 programs 3d. other 2 3 5 Sub-total 46 0 5 1 55 4 111 4. Making training systems work better for young people 4a. information 2 1 3 4b. credit (to individuals or 1 1 enterprises) 4c. financial incentives 3 1 4 (subsidies, vouchers) 4d. other 2 1 3 Sub-total 8 0 1 0 2 0 11 6. Improving labor market regulations to the benefit of 1 1 2 young people 7. Programs to promote overseas employment of young people 0 8. Comprehensive, multiple- 66 4 1 21 2 94 service approach 9. Other (e.g. voluntary 3 3 national service programs) Total 162 1 12 8 94 12 289 26 41412 Box 3: Entra 21 ­ An example of programs designed to provide skills training for young people (Category 3) Entra 21 is an initiative developed by the International Youth Foundation to prepare LAC youth, 16 to 29 years of age, for today's information-based economy. It has been widely implemented by local and central governments, NGOs, and local businesses to improve the employability of disadvantaged youths. The program started in 2002 with the goal of providing skills training in information and communication technology to 12,000 young workers in a 3-year period and to place at least 40 per cent of them in employment. Entra 21 programs are co-financed by the Multilateral Investment Fund of the Inter-American Development Bank. Other important partners are Microsoft Corporation, Lucent Technologies Foundation, Merrill Lynch, and USAID. Grants have been awarded in 18 countries, namely Argentina, Belize, Bolivia, Brazil, Chile, Colombia, Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Mexico, Nicaragua, Panama, Paraguay, Peru, Uruguay, and Venezuela. Entra 21 programs support youth through well-designed and coordinated lectures and internships. They offer life-skills training and continuous tutoring; these are central features of the intervention and key determinants of its success. There is also a financial scheme to provide an incentive for youth to register in the program. Programs last two years on average, and target mainly unemployed and underemployed disadvantaged young people who have completed high school (or are in the process of doing so). Gender is equally represented, as well as some minority groups (indigenous youths are particularly targeted by Entra 21 programs in Guatemala and Bolivia). There have been no net impact evaluations of Entra 21 programs but studies in El Salvador, Dominican Republic, Peru, Panama, Colombia, Paraguay, Bolivia, and Brazil have shown positive "gross" impact on employability of participants. Estimated job placement rates have ranged from 68 per cent in Peru to 41 per cent in Paraguay, with high satisfaction levels of employers and beneficiaries. Placement rates have been lower for women, especially in Panama, where 34 per cent of female participants got a job, compared to 64 per cent of male participants. On the other hand, in Sao Paulo, Brazil, both genders obtained the same placement rate. Regarding earnings effects, evaluations found that average monthly wages were at least as high as the minimum wage in Peru, Bolivia, Dominican Republic, Panama, Paraguay and Brazil. Most youth attained a job in the formal sector with at least one or more benefit, such as paid vacations, one month bonus and health insurance. Entra 21 programs are all given a rating of 2 for QOI based on QOE=1. Source: Pezzullo (2005) E. Quality of Evaluations of Programs in the Inventory Since an assessment of "what works" is one of the core objectives of the overall project, an important dimension of the inventory concerns the quality of the evidence that is available on the impact and cost-effectiveness of the interventions. In Section C, we introduced the "quality of evaluation" (QOE) variable that measures this for a given program. The classification defining this variable ranges from no information on outcomes or impact to estimates of net impact with cost-benefit analysis (recall Table 4). 27 41412 The quality of the evaluation evidence for the interventions included in the inventory is summarized in Figure 2. The general picture is that the level of program evaluation has been weak. Certainly, one strong conclusion drawn from our research is the need for major improvements in the quality of evidence available for youth employment interventions. In 39 per cent of all programs, there is no evaluation information at all on outcomes or impact. An additional 35 per cent have evaluations which cover only gross outcomes, and do not use a methodology (e.g., based on a control group) to estimate net impact. In other words, only about one-quarter of all programs included have some evidence on the net impact. And, of the programs that meet this evaluation standard, most (45 of 73) do not include any cost- benefit analysis. As Figure 2 indicates, then, only one in 10 programs have evaluations which measure both net impact and cost. Moreover, given the nature of the data collection process followed to compile the inventory, it is likely that Figure 2 overstates the actual quality of evaluations for youth programs.18 Figure 2: Summary of Quality of Evaluations (QOE) 9.7% No information on 15.6% outcomes (QOE=0) 39.4% Gross outcomes only (QOE=1) Net impacts, no cost analysis (QOE=2) Net impacts with cost analysis (QOE=3) 35.3% 18The reason for this is two-fold. First, it seems probable that programs with documentation (found on the internet or through other sources) and, thus, eligible for inclusion in the inventory would be more likely to have impact evaluations than "unobserved" programs (i.e., without any documentation on the internet or through other sources). Second, the data collection exercise did place some priority on including programs with solid evaluation evidence; for example, where researchers knew that an evaluation existed, they were more likely to search intensively for the documentation required to have the program included than where they knew or suspected no impact evaluation had been undertaken. 28 41412 The overall picture on the quality of evaluation evidence varies by type of program. Figure 3 shows the percentage of interventions by major category that have net impact evaluations (QOE=2 or 3) and have these evaluations with an analysis of costs (QOE=3). We have only included the four largest categories because the others have too few cases to draw valid conclusions. The incidence of net impact evaluations varies from 36 per cent for multiple-service programs down to only 9 per cent for entrepreneurship programs. These results are not surprising. The comprehensive programs tend to be large, highly visible interventions where resources available and interest in measuring results is likely to be high. On the other hand, entrepreneurship programs are often smaller and are not as easily evaluated as some of the other interventions included in the inventory. The relatively low incidence of net impact evaluations for skills training (20 per cent) is disappointing, especially given the prevalence of these programs. Finally, net impact evaluations with cost analysis are infrequent for all categories, especially entrepreneurship (only one of 33 programs) and skills training (8 of 111 programs). The complete distribution of the quality of evaluations by type of intervention is presented in Table 10 Figure 3: Percentage of Programs with Net Impact Evaluations by Category of Intervention 25% Total 10% 1. Making the labor market 31% work better for young people 17% 2. Improving chances for 9% young entrepreneurs 3% 3. Skills training for young 20% people 7% 8. Comprehensive, multiple- 36% service approach 13% Net impact (QOE=2 & 3) Net impact with cost analysis (QOE=3) 29 41412 Table 10: Quality of Evaluation (QOE)1 Evidence by Category of Intervention Category of intervention Quality of Evaluation (QOE) 0 1 2 3 Total 1. Making the labor market work better for young people 1a. counseling, job search skills 1 4 1 6 1b. wage subsidies 3 9 3 2 17 1c. public works programs 1 3 1 3 8 1e. other 2 1 1 4 Sub-total 7 17 5 6 35 2. Improving chances for young entrepreneurs 18 12 2 1 33 3. Skills training for young people 3a. vocational training including apprenticeship systems 46 34 11 7 98 3c. 2nd chance & equivalency programs 3 2 2 1 8 3d. other 3 1 1 5 Sub-total 52 37 14 8 111 4. Making training systems work better for young people 4a. information 3 3 4b. credit (to individuals or enterprises) 1 1 4c. financial incentives (subsidies, vouchers) 2 1 1 4 4d. other 2 1 3 Sub-total 7 1 2 1 11 6. Improving labor market regulations to the benefit of young people 1 1 2 8. Comprehensive, multiple-service approach 29 31 22 12 94 9. Other (e.g. voluntary national service programs) 3 3 Total 114 102 45 28 289 Notes: 1. for QOE specification, see Table 4. Box 4: Kenya's Jua Kali voucher program ­ An example of programs designed to make training systems work better for young people (Category 4) One of the best known programs under this heading is Kenya's Jua Kali voucher program, established in 1997 as a pilot program, under the auspices of the Micro and Small Enterprise Training and Technology Project. Under this type of program, vouchers are issued to unemployed youth, who can personally select a training provider based on their needs and objectives, rather than having them chosen by a bureaucratic institution. Vouchers for training have been used for some time in the UK and more recently in Germany as well as other countries. The voucher program intends to empower recipients with the capacity to buy training on the open market and thereby promote competition between private and public suppliers. The approach should improve the quality of training and bring down the costs, while at the same time ensuring a better match between the participant and the training course. Under the Jua Kali pilot program, anyone eligible for training is given a voucher which can be cashed in at the chosen training provider. Participants pay only 10 per cent of the cost of the voucher with the government subsidizing the remaining 90 per cent. Master craftsmen were the major providers of training, responding to demand from clients. Although the Jua Kali voucher scheme did not focus entirely on youth, the majority of those trained were young and disadvantaged. Under this program, 37,606 vouchers were issued to entrepreneurs and employees in enterprises with fifty workers or less over the 1997-2001 period. There is evidence that the scheme has had a positive impact on those who were trained and that it has boosted employment, assets, and business for enterprises which participated (in comparison with a control group). These findings relate to a small population served by the pilot program; there is no evidence of outcomes/impact in a large (national) sample. The scheme was complex and costly to establish, and it has proven to be difficult to phase out the subsidization of the vouchers. Lessons learned from the experience include the following: such schemes should be administered through the private sector rather than (as in Kenya) through a government ministry; the scheme should include provision for upgrading of training providers, especially those from small enterprises; and it should promote the willingness of clients to pay for training. An exit strategy is needed unless subsidies are to last forever. But, overall, the Jua Kali experience suggests that there is scope for the use of vouchers in a system more precisely targeted at the most vulnerable. This program was given a rating of QOI=2 based on a QOE=3. Source: Johnson and Adams (2004) 30 41412 Table 11 shows the distribution of evaluation quality by region. The highest incidence of impact evaluations is in the OECD, where 34.4 per cent had a QOE rating of 2 or 3. ECA (29.3 per cent) and LAC (25.0 per cent) were next, with programs in MENA, Asia, and SSA never or only very rarely being evaluated. When we consider only net impact evaluations with cost analysis (QOE=3), ECA surprisingly has the highest incidence. This largely reflects a major evaluation effort by the World Bank on a set of programs that accounts for a significant share of the relatively small ECA sample. Table 11: Percentage Distribution of Quality of Evaluation (QOE)1 by Region Region Quality of Evaluation Total % distribution numbe r 0 1 2 3 Europe & Central Asia 26.8 43.9 4.9 24.4 41 Latin America & 42.6 32.4 16.2 8.8 68 Caribbean Middle East & North 62.5 37.5 0.0 0.0 8 Africa OECD 39.3 26.2 25.4 9.0 122 South and East Asia & 23.8 76.2 0.0 0.0 21 Pacific Sub-Saharan Africa 55.2 37.9 3.4 3.4 29 World-wide 39.4 35.3 15.6 9.7 289 Notes: 1. See Table 4 for explanation of QOE specification. Within the OECD group, the Anglo-Saxon countries have had the stronger record regarding program evaluation. Figure 4, which breaks the OECD into the two country groupings used earlier, shows that there is substantially more information available on employment programs for youth in Anglo-Saxon countries than in other OECD countries (essentially continental Europe, with a few examples from Japan). In the former group of countries, 38 per cent of the programs we found had an impact evaluation (QOE= 2 or 3) and 14 per cent included a cost-benefit analysis (QOE=3). On the other hand, for programs in continental Europe, 47 per cent have no evaluation information available on outcomes or impact (QOE=0) and another 26 per cent measure only gross impact (QOE=1). We found no evaluations in this sub-region with cost-benefit analysis. In fact, the only evaluations in the OECD with cost-benefit analyses were in Canada, the U.K., and the U.S. The evaluation record is strongest for multiple-service programs; 48 per cent (20 out of 42) of all net impact evaluations carried out in the OECD region were in this category. These mixtures of 31 41412 classroom training, on-the-job training, counseling and subsidized employment, mostly financed by the governments, have been repeatedly evaluated in the Anglo-Saxon countries, mainly in the U.S., and less often in Australia, Canada, and the U.K. F. Quality of interventions in the inventory: Descriptive analysis Where possible, the interventions included in the inventory have been assessed in terms of their impact as well their cost-effectiveness. "Impact" is defined here as the effect of the programs on the future employment prospects of participants, as measured by post- program employment and/or earnings. On the basis of these indicators, programs are classified according to the "quality of intervention" (QOI) variable that was introduced in Section C (recall Table 3). This variable distinguishes between interventions with positive and those with negative or zero impact. The group of programs with a positive impact is then further divided into three sub-groups ­ those that are cost-effective, those that are not cost- effective, and those for which no cost evidence is available. Figure 4: Percentage Distribution of Programs by Quality of Evaluation (QOE) for OECD Country Sub-groups Total OECD 39% 26% 25% 9% Anglo-Saxon countries 35% 27% 24% 14% Continental Europe and other OECDs 47% 26% 28% No evaluation information Basic information on gross outcomes Net impact evaluations Net impact evaluations with cost-benefit analyses 32 41412 The assessment of the quality of an intervention is based on program documentation of these performance indicators and costs. The poor quality of the evaluation evidence, discussed in the previous section, must be kept in mind in interpreting these results. For almost 40 per cent of the programs in the YEI, no information on employment outcomes is available (i.e. QOE=0). In these cases, it was impossible to make any informed judgment on the quality of the intervention and the QOI variable was assigned a missing value. Another 35 per cent had information on gross outcomes, but not net effects (QOE=1). Of course, this level of evidence falls short of what is really required to assess a program's impact ­ i.e., a methodology that can isolate net impact by comparing the observed outcomes with what would have happened to the participants in the absence of the intervention (see Box 5). Nonetheless, in most of these cases where there is information only on gross outcomes, a "non-scientific" assessment of the labor market impact of the intervention was attempted (see section C above). Some readers may want to focus only on program assessments that are based on net impact studies (i.e., QOE=2 or 3). For this reason, the data on the quality of interventions are disaggregated by the quality of the underlying evaluation evidence. Also, the analysis of the effectiveness of programs puts more emphasis on results based on net impact evaluations.19 However, given that so many programs in the inventory do not have scientific evaluations, the judgment was made that too much information would have been lost had they been excluded altogether. 19Here we draw significantly from a background paper on lessons learned from the impact evaluations. See Puerto (2007a). 33 41412 Box 5: Evaluation within a cost/ outcome framework ­ a mini-manual Ideally, programs should be evaluated within a cost/ outcome framework ­ involving, as its name suggests, a comparison of the cost of a course of action with its outcome. Cost can be defined from various points of view. In the case, for instance, of an evaluation of a skills training course, which has not involved any capital expenditure, the cost to the individual of taking the course is the fee (if any) that has to be paid plus the value of whatever the individual has had to give up in order to participate in the training (principally, his/her after-tax earnings, which would be zero in the case of the unemployed). From the government point of view, the cost of the course is measured by its net implications for government expenditure. However, the relevant cost concept for social cost/outcome analysis is cost from the point of view of society as a whole, or social opportunity cost. This is defined as what society has to give up in order that the training should take place. In this case, cost will include not only actual expenditure on staff of all kinds, power, telephones, repair, maintenance, training materials, etc., but also the cost of resources for which no payment is involved, such as the time of volunteer teachers, trainees etc. if that time has an alternative productive use. It will also include the cost of indirect as well as direct inputs, such as the provision of special transport for participants. Inputs will be valued initially at market prices, then adjusted for inflation and for any differences between market prices and social opportunity cost (for instance, taxes should strictly be deducted from prices of inputs, and subsidies added to them). On the outcome side, again taking the example of a skills training program, the main interest is in what happens in the labor market to those who have received the training. Thus the impact of the course either on earnings or on employment has to be measured. This is commonly misunderstood. The impact of such a program on employment, for instance, should be measured not by the proportion of trainees who get jobs (the gross outcome) but by the difference the program makes to that proportion (the net impact). Thus, a comparison has to be made with a control group -- i.e., a group of people with all the same characteristics as the trainees (age, sex, education, social class, etc.) save that they did not participate in the program. The effectiveness of a training course should be measured by deducting the success rate of the control group (e.g., in obtaining jobs) from that of the trainees, to show what difference the training made. For example, a multi-service youth employment program in the Dominican Republic, offering training and private sector internships to disadvantaged young people, achieved a 57 per cent employment rate for participants ­ which looks good until it is revealed that the employment rate for the control group was 56 per cent (Card et al. 2006). Similarly, the benefit of a course, measured by impact on earnings, should be calculated by deducting the earnings of trainees over a defined period from those of a control group. Comparison of cost and outcome can take several forms. A relatively simple measure would be in terms of cost- effectiveness. For instance, in the case of training courses which are aimed at improving the chances of unemployed people of finding jobs, a relevant cost-effectiveness measure would be extent of improvement in employability per unit of spending. More ambitious would be some kind of cost/benefit calculation. Broadly speaking, this consists of comparing the stream of costs attributable to the training with the stream of benefits resulting from it. This comparison can take the form of a benefit/cost ratio, a net present value calculation, or an internal rate of return. The benefit/cost ratio is the discounted present value of the stream of benefits from the training (measured by its impact on the before-tax earnings of a trainee) divided by the discounted present value of the stream of costs (direct and indirect) attributable to the training. The net present value is the discounted present value of the stream of benefits minus the discounted present value of the stream of costs. The internal rate of return is the discount rate at which the present value of the stream of benefits is exactly equal to the present value of the stream of costs. Social cost/benefit analysis must always be supplemented by private cost/benefit analysis, which looks at costs and outcomes from the point of view of the individuals who participate in a program rather than from the point of view of government or society. Private pay-off can be measured in various ways. The simplest would be in terms of private cost-effectiveness. As before, the cost-effectiveness measure would be the extent of improvement in success rate in the job market per unit of cost, but this time with cost defined as private cost (see above). A private cost/benefit rather than a cost-effectiveness approach can also be tried, using the same three measures -- benefit/cost ratio, net present value or internal rate of return -- but seen from the private point of view. 34 41412 Figure 5 shows the overall distribution of the QOI variable for the programs in the inventory. It is based on 172 interventions where an assessment could be made regarding employment and/or earnings outcomes; these include both programs where only gross outcomes are available (QOE=1) and those where impact evaluations have been carried out (QOE = 2 or 3). The figure excludes those 117 interventions where an assessment of impact could not be made, either because no information was available on outcomes or impact (QOE = 0, 114 programs) or because a conclusion could not be drawn based on the documentation that was available (QOI=99, 3 programs). Of these 172 programs, 132 (78 per cent) were rated as having had a positive impact on the employment and/or earnings of participants. Again, it cannot be overemphasized that, in the case of many of these programs, the assessment has been made on the basis of gross-outcome data alone. The percentage of programs with a positive impact, on this basis, is higher than might have been expected but, as will emerge in the following paragraphs, a more complete assessment of the interventions leads to an estimated success rate that is considerably lower. Figure 5: Summary of Quality of Interventions (QOI) for all Programs with Evaluation Evidence of any Kind1 8.1% 22.1% Zero or negative impact (QOI = 0) Positive impact but not cost- effective (QOI =1) 6.4% Positive impact, no cost evidence (QOI =2) Positive impact and cost- effective (QOI =3 63.4% Notes: 1. Includes programs with data on gross and net outcomes (i.e., QOE=1,2, or 3) The weakness of the evaluation evidence underlying Figure 5 is given added importance by the finding that the assessed impact of an intervention is affected by the quality of the underlying evaluation evidence. This is shown in Table 12 which cross- 35 41412 tabulates QOI by QOE. When information on gross outcomes only is available (QOE=1), 90 of the 99 programs where a QOI assessment was made were judged to be positive. However, when a net impact evaluation has been carried out (QOE=2 or 3), the probability of finding a positive employment impact decreases significantly, to 60 per cent (44 of 73 programs). Table 12: Quality of Intervention (QOI) Disaggregated by Quality of Evaluation (QOE)1 Quality of Evaluation Quality of Intervention 0 1 2 3 99 Total 0 114 114 1 9 2 85 3 3 102 2 22 1 21 1 45 3 7 8 3 10 28 Total 38 11 109 14 117 289 Notes: 1. QOI and QOE values as described in Tables 3 and 4. Figure 6 summarizes the pattern of quality of interventions if confined to those where net impact evidence is available (i.e., QOE=2 or 3). Comparing this figure with the larger sample in Figure 5 above which also includes programs with only gross impact information underlines the point that the better the evaluation, the higher the likelihood of that the assessment will be unfavorable. Figure 6: Summary of Quality of Interventions (QOI) for Programs with Evidence on Net Impact1 15.1% Zero or negative impact (QOI = 0) 39.7% Positive impact but not cost- effective (QOI =1) Positive impact, no cost evidence (QOI =2) 32.9% Positive impact and cost- effective (QOI =3 12.3% Notes: 1. Includes programs with QOE=2 or 3. 36 41412 We can only speculate on the reasons for this finding but a likely possibility is that, even when serious and cautious attempts are made to estimate the impact of interventions on the basis of gross-outcome data alone, the real net impact will tend to be overestimated. This has two important implications. First, within the context of this study, an overall assessment of what interventions can do for the employment and earnings of young workers is much more favorable when the standard of acceptable evidence is relatively light than when a higher standard is set (i.e., net impact evaluation). Second, because of the lack of serious evaluations especially in developing countries, policy-makers ­ who tend to focus on gross outcome measures ­ are generally overestimating how useful their interventions are in helping young people find employment or increasing their earnings. Moreover, a complete estimate of the overall success of youth programs should consider the cost dimension, as well as their labor market impact. Unfortunately, in the vast majority of cases, it is not possible to determine whether interventions that have achieved positive impact did so cost-effectively. Of the 134 programs assessed to have positive employment impact, only 25 have a cost-benefit analysis. Of these, 14 were cost-effective (56 per cent) while 11 (44 per cent) were not. Using this information, we can simulate the overall success rate of interventions, where "success" is defined as a positive labor market impact and cost-effectiveness. To do this, we assume that the programs without cost information have the same probability of being cost effective as programs with cost information (56 per cent, as in the above paragraph). With this assumption, then we can estimate an overall success rate. Table 13 shows the result of this simulation for the 172 programs with evaluation evidence of any kind (QOE = 1, 2, or 3). Of these programs, we have already noted that 134 were assessed to have a positive employment impact. This is calculated as the sum of the three positive impact QOI ratings in the column of Table 13 labeled "Number of programs". We then apportion the 109 programs that had positive employment impacts but no cost information into "not cost effective" and "cost effective" using the assumption of a cost-effective rate of 56 per cent discussed above. This leads to the numbers in the "Adjusted number" and "Failures" and "Successes" columns. As the table shows, this methodology yields an estimate of 75 "successful" (positive labor market impacts and cost-effective) programs, which represents 43.6 per cent of the total of 172 programs. 37 41412 Table 13: Simulation of Overall Program Success Rate (positive impact, cost effective)for all Programs with Evaluation Evidence of any Kind QOI rating Number of Adjusted Failures2 Successes3 programs Number1 Negative or no impact (0) 38 38 38+59=97 Positive impact, not cost effective (1) 11 11+48=59 (56.4%) Positive impact, no cost data (2) 109 Positive impact, cost effective (3) 14 14+61=75 75 (43.6%) Notes: 1. Programs with positive impact but no cost information are allocated to positive impact, not cost effective or to positive impact, cost effective, based on cost-effectiveness distribution of programs with cost information. 56% are allocated to cost effective and 44% to cost ineffective. The numbers derived on this basis are in italics in the table. 2. Failures defined as either negative/no impact or positive impact but not cost effective. 3. Successes defined as positive impact with cost effectiveness. The same simulation of success rates can be made for the smaller group of interventions where net impact evaluations have been carried out. Of the 73 programs that meet this condition, 20 have information on costs and, of these, 11 (or 55 per cent) are cost- effective while 9 (45 per cent) are not. Note that these proportions are almost the same as those for the 172-program sample. As before, applying these proportions to programs with net impact evaluations but no evidence on costs, we estimate the proportion of interventions that are successful, both in having positive employment impact and being cost effective. As Table 14 shows, this results in a success rate of 33.2 per cent. The fact that the success rate is lower when we only consider programs with net impact evaluations reflects the less favorable assessments of impact when proper evaluations have been carried out. Table 14: Simulation of Overall Program Success Rate (positive impact, cost effective)for Programs with Evidence on Net Impact1 QOI rating Number of Adjusted Failures Successes programs Number Negative or no impact (0) 29 29 29+20=49 Positive impact, not cost effective (1) 9 9+11=20 (66.8%) Positive impact, no cost data (2) 24 Positive impact, cost effective (3) 11 11+13=24 24 (33.2%) Notes: 1. As for Table 13. In this case, 55% are allocated to cost effective and 45% to cost ineffective. Category of intervention. Table 15 presents the QOI ratings data by category of intervention. Although seven of the nine types of interventions in our framework are represented in the inventory, three of them ­ making training systems work better for young 38 41412 people; improving labor market regulations to the benefit of young people; and the residual (other) category -- do not have enough cases to allow any conclusions to be drawn about how well they generally work. As a result, our observations in this section are largely confined to the four most common types of interventions: making the labor market work better for young people; improving chances for young entrepreneurs; skills training for young people; and comprehensive multi-service programs. The relative lack of evaluation evidence is an especially limiting factor once we look at the inventory disaggregated in different ways. This is a particular problem for the entrepreneurship category where an assessment of the quality of the intervention could not be made in 55 per cent of the 33 cases in the inventory. But it is also serious for skills training, where 47 per cent of 111 programs could not be assessed, and comprehensive programs (31 per cent of 94). Table 15: Summary Rating of Quality of Intervention1 by Category of Intervention Category of intervention Quality of Intervention 0 1 2 3 99 Total 1. Making the labor market work better for young people 1a. counseling, job search skills 1 2 3 6 1b. wage subsidies 2 11 1 3 17 1c. public works programs 2 2 2 1 1 8 1e. other 1 1 2 4 Sub-total 5 2 16 3 9 35 2. Improving chances for young entrepreneurs 1 14 18 33 3. Skills training for young people 3a. vocational training including apprenticeship systems 11 2 35 3 47 98 3c. 2nd chance & equivalency programs 1 4 3 8 3d. other 1 1 3 5 Sub-total 13 2 40 3 53 111 4. Making training systems work better for young people 4a. information 3 3 4b. credit (to individuals or enterprises) 1 1 4c. financial incentives (subsidies, vouchers) 1 1 2 4 4d. other 1 2 3 Sub-total 2 2 7 11 6. Improving labor market regulations to the benefit of young people 1 1 2 8. Comprehensive, multiple-service approach 18 6 34 7 29 94 9. Other (e.g. voluntary national service programs) 2 1 3 Total 38 11 109 14 117 289 Notes: 1. QOI values as specified in Table 3. The QOI ratings data in Table 15 are summarized in Figure 7 which shows the percentage of programs with positive employment impact (regardless of cost) in each of these four intervention categories. As before, these results are shown both for all programs 39 41412 with evaluation evidence of any kind and just for those programs with net impact evaluations. Sample sizes in these categories become a consideration especially when we impose the restriction that programs need to have a net impact evaluation.20 The highest impact ratings are for entrepreneurship programs, although these are based on a small number of cases ­ all 15 with any outcome information were assessed as having a positive impact, only 3 of which had net impact evaluations. However, none of the entrepreneurship programs are shown to be cost-effective. There is not a great difference in the impact results for the other three intervention categories included in the figure. The clear majority of programs in each group were judged to have positive employment effects, although this proportion does decrease by 10-20 percentage points when we look only at programs with net impact evaluations. Note that the results for the skills training and comprehensive categories are based on reasonably large samples (total of 58 and 65 interventions, respectively; 22 and 34 with impact evaluations). Figure 7: Percentage of Programs with Positive Labor Market Impact, Regardless of Cost, by Category of Intervention1 120% 100% 100% 100% 81% 78% 80% 72% 64% 62% 60% 55% 40% 20% 0% LM w orks better Entrepreneur Training Comprehensive All programs w ith evaluation evidence of any kind Programs w ith evidence on net impact Notes: 1. Positive labor market impact when QOI = 1, 2, or 3. "All evaluated programs" includes 172 programs where QOE = 1, 2, or 3. "Programs with evidence on net impacts" includes 73 programs where QOE=2 or 3. Which categories are most "successful", once costs are taken into account? The methodology used to simulate success rates for all programs in Tables 13 and 14 can also be 20The number of interventions with impact evaluations ranges from 34 for comprehensive programs to just 3 for entrepreneurship. See Table 10. 40 41412 used to answer this question. However, much depends on the assumptions that are made. If, for instance, it is assumed that the positive-impact projects for which no cost data are available in a given category were cost-effective to the same proportion as those for which cost data are available in the same category, the ranking of our four main categories would be as follows: (1) making the labor market work better for young people (48 per cent "successful"); (2) skills training (46 per cent); (3) comprehensive multiple-service approaches (39 per cent ); (4) entrepreneurship (0 per cent). The problem with this assumption arises from the very small number of evaluations that included cost-effectiveness in each category. The likelihood of error in this estimator is too high. For example, there is only one cost-benefit analysis of an entrepreneurship program, which finds that the program is not cost-effective, but this is hardly a basis for assuming that all the others for which no cost information is available are cost-ineffective. The alternative is to apply the overall cost-effectiveness ratio to each category of intervention rather than to calculate a specific coefficient for each category. The results of this approach are shown in Table 16. The success rate is highest for entrepreneurship programs, at 52 per cent: however, as already emphasized, this estimate is based on 15 programs only. For the other three categories, where we do have more cases, between 40 and 46 per cent of programs are assessed as having both positive effects in the labor market and being cost-effective. Interestingly, whether the quality of interventions is judged by a simple impact rate or by a success rate that incorporates costs, the results, on these assumptions, show relatively little variation across programs. Table 16: Simulation of Program Success Rate (positive impact, cost effective), by Category of Intervention1 Category of intervention Number of Estimated percentage programs with with positive impact evaluation evidence and cost-effective of any kind Making labor market work better for young 26 46.0 people Improving chances for young entrepreneurs 15 52.3 Skills training for young people 58 43.8 Comprehensive, multi-service approach 65 40.1 All programs2 172 43.6 Notes: 1. Calculation of success rate follows methodology used in Tables 13 and 14 and discussed in text. 2. Includes programs in all intervention categories, including ones not reported in table. 41 41412 The inventory has accumulated a great deal of information on the specific programs included in the various intervention categories. This can be useful for policy-makers and others to "unpack" the statistical findings in this report and identify concrete factors associated with what works to support young workers. Given the scope of this synthesis paper, we can only briefly summarize below the key findings for the individual types of interventions. Only the four intervention categories with reasonable coverage are included and for these, we rely on the results of the available net impact evaluations.21 Readers are encouraged to consult the regional reports and a background paper on lessons from the inventory (Puerto 2007a) in order to get more complete and detailed information.22 Specific information on unit costs for a selection of the programs with cost information is shown in Annex C. Category 1: Making the labor market work better for young people. Relatively few interventions have been evaluated under this category -- 11 out of 35 programs included in the inventory. All evaluated programs are either from the OECD countries or ECA, so very little can be said about how well these interventions work in developing countries. Across sub-categories, wage subsidies are the most evaluated (5 out of 17 programs), then public works (4 out of 8 programs), followed by counseling and job search skills (1 out of 6), and "other" programs (a job placement program with sanctions is the only program evaluated among 4 in the sub-category). · Wage subsidies have generally had significant positive effects on improving employment outcomes for youth in transition and developed countries. The impact of this intervention in developing countries largely remains to be tested. Wage subsidies have been particularly successful in improving employment rates for youth, especially young women and the poorly-educated, in transition economies (i.e., Czech Republic and Poland), albeit with two caveats. The positive employment benefits did not extend to earnings, and cost-benefit analysis has not been carried out. In industrialized economies, programs in Belgium and the U.S. had statistically significant positive effects on 21Of the categories not included, the only one with more than a few interventions is category 4, making training programs work better for young people (information, funding, etc.). The only intervention in this category from a developing or transition economy is Kenya's Jua Kali voucher system, which has already been described in Box 4. The only other programs with impact evaluations in category 4 are two U.S. compulsory schemes to keep teenage parents on welfare out of unemployment. 22Original sources of the evaluation evidence for specific programs are not included in this synthesis but are available in the background paper on lessons learned. 42 41412 employment and earnings, although a program in Sweden found negative short-term and insignificant long-term effects on these outcomes. Often, wage subsidies have been targeted at certain categories of young workers (e.g., disadvantaged young blacks in the U.S., women in transition countries). Evidence from OECD countries stresses the benefits of directing subsidies to firms that also offer job training to subsidized workers. · Public works has a mixed record in terms of employment impact for the 4 programs in the inventory that compare outcomes for participants with a control group. Two studies indicate positive impact on future employment probability, ranging from 6 to 26 per cent in Bulgaria (Temporary Employment Program) and the U.S. (American Conservation and Youth Service Corps) respectively; however, programs in France (Contrat d'Emploi Solidarity) and Poland (Public Service Employment) show no effects at best, and even some negative impact on employment probability and wages. The U.S. program had a positive cost-benefit ratio but the Bulgarian one did not. In analyzing the total effects of public works programs, positive net benefits to society as a whole can sometimes be found if the value of the public goods and services produced by the program is included. Most public works programs do not target young people in particular and the cases in the inventory tend to be included because of high youth participation. · Counseling and job-search training are interventions that provide job search assistance (JSA). International studies have found that these relatively inexpensive services tend to be among the most successful of all active labor market programs, especially when costs are taken into account (OECD 2006; Betcherman, Olivas, and Dar 2004). However, there is very little evidence on the effectiveness of JSA programs in terms of helping young people. In fact, the inventory has only one intervention in this category with an impact evaluation, Portugal's Programa Inserção para a Juventude (InserJovem). However, a second program, the U.K.'s Restart ­ although classified in the "other" sub-category ­ also offers job search assistance and has been scientifically evaluated. InserJovem, which targets long-term unemployed youth, offers job-search assistance and short basic skills courses. The impact evaluation found a statistically (and economically) insignificant reduction in the average unemployment duration for participants, with no gains in wages. The Restart program evaluation found positive impact for male participants with unemployment rates 6 points lower than for those in the control group, although no long- 43 41412 term effects were observed for women. A major part of the Restart program includes sanctions (i.e., potential denial of welfare benefits for non-compliance with program rules), which may at least partly explain the evaluation results. While this combination of JSA services with such sanctions is now very prevalent in OECD countries, the relevance of this approach in developing countries is limited by the fact that most do not offer unemployment benefits. The contradictory findings of the InserJovem and Restart evaluations reinforce the point that the success of any intervention depends largely on how it is designed and implemented, as well as on the context in which it operates. Category 2: Improving chances for young entrepreneurs. The overall finding on these self-employment assistance programs, from a wide range of countries, is that they lead to positive outcomes. However, only three interventions in this category had a net impact evaluation: Bulgaria's Self-employment Program and two in Peru -- Formación Empresarial de la Juventud and Calificación de Jóvenes Creadores de Microempresas. In all cases, the evaluations found positive program effects, although their cost-effectiveness and long-term effects are in doubt. The Bulgaria program reported significant gains in employment for participants, with relatively greater effects on female young participants. However, costs per placement exceed those of training and subsidized employment programs. The programs in Peru aimed to increase earnings of participants through the creation of profitable small businesses and the development of trade skills. The evaluations found a positive impact on having a business (including formalization), on hiring employees, and significantly reduced unemployment and inactivity rates, while significantly increasing earnings. Key determinants of success were access to credit and a high frequency of counseling visits. Programs vary in terms of targeting. For example, in ECA, they have generally targeted unemployed people regardless of socio-demographic profile, while programs in LAC have often specifically targeted disadvantaged youth, with entrepreneurial skills or owning a small and/or informal business. One issue with entrepreneurship programs is an overall deficit of program 44 41412 performance indicators which is part of the reason for the lack of rigorous evaluation evidence. This lack of indicators also probably leads to higher business failure rates.23 Category 3: Skills training. Training is the most popular intervention for young people. However, these programs have not been well evaluated. Out of 111 included in the inventory, only 22 have been rigorously evaluated: 7 in ECA, 2 in LAC, and 13 in the OECD area. The category itself covers vocational training programs, including apprenticeships; second chance and equivalency programs; and a residual sub-category. The vocational training category is by far the major one and includes 18 of the 22 evaluated programs. There are 3 evaluations of second chance programs and one evaluation under the residual sub- category, featuring outcomes from a national training institution in LAC.24 Finally, it is useful to distinguish between programs that offer training only and other multiple service programs that include training as one of a number of interventions, which will be discussed separately. As Figure 7 (above) shows, when consideration is restricted to interventions with net impact evaluations, training programs have a somewhat lower incidence of positive employment impact than the three other categories with significant coverage in the inventory. This finding is consistent with other review studies, primarily in the OECD area, that have also found mixed results for youth training (e.g., Kluve 2006; Greenberg et al. 2003). A review by the Inter-American Development Bank of their own youth training programs found slightly more positive results (Ibarraran and Rosas 2006). 23A successful example of information systems for entrepreneurship schemes was developed in Colombia in the late 1990s. The Sistema de Evaluación de Impacto de los Programas de Apoyo a la Microempresa (Impact evaluation system of microentrepreneurship programs) was jointly sponsored by public and private institutions and was implemented in five cities across the country. It provided periodic and standardized information on programs outputs. Preliminary analyses show a reduction in the mortality rate of businesses after the introduction of the information system. 24 SENA is the biggest training provider in Colombia. It functions as a public university as well as a public training institute. Training activities comprise: (i) professional training courses for job seekers (long courses), and (ii) skill upgrading for workers (short courses). Additional resources are devoted to the development of entrepreneurship schemes and innovative business ventures. A recent impact evaluation compared labor market outcomes between SENA trainees and a control group drawn from a 1997 LSMS (i.e., Encuesta Nacional de Calidad de Vida). Net impact estimates suggest a negative effect on earnings and a negligible positive effect on employment. In particular, average wages of beneficiaries are 10 per cent lower than the comparison group, while participating in SENA's courses only increases the employment probability by one-fifth of a percentage point (Gaviria and Nuñez 2003). 45 41412 · Vocational training, including apprenticeship systems. It should be noted that this category is not meant to include formal vocational education, but is intended to cover training and apprenticeship programs for young people who have dropped out or completed formal schooling. Some programs in the inventory were designed to develop basic job readiness only, while others offer a comprehensive array of services that includes vocational classroom and on-the-job training. In many cases, governments are the direct providers of training, while others are open to the private sector, thus fostering competition among training institutions. The assessment of vocational training shows mixed results across regions, gender, and age. However, the evidence collected by the inventory does indicate better effects from training in transitional and developing countries than in advanced economies. Moreover, programs in the first two groups of countries tend to be less expensive, which enhances their relative performance even more. In the OECD area, 5 out of 8 training programs reported negative or zero impact on employment and earnings, and a sixth had positive impact but was shown to be cost- ineffective. It is important to understand that many of the youth training programs in OECD countries are for seriously disadvantaged young people (e.g., U.S. Supported Work Program), with major obstacles to overcome. Evidence on less developed economies, particularly in ECA and LAC, suggests a better record ­ 6 out of 8 evaluated programs reported positive labor market impact for participants, with some examples of cost-effectiveness (Table 17). Among programs with positive employment impact, the magnitude of the effect on likelihood of employment ranged from a minimum of 6 per cent in Hungary and a maximum of 57 per cent in Bosnia and Herzegovina. This wide range is mostly determined by gender and level of education: female participants and the low-educated tend to obtain higher gains from the programs than male participants and those with university degrees, respectively. In some cases, though not all, programs also had a positive effect on earnings. Cost-benefit analyses undertaken in Brazil, Bulgaria, and Poland concluded that programs were cost-effective in the first two, but not in Poland. 46 41412 · Second-chance and equivalency programs. These interventions are intended to bring school drop-outs up to an academic level equivalent to what they have lost by not completing their school programs. Of the 8 second-chance interventions in the inventory, only 3 have impact evaluations, all in the OECD area. The Danish Youth Unemployment Program aims to strengthen the employment possibilities for unemployed, low-educated youth and to provide motivation for them to return to education. Evaluation evidence indicates small but positive short-run effects on employment; however, this is largely due to an increase in the transition rate from unemployment to schooling rather than to employment. There is no data on cost-effectiveness. Two evaluations concern the U.S. JOBSTART Demonstration implemented in the mid-to-late 1980s which targeted school dropouts with poor reading skills. A nation-wide evaluation showed relatively disappointing results: employment rates among participants were not consistently above rates for the control group and earnings effects were either insignificant or negative. Any net gains to participants were outweighed by program costs. However, a specific evaluation of JOBSTART in San Jose concluded that the program resulted in substantial earnings gains for participants. Table 17: Assessment of Labor Market Impact of Selected Training Programs in Transition and Developing Countries Positive Impact on earnings and/or employment Country Program Negative or zero impact but cost- unknown and cost- ineffective cost- efficiency effective Brazil PLANFOR - National Plan of . Professional Education Bulgaria Government Re-training Program: . Guaranteed & Non-guaranteed Jobs Bosnia and Emergency Demobilization and Herzegovina Reintegration Project (EDRP) Hungary Government Re-training Program Romania Government (Public Employment . Offices) Re-training Program Poland Government Re-training Program Czech Rep. Government Re-training Program Turkey Government Re-training Program Source: Puerto (2007a) 47 41412 Category 8: Comprehensive-multi-service approaches. These programs involve some combination of training (i.e., job and/or life-skills training), job search assistance, entrepreneurial services, and a range of other social and employment-related support services. Comprehensive interventions are the most examined of all interventions. The inventory has documented 34 net impact evaluations, 14 in developing countries and 20 in industrialized economies. A number of comprehensive programs contain more than one entry in the inventory, reflecting the characteristics and results in different periods of time and using different analytical tools (Table 18). OECD countries have a long history of comprehensive programs, dating back to the 1960s. These types of interventions were introduced in developing countries in the early 1990s, with the Jóvenes programs, implemented first in Chile. Government-funded and training-oriented, Chile Jóven was quickly replicated in Argentina, Uruguay, Paraguay, Peru, Colombia, Dominican Republic and Venezuela (See Box 6). As Table 18 shows, the Jóvenes programs have been evaluated several times. In Sub-Saharan Africa, Uganda's Program for the Promotion of Children and Youth has been partially evaluated. Of the 34 net impact evaluations of comprehensive programs, 21 (62 per cent) reported positive net impact. The evaluations of the Jóvenes programs conclude that they have been largely, though not always, successful in improving job placement and earnings. In addition, the programs appear to have been relatively cost-effective. Even so, they are expensive and in some countries, the approach has been replaced by smaller, more focused, and less expensive interventions. On the other hand, the evaluation results for comprehensive programs in OECD countries have been less positive. A recent meta-analysis of eight U.S. programs found very moderate and often negative impact on the labor market. One comprehensive intervention, the Job Corps, has survived the scrutiny of evaluators. Two major evaluations of this program have been undertaken and the conclusion now is that Job Corps has positive impact for participants but is not cost effective. There have been some success stories outside the U.S. In Canada, the Employability Improvement Program had a significant impact on annual earnings due to an increase in weeks worked. In the U.K., young unemployed men are about 20 per cent more likely to gain jobs as a result of the New Deal for Young People program. Unfortunately, not much can be said about the impact and effectiveness of comprehensive 48 41412 programs in continental Europe. Only 3 programs in continental Europe have had net impact evaluations and none of them reported evidence of positive impact on the labor market prospects of young workers. Table 18: Comprehensive Programs with Net Impact Evaluations Country Program Evaluation year Industrialized countries Australia Closing the IT-Divide-Infochange and the Green PC, Victoria AU 2002 Canada Cooperative Education Option 1998 Employability Improvement Program (EIP) 1995 Youth Service Canada (YSC) 1999 France French Youth Employment Programs (1980's-1990's) 2002 Norway Active Labor Market Programs for Youth in Norway 2005 Sweden Labor Market Training Program in Sweden 2004 United Kingdom New Deal for the Young Unemployed 1999 ­ 2003 United States Comprehensive Employment and Training Act (CETA) 1984, 1986 and 1987 Job Corps 1982 and 2003 Job Training Partnership Act - Title II-A (JTPA) 1997 New Chance Demonstration 1997 New Hope Project 2003 National Evaluation of Welfare-to-Work Strategies (NEWWS) 2003 Youth Fair Chance 1996 and 1998 Meta-Analysis (sample of government-sponsored programs operated between 1962 and 1972) 1980 Meta-Analysis (sample of government-sponsored programs operated between 1962 and 1998) 2003 Developing countries Argentina Proyecto Jóven 2001, 2004 Brazil Programa Primeiro Emprego - Rio Grande do Sul 2004 Chile Chile Jóven 1997, 1999 and 2004 Colombia Proyecto de Servicios Integrados para Jóvenes 2002 and 2003 Dominican Rep. Programa Juventud y Empleo 2006 Peru PROJoven 1999, 2002 and 2003 Uruguay Opcion Jóven 2002 Uganda Promotion of Children and Youth in Uganda (PCY) 2003 and 2004 Source: Puerto (2007a) As a general rule, comprehensive programs implemented in industrialized countries have been relatively expensive. The U.K. New Deal stands as the least costly intervention among OECD programs with cost information available. Costs per participant in 2005 $US are around $1,000. On the other hand, many North American comprehensive programs have unit costs in the neighborhood of $10,000 with the U.S. Job Corps at around $17,000. 49 41412 Estimates for the Jóvenes programs range from the about $700 to about $2,000 per participant served. Costs for government can be kept down when firms cover the costs of on- the-job training and when service providers are selected through competitive cost bidding. Box 6: LAC's Jovenes Programs ­ An Example of Comprehensive Multiple-service Approaches (Category 8) The Jóvenes programs have represented a prototypical model of a multi-service intervention to improve youth employability and human capital in Latin America and the Caribbean since 1991. With the emphasis on demand, the model targets disadvantaged young workers, ages 16-29, with vocational training and numerous support services. The model was replicated in several countries across the region ­ first Chile and subsequently Venezuela, Argentina, Paraguay, Peru, Colombia, Panama, and the Dominican Republic. Few programs are currently operating; most have been adopted by national public training institutions or substituted by smaller interventions that have inherited several features from this model. Jóvenes' multi-service approach integrates classroom training and work experience in basic and specific trades, as well as life skills, job search assistance, counseling, and information. Both employers and beneficiaries receive financial incentives such as wage subsidies and daily stipends, respectively, to guarantee their participation. Training is offered through a competitive market where a public bidding system ensures quality and fosters private sector participation. Training institutions coordinate courses and internships, balancing the needs of the productive sector with the skills taught in the program. The main criteria in targeting are income levels, education, gender, and regional coverage (within countries). Participants are poor youth with low levels of education ­ high school at most, unemployed, or underemployed. Gender composition is also well balanced. Estimates of unit cost for the Jóvenes programs range from the high US$700s to about US$2,000 per participant served. Across programs, there is evidence of increased employment probability and earnings of participants upon graduation, compared to their control group. In Argentina, for instance, there is a 10 per cent increase in the employment probability of adult women, while in Chile the program increased the probability 21 percentage points, with significant results for youth 21 and younger, and women. Similarly, earnings increased about 10 percentage points in Argentina and Dominican Republic, with particularly favorable outcomes for young males and adult females; and about 26 per cent in Chile, with best results for the youngest. There have been varying estimates of costs relative to benefits. Early evidence from Peru indicates that the positive earnings effects need to last at least 7 years for PROJoven to yield a positive net gain. A recent longitudinal version of propensity score matching of PROJoven showed a positive internal rate of return, consistently above 4 per cent. In Dominican Republic, the investment on training is recuperated after 2 years. The Jóvenes programs have been assessed with a QOI=2, based on ratings of 1 or 2 for QOE. Sources: Aedo and Nunez (2001); Aedo and Pizarro (2004); Elias et al. (2004); Card et al. (2006); Nopo et al. (2002); and Diaz and Jaramillo (2006). Region and level of development. Table 19 presents the inventory data on QOI by region and Figure 8 summarizes the ratings in terms of program impact and cost- effectiveness. Overall, the QOI ratings indicate that the employment impact tends to be more favorable in developing and transition countries than in industrialized countries. While only 60 per cent of programs in the OECD region had a positive impact, the corresponding rates in 50 41412 Europe and Central Asia and Latin America and the Caribbean ­ the two other regions with significant samples ­ were 90 per cent and 92 per cent, respectively. Although the sample sizes are too small in South and East Asia and the Pacific and Sub-Saharan Africa to draw firm conclusions, the limited evidence in these regions offers additional support for the conclusion that youth programs have been more successful in developing countries. While regional differences in the incidence of programs with positive employment impact are evident, there is little variation in terms of cost-effectiveness. Much more detail on the interventions in the different regions is available in the regional reports. Table 19: Summary Rating of Quality of Intervention by Region1 Quality of Intervention Region 0 1 2 3 99 Total Europe & Central Asia 3 3 20 3 12 41 Latin America & Caribbean 3 3 30 3 29 68 Middle East & North Africa 1 1 1 5 8 OECD 29 4 33 6 50 122 South and East Asia & Pacific 1 15 5 21 Sub-Saharan Africa 1 10 2 16 29 Total 38 11 109 14 117 289 Notes: 1. QOI ratings as described in Table 3. Figure 8: Percentage of Interventions with Positive Labor Market Impact and With Cost- Effectiveness, by Region1 90% 92% 94% 92% 78% 67% 60% 15% 10% 8% 8% 8% 0% 0% Europe & Latin Middle East OECD South and Sub- Total Central Asia America & & North East Asia & Saharan Caribbean Africa Pacific Africa % QOI = 1, 2 or 3 % QOI = 3 Notes: 1. Missing values (QOI=99) are excluded the calculations. 51 41412 These differences in performance are also evident when we break the data down by level of income and by developing, transition, and industrialized status (Figure 9, Panels A and B). Whatever disaggregation is used, the results suggest that interventions are more likely to improve the employment and/or earnings of young people in non-industrialized countries than in industrialized ones. There are reasons to find this result surprising. Given their more extensive experience with employment programs, greater capacity and resources, more available information and analysis, and generally better functioning labor markets, industrialized countries might have been expected to have more effective interventions. Figure 9: Percentage of Interventions with Positive Labor Market Impact and with Cost-Effectiveness, by Income Level (Panel A) and Country Type (Panel B)1 Panel A Panel B 90% 91% 90% 90% 93% 60% 60% 5% 11% 6% 8% 7% 11% 8% Low Lower Middle Upper Middle High Developing Transition Industrialized % QOI = 1, 2 or 3 % QOI = 3 % QOI = 1, 2 or 3 % QOI = 3 Notes: 1. Missing values (QOI=99) are excluded in the calculations. So why do the inventory results suggest the opposite? First, it may be that this finding is due to measurement problems stemming from the fact that programs in industrialized countries tend to be more rigorously evaluated. We have seen that when programs are scientifically evaluated with a net impact methodology, they are less often found to have positive employment effects than when an assessment is based only on gross outcome results. So it may be that the differences evident in Figures 8 and 9 are due to the fact that the positive impact of interventions in non-industrialized countries is overstated and that, if they were evaluated as rigorously as OECD programs, these differences would disappear. However, the meta-analysis presented in the next section argues against this hypothesis. In this analysis, which includes only programs with net impact evaluations, the probability that a program has a positive impact declines as the country's income level rises. The observed differences, then, seem to reflect real differences in what youth programs can do in less developed compared to developed countries. For example, 52 41412 disadvantaged young people -- the dominant clientele of youth programs everywhere ­ may be at such a disadvantage in OECD countries, given the high average levels of human capital and the skill-intensity of labor demand, that employment interventions are simply not enough to compensate. In developing countries, on the other hand, where the proportion of young people forced to drop out before completing secondary school is much higher, these programs may give many of them a boost to realize their hidden potential. Unfortunately, we are not able to test this hypothesis in this study. Institutions or policies, such as employment protection laws (EPL), might also matter in explaining why certain groups of countries seem to have more successful programs than others. Employment protection rules, which affect hiring, contracting, and dismissal could limit the effectiveness of youth programs since it is well documented that, where such rules are strict, young people are likely to experience difficulty in entering the labor market (e.g., OECD 2004). Within the OECD region, the evidence is at least consistent with this hypothesis: youth programs have a higher positive impact rate in Anglo-Saxon countries (74 per cent) where EPL is more flexible than in the rest of the OECD, i.e., largely continental Europe (38 per cent) where rules are more protective.25 Using the World Bank's Doing Business "employment rigidity index" (2006) as a measure of the flexibility of EPL for individual countries, we looked at whether there was a link between this variable and the results of youth programs included in the inventory. As Figure 10 shows, the relationship with program impact is non-linear. The lowest rate of successful programs was in countries with rigid employment rules, but countries in the middle category had a higher proportion of programs with positive impact than countries with the most flexible rules. The incidence of cost-effective programs does decline with EPL rigidity. As we will see in the next sub- section, the meta-analysis does find that the probability of a program having positive impact does decrease as a country's employment protection rules become stronger. 25The differences between the Anglo-Saxon and continental Europe (and other) countries in the OECD are discussed in some detail in the OECD regional paper. 53 41412 Figure 10: Percentage of Interventions with Positive Employment Impact and with Cost- effectiveness, by Country's Rigidity of Employment1 90% 74% 67% 10% 8% 5% From 0 to 25 From 26 to 50 From 51 to 76 Rigidity of Employment Index % QOI = 1, 2 or 3 % QOI = 3 Notes: 1. Countries rated according to the Doing Business 2006 "rigidity of employment" index. Higher values indicate more rigidity. Missing values (QOI=99) are excluded from the calculations. Program targeting. In general, when interventions are oriented towards disadvantaged groups of young people, the results seem to be as good, if not better, than when there is no particular orientation. Programs classified as being oriented to one or more of these groups either had specific eligibility rules or had a high proportion of participants from a designated group. Table 20 summarizes the evidence on interventions oriented towards women, the disabled, particular ethnic groups, low-income youth, and the poorly educated. The first three are relatively rare but about one-half of all programs in the inventory are oriented to low- income and/or low-education youth. Compared to their share of the overall sample of interventions, programs oriented to women and the disabled are overrepresented in terms of cost-effective programs. Programs oriented to low-income youth are somewhat more likely than programs without any income orientation to have positive labor market impact although not when cost-effectiveness is taken into account. The most significant results from Table 19 relate to programs oriented to poorly-educated young people. While these programs account for 49 per cent of the total inventory, they represent 60 per cent of programs with positive impact on employment and 71 per cent of programs that are cost-effective. 54 41412 Table 20: Relative Impact and Cost-effectiveness of Interventions Oriented towards the Disadvantaged Orientation Share of all Share of positive-impact interventions Share of cost-effective interventions (QOI=1,2, or 3) interventions (QOI=3) Women 16% 18% 29% Disabled 11% 10% 29% Specific ethnicity 7% 7% 7% Low Income 52% 58% 50% Low Education 49% 60% 71% G. Quality of Interventions in the Inventory: Meta Analysis26 To identify the determinants of program outcomes more systematically, we have carried out a meta analysis based on the interventions collected by the inventory. A meta analysis uses econometric methods to quantitatively combine and synthesize results from individual studies in a common field in order to get an overall picture. We have chosen this approach in order to analyze what types of youth interventions work best and what are the key features in implementation design and targeting that explain variations in employment and earnings outcomes under different economic and institutional conditions. Meta analyses have been most widely used in fields such as education, medicine, and psychology and have only recently been applied to the study of labor market programs. One example is a recent study by Kluve (2006), who uses this approach to estimate the probability of success of a sample of 95 ALMPs in Europe, with special attention drawn to programs implemented since the late 1990s. About 25 per cent of these programs were youth-oriented. The probability of success (i.e., positive employment impact) was modeled by (i) the category of intervention, (ii) the study design, (iii) the institutional labor market context, and (iv) the prevailing country context. His results indicate that category of intervention is the only clear determinant of success of active labor market measures in Europe, and there is little if any evidence that study design or country-context factors explain the programs' effectiveness. Kluve (2006) defines the set of categories or program types as follows: labor market training, private-sector incentive programs (e.g., wage subsidies), direct employment programs in the public sector (e.g., public works programs), and services and sanctions (e.g., job search assistance and compulsory programs to maintain unemployment benefits). 26This section is based on Puerto (2007b). 55 41412 Programs were further disaggregated by target group, including youth. The findings suggest rather modest positive impact from training programs on employability. The model indicates significantly higher returns from private-sector incentive programs and services and sanctions programs; they increase the likelihood of positive labor market impact by 40 to 50 percentage points more than training programs do. On the other hand, relative to training programs, public sector employment programs are 30 to 40 per cent less likely to yield positive impact. On specific target groups, the model indicated that young people are the hardest to assist; when they are targeted, the probability of positive employment impact is reduced by 40 to 60 percentage points. Greenberg et al. (2003) used a meta analysis to synthesize findings from 15 publicly- funded training programs in the U.S. to measure the programs' effects on participants' earnings. Their model regresses the reported earning effects against (i) type of training, (ii) demographic characteristics of the target population, (iii) economic conditions of the area where the program was implemented, (iv) evaluation method, (v) number of years since training was received, and (vi) year in which the program was implemented. Of a total of 315 observations on earnings outcomes, 31 per cent were related to training programs for disadvantaged youth. Results suggest highly heterogeneous earning effects among assisted groups, i.e., men, women, and young people. The overall training effect on youth was negligible, but some control variables showed small positive effects: (i) across program components, classroom training yielded consistently better effects than on-the-job training, while (ii) gender and race controls suggested lower effectiveness of training for whites and female beneficiaries than for all other participants.27 1. Methodology The meta analysis is applied to two sub-samples of interventions in the inventory ­ (i) the set of programs with evidence on outcomes (i.e., QOE=1, 2, or 3); and (ii) only those programs with net impact evaluations (i.e., QOE=2 or 3). The former has the advantage of a larger number of interventions (n=172), while the latter, though smaller (n=73), includes a more reliable measure of program impact. The sample of interventions used in the meta analysis is summarized in Annex D, Table D.1. 27These findings are consistent with an early paper by Gay and Borus (1980). Their study identified net positive impact on earnings of out-of-school and black Neighborhood Youth Corps (NYC) beneficiaries; while there appeared to be significant negative effects for non-black NYC participants and all Job Corps beneficiaries. 56 41412 Based on the measures of intervention quality (or QOI, described in detail in Section C), a binomial variable has been constructed to identify the occurrence of positive labor market impact. This is the dependent variable of the model, which will measure the probability of program success. For the larger sample, this variable takes a value of 1 in 78 per cent of the cases ­ i.e., where the assessment is that effects on employability and/or earnings of beneficiaries was positive (i.e., QOI = 1, 2, or 3), and value 0 in the remaining 22 per cent of observations where negative or zero outcomes were reported (QOI=0). For the smaller sample including only programs with net impact evaluations, this variable has a value of 1 in 60 per cent of cases and 0 in 40 per cent. Ideally, we would have liked to test a model specification where the dependent variable incorporated cost-effectiveness; however, because of the lack of cost-benefit analyses, this specification could not be estimated. Explanatory variables can be organized into four groups: (i) category of intervention, (ii) evaluation quality, (iii) economic and institutional country context, and (iv) specific characteristics of the program. Category of Intervention. Given the evaluation evidence available in the set of 172 programs, our categories of intervention have been clustered into five types (see Annex D, Table D.2): Type 1 comprises interventions to make the labor market work better. Type 2 includes all entrepreneurship schemes. Training-related interventions (i.e. categories 3 and 4) are clustered under program type 3. Comprehensive programs are classified under program type 4. The last type clusters the remaining categories (categories 6 and 9) with fairly low evidence on outcomes. Program types are introduced in the model as five independent dummy variables, where training-related programs represent the omitted category. Evaluation Quality. Within the sample of interventions with evaluation evidence, a further distinction has been drawn between evaluations with only gross outcomes (i.e. QOE=1) and those with net impact evaluations (i.e. QOE= 2 or 3). This classification seeks to test whether the type of evaluation affects the reported labor market outcomes. It has been noted in the previous section that more rigorous evaluation designs tend to yield less positive results. Economic and institutional country context. The characteristics of the country have been considered in other analyses (e.g. Kluve 2006 and Greenberg et al. 2003) to capture the effect of the macroeconomic conditions and labor market regulations on labor market 57 41412 outcomes. We distinguish between developed and non-developed economies, in order to test whether the impact of youth employment programs is affected by the country's income level. About 58 per cent of evaluated interventions took place in non-developed countries (Annex D, Table D.3). In addition, we use the rigidity of employment index (as reported by the Doing Business, 2006) to measure the effect of employment regulations on program impact. Specific characteristics of the program refer mainly to the features of the target population, in particular whether there is a particular focus on women, the disabled, specific ethnic groups, and youth from low income families or with low levels of education. Dummy variables were created for each of these target groups to test whether targeting affects outcomes (Annex D, Table D.4). Whether programs were specifically targeted at youth or were open to workers of all ages is also included in the model specifications. Additional program characteristics included in the model are the decade when the intervention was first implemented and the current status of the program. Most interventions, nearly 72 per cent, have been implemented during the 1990s and 2000s, and over 60 per cent are already completed. The location of the program in rural and urban areas has also been considered in the model. The last variable considered is the program's primary source of financing, which takes value 1 for government-sponsored interventions (two-thirds of observations) and 0 for others. The analysis uses a probit model to estimate the effect of these explanatory variables on the probability that a youth employment program yields positive impact in the labor market for its participants. Probit is a binary choice model that estimates the probability of an event as a function of a set of attributes, assuming a normal distribution in the data. A formal definition of the model is presented in Box D.1, Annex D. 2. Results Table 21 reports the results for the two samples described above. The explanatory variables are the same, with the exception of the quality of evaluation variable which is not needed in the second specification. Marginal effects are displayed for each variable. These marginal effects report the change in the probability of a positive program impact for an infinitesimal change in each independent continuous variable or for a discrete change in the 58 41412 case of dummy variables. The models' estimated coefficients on which the marginal effects are based are presented in Annex D, Table D.5.28 Table 21: Probit Model Reporting Marginal effects of Youth Employment Programs Specification 1 Specification 2 QOE=1, 2, 3 QOE = 2, 3 Marginal Marginal effect z-stat effect z-stat Category of intervention1 Labor market work better -0.032 -0.19 0.011 0.04 Comprehensive -0.124 -1 -0.312 -1.41 Quality of the evaluation2 Net impact evaluation -0.347 -2.53 * Economic and institutional country context3 Non-developed countries 0.527 2.77 ** 0.791 2.61 ** Rigidity of employment index -0.013 -2.88 ** -0.021 -2.48 * Specific characteristics of the program Time period and status4 Program implemented before the nineties -0.422 -2.36 * -0.539 -1.7 Completed programs -0.348 -3.02 ** -0.683 -2.82 ** Targeting5 Programs target only youths -0.121 -1.11 -0.204 -0.92 Programs located in specific areas -0.328 -1.87 -0.549 -1.84 Programs focus on women -0.125 -0.75 -0.172 -0.71 Programs focus on specific ethnic groups 0.152 0.77 0.312 0.7 Programs focus on poor youth 0.47 2.33 * 0.753 2.21 * Programs focus on low-educated youth -0.232 -1.41 -0.539 -1.56 Financing Government-sponsored -0.107 -0.55 0.597 1.48 Observations = 95 Observations =59 Pseudo R2 = 0.46 Pseudo R2 = 0.42 Notes: 1. Training-related programs (including skills training and programs to make the training systems work better) are the omitted category. 2. Programs with evaluations reporting only gross outcomes are the omitted category. 3. Developed countries are the omitted category. The rigidity of employment index is a continuous variable. 4. On decade of implementation, programs implemented during the nineties and 2000s are the omitted category. On current status of the interventions, ongoing programs are the omitted category. 5. Omitted categories on targeting reflect none specific orientation toward disadvantage people within those groups. The values of the z-statistics are reported in the third column: * significant at 5%; ** significant at 1%. 28A logit model was also estimated to test whether a logistic distribution better fitted the data than a normal distribution. The logit regression reported very similar estimates than the probit. 59 41412 On the first set of variables regarding category of intervention, the estimates suggest there are no statistically significant differences among program types in terms of the likelihood that they deliver positive impact on the labor market. This result holds for both specifications. This indeterminate pattern of performance across categories of intervention was also reported by Heckman et al. (1999) for a sample of OECD programs. In the estimation process, two categories (entrepreneurship and others) were dropped due to collinearity effects of their small sample size on the predicted variable. The analysis confirms that evaluation quality matters. This is shown in Specification 1, where the statistically significant negative coefficient for quality of evaluation variable indicates that assessments of program impact are more likely to be negative when net impact studies have been carried out. Having a net impact evaluation reduces the likelihood of success by 35 percentage points. This reflects an over optimistic reading of results from evaluations with gross outcomes, and emphasizes the importance of conducting rigorous evaluations to capture the real effects of programs. Economic and institutional country context variables have highly significant effects on program impact. The regressions show that youth employment programs are more effective in developing countries and transition countries than in developed economies. The likelihood of success is between 53-79 percentage points (depending on specification) higher when the program is implemented in a developing or transitional setting. Given that the quality of evaluation is controlled for, this result cannot be explained by the fact that impact evidence is more rigorous in developed countries. As discussed above, another possible explanation, which cannot be tested with our models, is that the skills disadvantage of participants in developed countries may often be too much to overcome through employment programs, while in developing countries where skills are scarcer, programs may provide enough of a boost to make a measurable difference. A third hypothesis relates to institutional and policy-related factors, including, for example, the effect of employment protection laws on the effectiveness of the programs. In this regard, the significant negative coefficient for the employment rigidity index does suggest that economies with more flexible labor market regulations do get better outcomes from youth employment programs. However, note that the size of the coefficient in both specifications is very small ­ i.e., while the effect may be statistically significant, the 60 41412 importance seems minor. In any event, labor market flexibility cannot explain the difference in the performance of youth programs between developed and non-developed economies, since OECD countries report some of the lowest indices of rigidity around the world. Among program characteristics, the period of implementation and the current status of the program have significant effects on the probability of success. First, although statistical significance is borderline, the models suggest a learning process, where programs developed during the 1990s and after tend to yield better outcomes than older programs. This is the case in Latin America where there has been a move towards demand- oriented programs that match the needs of the productive sector, as well as open participation of the private sector and other agents in the provision and financing of programs. Second, both specifications indicate that ongoing programs perform better than completed programs. In terms of beneficiary orientation, programs targeting economically disadvantaged youth perform significantly better than programs without this orientation. This suggests that interventions do have promise for improving the labor market situation of low-income young people. Other considerations toward a particular gender, the disabled, specific ethnic groups, and youth with low education levels do not affect the outcomes. Similarly, the model tested whether publicly-funded programs perform better than otherwise, but the marginal effect of source of financing lacked statistical significance. To summarize, the meta analysis results indicate that program success is not determined by the type of intervention. This is contrary to what Kluve (2006) found for ALMPs in Europe, but is consistent with the OECD-wide results obtained by Heckman et al. (1999). On the other hand, country context seems to matter. An employment program implemented in a developing or transitional country has at least a 50 per cent higher probability of yielding positive impact for youth than a developed-country program. The analysis proved this is not a measurement problem, since the estimates hold even when the sample is constrained to studies with net impact evaluation. Other explanations may come into play, such as the human capital gap between these two groups of countries. Labor market institutions appear to have small but significant effects on program impact. The model shows that less flexible employment protection rules slightly lower the probability of obtaining positive outcomes from youth employment programs. Finally, certain characteristics of the programs show interesting effects. Ongoing programs and those 61 41412 carried out since the 1990s have significantly better performance than earlier interventions. Targeting interventions on economically disadvantaged youth appears to have substantial positive impact on participants' labor market prospects. Sensitivity tests show these results are stable under different specifications, particularly when the sample size is constrained to studies with net impact evaluations.29 H. Conclusions The Youth Employment Inventory has assembled information on a large number of programs implemented around the world to support young people in their early years in the labor market. Although the largest concentration of interventions included in the YEI are from OECD countries, there are also substantial numbers of programs introduced in the largely middle-income countries of Eastern Europe and Central Asia and Latin America and the Caribbean. In addition to assembling the inventory, which we hope can be regularly updated, this project has carried out analytical work to answer two questions: First, what sorts of interventions have been introduced in order to support youth in the labor market? Secondly, what appears to work, in terms of improving employment outcomes in a cost-effective manner? The evidence from the inventory on these questions is summarized in Table 22. While the "macro" statistical analysis presented in this report offers new insights, it should be recognized that this is only part of the total information base that policy-makers need to make solid decisions on interventions to help youth. This report is not intended to provide the more "micro" program and contextual detail on specific programs that is also important.30 29 Results for the model's Specification 1 were tested to ensure the best fit of the model and to rule out the possibility of outliers. The first test checked the stability of the explanatory power in a smaller sample. After splitting the sample randomly in two, the R-squared increases slightly form .46 to .51, suggesting a steady fit in the model. Marginal effects of this model are reported in Table D.6, Annex D. An additional test was performed to ensure the stability of the explanatory power by ruling out the possibility of outliers. Specification 1 is run iteratively by sequentially and randomly dropping one observation with replacement. Ninety-five models resulted from this exercise, and the R-squared reported ranged from .45 to .51 (Figure D.1., Annex D) verifying the stability of the specification's best fit, and eliminating the possibility of outliers. 30More of this type of information is available in the regional and background analytical reports. 62 41412 The major conclusions from our analysis are the following: 1. Training is the dominant form of intervention used to help young people improve their employment situation. Of the 289 cases included in the inventory, 38 per cent are training programs. Moreover, the second-largest category, comprehensive interventions (33 per cent), typically includes training as an important, if not the most important, component. Training interventions almost always involve the direct provision of courses. Interventions to improve the functioning of training "markets" through better information and financial instruments are relatively infrequently used. 2. Programs are often targeted at low-income or poorly-educated young people. The results of interventions oriented towards disadvantaged youth are as good, if not better, than programs with no particular orientation. The majority of programs are oriented to low-income and/or low-education young people, either through explicit ex ante targeting or ex post participant composition. Few programs are oriented to other forms of potential disadvantage (e.g., gender, disability, ethnicity) in the labor market. Both the descriptive and the meta-analysis find that the impact of programs oriented to disadvantaged categories of young people tends to be more positive than youth programs as a whole. 3. The overall evaluation evidence on youth employment programs is weak. One of the strongest conclusions of this report is the poor situation with regard to evaluation. For 40 per cent of the interventions included in the inventory, we could find no documentation of any sort regarding outcomes. Of the 60 per cent with such documentation, the majority have data on gross outcomes but nothing on net impact. Only one-quarter of the interventions in the inventory have had evaluations which use a control-group methodology to allow for the estimation of net impact. Less than 10 per cent have evaluations which measure both net impact and cost, which are required to assess cost- effectiveness. Moreover, our data collection methodology almost certainly has resulted in a bias towards the inclusion of well-evaluated programs in the inventory. Outside the OECD area (especially the Anglo-Saxon countries) and other than studies sponsored by international organizations, rigorous evaluations are rare. 4. Properly evaluated programs are less likely to lead to positive assessments of impact and effectiveness than judgments based on "non-scientific" methodologies. So where there is not a proper evaluation, program benefits are likely to be overestimated. Where 63 41412 possible, interventions included in the inventory were assessed according to their impact on the employment and earnings of participants. Even when only gross outcome data were available, we tried to judge impact as carefully as possible, according to a standardized methodology. However, despite these efforts, programs without net impact evaluations were 50 per cent more likely to be assessed (based on available information) as having a positive impact than programs with proper net impact evaluations. This suggests that, in the absence of such evaluations, policy-makers are likely to overestimate the benefit of their interventions and, as a result, allocate resources inefficiently. This is a particular concern in developing countries where resources are scarce and evaluations are uncommon. 5. Among programs with net impact evaluations, about 60 per cent were found to have positive effects on the employment and/or earnings of participants. However, when cost-effectiveness enters into the calculations, our estimation is that only about one- third of all programs are "successful". Our assessment of the impact of programs focuses on two indicators ­ post-program employment and earnings. Where these indicators for participants are compared with comparable measures for a control group of non-participants, about 60 per cent of programs demonstrated positive effects. But a complete judgment of the overall success of a program should incorporate not only results in the labor market, but also whether positive impacts were achieved in a cost- effective manner ­ i.e., where benefits were estimated as greater than program costs. Because of the scarcity of evaluations with cost-benefit analysis, we could only approximate the incidence of "successful" programs. Based on certain assumptions, the study concludes that about one-third of programs in the inventory realized positive labor market impact for participants while also being cost-effective. 6. There are no major differences across categories of interventions in terms of impact or cost-effectiveness. This suggests that no particular types of programs are inherently more successful than others, but that policy-makers should consider which type of intervention best addresses the problem of concern. The inventory collected information on a significant number of interventions in four categories: making the labor market work better for young people (job search assistance, subsidies, public works); support to 64 41412 entrepreneurs; skills training; and comprehensive, multi-service interventions. Evidence from evaluations suggests that between 55 per cent and 65 per cent of programs in each of these categories have positive net employment impact. The exception is support for entrepreneurs where all evaluated programs had positive results, but there were not enough scientific evaluations (i.e., with control groups) to have confidence in this result. When costs were taken into account, again there were not major differences across categories. The meta-analysis confirmed that there were no statistically significant relationships between type of intervention and probability of program success. The policy implication of this finding is that, since different categories of interventions address different issues, particular types of programs should not be favored but, rather, that interventions should be chosen based on the specific obstacles to employment that need to be overcome. Table 23 identifies the types of interventions that are appropriate for the most common problems relating to youth experiences in the labor market. 7. Interventions tend to be more successful in developing and transition countries than in advanced economies. The probability that programs will help young people in the labor market is greater in developing and transition countries than in industrialized ones. This is not due to the more rigorous evaluations in developed countries. The meta-analysis confirmed that the difference in program impact by level of development remained even after the quality of the evaluation evidence was taken into account. The study could not adequately explain this result, but it would be interesting to test two hypotheses. First, are disadvantaged youth so "disadvantaged" in developed countries that employment interventions are simply not enough to compensate? Second, are there institutions and policies that systematically differ by level of development that might explain the variation in program outcomes? 8. Youth programs have a lower likelihood of having positive impact in countries where labor markets are not flexible although the magnitude of the effect is small. In the OECD, for example, youth programs were almost twice as likely to have positive impact in Anglo-Saxon countries where labor markets are flexible as in continental Europe, where they are more rigid. Research has shown that protective employment rules create barriers for new entrants and our results suggest that employment programs do not significantly overcome these barriers. The meta-analysis finds that the rigidity of 65 41412 employment protection rules is associated with a lower probability of positive employment benefits to participants, although the magnitude of the effect is very small. In any event, policy-makers need to take a comprehensive approach to improving youth employment, implementing well-designed interventions and also, ensuring that labor market policies and institutions do not block access for young people. 66 41412 Table 22: Summary of Program Targets, Design and Risks, and Impacts, Youth Employment Inventory Target Design and Risks Impacts and Outcomes Developed countries Non-developed countries Making the labor market work better for young people Youth 14 to 30 years of Wage subsidies are provided upon hiring an Wage subsidies have positive outcomes for Wage subsidies have improved age. entitled unemployed worker during a specified youth, increasing employment rates, employment outcomes with net Also open to workers of period of time. duration and earnings. Successful employment effects from 12 to 15.6 %. all ages (i.e. public works Public works programs offer temporary examples: U.S. YIEPP and the Belgian Young women and low educated programs). employment, mainly in the public sector. They are Employment Plan. participants tend to benefit the most. The Unemployed workers in not youth-specific in general, but can be designed Public works present mixed results. impact on monthly earnings is slightly advanced and transition to pay particular attention to young people. Positive outcomes indicate greater negative. Successful examples: Czech countries; and poor youth It is key to target firms and sectors with potential employment probability of about 26% with Republic's Wage Subsidy Program and in developing countries. to create human capital accumulation among the respect to the control group. Successful Poland's Intervention Works Program. Low levels of education young. examples: American Conservation and Public works present mixed results. are common among There is a risk of increasing welfare dependency Youth Service Corps. Positive outcomes indicate greater beneficiaries. among beneficiaries. employment probability of about 6% with Rural and urban focus. respect to the control group. Cost- Some orientation towards effectiveness remains to be tested. women in developing and Successful examples: Bulgaria's Temporary transition economies. Employment Program. Improving chances for young entrepreneurs Youth 14 to 35 years of Entrepreneurship schemes go from basic training There is no evaluation evidence in OECD Evidence from countries in transition age. on managerial skills and the creation of business countries. shows positive effects on employment and Unemployed workers in plans, to more comprehensive programs including cost-effectiveness. Successful examples: advanced and transition further training in accounting, taxes, sales, Bulgaria's Self-employment Programme. countries; and poor youth internships in local businesses and start-up loans. Evidence from developing countries show in developing countries. Credit market failure limits entrepreneurial an increase of 7.8 percentage points in the Low levels of education possibilities among the young due to lack of credit probability of having a business operating, are characteristic in history, collateral, etc. and an 8%-increase in the beneficiaries' developing countries. There is great and increasing participation of average income. Successful examples: Rural and urban focus. NGOs in design and implementation. Peru's Formación Empresarial de la Some orientation towards The lack of success/failure indicators (i.e. Juventud and Calificación de Jóvenes women in developing information systems and long-term evaluation Creadores de Microempresas. economies. evidence) may lead to budget cuts, hindering programs sustainability. 67 41412 Table 22: Summary of Program Targets, Design and Risks, and Impacts, Youth Employment Inventory (cont'd) Target Design and Risks Impacts and Outcomes Developed countries Non-developed countries Skills training for young people Youth 14 to 30 years of Comprises non-formal vocational skills training, Cross-country evaluations in OECD countries There are positive impacts from training with age. second chance programs and apprenticeship suggest non-significant labor market impacts. relatively proven cost-effectiveness. The Unemployed and systems. There are some positive effects for adult programs increased the likelihood of disadvantaged youth with Training systems include public-private alliances women and educated men, but in general employment among the young between 6 and low levels of education in the design and provision of services, creating negligible and negative effects for youth. 57%. This wide range of effects on (i.e. school dropouts). cost-sharing structures and allowing consistency Successful examples: Finland's Active labor employment is mostly determined by gender There is a distinct urban between courses and skills demanded by the market policy and the U.S. Summer Youth and level of education: female participants and focus in developing market. Employment and Training Program. the low-educated tend to obtain higher gains countries. Sanction schemes have been designed to reduce than the rest. Successful examples: Brazil's Some orientation towards the probability of dropping out. PLANFOR and Bulgaria's Re-training women in transition Program (Guaranteed & Non-guaranteed economies. Jobs). Making training systems work better for young people Disadvantaged and These programs offer information networks, Programs report positive but no lasting There is no solid evaluation evidence in unemployed youth with vouchers and subsidies to allow young people to impacts on the labor market. developing countries. Kenya's Jua Kali Pilot low levels of education. acquire training. Voucher Program reported net improvements Rural and urban focus. The lack of evaluation evidence in developing in terms of job creation, productivity and There is a wide countries may lead to budget cuts, hindering business profits, but its overall effectiveness orientation towards programs sustainability. remains to be tested. women, particularly There is a risk of increased welfare dependency. teenage mothers in developed countries. Programs with comprehensive interventions Youth 14 to 30 years of Encompasses job and life skills training (in Evidence from OECD countries suggests Comprehensive programs reported positive age. classroom and/or on-the-job), apprenticeship and mixed effects from comprehensive programs. outcomes on employment and earnings. Un/underemployed youth, entrepreneurship schemes, information, A cross-program study in the U.S. found very Evidence from LAC shows 10 to 21% with low income and counseling/placement, financial incentives (to moderate and often negative impacts on the increase in the employment probabilities, and education level. employers and beneficiaries) and other services. labor market. When impacts were positive about 10 to 26% net increase in earnings. The Rural and urban areas are Most programs are publicly-sponsored. they were surpassed by program costs. In most benefited are young youths and women. equally served with some Quality and relevance of training is key to ensure other countries (Canada and the U.K.) Programs are also cost-effective. Successful focus in the main cities in success and sustainability. programs increased annual earnings and the examples: Jóvenes Programs. developing countries. Very large scale programs may have coordination likelihood of getting a job after graduation. Some orientation towards problems between local and central agencies. Successful examples: Canada's Employability women in developing Excessive costs may defer the returns of positive Improvement Program, U.K. New Deal for economies. net gains and hinder sustainability. Young People and the U.S. Job Corps. 68 41412 Table 23: What Program for which Problem? Program categories: Making the labor Improving Skills training for Making training Improving labor Comprehensive Other programs market work better for chances for young young people systems work better for market regulations programs young people entrepreneurs young people to the benefit of young people Nature of problem: High unemployment rates - Counseling, based - Micro- - Literacy & 2nd - Reform of - Training, among less-educated youth, on accurate labor finance chance employment job search & large numbers of out-of- market programs programs protection assistance, school youth outside the labor information regulations support force - Wage subsidies services etc. - Public works programs High unemployment rates - Counseling, based - Information - Voluntary among more-educated youth on accurate labor about high-return national service market training programs information opportunities Over-representation of young - Wage subsidies - Micro- - Literacy & 2nd - Reform of people in low-paid & unpaid finance chance employment family work programs programs protection regulations Apparent skills mismatch - High-return - Information vocational about high-return training training programs opportunities Low take-up of training - Credit, subsidies, vouchers for training Severe disadvantage for some - Anti- - Targeted - Literacy & 2nd - Training system - Targeted - `Social categories of young people discrimination micro- chance with pro- training, job business' legislation finance programs disadvantaged search programs programs - Targeted high- bias assistance, targeted at dis- return support advantaged vocational services etc. training programs 69 41412 References Aedo, Cristian and Marcelo Pizarro. 2004. "Rentabilidad económica del programa de capacitación laboral de jóvenes Chile Joven". Mimeo. Aedo, Cristian and Sergio Núñez. 2001. "The impact of training policies in Latin America and the Caribbean: The Case of Programa Joven". ILADES and Georgetown University. May 2001. Betcherman, Gordon, Karina Olivas, and Amit Dar. 2004. "Impact of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and transition Countries." Washington, D.C.: World Bank, Social Protection Discussion Paper Series 0402. BMZ. 2006. Cornerstones of Youth Employment Promotion in Development Cooperation. Federal Ministry for Economic Cooperation and Development, Division of Development Education and Information. Germany. May 2006. Boeri, Tito. 2002. Social Policy: One For All? Centre d'Etudes Prospectives et d'Informations Internationales (CEPII), Paris. Card, David; Pablo Ibarraran; Ferdinando Regalia; David Rosas; and Yuri Soares. 2006. "Labor Market Impacts of Youth Training in the Dominican Republic: Evidence from a Randomized Program". Inter-American Development Bank, Washington, DC. Dar, Amit and P. Zafiris Tzannatos. 1999. "Active Labor Market Programs: A Review of the Evidence from Evaluations," Social Protection Discussion Paper no. 9901, January. The World Bank. Washington, D.C. de Moura Castro, Claudio and A. Verdisco. 1998. "Training Unemployed Youth in Latin America: Same old sad story?" Inter-American Development Bank, Washington, DC. Díaz, Juan José and Miguel Jaramillo. 2006. "Evaluation of the Peruvian "Youth Labor Training Program - ProJoven". Working Paper, October. Grupo de Analisis para el Desarrollo, GRADE, Lima. Elias, Victor, F. Ruiz-Nunez, R. Cossa, and D. Bravo. 2004. "An econometric cost-benefit analysis of Argentina's Youth Training Program". Inter-American Development Bank, Research Network Working Paper #R-482. European Commission. 2005. European Employment Observatory, Review: Autumn 2004. Employment and European Social Fund. Luxembourg. European Commission. 2006. European Employment Observatory, Review: Spring 2005. Directorate-General for Employment, Social Affairs and Equal Opportunities. Luxembourg. 70 41412 Fretwell, D.H., J. Benus, and C.J. O'Leary. 1999. "Evaluating the Impact of Active Labor Market Programs: Results of Cross Country Studies in Europe and Asia." Social Protection Discussion paper No. 9915, Washington: The World Bank. Gaviria Alejandro and Jairo Nuñez. 2003. Evaluating the impact of SENA on earnings and employment. Departamento Nacional de Planeación. January 2003 Gay, Robert, and Michael Borus. 1980. "Validating performance indicators for employment and training programs," Journal of Human Resources. Winter 1980, 15, 29-48. Godfrey, Martin, 2003, "Youth Employment Policy in Developing and Transition Countries ­ Prevention as well as Cure" World Bank Social Protection Discussion Paper Series No. 0320, Washington D.C., World Bank. Greenberg, David H.; Charles Michalopoulos; Philip K. Robins. 2003. "A Meta-Analysis of Government-Sponsored Training Programs". Industrial & Labor Relations Review. Volume 57, Issue 1 2003 Article 2. Heckman, J.J., R.J. LaLonde and J.A. Smith (1999), "The economics and econometrics of active labour market programs", in O. Ashenfelter and D. Card (eds.), Handbook of Labor Economics 3, Elsevier, Amsterdam. Ibarraran, Pablo and David Rosas. 2005. IDB's Labor Training Operations: Ex-Post Thematic Evaluation Approach Paper. Office of Evaluation and Oversight, OVE. Inter-American Development Bank. Washington, D.C. June 15, 2005 International Finance Corporation and the World Bank. 2006. Doing Business Database. International Labor Office, 2004, Global Employment Trends for Youth, Geneva, ILO. International Labor Office, 2005, Youth: Pathways to Decent Work, Background Report of the International Labor Conference, 93rd session, Geneva, ILO. International Labor Office, 2006, Global Employment Trends for Youth, Geneva, ILO. Johanson, Richard K., and Arvil Van Adams. 2004. Skills Development in Sub-Saharan Africa. Washington, DC: World Bank. Kluve, J. 2006. "The Effectiveness of European Active Labor Market Policy", IZA Discussion Paper, No. 2018, Bonn. 71 41412 Kluve, J. and C.M. Schmidt (2002), "Can training and employment subsidies combat European unemployment?" Economic Policy 35, 409-448. Martin, J.P. and D. Grubb (2001), "What works and for whom: a review of OECD countries' experiences with active labour market policies", IFAU Working Paper 2001:14. Nopo, Hugo, M. Robles, and Jaime Saavedra. 2002. "Una Medición del Impacto del Programa de Capacitación Laboral Juvenil PROJoven". Documento de Trabajo 36, Grade. Perú. OECD. 2002. Employment Outlook, Paris. OECD. 2004. Employment Outlook, Paris. OECD. 2006. Employment Outlook, Paris. O'Leary, Christopher J. 1998. Evaluating the Effectiveness of Active Labor Programs in Poland. Upjohn Institute Technical Report No. 98-012. June 1998 Pezzullo, Susana, 2005, Project Effectiveness and Impact: Youth Employability and Job Placement, Baltimore, International Youth Foundation. Puerto, Olga Susana. 2006. Interventions to Support Young Workers in Latin America and the Caribbean. World Bank: Washington D.C. Puerto, Olga Susana. 2007a. Learning from International Experiences, The Youth Employment Inventory; Background paper for the Sierra Leone Youth and Employment ESW, World Bank, Washington D.C. Puerto, Olga Susana. 2007b. Labor Market Impacts for Youth: a meta-analysis of the Youth Employment Inventory. World Bank: Washington D.C. Quintini, Glenda, and Sébastien Martin, 2006, Starting Well or Losing Their Way? The Position of Youth in the Labor Market in OECD Countries, OECD Social, Employment and Migration Working Papers, No. 39, Paris, OECD. Rother, Friederike and Olga Susana Puerto. 2007. Interventions to Support Young Workers in OECD countries. World Bank: Washington D.C. Rother, Friederike. 2007. Interventions to Support Young Workers in Sub-Saharan Africa. World Bank: Washington D.C. Stavreska, Antoneta. 2006. Europe and Central Asia, Youth Employment Inventory Summary Report. World Bank: Washington D.C. 72 41412 Stavreska, Antoneta. 2006. Interventions to Support Young Workers in South and East Asia and the Pacific. World Bank: Washington D.C. World Bank. 2006. World Development Report 2007: Development and the next generation. New York: Oxford University Press. 73 41412 Annex A: Inventory template and coding system YOUTH EMPLOYMENT INVENTORY Worksheet for Assembling Inventory PROGRAM NAME: A. Intervention category H. Impact and performance indicators 1 Primary Category CODE 1 Outcome (number who got jobs, the number who got waged jobs, their average earnings etc.) Comments: 2 Secondary Category CODE Comments: 2 Impact (measured by e.g. the number who got jobs, the number who got waged jobs, their average 3 Tertiary Category and others CODE earnings, compared with a control group who were not affected by this intervention). Comments: B. Country 3 Cost Comments: C. Time period of the intervention to Society: to Government: D. Status of the Project (Completed or Ongoing) CODE Comments: to Individual Participants: E. Problem addressed 4 Impact in relation to Cost (benefit/cost ratio, net present value, internal rate of return, cost-effectiveness) Comments: F. Nature of the Program/Policy and/or Stated objective to Society: to Government: G. Program content details 1 Description to Individual Participants: 2 Target beneficiaries 3 Age group (Only young people or all ages but mainly young people) CODE Comments: I. Summary rating of quality of evaluation 4 Location (Urban, rural or both) CODE Comments: CODE Comments: 5 Access for disadvantaged a. Gender (Positive, Neutral, Negative, Not known) CODE Comments: J. Summary rating of quality of intervention b. Disability (Positive, Neutral, Negative, Not known) CODE Comments: CODE Comments: c. Ethnicity (Positive, Neutral, Negative, Not known) CODE Comments: d. Income group (Positive, Neutral, Negative, Not known) CODE K. Sources of further information Comments: e. Education (Positive, Neutral, Negative, Not known) CODE Comments: f. Financing. (Government, Beneficiaries, Employers, NGO or Other) CODE Comments: 74 41412 Annex B: Template Reader Manual This Annex contains detailed instructions to guide the creation of electronic databases for the inventory as well as standardized informative tables for analysis. Microsoft Excel serves as software platform for this process. The automatic compilation of files and tables are done through Visual Basic macros. Key files The Template_Reader.zip contains the following three files: 1. MACROS.xls 2. Countrydata_Inventory.xls 3. Codebook.xls MACROS.xls contains the Visual Basic code that will allow you to create a database from a collection of formatted templates. Countrydata_Inventory.xls contains some macroeconomic information that will be added to the database you create in order to organize the information in tables. Codebook.xls contains the variable definitions of the generated database. It also contains the location of each variable in the formatted templates, as well as the Visual Basic code needed to update the macros in case the template format is updated or variables are added. How to use the Template Reader Macro: 1) Create a folder called TEMPLATES anywhere in your hard drive. 2) Inside TEMPLATES create two folders with the following names: a. FilledTemplates: this folder will contain all the templates that you want in your database. b. Country_Data: this folder will contain the file countrydata_Inventory.xls which is used to add macro statistics and region codes to the database. 3) Place all templates in the folder FilledTemplates and place countrydata_Inventory.xls in the Country_Data folder. 4) Place the MACROS.xls file in the TEMPLATES folder and open it by double clicking on its icon. Excel will not display the contents of this file. If you want to access it, click on the Visual Basic Editor button in Excel (or click on Tools menu, Macro, and Visual Basic Editor) 5) Create an empty excel spreadsheet and save it in the TEMPLATES folder with the name OUTPUT. In OUTPUT.xls create an empty worksheet and name it DATA. Save 75 41412 OUTPUT.xls. 6) Place the cursor in any cell within worksheet DATA, in OUTPUT.xls. In Excel go to Tools menu, select Macro and then Macros (or press Alt+F8). Choose the CREATE_DATABASE macro and click in Run it. This will take a few minutes while the macro reads each file and builds the database. 7) Once the database has been created you can proceed to create the tables. For this, simply go to the Tools menu, select Macro and then Macros (or press Alt+F8). Choose the TABLES macro and run it. 76 41412 Annex C: Unit Costs of a Sample of Youth Employment Programs31 Table C.1. Unit Costs of Wage Subsidies Programs Country Program Unit Cost Units 2005 USD Czech Republic Government ALMP: Wage Subsidy $ 885 1996 USD $ 1,438 Poland a Government ALMP: Intervention Works Program $ 1,782 1996 LCU $ 891 United States Youth Incentive Entitlement Pilot Projects (YIEPP) $ 1,472 1981 USD $ 1,475 a: Includes the direct cost of operating the program per participant and the administrative cost of program per participant Table C.2. Unit Costs of Public Works Programs Country Program Unit Cost Units 2005 USD Bulgaria Government's Temporary Employment Program $ 322 2000 LCU $ 252 Poland a Public Service Employment $ 2,436 1996 LCU $ 1,218 a: Includes the direct cost of operating the program per participant and the administrative cost of program per participant Table C.3. Unit Costs of Entrepreneurship Schemes Country Program Unit Cost Units 2005 USD Bulgaria Government's Self- Employment Program $ 594 2000 LCU $ 465 Peru Calificación de Jóvenes creadores de . microempresas $ 536 2005 USD $ 536 Table C.4. Unit Costs of Training Programs Country Program Unit Cost Units 2005 USD U.S. National Supported Work Demonstration $ 6,800 - $ 12,132 - $ 9,100 a 1982 USD $ 16,235 a Summer Youth Employment and Training Program (SYETP) $ 1,362 1993 USD $ 2,337 Brazil . PLANFOR - National Plan of Professional Education $ 170 2000 LCU $ 110 Bulgaria Government Re-training Program: Guaranteed & . Non-guaranteed Jobs $ 50 2000 LCU $ 39 Czech Rep. Government Re-training Program $ 265 1996 USD $ 431 Hungary Government Re-training Program $ 500 1996 USD $ 818 Poland Government Re-training Program $ 997 1996 LCU $ 498 Turkey Government Re-training Program $ 200 1996 USD $ 286 a: The upper bound is the program cost per AFDC (Aid to Families with Dependent Children) participant and the lower bound is the cost for other target groups. 31This Annex is based on the review and calculations in Puerto (2007a, forthcoming). 77 41412 Table C.5. Unit Costs of Second-chance Programs in the U.S. Country Program Unit Cost Units 2005 USD U.S. Jobstart Demonstration $ 4,548 1986 USD $ 7,140 Jobstart Demonstration - CET Project in San Jose $ 2,034 1986 USD $ 3,193 Table C.6. Unit Cost Estimates across Teenage Parent Demonstration Programs in the U.S. Camden Newark Chicago 1989 current prices Average cost per person-month of AFDC receipt $344 $292 $206 Average annual cost per person $3,130 $2,657 $1,730 2005 USD Average cost per person-month of AFDC receipt $490 $416 $293 Average annual cost per person $4,454 $3,781 $2,462 Table C.7. Unit Cost Estimates for Jóvenes Programs Country Program Unit cost Units 2005 USD Argentina Proyecto Jóven $ 2,000 1998 USD $ 1,159 Chile Chile Jóven $ 730 - $ 930 1998 USD $ 825 - $ 1.051 Peru PROJoven $ 691 2005 USD $ 691 Table C.8. Unit Cost Estimates for Comprehensive Programs in the OECD Area Country Program Unit cost Units 2005 USD Canada Youth Service Canada (YSC) $ 8,277 1996 LCU $ 8,169 U.K. New Deal for the Young Unemployed a $ 454 - $790 1999 LCU $ 950 - $1.653 U.S. Sample of government-sponsored programs b $ 8,782 1999 USD $10,032 Job Corps $ 14,128 1995 USD $17,151 Job Training Partnership Act - Title II-A (JTPA) c $ 2,377 1988 USD $ 3,511 New Chance Demonstration $ 9,000 1992 USD $11,645 New Hope Project d $ 5,300 1996 USD $ 6,314 a: estimates based on a total cost estimate of 68.1 million, and a number of participants ranging between 86,200 and 150,000. b: estimated for a meta-analysis of 8 comprehensive training programs, based on individual program evaluations (Greenberg et al., 2003). c: proxy estimate using the data for JTPA adult trainees. d: unit cost per family. The average family consisted of one adult and two or three children 78 41412 Annex D: Further Results from the Meta-analysis Table D.1.: Sample of Interventions for the Meta-analysis Quality of Evaluation Quality of Intervention 0 1 2 3 Total 1 9 2 85 3 99 2 22 1 21 1 45 3 7 8 3 10 28 Total 38 11 109 14 172 Note: QOI and QOE values as described in Tables 3 and 4 in the Synthesis Report. Table D.2: Classification of Categories of Intervention by Labor Market Impact (for a sample of programs with QOE=1, 2, 3) Type Category of intervention Negative or Zero Positive impact impact Total 1 1. Making the labor market work better for young people 5 21 26 2 2. Improving chances for young entrepreneurs 0 15 15 3 3. Skills training for young people 13 45 58 4. Making training systems work better for young people 2 2 4 4 8. Comprehensive approach 18 47 65 5 6. Improving labor market regulations 0 1 1 9. Other 0 3 3 Total 38 134 172 Table D.3: Classification of Countries' Level of Development by Labor Market Impact (for a sample of programs with QOE=1, 2, 3) Negative or Positive Zero impact impact Total Developing and Transition Countries 9 91 100 OECD Countries 29 43 72 Total 38 134 172 Table D.4: Number of Interventions Targeting Disadvantaged Youths by Labor Market Impact (for a sample of programs with QOE=1, 2, 3) Negative or Positive Zero impact impact Total % Women 6 24 30 17% Disabled 1 13 14 8% Ethnicity 1 9 10 6% Income 18 78 96 56% Education 23 81 104 60% 79 41412 Box D.1. The Probit Model Following Hayashi (2000), in the probit model, a scalar dependent variable yt is a binary variable, yt 0, 1 . In our case yt = 1 indicates that a certain program reported positive labor market { } impact on youth, while yt = 0 indicates negative or zero impact. This event is determined by a vector of regressors xt , namely category of intervention, evaluation quality, country characteristics and program characteristics. As a result, the conditional probability of yt given xt is given by ff (yt =1| xt;0)= ' (x ), ' (yt = 0| xt;0)=1-(txt 0), 0 where . ( ) is the cumulative density function of the standard normal distribution. Given the binary features of yt , this can be written compactly as f (yt | xt;0 ) = xt 0 ( ' ) [1 yt - xt 0 ( ' )]1-yt The maximum-likelihood estimator of 0 for the specification above is given by the function m(wt;) = log f (yt | xt;) = yt log xt + 1- yt )log1- xt , ( ' ) ( [ ( ' )] wherewt is the t-th observation in the dataset. Source: Hayashi 2000. 80 41412 Table D.5: Probit Model: Simple Coefficients (Table 19, Synthesis Report) Specification 1 Specification 2 QOE=1, 2, 3 QOE = 2, 3 Marginal Marginal effect z-stat effect z-stat Category of intervention1 Labor market work better -0.118 -0.19 0.029 0.04 Comprehensive -0.464 -1 -0.811 -1.41 Quality of the evaluation2 Net impact evaluation -1.586 -2.53 * Economic and institutional country context3 Non-developed countries 2.149 2.77 ** 2.808 2.61 ** Rigidity of employment index -0.051 -2.88 ** -0.053 -2.48 * Specific characteristics of the program Time period and status4 Program implemented before the nineties -1.438 -2.36 * -1.484 -1.7 Completed programs -1.848 -3.02 ** -2.441 -2.82 ** Targeting5 Programs target only youths -0.500 -1.11 -0.528 -0.92 Programs located in specific areas -1.129 -1.87 -1.538 -1.84 Programs focus on women -0.426 -0.75 -0.435 -0.71 Programs focus on specific ethnic groups 0.896 0.77 0.983 0.7 Programs focus on poor youth 1.583 2.33 * 2.359 2.21 * Programs focus on low-educated youth -0.982 -1.41 -1.769 -1.56 Financing Government-sponsored -0.459 -0.55 2.184 1.48 Constant 5.120 3.15 ** 1.609 0.83 Observations = 95 Observations =59 Pseudo R2 = 0.46 Pseudo R2 = 0.42 Notes: 1. Training-related programs (including skills training and programs to make the training systems work better) are the omitted category. 2. Programs with evaluations reporting only gross outcomes are the omitted category. 3. Developed countries are the omitted category. The rigidity of employment index is a continuous variable. 4. On decade of implementation, programs implemented during the nineties and 2000s are the omitted category. On current status of the interventions, ongoing programs are the omitted category. 5. Omitted categories on targeting reflect none specific orientation toward disadvantage people within those groups. The values of the z-statistics are reported in the third column: * significant at 5%; ** significant at 1%. 81 41412 Table D.6: Probit Model: Specification 1 Dropping Randomly 50 Per Cent of the Sample Marginal effect z-stat Category of intervention1 Labor market work better -0.017 -0.79 Comprehensive -0.005 -1.28 Quality of the evaluation2 Net impact evaluation -0.010 -1.62 Economic and institutional country context3 Non-developed countries 0.594 1.46 Rigidity of employment index 0.000 -1.53 Specific characteristics of the program Time period and status4 Program implemented before the nineties -0.004 -0.69 Completed programs -0.001 -0.38 Targeting5 Programs target only youths 0.004 0.93 Programs located in specific areas 0.001 0.47 Programs focus on women -0.164 -1.99 * Programs focus on poor youth 0.019 0.84 Programs focus on low-educated youth -0.274 -1.1 Financing Government-sponsored 0.000 0.05 Observations = 47 ; Pseudo R2 = 0.5112 Notes: 1. Training-related programs (including skills training and programs to make the training systems work better) are the omitted category. 2. Programs with evaluations reporting only gross outcomes are the omitted category. 3. Developed countries are the omitted category. The rigidity of employment index is a continuous variable. 4. On decade of implementation, programs implemented during the nineties and 2000s are the omitted category. On current status of the interventions, ongoing programs are the omitted category. 5. Omitted categories on targeting reflect none specific orientation toward disadvantage people within those groups. The values of the z-statistics are reported in the third column: * significant at 5%; ** significant at 1%. 82 41412 Figure D.1: R-squares of 95 Models Featuring Specification 1 (Repeatedly dropping observations with replacement) 2 .5 .5 q 8 rs .4 6 .4 4 .4 0 20 40 60 80 100 pcount 83 41412 41412 Social Protection Discussion Paper Series Titles No. Title 0715 A Review of Interventions to Support Young Workers: Findings of the Youth Employment Inventory by Gordon Betcherman, Martin Godfrey, Susana Puerto, Friederike Rother, and Antoneta Stavreska, October 2007 0714 Performance of Social Safety Net Programs in Uttar Pradesh by Mohamed Ihsan Ajwad, October 2007 0713 Are All Labor Regulations Equal? Assessing the Effects of Job Security, Labor Dispute and Contract Labor Laws in India by Ahmad Ahsan and Carmen Pagés, June 2007 0712 Convention on the Rights of Persons with Disabilities: Its Implementation and Relevance for the World Bank by Katherine Guernsey, Marco Nicoli and Alberto Ninio, June 2007 0711 Reaching the Poor and Vulnerable: Targeting Strategies for Social Funds and other Community-Driven Programs by Julie Van Domelen, May 2007 0710 The Macedonia Community Development Project: Empowerment through Targeting and Institution Building by Caroline Mascarell, May 2007 0709 The Nuts and Bolts of Brazil's Bolsa Família Program: Implementing Conditional Cash Transfers in a Decentralized Context by Kathy Lindert, Anja Linder, Jason Hobbs and Bénédicte de la Brière, May 2007 (online only) 0708 Globalization and Employment Conditions Study by Drusilla K. Brown, April 2007 0707 The Kosovo Pension Reform: Achievements and Lessons by John Gubbels, David Snelbecker and Lena Zezulin, April 2007 (online only) 0706 Measuring Disability Prevalence by Daniel Mont, March 2007 41412 0705 Social Safety Nets in World Bank Lending and Analytic Work: FY2002- 2006 by Annamaria Milazzo and Margaret Grosh, March 2007 (online only) 0704 Child Labor and Youth Employment: Ethiopia Country Study by Lorenzo Guarcello and Furio Rosati, March 2007 0703 Aging and Demographic Change in European Societies: Main Trends and Alternative Policy Options by Rainer Muenz, March 2007 (online only) 0702 Seasonal Migration and Early Childhood Development by Karen Macours and Renos Vakis, March 2007 0701 The Social Assimilation of Immigrants by Domenico de Palo, Riccardo Faini and Alessandra Venturini, February 2007 (online only) 0616 Pension Systems in Latin America: Concepts and Measurements of Coverage by Rafael Rofman and Leonardo Lucchetti, November 2006 (online only). Also available in Spanish. 0615 Labor Market Outcomes of Natives and Immigrants: Evidence from the ECHP by Franco Peracchi and Domenico Depalo, November 2006 (online only) 0614 The Relative Merits of Skilled and Unskilled Migration, Temporary and Permanent Labor Migration, and Portability of Social Security Benefits by Johannes Koettl under guidance of and with input from Robert Holzmann and Stefano Scarpetta, November 2006 (online only) 0613 The Limited Job Prospects of Displaced Workers: Evidence from Two Cities in China by Gordon Betcherman and Niels-Hugo Blunch, October 2006 0612 Unemployment Insurance in Chile: A New Model of Income Support for Unemployed Workers by Germán Acevedo, Patricio Eskenazi and Carmen Pagés, October 2006 0611 Evaluating Social Fund Impact: A Toolkit for Task Teams and Social Fund Managers by Sarah Adam, October 2006 0610 Risk and Vulnerability Considerations in Poverty Analysis: Recent Advances and Future Directions by Carlo Cafiero and Renos Vakis, October 2006 41412 0609 Comparing Individual Retirement Accounts in Asia: Singapore, Thailand, Hong Kong and PRC by Yasue Pai, September 2006 (online only) 0608 Pension System Reforms by Anita M. Schwarz, September 2006 (online only) 0607 Youth Labor Market in Burkina Faso: Recent Trends by Daniel Parent, July 2006 0606 Youth in the Labor Market and the Transition from School to Work in Tanzania by Florence Kondylis and Marco Manacorda, July 2006 0605 Redistributing Income to the Poor and the Rich: Public Transfers in Latin America and the Caribbean by Kathy Lindert, Emmanuel Skoufias and Joseph Shapiro, August 2006 (online only) 0604 Uninsured Risk and Asset Protection: Can Conditional Cash Transfer Programs Serve as Safety Nets? by Alain de Janvry, Elisabeth Sadoulet, Pantelis Solomon and Renos Vakis, June 2006 0603 Examining Conditional Cash Transfer Programs: A Role for Increased Social Inclusion? by Bénédicte de la Brière and Laura B. Rawlings, June 2006 (online only) 0602 Civil-service Pension Schemes Around the World by Robert Palacios and Edward Whitehouse, May 2006 (online only) 0601 Social Pensions Part I: Their Role in the Overall Pension System by Robert Palacios and Oleksiy Sluchynsky, May 2006 (online only) 0543 Complementing Natural Disasters Management: The Role of Social Protection by Renos Vakis, February 2006 0542 Brazil's Bolsa Escola Program: The Role of Local Governance in Decentralized Implementation by Alain de Janvry, Frederico Finan, Elisabeth Sadoulet, Donald Nelson, Kathy Lindert, Bénédicte de la Brière and Peter Lanjouw, December 2005 (online only) 0541 Japan's Pension Reform by Junichi Sakamoto, December 2005 (online only) 41412 0540 Demographic Alternatives for Aging Industrial Countries: Increased Total Fertility Rate, Labor Force Participation, or Immigration by Robert Holzmann, December 2005 (online only) 0539 Disability, Poverty and Schooling in Developing Countries: Results from 11 Household Surveys by Deon Filmer, November 2005 (online only) 0538 Food Aid and Food Security in the Short- and Long Run: Country Experience from Asia and sub-Saharan Africa by Carlo del Ninno, Paul A. Dorosh and Kalanidhi Subbarao, November 2005 (online only) 0537 The New Pensions in Kazakhstan: Challenges in Making the Transition by Richard P. Hinz, Asta Zviniene and Anna-Marie Vilamovska September 2005 (online only) 0536 Household's Vulnerability to Shocks in Zambia by Carlo del Ninno and Alessandra Marini, September 2005 (online only) 0535 Linking Community Empowerment, Decentralized Governance, and Public Service Provision through a Local Development Framework by Louis Helling, Rodrigo Serrano, David Warren, September 2005 (online only) 0534 Youth Employment in the MENA Region: A Situational Assessment by Nader Kabbani and Ekta Kothari, September 2005 0533 Measuring Risk Perceptions: Why and How by Joachim De Weerdt, July 2005 0532 Implementing Means-Tested Welfare Systems in the United States by Kathy Lindert, June 2005 (online only) 0531 La Focalización En El Programa De Desarrollo Humano Oportunidades De Mexico by Mónica Orozco y Cecilia Hubert, June 2005 (online only) 0530 Sistema de Identificación de la Población Objetivo: SIPO en Costa Rica by Roxana M. Víquez, June 2005 (online only) 0529 Targeting Social Spending To the Poor with Proxy­Means Testing: Colombia's SISBEN System by Tarsicio Castañeda, June 2005 (online only) 41412 0528 Focalización De Programas En Chile: El Sistema CAS by Osvaldo Larrañaga, June 2005 (online only) 0527 Reforming Brazil's Cadastro Único to Improve the Targeting of the Bolsa Família Program by Bénédicte de la Brière and Kathy Lindert, June 2005 (online only) 0526 Designing and Implementing Household Targeting Systems: Lessons from Latin American and The United States by Tarsicio Castañeda and Kathy Lindert, with Bénédicte de la Brière, Luisa Fernandez, Celia Hubert, Osvaldo Larrañaga, Mónica Orozco and Roxana Viquez, June 2005 (online only). Also available in Portuguese. 0525 Unequal Prospects: Disparities in the Quantity and Quality of Labour Supply in sub-Saharan Africa by John Sender, Christopher Cramer and Carlos Oya, June 2005 0524 Pension Supervision: Understanding International Practice and Country Context by Richard P. Hinz and Anca Mataoanu, May 2005 0523 Social Security Coverage in Latin America by Rafael Rofman, May 2005 0522 A Technology White Paper on Improving the Efficiency of Social Safety Net Program Delivery in Low Income Countries: An Introduction to Available and Emerging Mobile Technologies by Mike Gallaher, May 2005 (online only) 0521 Aging and Poverty in Africa and the Role of Social Pensions by Nanak Kakwani and Kalanidhi Subbarao, May 2005 (online only) 0520 Delivery Mechanisms of Cash Transfer Programs to the Poor in Bangladesh by Shaikh S. Ahmed, May 2005 (online only) 0519 Portability Regimes of Pension and Health Care Benefits for International Migrants: An Analysis of Issues and Good Practices by Robert Holzmann, Johannes Koettl and Taras Chernetsky, May 2005 (online only) 0518 Who Has the Yam, and Who Has the Knife? Social Action Funds and Decentralization in Malawi, Tanzania, and Uganda by N. Mungai Lenneiye, May 2005 0517 Household Vulnerability and Children's Activities: Information Needed from Household Surveys to Measure their Relationship by Diane Steele, May 2005 (online only) 41412 0516 The Effect of Child Labor on Mathematics and Language Achievement in Latin America by Mario A. Sanchez, Peter F. Orazem, and Victoria Gunnarsso, May 2005 0515 The Inter-Generational Persistence of Child Labor by Patrick M. Emerson and André Portela Souza, May 2005 0514 How Does Working as a Child affect Wage, Income and Poverty as an Adult? by Nadeem Ilahi, Peter F. Orazem, and Guilherme Sedlacek, May 2005 0513 Dynamics of Child Labor: Labor Force Entry and Exit in Urban Brazil by Suzanne Duryea, Jasper Hoek, David Lam and Deborah Levison, May 2005 0512 The Responses of Child Labor, School Enrollment, and Grade Repetition to the Loss of Parental Earnings in Brazil, 1982-1999 by Marcelo Côrtes Neri, Emily Gustafsson-Wright, Guilherme Sedlacek and Peter F. Orazem, May 2005 0511 Child Labor, Schooling, and Poverty in Latin America by Guilherme Sedlacek, Suzanne Duryea, Nadeem Ilahi, and Masaru Sasaki, May 2005 0510 Changing Patterns of Child Labor around the World since 1950: The Roles of Income Growth, Parental Literacy and Agriculture by Victoria Gunnarsson, Peter F. Orazem, and Guilherme Sedlacek, May 2005 0509 Disability and Social Safety Nets in Developing Countries by Sophie Mitra, May 2005 0508 Social Funds: A Review of Public Sector Management and Institutional Issues by Mukhmeet Bhatia, May 2005 0507 Pension Reform in El Salvador by Rodrigo Acuña, April 2005 0506 How Changes in Benefits Entitlement Affect Job - Finding: Lessons from the Slovenian "Experiment" by Jan van Ours and Milan Vodopivec, April 2005 0505 A Guide to Multisector Labor Market Models by Gary S. Fields, April 2005 41412 0504 Multinational Enterprises and Training Revisited: Do International Standards Matter? by Niels-Hugo Blunch and Paula Castro, March 2005 0503 Community-Based Health Insurance & Social Protection Policy by Steven Tabor, March 2005 (online only) 0502 Towards a Better Understanding of the Nature, Causes and Consequences of Youth Labor Market Disadvantage: Evidence for South-East Europe by Alexandre Kolev and Catherine Saget, March 2005 (online only) 0501 A Lecture on the Political Economy of Targeted Safety Nets by Lant Pritchett, January 2005 To view Social Protection Discussion papers published prior to 2005, please visit www.worldbank.org/sp. 41412 Summary Findings This paper summarizes the main findings from the Youth Employment Inventory (YEI), a compilation of 289 youth employment interventions from 84 countries in all regions of the world. The YEI was assembled to strengthen the empirical basis for policy-makers to make decisions about how to address the problem of youth employment. This paper describes the methodology for compiling the Youth Employment Inventory and then presents an analysis of the programs themselves in order to (i) document the types of interventions that have been implemented to support young workers; and (ii) identify what appears to work in terms of improving employment outcomes for youth. HUMAN DEVELOPMENT NETWORK About this series... Social Protection Discussion Papers are published to communicate the results of The World Bank's work to the development community with the least possible delay. The typescript manuscript of this paper therefore has not been prepared in accordance with the procedures appropriate to formally edited texts. The findings, interpretations, and conclusions expressed herein are those of the author(s), and do not necessarily reflect the views of the International Bank for Reconstruction and Development / The World Bank and its affiliated organizations, or those of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. For free copies of this paper, please contact the Social Protection Advisory Service, The World Bank, 1818 H Street, N.W., Room G7-703, Washington, D.C. 20433 USA. Telephone: (202) 458-5267, Fax: (202) 614-0471, E-mail: socialprotection@worldbank.org or visit the Social Protection website at www.worldbank.org/sp.