SOCIAL PROTECTION & JOBS DISCUSSION PAPER No. 2103 | JANUARY 2021 Review of the Evidence on Short-Term Education and Skills Training Programs for Out-of-School Youth with a Focus on the Use of Incentives Marguerite Clarke, Meghna Sharma, and Pradyumna Bhattacharjee © 2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: +1 (202) 473 1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: +1 (202) 522 2625; e-mail: pubrights@worldbank.org. Abstract retro geometric background: © iStock.com/marigold_88 Project 41595 Review of the Evidence on Short-Term Education and Skills Training Programs for Out-of-School Youth with a Focus on the Use of Incentives Marguerite Clarke, Meghna Sharma, and Pradyumna Bhattacharjee Abstract: Short-term education and skills training programs are a popular way to meet the needs of unemployed, out-of-school youth by providing them with an opportunity to quickly acquire qualifications and skills that can lead to productive employment. This paper reviews the global evidence to identify which programs are most effective at delivering results. How incentives for stakeholders are incorporated into the program design is given particular attention. Based on the findings, recommendations are offered for the future design of these training programs. JEL Codes: C18, C93, I25, J24, M52, M53, O15 Keywords: Education, short-term education, second chance education, skills, skills training programs, youth, employment, incentives, results-based financing (RBF), human capital, human development, program evaluation, impact evaluation. Acknowledgments: This paper was written under the supervision of Cristian Aedo, Practice Manager (HSAED) and Omar Arias, Practice Manager (HEDGE), with the support of a grant (TF0A2998) from the REACH Trust Fund. The authors are grateful for the advice and support received from Samira Halabi and Samer Al-Samarrai (current and former REACH Trust Fund managers), feedback received from peer reviewers, Victoria Levin, Shubha Chakravarty, and Alexandria Valerio, and comments received from Jessica Ling. Any errors or omissions are entirely the responsibility of the authors. Executive Summary Short-term education and skills training programs are a popular way to meet the needs of unemployed and out-of-school youth by providing them with an opportunity to acquire qualifications and skills that can lead to productive employment. The COVID-19 pandemic, which resulted in the shutdown of schools and places of work around the world, has increased the numbers of these unemployed and out-of-school youth as well as the need to provide them with a way to quickly reengage with education and employment. Given that short-term training programs can be a costly investment for governments, this paper reviews some of the global evidence to try to identify which programs are effective at delivering results. The way incentives for different stakeholders are incorporated into the program design is given particular attention because these programs can only succeed if they are able to (i) motivate youth to engage with the education and/or skills training in a sustained manner and (ii) motivate providers to deliver the resources and support needed to ensure that enrolled youth graduate with the right qualifications and skills and are able to find employment. Results-based financing (RBF), which makes funding available on the achievement of predefined results, is viewed by many governments as a particularly promising way to incentivize these desired outcomes. This review pays particular attention to whether there is evidence to demonstrate its effectiveness. Specifically, this paper addresses three main questions: • What kinds of short-term education and skills training programs for out-of-school-youth have been implemented around the world and what is the evidence on their effectiveness? • How do these programs incentivize participants and providers, and what do we know about the relative effectiveness of those that use an RBF approach? • What are some implications for the future design of these programs? The findings indicate that a wide array of short-term education and skills training programs for out-of- school youth have been implemented around the world, but that few have undergone any kind of rigorous evaluation. Of those programs for which impact data exist, most demonstrate a net positive impact on beneficiaries, at least in the shorter-term and in lower-income country contexts. The benefits to female enrollees – in the form of post-program employment and earnings – typically exceed those for males. One of the explanations for this is that women usually are starting from a lower base and thus have more room to grow by participating in these schemes. The longer-term impact of these training programs is less clear, however, mainly due to a paucity of data. Where longitudinal data are available, the impact seems positive, at least for programs carried out in a developing country context. In order to encourage desired behaviors or outcomes on the part of beneficiaries and providers, many training programs build incentives into their design. Unfortunately, the scarcity of appropriate evaluations makes it difficult to conclude whether RBF approaches are better than others for incentivizing these desired behaviors and outcomes. For now, it may be safest to conclude that they are as effective as existing alternatives. While this review does not find evidence to conclusively demonstrate the superiority of RBF approaches, it does find useful insights on incentives in general. Based on this, five recommendations are offered to policymakers and practitioners working on the design of short-term training programs for out-of-school youth. Table of Contents A. Background and Rationale……………..…………………………………………….….……………………………………………..5 B. Research Questions and Methodology….…………………………………………………………………………………………8 C. Findings……………………………..………………………………………………………………………………………………………….11 (i) What kinds of short-term education and skills training programs for out-of-school youth have been implemented around the world and what is the evidence on their effectiveness?............................................................................................................11 (ii) How do these programs incentivize participants and providers, and what do we know about the relative effectiveness of those that use an RBF approach?............................17 (iii) What are some implications for the future design of these programs?.........................27 D. Conclusion…………….………………………………………………………………………………………………………………………33 References………………………………………………………………………………………………………………………………………..34 Annexes…………………………………………………………………………………………………………………………………………….40 A. Background and Rationale 1. According to the 2019 World Development Report, the most significant investment that governments can make is in enhancing their country’s human capital. 1 This is because a basic level of human capital, such as literacy and numeracy skills, is needed for an individual’s economic survival, and because all types of jobs increasingly demand more advanced cognitive skills. The individual who leaves school without qualifications, but still finds economic success is increasingly the story of a bygone era. So too the notion of a country relying on natural resources alone to propel its economic development. 2. In particular, countries need to invest in the human capital development of their children and youth because these are the future workers and drivers of economic prosperity, and because it is easier to build human capital in the early years of a person’s development. 2 While much of this capital formation happens through a country’s formal education system, not all children and youth are able to avail of this, either because they were never enrolled in school or because they dropped out. Girls are particularly at risk of not being enrolled in school or of being pulled out early. 3 3. In fact, globally, around 259 million children and youth between the ages of 6–17 years are currently out of school. 4 Of these, about 78 percent are of lower- and upper-secondary school age, most of whom reside in Sub-Saharan Africa and South Asia. With the recent COVID-19 pandemic and closure of schools in more than 180 countries, it is highly likely that many of the 1.6 billion children and youth affected by these shutdowns 5 (who are not included in the aforementioned 259 million) will not return to full-time schooling, thereby swelling the numbers of out-of-school children and youth lacking qualifications. 4. The consequences of being out of school are dire and include long-term economic, social, and health costs that impact not just the individual, but also their families and communities. Given ongoing technological advances and the changing nature of work, failure to educate and train this next generation of workers will only fuel the intergenerational persistence of poverty, lock in gender disparities, and undercut the effective utilization of human capital for ensuring the prosperity and security of nations. 6 5. Given the steep costs to the individual and the system, governments are increasingly trying to tackle the issue of out-of-school children and youth. One strategy is to try to prevent the problem from occurring in the first place by investing early on to retain students. While these efforts are important, they do not address the needs of those who are already out of school. Governments, therefore, are adopting a multi-pronged approach that includes not only preventive measures focused on students still in the school system, but also curative measures that address the consequences of having left the system early. While preventive measures are particularly likely to focus on younger children and early 1 World Bank 2019. 2 World Bank 2018. 3 Ibid. 4 This is based on UNESCO Institute for Statistics data for the school year ending in 2018. The total includes 59 million children of primary school age, 62 million of lower secondary school age, and 138 million of upper secondary age. See http://uis.unesco.org/en/topic/out-school-children-and-youth. 5 https://www.worldbank.org/en/topic/education/coronavirus 6 World Bank 2019. 5 adolescents, curative measures often target youth between the ages of 15 and 24 years who have either just dropped out of school or who dropped out several years prior but still lack a formal qualification and are under or unemployed. 7 6. The most common strategy that governments use to address the needs of these out-of-school youth is short-term training programs. 8 Some of these programs focus on providing enrollees with education to address the missing years of schooling (e.g., second chance/equivalency programs or basic literacy training). Others focus on skills training as a way to directly link enrollees to productive employment. However, governments are increasingly implementing hybrid programs that offer both education and skilling, along with other services. This combined approach helps tackle education qualification or cognitive skills gaps that may be holding youth back from pursuing certain careers while at the same time addressing their immediate need for marketable skills in order to obtain a job and secure a regular income. Indeed, research shows that although skills training programs are a useful instrument to facilitate entry into the labor market, the amount of human capital that can be provided through these programs alone remains modest, and educational attainment and qualifications continue to shape employment outcomes and the long-term productivity of youth. 9 7. Regardless of whether programs focus on education or skills training or both, they can only succeed if they are able to (i) motivate youth to enroll and engage in a sustained manner and (ii) motivate providers to deliver the resources and support needed to ensure that enrolled youth graduate with the right qualifications and skills and are able to find employment. Traditionally, many training programs have focused more on inputs, 10 such as the number of individuals enrolled, salaries paid, and buildings rented, with providers only expected to account for how funding was used in these areas. This meant that providers did not necessarily have strong incentives to focus on outcomes, including motivating youth to attend and complete the program. An implicit assumption was that these youth would be self-motivated to attend and complete the program given a presumed self interest in receiving qualifications and the possibility of employment. As research on these training programs has shown, however, this theory does always not hold up in practice, with at-risk and female youth in particular more likely to drop out before completion (exactly the groups that would benefit most from this training). 11 7 There is no universally agreed definition of the youth age group. For statistical purposes, however, the United Nations defines ‘youth’ as persons between the ages of 15 and 24 years (https://www.un.org/en/sections/issues- depth/youth-0/). Today, there are 1.2 billion young people aged 15 to 24 years, accounting for 16 per cent of the global population. By 2030—the target date for the Sustainable Development Goals (SDGs) that make up the 2030 Agenda—the number of youth is projected to have grown to nearly 1.3 billion. The realization of targets in the areas of quality education (SDG 4) and employment/decent work (SDG 8) have been singled out as fundamental to overall youth development. 8 Training programs make up about 79 percent of over 600 cases included in the World Bank’s Youth Employment Inventory database. Traditionally, OECD countries and those in Latin America have been the main implementers of these youth training programs, but countries in other parts of the world, including Sub-Saharan Africa and Asia, are increasingly adopting them in response to a large youth bulge, high dropout rates at the secondary level, and a mismatch between the education and skills profile of their out-of-work youth and young adults and what employers are looking for. 9 World Bank 2019. 10 Ziderman 2016. 11 Rajasekaran and Reyes 2019. Cho et al. (2013) note that several published evaluations of job training programs report that a large fraction of beneficiaries fail to show up or discontinue training after a short period. Heckman et 6 8. More recently, training programs have paid more attention to the issue of incentives, perhaps influenced by a general shift towards more output-oriented approaches in service delivery that prioritize what was achieved with the allocated funding. 12 One of the more popular of these output- oriented approaches is Results-Based Financing (RBF), which makes funding available only on the achievement of predefined results.13 The rationale is that by linking financing directly to the achievement of specific processes, outputs, or outcomes, stakeholders will be more incentivized to behave in ways that improve program effectiveness. RBF has been extensively applied in the K-12 education sector, albeit with mixed results (Box 1). To our knowledge, there has been no systematic review of its effectiveness in the area of short-term education and skills training. Box 1. Results-Based Financing in K-12 Education The available evidence on the use of RBF in K-12 education suggests that cash incentives for teachers may improve their attendance and their students’ learning, albeit the design of the incentive scheme and the context matter. The effects are larger and more positive in developing country contexts. The evidence base for the effectiveness of performance-based grants for schools is more limited. In some cases, these grants can work, especially when combined with other interventions such as capacity building (for example, for principals and school committees) or when money is spent on inputs that affect learning outcomes. The bulk of the evidence on RBF in K-12 education pertains to financial incentives for students and families, such as conditional cash transfers (CTT) or vouchers. These have a good track record of reducing school dropout and increasing school attendance. One of the first CCT programs was PROGRESA in Mexico, which resulted in less grade repetition, better grade progression, lower drop-out rates, and higher school re-entry among dropouts (Behrman, Sengupta, and Todd, 2005; Schultz, 2004). In Colombia, two voucher schemes, namely PACES and Familias en Acción, randomly assigned vouchers covering half the cost of secondary school in exchange for adequate academic progress (although the requisites were so low that “progress” essentially meant school attendance). It was found that students who received the voucher were more likely to finish eighth grade and scored higher on achievement tests (Angrist, Bettinger, and Kremer, 2006; Angrist, Bettinger, Bloom, King, and Kremer, 2002). In Bangladesh, a stipend for girls in secondary school substantially increased their enrollment, particularly in rural areas (World Bank, 2001). The evidence on financial incentives offered for academic performance is rather mixed. A meta-analysis by Snilstveit et al. (2015) shows that providing financial incentives does not necessarily improve learning outcomes. Fryer (2010) conducted randomized experiments in public schools in four urban districts—Chicago, Dallas, New York City, and Washington—during 2007–2008 and 2008–2009 by creating variations in educational inputs (like attendance or reading a book) or outputs (like grades) and how often and how much students were paid. A surprising finding was that financial incentives offered for educational outputs, such as better grades, were less effective than incentives for educational inputs, such as attendance, good behavior, or wearing uniforms. al. (2000), reviewing five different experimental evaluations of employment and training programs in the U.S., report drop-out rates as high as 79 percent. The Card et al. (2011) Dominican Republic study reports that 17 percent of their treatment group failed to attend training. 12 https://www.oecd.org/dac/peer-reviews/Results-based-financing-key-take-aways-Final.pdf. 13 Results for Development Institute 2016. 7 9. While short-term training programs for out-of-school youth are expensive, and governments are increasingly under scrutiny to account for the money spent on them, the degree to which they have been formally evaluated is surprisingly limited. 14 In fact, outside of the OECD, and other than studies sponsored by international organizations, rigorous evaluations are rare. 15 The general lack of evaluations is concerning for many reasons, including that it represents a missed opportunity to understand the effects of different designs on program outcomes. B. Research Questions and Methodology 10. This review takes a closer look at some of the available evidence on short-term education and skills training programs for out-of-school youth in order to better understand which deliver the best results. The review examines a limited, yet representative, array of short-term training programs in North and South America, Europe, Africa, and Asia for which impact data were available and addresses three main questions: • What kinds of short-term education and skills training programs for out-of-school youth have been implemented around the world and what is the evidence on their effectiveness? • How do these programs incentivize participants and providers, and what do we know about the relative effectiveness of those that use an RBF approach? • What are some implications for the future design of these programs? The findings should be of particular interest to those involved in designing, implementing, or evaluating short-term training programs for out-of-school youth as well as those involved in education or skills training programs in general. 16 11. The main methodology used to answer the three questions was a desk-based review of the global literature on short-term education and skills training programs for out-of-school youth. The bulk of 14 Betchermann 2007. 15 For example, Cho et al. (2013) note that only around 10 percent of evaluations in the developed world have used randomized controlled trial methodology. 16 For example, we hope that this paper will help to inform the design of short-term training programs in India, a place where all three of the authors of this paper currently work. The number of training programs for out-of-school youth has expanded considerably over the last decade. There are now around forty different national schemes for skills training implemented by eighteen different Ministries and Departments of the Government of India. Traditionally, these programs have relied on supply-driven financing focused on inputs. However, newer programs have a stronger focus on results, requiring training providers to focus on establishing links with potential employers and to offer education and skills training that match labor market demand. An example of one of these newer, results-oriented programs is the Nai Manzil (New Horizons) Scheme, which was launched in 2015 to support second- chance education and skills training for out-of-school religious minority youth. The scheme funds results-based contracts between the Ministry of Minority Affairs and private education and skills training providers across India. Payments to providers are based on the achievement of key performance milestones, including (i) the number of youth enrolled in open schooling; (ii) the number obtaining upper primary or secondary education certification through open schooling; (iii) the number obtaining skills training certification; and (iv) the percentage placed in jobs. 8 the literature search was conducted between March and September 2019, with additional searches conducted between April and June of 2020. 12. The main criteria used to select studies were: a) it should be a short-term program that uses education and/or skills training to improve the educational outcomes or workforce participation of out-of-school youth, preferably those between 15 and 24 years who did not complete secondary schooling; b) there should be reported data on program outputs and outcomes; and c) there should be evidence of program impact through the use of treatment and control groups (preferably in the form of a randomized control trial 17). 13. Out of an initial group of 30 short-listed studies, only ten ended up meeting all three of these selection criteria. 18 These included youth training programs in Brazil, Colombia, the Dominican Republic, Liberia, Nepal, Uganda, the United Kingdom, and the United States, many of which are among the most widely quoted programs in the literature. 14. Because none of the selected studies was from East Asia and the Pacific, the treatment/control group requirement (criterion (c) in the above list) was relaxed for studies from that region. Twenty- six programs were initially identified using this approach. However, after further checking, including whether the program had sufficient information on outcomes, the twenty-six were winnowed to five programs from China, East Timor, the Philippines, and Timor Leste. These studies were analyzed separately from the ten that met criteria (a), (b), and (c). 19 15. Limitations of this review include the possibility that relevant studies may have been omitted, either due to not being picked up by the literature search, or due to publication bias, whereby studies with null findings or that duplicate earlier findings may not be published or receive as wide a dissemination. In addition, this review does not do justice to the extensive literature on motivation or incentives, although it recognizes the relevance of many of the findings from this literature for the future design of training programs for out-of-school youth (Box 2). Instead, the approach is more of an inductive 17 A randomized control trial (RCT) is a type of research study in which participants are randomly assigned to one of two (or more) interventions. The RCT is regarded as the gold standard for evaluating the effectiveness of interventions. A sample of the population of interest is randomly assigned to one or another intervention and the two groups are followed for a specified period of time. Apart from the interventions being compared, the two groups are treated and observed in an identical manner. At the end of the study, the groups are analyzed in terms of outcomes defined at the outset. As the groups are treated identically apart from the intervention received, any differences in outcomes are attributed to the intervention. 18 The starting point for this review was the reference lists of existing systematic reviews of the literature on active labor market programs as well as online searches of Google Scholar, the World Bank Open Knowledge Repository, and the Solutions For Youth Employment website and project repository (https://www.s4ye.org/). From these sources, we culled studies pertaining specifically to short-term training programs for out-of-school youth. This resulted in a set of 30 potential studies. 19 East Timor and Timor Leste refer to the same country. However, in this paper the two names are used to distinguish between two programs, East Timor: Prepara Ami ba Serbisu (PAS)/Preparing Us for Work and Timor Leste: A Second Chance at Education. 9 analysis of existing short-term training programs to identify and describe what is currently being done to incentivize targeted youth on the one hand and providers on the other. Future research is needed to make more explicit connections between these current practices and the theoretical and empirical literature on motivation and incentives. Another limitation of this review is the huge challenge in trying to make any kinds of comparisons across short-term skills training programs given the immense heterogeneity of participant profiles, program characteristics, and outcomes targeted and measured, as well as the fundamental role that contextual factors play (economic and social) in mediating program outcomes. The authors have tried to reduce some of this heterogeneity by focusing in on training programs specifically meant for out-of-school youth between the ages of 15 and 24 years, but nonetheless considerable heterogeneity remains. Box 2. Some key findings from the literature on motivation and incentives While this paper does not delve deeply into the extensive literature on motivation and incentives, it is worth noting some of the broad conclusions from this literature that have particular relevance for training programs for out-of-school youth. One of the key concepts from this literature is Self-Determination Theory, which aims to explain individuals’ goal-directed behavior (Deci & Ryan 1985; Ryan & Deci 2000). It notes that people feel motivated by activities that allow them to satisfy three universal needs: (i) Competence (self-belief in one’s ability to perform well in an activity); (ii) Autonomy (when individuals feel they are engaging in an activity because they choose to do so, not because they are being pressured by other people or external factors); and (iii) Psychological relatedness (people are motivated by activities which allow them to form and enjoy good relationships). Originally, Self-Determination Theory differentiated between intrinsic and extrinsic motivations. Later, a continuum with varying degrees of individual autonomy was proposed (Deci & Ryan, 2000). Intrinsic motivation represents the most autonomous behavior regulation by inherent interest, enjoyment, and satisfaction. Extrinsic or controlled motivation characterizes those activities that yield specific outcomes in terms of rewards or avoided punishments whereas perceived autonomy is low. Incentive Theory argues that behavior is primarily extrinsically motivated: people are more motivated to perform activities if they receive a reward afterward, such as money or a promotion, rather than simply because they enjoy the activities themselves. Social and emotional incentives like praise and attention are also extrinsic motivators since they are bestowed on the individual by another person. Some studies have shown that the effectiveness of extrinsic motivators varies depending on factors like self-esteem, locus of control (the extent to which someone believes they can control events that affect them), and self-efficacy (how someone judges their own competence to complete tasks and reach goals). For example, praise might have less effect on behavior for people with high self-esteem because they would not have the same need for approval that would make external praise reinforcing. On the other hand, someone who lacks confidence may work diligently for the sole purpose of seeking even a small amount of recognition. Meta-analyses of the research have come to different conclusions on whether and how external rewards or incentives might undermine intrinsic motivation. Cognitive Evaluation Theory (Deci & Ryan, 1985) predicts that rewards only reduce intrinsic motivation when they are perceived as controlling the behavior and might actually enhance intrinsic motivation if they engender feelings of competence. In line with this 10 prediction, no reduction of intrinsic motivation is found if the reward is not tangible, for example, if it comprises verbal praise, and if it is unexpected (Cameron & Pierce, 1994; Deci et al., 1999). C. Findings (i) What kinds of short-term education and skills training programs are being implemented around the world and what is the evidence on their effectiveness? 16. The ten training programs for which rigorous impact data exist cover South and North America, Europe, Africa, and Asia (see Table 1; additional details are provided in Annex 1). Eight of the ten programs were implemented between 2000 and 2015. The other two were implemented in the 1980s (United Kingdom) and 1990s (United States) and represent landmark early programs in this area that inspired subsequent initiatives. 17. The beneficiaries of these programs have varied profiles. The age range runs from 16 years to adult, with most programs setting an upper limit of 30 years. This cap goes beyond the typical definition of youth (15-24) but is a response by governments to the need to reach those who still have many potential years of productive employment before them. Beneficiaries typically come from poor urban or minority neighborhoods or meet a minimum income requirement. They tend to be unemployed or underemployed. While many are high-school dropouts, beneficiaries can range from functionally illiterate to possessing basic literacy and numeracy skills. Most programs cater to both male and female youth. 18. The ten programs differ in the degree to which they emphasize education, skilling, or both. The Galpão Aplauso Program in Brazil offers both education and skills training. Other programs, like the Economic Empowerment of Adolescent Girls and Young Women Project in Liberia, do not have an explicit education component and instead focus on providing business/jobs and general life skills training. Still others, like the Youth Opportunities Program in Uganda, take a more unstructured approach in the form of direct cash grants to youth groups that can be used for a variety of education and skills training needs according to their preference. 19. Among the programs with a strong skills-training emphasis, the standard approach is a combination of classroom-based training followed by work experience. For example: • Colombia’s Youth in Action Program offers three months of classroom-based skills training and three months of on-the-job training; • The Youth and Employment Program in the Dominican Republic divides its training into 75 hours (about three weeks) of basic or life skills training and 150 hours (about six weeks) of technical or vocational training, followed by an internship in a private sector firm; • Liberia’s Economic Empowerment of Adolescent Girls and Young Women (EPAG) Program provides six months of classroom-based training in job skills or business services and six months of job placement support for either self- or wage-employment. 20. The decision to include an education component in these training programs is typically linked to the desire to reach more vulnerable youth. Programs with a skills-only emphasis tend to assume that youth already have a basic level of literacy and numeracy skills. This effectively excludes more 11 vulnerable youth who need support in these basic areas before they can benefit from skills training. In recognition of the potential exclusionary effects of a sole emphasis on skilling, Liberia’s EPAG Program, which originally focused only on skills training, subsequently expanded to also offer education training (basic literacy and numeracy skills) so that less educated youth could benefit from the program’s offerings. It is important to point out that the education components of these short- term training programs do not always have the same expected outcomes. While some are meant only to provide informal remedial support, others lead to formal certification, and some even allow the beneficiary to transfer credits towards higher education. 21. Among the programs with a combined skills-and-education focus, the typical approach is to start with education (usually basic skills such as reading, writing, and math), followed by skills training and job placement or work experience. Work experience is usually provided by partner firms through paid or unpaid internships, or through on-the-job training. Job placement support is provided through a range of mechanisms – from aligning programs to the needs of local employers, to formal or ad-hoc alliances with private sector actors for employment, internship, and beneficiary sponsorship. Different programs combine these elements in different combinations: • The Galpão Aplauso Program in Brazil is a six-month program (five hours a day, five days a week) that teaches academic and basic skills (including remedial courses in mathematics and Portuguese), life skills, and vocational skills. It also provides placement services, with most program graduates being placed in jobs in the area of manufacturing and construction. • The Year Up Program in the United States provides enrollees with instruction and assignments that improve the skills needed in college, including reading and writing, along with six months of training in the fields of information technology and investment operations. This is followed by a six-month internship with companies that are corporate partners of the program. Students also receive assistance with the job search process and with college enrollment. • The Job Training Partnership Act in the United States provides for a six-month program of either basic education (leading to the General Education Development test, which provides a high school equivalency credential) or classroom training in occupational skills, combined with on-the-job training in private firms and job search assistance. 22. Regardless of whether these programs emphasize education, skilling, or both, their underlying theory of change is that the lack of certain skills and qualifications (whether technical or academic in nature) is directly related to the poor labor market and life outcomes of the targeted youth, and that short-term courses will suffice to address these gaps. There also is an assumption that the economy has vacancies that can be filled by graduates of these programs or that tertiary education providers will recognize graduates’ education experience or qualifications and admit them for further studies. In line with this logic, the key indicators used to evaluate the success of these programs primarily focus on employment and earnings, and to a lesser extent, socio-emotional or behavioral skills, college attendance, and the acquisition of college credits or degrees. 23. The available impact data (see Table 1 and also Annex 1) suggest that this theory of change holds up in some respects. All ten programs reviewed for this study generated some short-term positive impacts on their target beneficiaries in the areas of employment or earnings, and to a lesser extent, 12 socio-emotional and behavioral skills. The effects on future education attainment or college attendance were more muted. Of the four programs that included an explicit education component, only two showed clear positive impacts on education outcomes (I-BEST and JTPA). It should not be surprising that the programs all showed some positive impacts since they embody several of the key ingredients of successful programs as identified by the literature in this area, such as such as being responsive to local market conditions; having a multi-component design (education and training, career-related guidance, life and leadership skills); and making strong efforts to place people in jobs or work-based learning activities. 20 24. The benefits to female enrollees typically exceed those for males. For example, the Youth in Action Program in Colombia and the AGEI in Nepal had larger positive impacts on employment rates and earnings for women than for men. The Youth and Employment Program in the Dominican Republic and the Youth Opportunities Program in Uganda also demonstrated stronger positive impacts for women’s earnings as well quality of jobs. In the United Kingdom, women were more likely than men to obtain a good job after graduating from the Youth Training Scheme. One of the explanations given for this greater female effect is that women are usually starting from a lower base and thus have more room to grow by participating in these programs. 25. The benefits to trainees in developing countries typically exceed those for developed countries. For example, the impact of the JTPA in the United States and the Youth Training Scheme in the United Kingdom could be characterized as minimally, if at all, positive, while Liberia’s EPAG program and Uganda’s Youth Opportunities Program had life changing effects on participants. One explanation for this differential impact is that one can expect returns to be higher where the skill levels of the population are very low to begin with. 26. In general, the findings from these programs echo the broader literature on short-term training and active labor market programs, particularly in terms of the more positive effects in developing country contexts and for women. The findings also mirror this literature in terms of the tendency for positive impacts to grow over time, 21 albeit this finding is limited to three of the ten programs for which long- term tracking of graduates was carried out. Specifically, the evaluators of the EPAG Program in Liberia tracked graduates for 30 months after program completion and found that the program’s initial positive effects persisted at the same or higher levels over that time period. An analysis of the long- 20 Ibarraran and Rosas. 2009; Ivry and Doolittle. 2003. 21 Card et al.’s (2015) meta-analysis of active labor market programs finds that the impact of these programs increases over time, particularly for programs that focus on human capital accumulation. Card et al. summarize the estimates from over 200 recent studies of active labor market programs. They classify the estimates by type of program and participant group and distinguish between three different post-program time horizons. Using regression models for the estimated program effect (for studies that model the probability of employment) and for the sign and significance of the estimated effect (for all the studies in their sample), they conclude that: (1) average impacts are close to zero in the short run, but become more positive 2-3 years after completion of the program; (2) the time profile of impacts varies by type of program, with larger average gains for programs that emphasize human capital accumulation; (3) there is systematic heterogeneity across participant groups, with larger impacts for females and participants who enter from long term unemployment; (4) active labor market programs are more likely to show positive impacts in a recession. 13 term impacts of the Dominican Republic’s Youth and Employment Program on graduates six years after program completion found significant impacts on the formality of employment, particularly for men, and impacts for both men and women in Santo Domingo, the capital of the Dominican Republic. 22 Follow-up on Colombia’s Youth in Action trainees and their relatives for more than a decade after the program found positive long-term educational impacts, not just on program participants but also their relatives. 23 Table 1. Overview of training programs for out-of-school youth Program Beneficiaries Education/Skills training Results Brazil: Galpão Male and female youth 180 hours of academic and Employment: Increase in employment Aplauso (2009 – under 29 from households basic skills, including remedial prospects 4-5 months post program. 2013) with monthly incomes courses in Mathematics and Earnings: Increase in earnings 4-5 months post below minimum wage Portuguese. 300 hours of program. who were screened vocational training tailored to Personality traits: No change in performance through interviews and beneficiaries and 120 hours of on standardized psychological measures of math and language tests. life skills. Most placed in socio-emotional skills although anecdotal positions related to evidence suggests program youth display better manufacturing and performance in skills valued by labor market construction. (punctuality, responsibility). Colombia: Unemployed male and 3 months of classroom training Employment: Increase in employment for both Jóvenes en Acción female youth between 18 on vocational skills by public men and women, with larger effects for women. (Youth in Action) and 25 from lowest socio- and private training Earnings: Increase in earnings for both men and (2001 – 2005) economic strata (mostly institutions. On-the-job training women, with larger effects for women. Women high-school dropouts) in provided by legally registered offered training earn about 18 percent more urban areas. companies, which provide than those not offered training; men offered unpaid internships to training earn about 8 percent more than those participants in manufacturing, not offered training. retail and trade, and services. Dominican Male and female high Basic skills training to Labor market outcomes: No significant impact Republic: school dropouts between strengthen self-esteem and on overall employment. Positive impact on Juventud y 16 and 29; living in poor work habits; vocational training quality of employment for males and on Empleo (Youth neighborhoods; to address needs of local monthly earnings for women. and Employment unemployed, employers. Program duration Youth behavior and life expectations: Effective Program) (2001 – underemployed, or not specified. in reducing teenage pregnancy. Positive impact Not specified) inactive; not holding an on beneficiaries’ perceptions of current identity card. situation and expectations regarding future; stronger for females and younger individuals. Socio-emotional skills: Positive impacts on measures of non-cognitive skills. Liberia: Economic Women aged 16-27 with 6 months of classroom-based Economic: Strong impacts on employment and Empowerment of basic literacy and training (Job Skills Track or earnings, particularly for Business Development Adolescent Girls numeracy skills, not Business Development Services Services Track. and Young enrolled in school, and Track) and 6 months of job Empowerment: Positive effects on access to Women (EPAG) living in one of nine target placement support for either and control over monetary resources. (2010 – 2014) communities. self- or wage-employment. Household: Improved food security and shifting attitudes to gender norms. 22 Ibarraran et al. 2015. 23 Kugler et al. 2015 (updated in 2019). 14 Nepal: More than 4000 young Basic life skills, Employment and earnings: Positive impact on Adolescent Girls women between the ages technical/vocational skills, and employment rates and earnings; courses in Employment of 16 to 24 over 3-year basic business skills. Program electronics, beautician services and tailoring Initiative (AGEI) period. Carried out as part duration not specified. were most effective. Impact larger for women. (2010-2012) of broader Employment Empowerment and self-confidence: No Fund Project (2008 – Not consistent impact on empowerment; only specified). significant impact on control over earnings for 2011 cohort. Risky behaviors: Few impacts on reproductive health Impact on household: Few impacts on household level outcomes like remittances received by households of trainees. Uganda: Youth Poor underemployed male Cash grants offered to groups Earnings: Groups assigned to grants had higher Opportunities and female youth, aged for self-employment. Part could earnings. Incomes of treatment women were 73 Program (YOP) 16-35, from Uganda's be used for investment in percent greater than control women, compared (2006 – 2008) conflict-affected North. training. 68 percent of the to a 29 percent gain for men. Many were rural farmers treatment group enrolled in Capital/assets: Groups assigned to grants had and functionally illiterate. vocational training. greater capital stocks. Tendency to practice skilled trade: Groups assigned to grants twice as likely to practice a skilled trade. United Kingdom: 16- to 17-year-old Apprenticeship training in Duration to any job: Participants took longer to Youth Training unemployed youth industry. Program duration not find employment than those who were Scheme 1983 – specified. unemployed at the start of survey period. 1989) Duration to good job (involving training/apprenticeship and falling above a low pay threshold): Women had increased chance of a good job; no such effect for men. United States: Adult basic education Enables basic skills students to Educational outcomes: Positive impacts on Integrated Basic students enrolling in enroll directly in college-level education outcomes (except persistence). As Education and career and technical coursework. Basic skills compared to baseline, students were more Skills Training education offered by instruction may include Adult likely to earn college credits, and attain Program (I-BEST) community and technical Basic Education or General certificate or degree. (2004 – Not colleges. (Adult basic skills Education Development Labor market outcomes (change in wages and specified) students are adults who programs. College-level hours worked): No significant effect found on lack high school-level occupational courses taught by wages and average quarterly hours worked. skills) basic skills instructor and professional-technical faculty member. Program duration not specified. United States: Unemployed individuals Basic education leading to the Earnings: No significant increase in earnings; Job Training (above age 16) who met General Education modest positive impact on incremental Partnership Act income eligibility limits or Development exam that earnings (difference between earnings of (Title II-A) had low family income in provides a high school treatment and control groups) for adult men (1982)/Workforce the 6 months preceding equivalency credential. and women. Negative impact on earnings for Investment Act application to the Classroom training in male youth. (1998) program. occupational skills combined Educational attainment: Positive impact on with on-the-job training in educational attainment for female youth, adult private firms and job search women, and adult males who were school assistance. Program duration dropouts. not specified. Welfare receipt: No significant impact on reducing welfare dependence. 15 United States: Young male and female Instruction and assignments Earnings: Students experience larger earning Year Up (2000 – adults aged 18 to 24 from that improve skills needed in gains after year in labor market compared to Not specified) low-income urban college, including reading and control group. communities. writing. Students receive College attendance: Program participants just credits they can transfer to as likely to enroll in postsecondary education as more than 2,000 higher control group. education institutions for degree programs. 6 months of training in information technology and investment operations fields followed by 6- month internship with companies that are corporate partners of program. 27. Most of the training programs shown in Table 1 are government-funded, although in at least three instances (Colombia, Liberia, and Nepal), the original funding source is a loan or grant from an external development agency. The Year Up Program in the United States has the most diverse funding base, and includes support from private foundations and corporations, individual contributions, public funds, and contributions from internship partners. In the case of Brazil’s Galpao Aplauso Program, the program began as a publicly funded venture, with a large outreach, but changed to a private-sector oriented strategy that was much more selective in its approach, including in its targeting of beneficiaries. This suggests a possible link between program funding sources and the ability to target more vulnerable youth that may be worthy of more study. 28. Program costs were available for five of the ten programs (Table 2). Costs varied due to multiple factors, including the years in which the program operated, program length, and the cost of living in a particular country. Where cost-benefit data are available, they indicate a reasonable return on investment over a medium- to longer-term horizon (roughly five to six years). Some of the documentation on these programs also indicates that costs tend to be higher when a program is being piloted or in the early years of implementation and tend to go down as service providers gain more experience and as learnings from impact and other evaluations are incorporated into the program design. 24 Table 2. Program costs Program Overall cost per Other cost information youth Brazil: Galpão Aplauso USD 2,225 The average cost per youth is R$ 810 (USD 385) a month, or R$ 4,680 (USD 2,225) for the entire curriculum. Colombia: Jóvenes en USD 875 Depending on the assumptions used, the Internal Rate of Acción Return varies from 13.5 to 25 percent for women and 4.5 to 16 24 For example, after tracking Colombia’s Youth in Action trainees and their relatives for more than a decade and finding positive long-term educational impacts, Kugler et al. (2015; updated in 2019) concluded that accounting for these long-term impacts dramatically increases the program’s Internal Rate of Return (IRR). Focusing only on labor- market outcomes and not accounting for these long-term tertiary education impacts on participants substantially understated the social desirability of the Colombian program. By contrast, including tertiary education impacts on participants increases the program’s IRR for women from 22.2 to 23.5 percent and for men from 10.2 to 20.5 percent. 16 percent for men. Cost-benefit analysis of these results suggests that the program generates a large net gain, especially for women. Dominican Republic: USD 700 The program cost would be recovered in about 50 months Juventud y Empleo (average income of control group was about USD 130). Liberia: Economic USD 1200-1350 Average monthly increase of USD 11 for Job Skills and USD 44 for Empowerment of (Business Skills Business Skills. It would take 12 years to recover the costs for the Adolescent Girls and Young training) and USD former and just 3 years for the latter. Women 1650 (Job Skills training) Uganda: Youth USD 382 The program could be paid back in 4.7 years at a real interest Opportunities Program rate of 15% if it were a loan and in 7.3 years at 25%, assuming the full earnings increase went to repay it. 29. As previously mentioned, given the lack of impact evaluations for youth training programs in East Asia and Pacific countries, it was decided to look at the findings from other kinds of evaluations of these programs in order to gauge whether outcomes were similar to the ten programs discussed above. A list of five youth training programs for which outcomes data exist is shown in Annex 2. These programs cover China, East Timor, the Philippines, and Timor Leste. Similar to the ten programs already discussed, reported outcomes were generally positive, particularly for women (East Timor, the Philippines). This included participants using the skills they acquired through the program to generate income through self- or wage employment (China, East Timor, the Philippines); increased confidence levels (China); participants returning to formal schooling or further training after completing the program (East Timor, Philippines); and participants gaining academic credentials (Philippines, Timor Leste). At the same time, the reported data tended to be more qualitative and short-term in nature, with a focus on outcomes immediately at, or shortly after, graduation from the program. There also was no real analysis of cost, making it difficult to determine if the program represented value for money. The exception to this was the Alternative Learning System Program in the Philippines, which estimated the net economic gain from participating in the program and concluded that only relatively young enrollees (under 26) were likely to garner an increase in future earnings that exceeded the income they did not earn while participating in the program. (ii) How do these programs incentivize participants and providers, and what do we know about the relative effectiveness of models that use an RBF approach? 30. Ideally, youth training programs should be designed so that providers and beneficiaries are incentivized to achieve the desired results at each stage. On the beneficiary side, the basic content and goals of the program should be sufficiently attractive that they are motivated to enroll. This, however, is usually not enough to ensure that they will also complete the program given that these programs tend to target more disadvantaged youth whose ability to engage on an ongoing manner is likely to be compromised by economic, social, and other pressures in their environment. The program design needs to anticipate these pressures and address them in ways that help engage and retain these youth. Another challenge is how to ensure that providers are incentivized to enroll the neediest students, offer high-quality education and skills training, and encourage students to attend and complete. This is a significant task given that providers may have competing motivations to enroll less 17 needy students who are easier to educate/train, offer cheaper courses, and not invest in resources to support students at risk of dropping out. 31. In order to understand how training programs deal with these issues, the ten programs included in this study were analyzed to identify whether and how providers and beneficiaries were being incentivized at different stages of the program. Incentives are defined as specific aspects of the program design that directly address the pressures and competing motivations mentioned above that might lead a provider or trainee to be less motivated to focus on achieving desired program results. Incentives were catalogued for three key program stages: (i) beneficiary recruitment/enrollment; (ii) beneficiary attendance/completion; and (iii) beneficiary employment/post program. The results are shown in Table 3. The use of incentives at a particular stage of a program is indicated by a tick mark. Tick marks that appear in shaded cells represent the use of RBF as an incentive. Table 3. Use of incentives at difference program stages Program Recruitment Attendance Employment Training Trainee Training Trainee Training Trainee Provider Provider Provider Brazil: Galpão Aplauso Dominican Republic: Juventud y Empleo Colombia: Jóvenes en Acción Liberia: Economic Empowerment of Adolescent Girls & Young Women Nepal: Adolescent Girls Employment Initiative Uganda: Youth Opportunities Program United Kingdom: Youth Training Scheme United States: Integrated Basic Education & Skills Training Program United States: Job Training Partnership/Workforce Investment United States: Year Up Note: A tick indicates use of an incentive. A shaded cell indicates that the incentive was RBF. 32. Apart from Brazil’s Galpão Aplauso program, all of the programs used some form of incentive at one or more of the key stages. The Dominican Republic and Nepal used incentives for training providers or beneficiaries at all three stages of program implementation. Looking across the ten programs and three stages, it appears that training programs mainly provide incentives to beneficiaries during the program itself (i.e., the attendance/completion phase), as a way to retain them, whereas they mainly provide incentives to training providers at the beginning and end of the program (i.e., the recruitment/enrollment and employment/post program phases), as a way to influence the types of beneficiaries they enroll and the results they achieve. 18 33. Five of the ten programs used some form of RBF (the shaded cells in Table 3 and the bolded text in Tables 4-6). Of the ten programs reviewed, those in Liberia, Nepal, and the United States (JTPA) had the strongest RBF elements (see Table 3 and bolded text in Tables 4-6), and these were mainly used to direct the behavior of the training providers. 25 However, while all three of these programs achieved good outcomes (see Table 1 and Annex 1), so too did programs with apparently weaker links between funding and results such as the Galpao Aplauso Program in Brazil, the Youth Opportunities Program in Uganda, and the Year Up Program in the United States. Since none of the ten programs treated incentives as a design variable or collected data on the effectiveness of the different kinds of incentives used, it is difficult to draw any firm conclusions about the relative effectiveness of RBF. Given the available evidence, it may be safest to conclude that RBF is at least as effective as existing alternatives. Despite this, there is a lot to learn from looking at the different types of incentives used by the ten programs. Further details are provided below, organized according to the three program stages. Recruitment/enrollment stage: 34. The primary aim at the recruitment/enrollment stage of the program is to incentivize training providers to recruit youth from the target populations and to incentivize youth from these populations to enroll (Table 3). Usually, providers have undergone a screening/competitive selection process prior to the start of the program and are on board with the general program design and goals. However, given that they will be accountable for the training, and possibly also placement, of their enrollees, they typically are given the freedom to select the candidates, or a pool of potential candidates, from the target population. This provides them with some sense of control over who they will be expected to train, which in turn makes them more likely to be motivated to train these individuals. For example, training providers for the Dominican Republic’s Youth and Employment Program were allowed to identify a pool of potential enrollees from which the final participants were selected. Similarly, training providers for Colombia’s Youth in Action Program, were allowed to select who they wanted to train from the applicant pool.26 35. This freedom to choose potential trainees needs to be balanced, however, against the need for providers to enroll more disadvantaged youth from the target population. These more disadvantaged youth tend to be more difficult to train, making many providers reluctant to take on the challenge. Nepal’s approach to addressing this issue involved allowing providers to recruit and select their own trainees according to the program’s guidelines but applying a differential pricing formula that awarded higher financial incentives to training providers who agreed to train youth from 25 RBF ties a high-value reward – money – to desired results. As mentioned previously, these desired results are typically in the form of targets to be achieved in key process, output, or outcome areas. What differentiates the RBF modality from other approaches is that the disbursement of program funds is contingent on the achievement of these specific, predefined results (i.e., if you do not achieve the result, you do not receive the funding). The belief is that this will incentivize the provider or the beneficiary to behave in a certain way so that the desired results are more likely to be achieved. 26 These private training institutions were also allowed to decide what courses were offered, and how they were marketed and designed. 19 more disadvantaged groups. Similarly, under the USA’s JTPA Program, while the selected training centers were given a monopoly on providing JTPA-funded services in their areas, many states created financial incentives to encourage centers to offer their services to needier groups within their catchment area. No evidence is provided, however, on the impact of these approaches on either programs’ ability to enroll more youth from these desired groups. 36. On the trainee side, issues that might make them less inclined to enroll include a lack of confidence 27 in their ability to succeed in the program, economic or cultural barriers, and lack of family support. Many programs address the economic barriers by providing a stipend to participants during the attendance stage. The promise of such an allowance can be a major incentive to sign up. Cultural barriers or lack of family support can be particular issues for female youth and may require the development of informational messages and recruitment strategies specifically targeted towards them. For example, Nepal’s AGEI Program struggled to recruit young women in the first year of operation, but thereafter implemented a specialized communications and outreach strategy to sensitize young women and their families to the program. Radio and television advertisements also were used to incentivize women to sign up for the more non-traditional trades being offered under the program (the logic being that these were more lucrative than the trades women typically trained for). In addition, the program partnered with community-based organizations to attract applications from women. If a referred applicant gained entry to a sponsored training course, the partner organization was paid a small finder’s fee. 37. The Uganda Youth Opportunities Program is an interesting case because all incentives under the program were directed towards applicants and focused on the recruitment/enrollment stage of the program. The incentives were in the form of an upfront cash grant to applicants that was not contingent on the achievement of specific results. Instead, the program designers used social accountability mechanisms to increase the probability of achieving the desired program results. This rested on the requirement for people to apply for a cash grant in groups (typically village- or community-based) and to submit group plans for what they planned to do with the money.28 In the absence of formal monitoring, officials correctly surmised that groups would be more likely than individuals to implement their proposals given the inbuilt social pressures and accountability. 27 Lack of confidence can be a debilitating issue for youth who have been out of school for a long time and are unused to studying. Liberia’s EPAG Program tried to build confidence during program implementation by instituting a system of coaching and peer support. Girls received emotional and learning support throughout the program from volunteer coaches who attended the classroom training and followed up with girls in their communities during the placement phase. Girls also were organized into peer groups for social and learning support. It’s unclear to what extent these design features were communicated to girls during the recruitment stage (and hence may have affected their decision to enroll), but they reportedly contributed to participants’ feelings of increased confidence by the end of the program in comparison to the control group. 28 Villages typically submitted one application. Village officials passed applications up to districts, which verified the minimum technical criteria (such as group size and a complete proposal) and were supposed to visit projects they planned to fund. To minimize the chances of corruption the central government also sent out audit teams to visit and verify each group. The government disqualified about 70 applications out of 625, mainly for incomplete information or ineligibility (e.g. many group members over 35 years, or a group size more than 40). 20 Table 4. Recruitment/enrollment stage Program Training provider/facilitator Trainee Brazil: Galpão Aplauso • None specified. • None specified. According to interviews with program administrators, cohorts are over-subscribed and the program does not need to carry out information or enrollment campaigns. Demand for the program is produced by word-of-mouth, relying mainly on former participants, teachers, and private sector partners to announce the opening of each of the cohorts. Dominican Republic: • Training providers are allowed to identify • None specified. Juventud y Empleo a pool of potential enrollees for their programs. Colombia: Jóvenes en • The private training institutions decide • None specified. Acción what courses are offered, how they are marketed, and how they are designed. They also are allowed to select who they want to train from the applicant pool. Liberia: Economic • None specified. • None specified Empowerment of Adolescent Girls and Young Women Nepal: Adolescent Girls • The chosen providers can recruit and • There is a specialized communications and Employment Initiative select their own trainees according to the outreach strategy to sensitize young program’s guidelines. women and their families to the program. • There is a differential pricing mechanism Radio and television advertisements are that awards higher financial incentives used to encourage women to sign up for to training providers who agree to train more non-traditional trades. more disadvantaged groups. • The program organizers partnered with community-based organizations to attract applications from women. If a referred applicant gained entry to a sponsored training course, the partner organization was paid a small finder’s fee. Uganda: Youth • None specified • Successful proposals received one-time Opportunities Program unsupervised grants worth USD 7,500 on average—about USD 382 per group member, roughly their average annual income – with no government monitoring thereafter. • People had to apply as a group. In the absence of formal monitoring, officials believed that groups would be more incentivized to implement proposals. United Kingdom: Youth • None specified • None specified Training Scheme United States: Integrated • None specified • None specified Basic Education and Skills Training Program (I-BEST) United States: Job Training • Selected training centers were given a • None specified Partnership/Workforce local monopoly on providing JTPA-funded Investment Act services. 21 • Many states added additional measures that provided financial incentives targeted at services for particular groups within the JTPA-eligible population. United States: Year Up • None specified. • None specified Note: Bolded text in the ‘Training provider’ or ‘Trainee’ columns indicates the use of RBF as an incentive. Attendance/completion stage: 38. At this stage, the primary aim is to incentivize training providers on the one hand to deliver a high- quality training program and participants on the other hand to attend and complete the program (Table 4). Based on the ten programs reviewed for this study, the majority of incentives at this stage are focused on the trainees, the most common being the use of stipends to ensure that they attend and complete the program. Stipends help address attendance and completion through a couple of routes. First, they help cover the opportunity cost of students attending the training program. For many trainees, attending the program means giving up an existing source of employment. The lost funds, however small, need to be replaced in order to allow the trainee to cover living costs while they attend the training. Second, if payment is tied to program attendance or completion, it helps incentivize participants to attend on a regular basis. While stipends obviously have many advantages, the experience of Colombia’s Youth in Action Program illustrates the need to give careful thought to the optimal stipend amount and to consider potential differential impacts by gender or other key groups. In the case of Colombia’s program, the average monthly salary before training for women was 86,716 Colombian pesos or USD 42, while for men it was 124,647 Colombian pesos or USD 60.57. Women got a stipend of USD 2.20 daily under the program if they did not have children under seven years of age, and USD 3.00 if they had young children. Under either scenario, this more than compensated for the monthly salary they would have gotten without the training. For men, however, the daily stipend of USD 2.20 implied a monthly transfer of USD 44 so the opportunity cost due to the three months of lost employment was USD 16.57 per month or USD 49.71 for the entire period. This ‘lost income’ was seen as contributing to the slightly higher drop-out rate for men from the program. 39. The Year Up Program in the United States is an interesting case where stipends formed part of an overall incentives package for students that combined both rewards and punishments. Stipends provided to students under the Year Up Program helped cover their expenses while also providing an incentive to abide by the program’s attendance and other rules. Students had to maintain high attendance rates, be on time, and complete assignments in order to receive their full stipend. However, students who did not meet these expectations not only failed to receive the reward of a stipend, they also were punished by being ‘fired’ from the program. 29 It’s unclear, however, exactly how many students were actually fired as a result of failing to meet these expectations. 29 64 percent of the Year Up participants completed the program and graduated on time in July 2008. Graduates earned significantly higher wages than participants who dropped out of the program (USD 13.54 an hour versus USD 10.96 an hour). 22 40. In the case of the I-BEST community college program in the USA, the economic pressures experienced by students were assuaged not through the use of a stipend, but through a deliberate strategy of program administrators supporting students to apply for financial aid to cover the cost of their classes. Because of this, students enrolled in the program received financial aid at significantly higher rates than basic skills students enrolled elsewhere. There was no link between the achievement of particular I-BEST program results and access to financial aid by program participants, albeit this aid usually comes with its own stipulations and conditions. 41. Non-financial incentives are also frequently used to encourage trainees to attend and complete training programs. For example, in Liberia, in addition to attendance-linked stipends and a completion-linked bonus, students were incentivized to attend through the prevision of free childcare at every site, attendance prizes, and business plan competitions. Similarly, in the USA, JTPA training centers offered transportation, childcare, and clothing allowances to participants. 42. There seems to be less focus on incentives for training providers at this stage of the program. Where incentives do exist, they are focused on motivating providers to ensure that trainees complete the program. For example, in Colombia, training institutions were paid conditional on completion of training by the program participants while in Nepal, training and employment providers received 40 percent of their payment when the trainee successfully completed the program. There does not seem to be a specific emphasis on incentivizing providers to offer high-quality training although this may be dealt with through other means such as the criteria used to screen providers upfront. In the case of Liberia’s EPAG Program, quality of training was assured through the use of a robust monitoring and evaluation framework that included frequent and unannounced visits to ensure that service providers maintained a high-quality learning environment. Table 5. Attendance/completion stage Program Training Provider Trainee Brazil: Galpão Aplauso • None specified • None specified Dominican Republic: Juventud y • None specified • Participants receive USD 200 as a stipend for Empleo transportation and meals. Colombia: Jóvenes en Acción • Training institutions paid according • Stipend of about USD 2.20 per day to male to market prices and conditional on and female trainees without young children completion of training by the to cover transportation and lunch. About program participants. USD 3.00 per day for women with children under 7 years to help cover childcare expenses. Liberia: Economic Empowerment of • None specified • Every site offers free childcare. The costs are Adolescent Girls and Young Women budgeted directly into service provider contracts. • Participation is incentivized through attendance prizes, business plan competitions, etc. • Trainees receive small stipends contingent upon classroom attendance. • Girls who complete the training are awarded a small completion bonus (USD 20). 23 Nepal: Adolescent Girls • Training and employment • None specified Employment Initiative providers receive 40 percent of their payment when the trainee successfully completes the training. Uganda: Youth Opportunities • None specified • None specified Program United Kingdom: Youth Training • None specified • None specified Scheme United States: Integrated Basic • None specified • I-BEST students received financial aid at Education and Skills Training significantly higher rates than other basic Program (I-BEST) skills students. This is because I-BEST program administrators are encouraged to help their students apply for financial aid, since many of them have low incomes. United States: Job Training • None specified • Training centers offered special services, Partnership/Workforce Investment such as transportation, child care, and Act clothing allowance. United States: Year Up • None specified • Stipends help students cover some of their expenses while providing an incentive for abiding by the program’s attendance and other rules, which is required for students to receive the full amount. • Students must maintain high attendance rates, be on time, and complete assignments. Students who repeatedly fail to meet these expectations end up “firing themselves” from the program. Note: Bolded text in the ‘Training provider’ or ‘Trainee’ columns indicates the use of RBF as an incentive. Employment/post-program stage: 43. At this stage of the program, the primary aim is to incentivize training providers to place their graduates in (or support them in finding) suitable employment or further education or skills training opportunities (Table 5). At the same time, there is a need to incentivize graduates to seek out or take on these employment or entrepreneurial opportunities in a productive and sustained manner. Based on the ten training programs reviewed for this study, most of the incentives at this stage are focused on the provider side of the equation. 44. For example, in Liberia, performance bonuses were awarded to training providers for successfully placing graduates in jobs or micro-enterprises. Providers received the highest bonus for trainees who successfully completed the classroom training and had an ongoing business for at least 3 months after the training. In Nepal, training providers received the remaining 60 percent of their payment only if the graduate was gainfully employed. Trainees’ employment status was verified three months and six months after completion of the training. Upon verification, providers received a payment that was higher for trainees who were employed. The highest incentive was awarded for placing the most disadvantaged, and incentives were gradually lowered for less prioritized groups. 45. The JTPA Program in the United States is an interesting example of a cascade approach to incentivizing employment outcomes. Under the program, states set the rules that determined the amount of the payoff to training providers as a function of their performance relative to set standards 24 or to each other. Usually, the greatest weight was given to employment and wage rate measures. A center could receive nothing or, if it exceeded the standards, it could receive a bonus amounting to as much as 20 to 30 percent of its regular budget. At the same time, the federal government expected states to meet set goals under the program every year. A state that failed to meet its expected performance level for two consecutive years might be subject to a 5 percent reduction in its annual grant. If states met or exceeded expected performance levels, they were eligible to receive incentive grants that ranged from USD 750,000 to USD 3 million. 46. Some programs also give particular attention to incentivizing the employers who are supposed to hire program graduates. In the case of the Dominican Republic’s Youth and Employment Program, the vocational training component was explicitly designed to address the needs of local employers in order to incentivize them to offer internships to program participants once they graduated. In the United Kingdom, the co-operation of employers and young people under the Youth Training Scheme was secured by payment of an implicit subsidy, whereby trainees were paid an allowance (set slightly above the unemployment benefit level) and firms could employ trainees without incurring any further wage costs for a maximum of one year. Young people would gain from the work experience and training and increase the chance of permanent employment with that firm. Firms gained from cheap labor but also had a period to evaluate the productivity of trainees. In the case of the Year Up Program in the United States, the entire program design was built around the needs of its corporate partners as well as their commitment to sponsor and provide on-the-job training to the student interns. Many of these interns obtained regular jobs with these employers after program completion. Under this model, the incentives for training providers to produce job-ready candidates and for employers to take on these candidates were tightly aligned. 30 Table 6. Employment/Post Program Program Training provider/employer Trainee Brazil: Galpão Aplauso • None specified. ad hoc alliances with • None specified. ad hoc alliances with particular private sector actors, but particular private sector actors, but youth youth were mostly left on their own to were mostly left on their own to find jobs. find jobs. Dominican Republic: Juventud y • The vocational training is specifically • None specified Empleo designed to address the needs of local employers, making them more open to offering internships to program participants. • Training is followed by an internship in a private sector firm, which should be contacted by the training provider in order to develop training programs tailored to the firm´s labor demand. Colombia: Jóvenes en Acción • None specified • None specified 30 Overall, one quarter of the Year Up participants (44 percent of program graduates) were hired either by their internship employer or by another employer partner. The Year Up participants who were hired by the program’s employer partners were significantly more likely to work in information technology or investment operations positions than those who obtained jobs elsewhere (94 percent versus 12 percent, respectively). 25 Liberia: Economic Empowerment • Performance bonuses awarded to • None specified of Adolescent Girls and Young training providers for successfully Women placing graduates in jobs or micro- enterprises. Highest bonus for trainees who successfully completed the classroom training and have an ongoing business and regular account books for at least 3 months after the training. Nepal: Adolescent Girls • Training and employment providers • None specified Employment Initiative receive the remaining 60 percent of their payment only if the graduate is gainfully employed. Trainees’ employment status is verified three months and six months after completion of training. Upon verification, providers receive a payment that is higher for trainees who are employed. The highest incentive is awarded for placing the most disadvantaged, and incentives are gradually lowered for less prioritized groups. Uganda: Youth Opportunities • None specified • None specified Program United Kingdom: Youth Training • The co-operation of employers and • The co-operation of employers and young Scheme young people was secured by the people was secured by the payment of an payment of an implicit subsidy, implicit subsidy, whereby trainees were whereby trainees were paid an paid an allowance (set slightly above the allowance (set slightly above the unemployment benefit level) and firms unemployment benefit level) and firms could employ trainees without incurring could employ trainees without any further wage costs for a maximum of incurring any further wage costs for a one year. Young people would gain from maximum of one year. Young people the work experience and training and would gain from the work experience increase the chance of permanent and training and increase the chance of employment with that firm. Firms gain permanent employment with that firm. from cheap labor but also had a period to Firms gain from cheap labor but also evaluate the productivity of trainees. had a period to evaluate the productivity of trainees. United States: Integrated Basic • None specified • None specified Education and Skills Training Program (I-BEST) United States: Job Training • States were given latitude in setting • None specified Partnership/Workforce the rules that determined the payoff Investment Act to the providers as a function of their performance relative to the standards or to each other. Usually, the greatest weight was given to employment and wage rate measures. A center could receive nothing or, if it exceeded the standards, could receive a bonus amounting to as much as 20 to 30 percent of its regular budget. States are expected to meet set goals every year. A state that fails to meet its performance level for two consecutive years may be subject to a 5-percent 26 reduction in its annual grant. If states meet or exceed expected performance levels, they are eligible to receive incentive grants that range from USD 750,000 to USD 3 million. United States: Year Up • Year Up designed a curriculum that • None specified meets the needs of its corporate partners. It also obtains employer commitments to sponsor and provide on-the-job training to student interns, many of whom obtain regular jobs with their employers after program completion. Note: Bolded text in the ‘Training provider’ or ‘Trainee’ columns indicates the use of RBF as an incentive. 47. The five programs from East Asia and the Pacific were also reviewed to determine the kinds of incentives used at each stage. Surprisingly, only one of the programs – the Alternative Learning System Program in the Philippines – had any documented discussion of incentives. This program is implemented by two kinds of training providers – learning facilitators from the Department of Education and external contractors. According to the program evaluation, these providers were differentially motivated by the program design to deliver quality training. Specifically, learning facilitators in Department of Education-delivered programs had strong incentives to deliver high- quality work, even without frequent monitoring, because their careers at the Department were linked to their performance. Conversely, facilitators in Department of Education-procured programs, all of whom were external contractors, were not subject to the same institutional incentive structure and also did not seem to be subject to any other consequences related to their performance. As a result, they tended to deliver lower-quality training than the Department of Education employees. (iii) What are some implications for the future design of these programs? 48. The findings of this review indicate that a wide array of short-term education and skills training programs have been implemented around the world in response to the growing numbers of out-of- school youth. Of those programs for which impact data exist, the majority demonstrate a net positive impact on their beneficiaries, at least in the short-term and in developing country contexts, with somewhat less evidence available on the longer-term impact and overall cost efficiency. 49. In order to encourage desired behaviors or outcomes on the part of beneficiaries and providers, many training programs build incentives into their design. These take various forms, including RBF approaches that link funding to results as a way to enhance the efficiency and effectiveness of program spending. 50. This review concludes that while there is enthusiasm for the idea of RBF, there is currently limited evidence to indicate that RBF approaches are superior to others in the results that they generate. This is for two main reasons. First, most training programs tend not to undergo any kind of rigorous evaluation, and thus the pool of available evidence is very small. Second, even when these programs are evaluated, the evaluations typically are not designed to isolate the specific impact of RBF or other 27 incentives on program outcomes. Given what we know about the use of RBF from other contexts, it is highly likely that it is a more effective means of improving program outcomes, particularly when used in combination with other forms of motivation and incentives. For now, however, it may be safest to conclude that RBF approaches are as effective as existing alternatives when it comes to short- term training programs for out-of-school youth. 51. These findings suggest several recommendations for the future design of training programs for out- of-school youth. Five recommendations for program designers are discussed here. Recommendation #1. Program designers should pay attention to how incentives for training providers shape which youth they enroll. 52. The target population for youth training programs is typically out-of-school youth from low-income families who face different barriers to education and employment. Within this defined universe, there exists great diversity in terms of the degree of disadvantage and the types of barriers faced. The default preference for many training providers is to enroll those from among this target population who will be easiest to train and place since this will allow them to optimize costs and efficiency. This is particularly the case if most of the provider’s funding is tied to targets for program completion and job placement. 31 Program designers need to consider how to devise incentives that will encourage providers to enroll more disadvantaged youth from the target population since it is these youth who are more in need of, and thus more likely to benefit from, the training. • An effective approach used by several of the programs in this review was to allocate more funding to providers who enrolled the most disadvantaged applicants. This strategy also could be extended to program completion and placement rates for these groups. • Another strategy is to choose performance measures that recognize a variety of legitimate outcomes for youth graduating from these programs. One way that programs have done this is by rewarding placement in further education on the same level as employment in recognition of the fact that additional education leading to a certification/qualification may be a better predictor of longer-term employment and earnings for some youth (particularly those who need to make up for many years of being out of school) than immediate placement in a lower-wage job. This takes the pressure off providers to immediately move all graduates into employment. • Finally, program designers may wish to consider giving extra points in the provider selection process to those who have a history of working with more disadvantaged youth. This may create a virtuous cycle of providers taking on more disadvantaged youth in each training cycle, knowing that this will benefit them in the selection process for the next cycle. 31 This phenomenon, often termed as ‘cream skimming’, has been defined by Heckman, Heinrich, and Smith (2002) as selecting individuals who help achieve short-run goals rather than selecting those who are expected to have long-term benefits. 28 Recommendation #2. Program designers should pay attention to how incentives for youth shape whether they enroll. 53. This review demonstrates that many out-of-school youth need special outreach efforts to overcome the potential barriers to their participation in training programs. These barriers are not just cultural and economic, but also psychological because such youth have frequently become alienated from mainstream institutions like schools due to earlier negative experiences. Cultural barriers typically pertain to women and minority groups (religious or ethnic) and manifest as expectations about what constitute appropriate and inappropriate roles and responsibilities for these groups in society. • Training programs need to identify such groups upfront and devise targeted informational and outreach campaigns that specifically address these assumptions. The Nepal AGEI Program is an example of an effective communications and outreach strategy to encourage women to overcome cultural expectations about their roles in society so that they can enroll in the program. The strategy involved sensitizing both young women and their families to the program. It used radio and television advertisements and also partnered with community-based organizations to attract applications from women. • Economic barriers are most frequently addressed through the use of stipends. As previously discussed, stipends help address the opportunity cost of students attending the training program, and also help incentivize participants to attend on a regular basis, particularly if tied to program attendance or completion. The caveat (based on the experience of Colombia’s Youth in Action Program) is the need to give careful thought to the optimal stipend amount and to consider potential differential impacts by gender or other key groups. • The psychological barriers to participation may be the most difficult of all, but there is some useful research on how to lower the mental burden for disadvantaged youth to engage with these training programs. 32 As mentioned above, many programs prefer to work with the most motivated and able participants in order to meet performance standards set by funders, and therefore use extensive screening processes to recruit participants. If young people cannot follow through with all the steps, programs may interpret this to mean that they are not motivated or ready for the program, which is not necessarily the case. Rather, the increasing hassle factors associated with enrolling in a program may be simply overwhelming young people who are already facing many barriers. Their inability to follow through does not necessarily mean that they are not ready to change their lives. Research indicates that behaviorally informed enrollment and engagement practices that reduce the cognitive load on young people by decreasing unnecessary hassle factors and that emphasize each person’s strengths may actually inspire and motivate them to take action. This is a promising area for future program design. • It is also possible to influence and inspire beneficiaries from disadvantaged groups to enroll in training programs by providing them with concrete information on the benefits of these programs in easily understood terms. For example, beneficiaries could be informed if the program curriculum is aligned to market needs and will facilitate a quick 32 Rajasekaran and Reyes 2019. 29 transition to employment. Or, they could be informed that training partners have industry connections and that there are commitments for internships, or apprenticeships, or full- time roles. In cases where the future wages are considerably better than current wages, information on wages could also be hugely motivating. Such information could act as incentives for not just course enrolment, but also engagement and completion. Recommendation #3. Program designers should consider including an education component. 54. Training programs typically require a certain minimum level of education and skills among their target populations. Apart from Uganda’s Youth Opportunities Program, none of the programs reviewed for this study included illiterate youth among their target population. • This suggests the possible need for a new breed of training programs that target more vulnerable youth, albeit these programs likely will be more challenging and costly to design and implement. • Even within the current range of training programs, however, there is room to create designs more suited to vulnerable youth. One way is to ensure that these programs include an education component that provides basic literacy and numeracy training or second chance education leading to a certification. The effects of such an addition to the program design on the profile of the applicant pool was most clearly seen among Liberia’s EPAG Program participants. The initial program design, which did not include an education component, but rather focused on skills training, was found to be most effective for girls in the middle of the wealth distribution or with moderate education, with the greatest impact seen among those who had completed high school. However, this design effectively excluded poorer and less educated girls. In recognition of these exclusionary effects, the program subsequently expanded to also offer education training (basic literacy and numeracy) so that less educated youth could benefit from the program’s offerings. Recommendation #4. Program designers should use funding-related incentives judiciously. 55. Funding can be a major incentive to direct both provider and beneficiary behavior, but it is important to think carefully about what that funding will be tied to and how that might direct behavior for better or worse. • The Alternative Learning System Program in the Philippines offers an interesting example. In order to shed light on the potential impact of performance-based payments, the program developers asked facilitators to choose which of the following payment options they would prefer: (i) a one-year contract for a fixed amount of PHP 50,000, or (ii) a one-year contract for a minimum amount of PHP 25,000, plus an additional amount determined by their participants’ education certification pass rates. Under the second option, each facilitator’s salary would be PHP 25,000 + PHP 50,000 × the education certification pass rate. The survey question assumed an average pass rate of 50 percent, or 0.5 in the payment formula. The results revealed a positive and statistically significant 30 correlation between the existing performance of facilitators and their preference for option (ii). In other words, high-performing facilitators consistently preferred performance-based payments linked to their participants’ education certification pass rates. The evaluators concluded that the introduction of such an incentive system could improve overall facilitator performance. However, they also noted that this might incentivize facilitators to work with more able groups of participants and create a disincentive to work with more challenging groups. In order to counter this, a regression- based formula was created that adjusted for the impact of various individual and contextual variables on pass rates so that ‘like’ could be compared with ‘like’. • The JTPA Program offers a cautionary tale about the need to carefully consider the relationship between performance-based funding formulas linked to short-term outcomes, and provider behavior. There is a general preference to focus on short-term over long-term outcomes when evaluating programs because the former rely on straightforward calculations using administrative data. This helps keep measurement costs low. It also provides a way to give training providers rapid feedback on their performance. These factors led JTPA designers to choose a set of short-term labor-market measures based on an enrollee’s employment status, wage, and earnings. This choice of performance measures had a less-than optimum influence on provider behavior, however. Evaluations of the program indicated that while the short-term outcome measures tied to financial incentives were likely to improve, other activities under the program that were not linked to funding might be performed in less-efficient ways that were not beneficial. 33 Nor was it clear whether these short-term outcome measures reflected the longer-term outcomes that presumably were the ultimate goal of the JTPA Program. In fact, researchers found that the short-term outcome measures used under the program were weakly and negatively related to participants’ longer-term earnings and employment gains. 34 The conclusion was that it was important to carefully examine the relationship between desired short-term and longer-term outcomes for these training programs and to identify strategies and incentives that are most successful in translating one into the other. Recommendation #5. Program designers should build impact evaluation into the design of the program and use the findings to inform future program iterations. 56. The sheer number of youth training programs that have been implemented without any kind of impact evaluation represents a huge waste of knowledge and savings. Conversely, programs that have employed impact evaluations as part of their design demonstrate the huge amount of learning and efficiencies that can be achieved. More generally, the literature on this issue notes that studies with proper impact evaluations tend to show less optimistic outcomes than studies with basic information on gross outcomes. This suggests that the absence of rigorous evaluations may lead to an overestimation of program impacts and misguide policy decisions. 33 Holmstrom and Milgrom 1991. 34 Heckman, Heinrich, and Smith 2002. 31 • The implication is that impact evaluation (Box 3) should be viewed as a core element of the design of any training program and not as an optional extra. The potential value of this approach is clearly illustrated in the cases of the training programs carried out in the Dominican Republic and Liberia. In the case of the Juventud y Empleo Program in the Dominican Republic, an impact evaluation was built into the program design from its inception in 1999. This enabled the program managers to learn from program implementation and use the findings to improve subsequent phases. The ongoing use of impact evaluation, not just at the pilot stage, but throughout program implementation, allowed the program to show incremental improvements in impact due to the refined and road-tested program design. In the case of the Liberia program, the evaluation findings from each of its four rounds of implementation, were used to inform the design for subsequent rounds. Some of the learnings from earlier rounds that were incorporated later on included a greater emphasis on business development skills training due to its superior impact on employment and earnings post program; the inclusion of more rural areas due to the large unmet need in these areas; and a reduction in program costs through condensing the training timetable (from 6 months to 4 months), nationalizing the project team, and using more local service providers. These programs also demonstrate the value of tracking program outcomes over several years so that there is evidence of longer-term impacts and so that better data are available to conduct cost-benefit analyses. Thus, program designers should plan to follow both treatment and control groups over time. • Several of the evaluations reviewed for this study noted the need to disentangle the relative contribution of different design elements to the program’s results in order to come up with better recommendations for future program design. This includes creating designs that allow for disentangling the effects of different kinds of incentives, including RBF, on program outcomes. For example, the evaluators of Brazil’s Galpao Aplauso Program noted that the selective nature of the program introduced a complexity in the interpretation of the results. Were employment gains being driven by human capital formation under the program or were they driven by the program’s ability (because of its excellent reputation) to signal to “higher quality” youth who it was then able to enroll? In the case of the I-BEST Program in the USA, the evaluators noted that it was impossible to determine whether the positive effects of the program were due to its content or structure or to the improved access to financial aid that allowed students to progress. In order to disentangle the relative contributions of RBF and other design elements, impact evaluations may need to employ more complex designs, including factorial approaches and multiple treatment arms. 35 For example, a training program might have two different treatments – one where providers receive funding contingent on the achievement of specific targets for trainees at different stages (enrollment, attendance, completion, and placement) and the other where providers receive funding at each of these program stages regardless of the results achieved. These treatments in turn would be compared with a control group. 35 Mukerjee and Wu 2006. 32 Box 3: Main Types of Evaluations Performance Monitoring: These evaluations provide information on how a system or program is operating, and the extent to which specified program goals are being achieved. Examples of performance indicators include the proportion of program participants employed, monthly earnings of program participants, and program cost. Results are used by policymakers to assess program performance and accomplishments. Impact Evaluations: These evaluations focus on the question of causality. Did the program have its intended effects? What was the magnitude of the effect? Did the program have any unintended consequences: positive or negative? These evaluations, when done well, can be used to compare program outcomes with some measure of what would have happened without the programs. Cost and Cost-Benefit Evaluations: These evaluations address the program costs, preferably in relation to alternative uses of the same resources and to the benefits being produced by the program. Process Evaluations: These answer questions about how the program operates, and document the procedures and activities undertaken in service delivery. Such evaluations help identify problems faced in delivering services and strategies for overcoming these problems. Source: Dar and Tzannatos (1999) D. Conclusion 57. This paper described the kinds of incentives currently being used in short-term training programs for out-of-school youth, drawing on ten programs as representative examples. It offers this as a starting point for future research. While the review did not find evidence to demonstrate the superiority of RBF approaches, it does offer useful insights on incentives in general. These include that incentives are a key aspect of many training programs, albeit they can take many forms. It demonstrates the importance of paying attention to incentives at different stages of a program in order to ensure that providers and participants maintain a focus on desired outcomes, and it brings home the message that evaluation is key to understanding and improving on the effectiveness of these and other aspects of a program’s design. It is hoped that these findings will be instructive to those involved in designing, implementing, or evaluating short-term training programs for out-of- school youth around the world. 33 General References Almeida, Rita K., and Truman G. Packard. 2018. Skills and Jobs in Brazil: An Agenda for Youth. International Development in Focus. Washington, DC: World Bank. Angrist, Joshua, Eric Bettinger, Erik Bloom, Elizabeth King, and Michael Kremer. 2002. “Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment.” American Economic Review, 92(5): 1535–58. Angrist, Joshua, Eric Bettinger, and Michael Kremer. 2006. “Long-Term Educational Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia.” American Economic Review, 96(3): 847–62. Behrman, Jere Richard, Piyali Sengupta, and Petra Todd. 2005. “Progressing through PROGRESA: An Impact Assessment of a School Subsidy Experiment.” Economic Development and Cultural Change, 54(1): 237–75. Betcherman, Gordon, Karina Olivas, and Amit Dar. 2004. “Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries.” Social Protection Discussion Paper Series No. 0402, Washington, DC: World Bank. Betcherman, Gordon, Martin Godfrey, Susan Puerto, Friederike Rother, and Antoneta Stavreska. 2007. “A Review of interventions to Support Young Workers: Findings of the Youth Employment Inventory.” Social Protection Discussion Paper No. 0715, Washington, DC: World Bank. Blattman, Christopher and Laura Ralston. 2015. “Generating Employment in Poor and Fragile States: Evidence from Labor Market and Entrepreneurship Programs.” Available at SSRN: https://ssrn.com/abstract=2622220 or http://dx.doi.org/10.2139/ssrn.2622220. Cameron, Judy, and W. David Pierce. 1994. “Reinforcement, Reward, and Intrinsic Motivation: A Meta- Analysis.” Review of Educational Research, 64(3), 363-423. Card, David, Jochen Kluve, and Andrea Weber. 2010. “Active Labor Market Policy Evaluations: A Meta- Analysis.” Economic Journal, 120(548), 2010, F452–77. Card, David, Jochen Kluve, and Andrea Weber. 2015 [revised 2017]. “What Works? A Meta- Analysis of Recent Active Labor Market Program Evaluations.” Working Paper 21431, Cambridge, MA: National Bureau of Economic Research. Card, David, Pablo Ibarraran, Ferdinando Regalia, David Rosas Shady, and Yuri Soares. 2011. "The Labor Market Impacts of Youth Training in the Dominican Republic." Journal of Labor Economics 29 (2): 267-300. Cho, Yoonyoung, Davie Kalomba, Ahmed Mushfiq Mobarak, and Victor Orozo. 2013. “Gender Differences in the Effects of Vocational Training: Constraints on Women and Drop-Out Behavior.” Policy Research Working Paper 6545, Washington, DC: The World Bank. 34 Dar, Amit, and Zafiris Tzannatos. 1999. “Active Labor Market Programs: A Review of the Evidence from Evaluations.” Social Protection Discussion Series No. 9901, Washington, DC: World Bank. Datta, Namita, Angela Elzir Assy, Johanne Buba, Sara Johansson de Silva, Samantha Watson, et al. 2018. “Integrated Youth Employment Programs. A Stocktake of Evidence on What Works in Youth Employment Programs.” World Bank, Washington, DC. de Hoyos, Rafael, Halsey Rogers, and Miguel Székely. 2016. Out of School and Out of Work: Risk and Opportunities for Latin America’s Ninis. Washington, DC: World Bank. Deci, Edward and Richard Ryan. 1985. Intrinsic Motivation and Self-Determination in Human Behavior. New York: Plenum Press. Deci, Edward, and Richard Ryan. 2000. “The ‘What’ and ‘Why’ of Goal Pursuits: Human Needs and the Self-determination of Behavior.” Psychological Inquiry, 11, 227-268. Deci, Edward, Richard Koestner, and Richard Ryan. 1999. “A Meta-analytic Review of Experiments Examining the Effects of Extrinsic Rewards on Intrinsic Motivation.” Psychological Bulletin, 125, 627-668. Fares, Jean, and Olga Susana Puerto. November 2009. “Towards Comprehensive Training.” Social Protection Discussion Paper No. 0924, Washington, DC: World Bank. Fryer, Roland G. 2010. “Financial Incentives and Student Achievement: Evidence from Randomized Trials.” NBER Working Paper 15898, National Bureau of Economic Research, Cambridge, MA. Heckman, James J., Carolyn Heinrich and Jeffrey Smith. "The Performance Of Performance Standards," Journal of Human Resources, 2002, v37(4,Fall), 778-811. Heckman, James J., Neil Hohmann, Jeffrey Smith, and Michael Khoo. 2000. "Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment." The Quarterly Journal of Economics 115 (2): 651-694. Bengt Holmstrom and Paul Milgrom. 1991. “Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design.” Journal of Law, Economics, and Organization, 1991, vol. 7, issue 0, 24-52. Holzer, Harry J. 2008. “Workforce Development as an Antipoverty Strategy. What Do We Know? What Should We Do?” IZA Discussion Papers No. 3776, Bonn: Institute for the Study of Labor (IZA). Ibarrarán, Pablo, and David Rosas. 2009. “Evaluating the Impact of Job Training Programs in Latin America: Evidence from IDB funded operations.” Journal of Development Effectiveness 2, 195-216. Ivry, Robert and Fred Doolittle. 2003. “Improving the Economic and Life Outcomes of At-Risk Youth.” MDRC. https://www.mdrc.org/sites/default/files/full_403.pdf 35 Inoue, Keiko, Emanuela di Gropello, Yesim Sayin Taylor, and James Gresham. 2015. Out-of-School Youth in Sub-Saharan Africa: A Policy Perspective. Directions in Development, Washington, DC: World Bank. Lee, Ling Jessica Diana; Medina Pedreira, Octavio. 2019. “Results-based Financing in Education: Learning from What Works.” Washington, D.C.: World Bank. Mukerjee, R., & Wu, C. F. J. (2006). A Modern Theory of Factorial Designs. New York: Springer Publishing. Rajasekaran, Subhashini, and Joel Reyes. 2019. Back to School: Pathways for Reengagement of Out-of- School Youth in Education. International Development in Focus, Washington, DC: World Bank. Results for Development Institute. 2016. “Paying for Performance: An Analysis of Output-based Aid in Education.” Washington, DC: Results for Development Institute. Roder, Anne, and Mark Elliott. 2014. “Sustained Gains: Year Up’s Continued Impact on Young Adults’ Earnings.” New York: Economic Mobility Corporation. Ryan, Richard and Edward Deci. 2000. “Self-determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-being.” American Psychologist, 55, 68-78. Schultz, T. Paul. 2004. “School Subsidies for the Poor: Evaluating the Mexican PROGRESA Poverty Program.” Journal of Development Economics, 74(2): 199–250 Snilstveit, Birte, Jennifer Stevenson, Daniel Phillips, Martina Vojtkova, Emma Gallager, Tanja Schmidt, Hannah Jobse, Maisie Geelen, Maria Grazia Pastorello, and John Eyers. 2015. “Interventions for Improving Learning Outcomes and Access to Education in Low and Middle-income Countries: A Systematic Review.” 3ie Systematic Review 24, London: International Initiative for Impact Evaluation (3ie) United States Department of Labor, Employment and Training Administration. 2015. “The Workforce Innovation and Opportunity Act: Fact Sheet: Youth Program.” http://www.doleta.gov/wioa/pdf/WIOA_YouthProgram_FactSheet.pdf. United States Government Accountability Office. 2006. “Workforce Investment Act: Employers Found One-Stop Centers Useful in Hiring Low-Skilled Workers; Performance Information Could Help to Gauge Employer Involvement.” Report to Congressional Requesters, Washington, DC: United States Government Accountability Office. Valerio, Alexandria, and Viviana Roseth. 2016. “Technical Note. Short Term Training Programs: A Review of the International Evidence.” Washington, DC: The World Bank. Vansteenkiste, M., Simons, J., Lens, W., Sheldon, K. M., & Deci, E. L. (2004). “Motivating Learning, Performance, and Persistence: The Synergistic Effects of Intrinsic Goal Contents and Autonomy- Supportive Contexts.” Journal of Personality and Social Psychology, 87, 246-260. 36 World Bank. 2001. “A Global Directory of Early Child Development Programs Supported by the World Bank.” Washington, D.C: World Bank. World Bank. 2018. World Development Report 2018. Learning to Realize Education’s Promise. Washington, DC: The World Bank. World Bank. 2019. World Development Report 2019. The Changing Nature of Work. Washington, DC: The World Bank. Ziderman. 2016. “Funding Mechanisms for Financing Vocational Training: An Analytical Framework.” IZA Policy Paper No. 110, Institute for the Study of Labor, Bonn. Impact Evaluations [Brazil] Calero, Carla, Carlos Henrique Corseuil, Veronica Gonzales, Jochen Kluve, and Yuri Soares. 2014. “Can Arts-Based Interventions Enhance Labor Market Outcomes among Youth? Evidence from a Randomized Trial in Rio de Janeiro.” IZA Discussion Paper 8210, Institute for the Study of Labor, Bonn. [Colombia] Attanasio, Orazio, Adriana Kugler, and Costas Meghir. 2008. “Training Disadvantaged Youth in Latin America: Evidence from a Randomized Trial.” NBER Working Paper 13931, National Bureau of Economic Research, Cambridge, MA. Kugler, Adriana, Maurice Kugler, Juan Saavedra, and Luis Omar Herrera Prada. 2015 [revised 2019]. “Long-term educational consequences of vocational training in Colombia: Impacts on young trainees and their relatives.” Working Paper 21607, Cambridge, MA: National Bureau of Economic Research. [Dominican Republic] Ibarrarán, Pablo, Laura Ripani, Bibliana Taboada, Juan Miguel Villa, and Brigida Garcia. 2014. “Life Skills, Employability and Training for Disadvantaged Youth: Evidence from a Randomized Evaluation Design.” IZA Journal of Labor & Development 3 (1): 1–24. Ibarraran, Pablo, Jochen Kluve, Laura Ripani, and David Rosas Shady. 2015. “Experimental Evidence on the Long-Term Impacts of a Youth Training Program.” IZA Discussion Paper No. 9136, Bonn, Germany: IZA. [Liberia] Adoho, Franck, Shubha Chakravarty, Dala T. Korkoyah, Mattias Lundberg, and Afia Tasneem. 2014. “The Impact of an Adolescent Girls Employment Program: The EPAG Project in Liberia.” Policy Research Working Paper 6832, World Bank, Washington, DC. [Liberia] Ministry of Gender and Development. 2014. “Can Skills Training Increase Employment for Young Women? The Case of Liberia. Adolescent Girls Initiative.” Ministry of Gender and Development. [Nepal] Chakravarty, Shubha, M. Lundberg, P. Nikolov, and J. Zenker. 2015. “The Role of Training Programs for Youth Employment in Nepal: Impact Evaluation Report on the Employment Fund.” Report No: ACS14600, Washington, DC: World Bank. 37 [Nepal] World Bank. 2015. "The Role of Skills Training for Youth Employment in Nepal: An Impact Evaluation of the Employment Fund (English).” Adolescent Girls Initiative (AGI) Results Series, Washington, DC: World Bank Group. [Uganda] Blattman, C., N. Fiala, and S. Martinez. 2013. “Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda.” Quarterly Journal of Economics. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2268552. [United Kingdom] Dolton P.J., G. H. Makepeace, and J. G. Treble. (1994) "The Youth Training Scheme and the School-to-Work Transition," Oxford Economic Papers, Oxford University Press, 46(4):629–657. [USA, I-BEST] Jenkins, D., M. Zeidenberg, and G. S. Kienzl (2009). “Educational outcomes of I-BEST, Washington State Community and Technical College System’s Integrated Basic Education and Skills Training Program: Findings from a Multivariate Analysis.” CCRC Working Paper No.16, New York, NY: Columbia University, Teachers College, Community College Research Centre. [USA, I-BEST] Wachen, J., D. Jenkins, and M. Van Noy. (with K. Kulongoski, K. Kurien, A. Richards, L. Sipes, M. Weiss, and M. Zeidenberg). 2010. “How I-BEST Works: Findings from a Field Study.” New York, NY: Columbia University, Teachers College, Community College Research Center. [USA, I-BEST] Zeidenberg, Matthew, Sung-Woo Cho, and Davis Jenkins. 2010. “Washington State’s Integrated Basic Education and Skills Training Program (I-BEST): New Evidence of Effectiveness.” CCRC Working Paper No. 2, Community College Research Center, Teachers College, Columbia University. [USA, Year Up] Roder, Anne, and Mark Elliott. 2011. “A Promising Start: Year Up’s Initial Impacts on Low Income Young Adults’ Careers.” Economic Mobility Corporation, New York. Course Catalogue: https://www.yearup.org/wp-content/uploads/2014/03/Year-Up-Course-Catalog-Oct2016.pdf [USA, JTPA] Bloom, H., L. Orr, S. Bell, G. Cave, F. Doolittle, W. Lin, and J. Bos. 1997. “The Benefits and Costs of JTPA Title II-A Programs: Key Findings from the National Job Training Partnership Act Study.” Journal of Human Resources, 32(3), 549–576. [USA, JTPA/WIA] Kirby, Sheila Nataraj, and Brian Stecher, editors. 2004. “The Job Training Partnership Act and the Workforce Investment Act.” Organizational Improvement and Accountability: Lessons for Education from Other Sectors, 1st ed., RAND Corporation, Santa Monica, CA; Arlington, VA; Pittsburgh, PA, pp. 51–64. Other Evaluations [China] UNICEF. September 2001. “Second Chance Education for Out-of-School Adolescents in China.” Project Impact Assessment Report, China: UNICEF [Philippines] Education Development Center. “Implementing Education and Youth Development Programming in the Fragile Context of Mindanao, Philippines.” [Philippines] World Bank Group. May 2018. “Philippines Education Note. Education Policy Note. A Second Chance to Develop the Human Capital of Out-of-school Youth and Adults: The Philippines Alternative Learning System.” 38 [Timor Leste] Education Development Center. “Prepara Ami ba Serbisu.” 39 Annex 1. Overview of Training Programs for Out-of-School Youth (with Impact Evaluations) Brazil: Galpão Aplauso (2009-2013) Beneficiaries Education/Skills Training Evaluation Results Male and female youth Program duration was approximately 6 Randomized assignment Employment: 33.3 percent increase in the probability of being under 29 from households months, 5 hours a day, 5 days a week, of youth applying for 2012 employed four to five months post program. However, the in Rio de Janeiro with delivered in three shifts—morning, afternoon, cohort into program and program had no impact on the formality of jobs. monthly incomes below and evening. 180 hours of academic and basic control groups. Baseline Earnings: 23.6 percent increase in earnings four to five months minimum wage who were skills, including remedial courses in interviews and follow-up post program. screened through Mathematics and Portuguese. 300 hours of survey by firm. Personality traits: No change in performance on standardized interviews and math and vocational training (mainly construction- psychological measures of socio-emotional skills although language tests. Those who related, soldering, wood shop), tailored to anecdotal evidence suggested program youth displayed better performed best on the beneficiaries, and 120 hours of life skills. Most performance in skills valued by labor market (i.e. punctuality, math and language tests were placed in positions related to responsibility, dedicated to their work). were invited to enroll in manufacturing and construction. Within the the program according to life skills component, youth attended sessions the number of slots on basic principles of “social harmony”, which available. emphasized civics and certain shared values, along with socio-emotional development. The pedagogic model employed made extensive use of arts and theatre as training mechanisms. Colombia: Jóvenes en Acción (Youth in Action) (2001-2005) Beneficiaries Education/Skills Training Evaluation Results Unemployed male and Three months of classroom training on Training institutions in the Employment: Increase in employment for both men and female youth between 18 vocational skills by public and private training seven largest cities of the women, with larger effects for women. In particular, being and 25 from lowest socio- institutions. Three months of on-the-job country chose the courses offered training increased paid employment by about 14% for economic strata (mostly training provided by legally registered to be taught and received women and increased their days and hours worked by about high-school dropouts) in companies, which provided unpaid internships applications. For the 2005 11%. urban areas of Colombia. In to the participants in manufacturing, retail and cohort, each was asked to Earnings: Increase in earnings for both men and women, with total, 80,000 youth were trade, and services. select more applicants larger effects for women. Women offered training earned about trained under the scheme, than they had room for. 18 percent more than those not offered training, while men or 50% of the target The program organizers offered training earned about 8 percent more than those not population. subsequently randomly 40 offered training to as offered training. Much of these earnings increases were related many people as there to increased employment in formal sector jobs. were slots. The remainder were used as a control group. Baseline survey conducted before program and follow-up one year after completion. Dominican Republic: Youth and Employment Program, Juventud y Empleo (2001 – Not specified) Beneficiaries Education/Skills Training Evaluation Results Male and female high Basic skills training to strengthen self-esteem Random assignment of Labor market outcomes: No significant impact on overall school dropouts between and work habits; vocational training to address participants who met the employment. Positive impacts on quality of employment for 16 and 29; living in poor needs of local employers. eligibility criteria into males, measured as employer provided health insurance or as neighborhoods; treatment and control having a written contract. Males assigned to the treatment unemployed, groups (2008 cohort). group were four percentage points more likely to get a job in the underemployed, or Follow-up household formal sector than males assigned to the control group. This inactive; not holding an survey done for both represents an impact of 17 percent over the control group identity card. groups 18-24 months average. Positive impact on monthly earnings for women (and after graduation. for men in Santo Dominigo). Youth behavior and life expectations: Effective in reducing teenage pregnancy. Females in the treatment group were on average two percentage points less likely to be pregnant than females in the control group. Positive impact on beneficiaries’ perceptions of current situation and expectations regarding future; stronger for females and younger individuals. Socio-emotional skills: Positive impacts on non-cognitive skills as measured by three different scales. Scores improved between 0.08 and 0.16 standard deviations with the program. 41 Liberia: Economic Empowerment of Adolescent Girls and Young Women (EPAG) (2010-2014) Beneficiaries Education/Skills Training Evaluation Results Women aged 16-27 with Six months of classroom-based training (Job Impact evaluation based Economic: Strong impacts on employment and earnings basic literacy and Skills Track (JS) or Business Development on 2010 and 2011 outcomes of program participants relative to the control group. numeracy skills, not Services Track (BDS) and six months of job cohorts. Control and The EPAG program increased employment by 47 percent and enrolled in school, and placement support for either self- or wage- treatment groups. earnings by 80 percent. The positive employment outcome was living in one of nine target employment. The JS track provided training in Baseline, midline, and stronger among the BDS trainees. By the end-line, just over 70% communities. six areas: 1) hospitality, 2) professional endline data collected via of BDS graduates were engaged in at least one income- cleaning / waste management, 3) office / quantitative surveys and generating activity (wage or self-employment), compared to just computer skills, 4) professional house / office focal group discussions. under 60% of JS graduates. painting, 5) security guard services, and 6) Empowerment: Positive effects on access to and control over professional driving. These were determined monetary resources. based on independent labor market Household: Improved food security and shifting attitudes to assessments. All JS trainees received training gender norms. in entrepreneurship skills as well. The BDS Other: Impacts sustained one year after project completion. training taught young women how to identify micro-enterprise opportunities based on an assessment of market needs, and how to grow and manage any existing businesses they already had. The curriculum included entrepreneurship principles, market analysis, business management, customer service, money management, and record-keeping. Nepal: Adolescent Girls Employment Initiative (AGEI) (2010-2012) Beneficiaries Education/Skills Training Evaluation Results 4,410 young women Basic life skills, technical/vocational skills, and 3,142 trainees from 2010 Employment and earnings: Positive impact on employment between the ages of 16 to basic business skills. and 2011 cohorts rates and earnings; courses in electronics, beautician services, 24 over three-year period. compared with similar and tailoring were most effective (more effective than Carried out as part of individuals who applied construction, poultry rearing, handicrafts, and food preparation broader Employment Fund but were not selected. and hospitality). Impact larger for women; no significant impact Project (skills training and Sample included both for males. Impact for younger women not different from older employment placement males and females women. Women selected for training in 2010 to 2012 services) which aims to between 16-35 years to experience overall and non-farm employment gains of 13 and 19 allow for comparison of percentage points respectively, while the corresponding impacts 42 reach more than 15,000 AGEI beneficiaries with for men are 2 and 10 percent. However, there were no Nepalese youth annually. men and older women significant differences by gender on other economic indicators (16-35) participating in such as earnings, hours worked, trade-specific employment, or broader Employment savings and loans. Fund Project. Surveys Empowerment and self-confidence: Trainees had more control conducted at baseline and over economic resources and stronger self-assessed self- one year later. confidence. Risky behaviors: Few impacts on reproductive health Impact on household: Few impacts on household level outcomes like remittances received by households of trainees. Uganda: Youth Opportunities Program (YOP) (2006 – 2008) Beneficiaries Education/Skills Training Evaluation Results Poor underemployed male Unsupervised cash grants offered to groups Funding randomly Earnings: Groups assigned to grants had higher earnings after and female youth, aged 16- for self-employment (approximately US$ 382 assigned among 535 four years. Many also formalized their enterprises and hired 35, from Uganda's conflict- per member). Part could be used for screened and eligible labor. Incomes of treatment women were 73 percent greater affected North. Many were investment in training. 68 percent of the applicant groups in 2008. than control women, compared to a 29 percent gain for men. rural farmers and treatment group enrolled in vocational There were 265 treatment functionally illiterate. training. Groups selected their own trainers, groups and 270 control Capital/assets: Groups assigned to grants had greater capital typically a local artisan or small institute. groups. Treatment and stocks. Relative to the control group, the program increased Groups would travel to be closer to trainers, control groups were business assets by 57%, work hours by 17%, and earnings by 38%. or paid transport and upkeep for trainers to surveyed two and four Tendency to practice skilled trade: Groups assigned to grants come to them. years after disbursement. were more than twice as likely to practice a skilled trade. Other: There was no impact on social cohesion, anti-social behavior, or protest. United Kingdom: Youth Training Scheme (YTS) (1983-1989) Beneficiary Education/Skills Training Evaluation Results 16- to 17-year-old Apprenticeship training in industry. YTS began Impact of training Duration to any job: YTS participants took longer to find unemployed youth. by offering subsidized training for a maximum measured using duration employment than those who were unemployed at the start of the of one year, although this period was models. Sample was taken survey period. increased to two years in April 1986. from YTS III survey that Duration to good job (involving training/apprenticeship and recorded data for falling above a low pay threshold): Female participants on YTS individuals who obtained good jobs at a faster rate than non-participants, 43 completed compulsory although male participants obtained good jobs at the same rate education in 1985-86 over as non-trainees. a period of 30 months from September 1986 to February 1989. The sample included men and women who left full-time education during the survey period. United States: Year Up (2000 – Not specified) Beneficiaries Education/Skills Training Evaluation Results Young male and female Instruction and assignments that improve Evaluation carried out in Earnings: During the second year after random assignment—the adults aged 18 to 24 from skills needed in college, including reading and 2007-2008. Treatment year after the program took place—the annual earnings of the low-income urban writing. Students receive credits they can and control group Year Up participants were 30 percent greater, on average, than communities. transfer to more than 2,000 higher education comparison. Control those of control group members. Average wages among the institutions for degree programs. Six months group were applicants Year Up participants were US$ 14.79 an hour in information of training in information technology and placed on waiting list who technology jobs and US$ 15.72 an hour in jobs in the investment investment operations fields followed by a six- were told they could re- operations field, compared to US$ 10.68 an hour in all other month internship with companies that are apply after ten months. occupations. The hourly wages of Year Up participants who corporate partners of program. worked in occupations other than information technology and investment operations did not differ significantly from the wages of control group members. Therefore, the overall differences in wages resulted from Year Up participants’ ability to access jobs in the targeted sectors. Employment: Year Up participants’ current or most recent jobs were more likely to be full-time (35 hours or more per week) than control group members’ jobs. There were no statistically significant differences between the groups in the availability of employer-provided medical benefits or tuition assistance. College attendance: Program participants were just as likely to enroll in postsecondary education as control group. Other: Outcomes compared two years after random assignment took place. A future report was supposed to examine whether the participants sustained the earning gains four years after program application and whether they made greater progress in pursuing a postsecondary education degree. 44 United States: Job Training Partnership Act (1982)/Workforce Investment Act (1998) Beneficiaries Education/Skills Training Evaluation Results Unemployed individuals Basic education leading to the General Randomized experiment Earnings: No significant increase in earnings; modest positive (above age 16) who met Education Development (GED) exam that with participants, of impact on incremental earnings (difference between earnings of income eligibility limits or provides a high school equivalency credential. which two thirds were treatment and control groups) for adult men and women. Negative had low family income in Classroom training in occupational skills assigned to treatment and impact on earnings for male youth. the six months preceding combined with on-the-job training in private one third assigned to Educational attainment: Positive impact on educational application to the firms and job search assistance. control. Random attainment for female youth, adult women, and adult males who program. Each beneficiary assignment lasted for were school dropouts, but no impact on male youth. was assigned one of three about 15 months. Surveys Welfare receipt: No significant impact on reducing welfare services offered. of treatment and control dependence. groups took place 30 months after random assignment. United States: Integrated Basic Education and Skills Training Program (I-BEST) (2004- Not specified) Beneficiaries Education/Skills Training Evaluation Results Adult basic education Enables basic skills students to enroll directly Comparison between Educational outcomes (college credit of any kind, occupational students enrolling in in college-level coursework. Basic skills basic skills students who college credit, persistence to the following year after initial career and technical instruction may include Adult Basic Education, enrolled in I-Best and enrollment, earning certificate or degree, point gains on basic skills education offered by English as a Second Language, and Adult those who took an tests): I-BEST students were more likely to earn college credits, community and technical Secondary Education or General Education occupational course on including occupational college credits; more likely to attain a colleges. (Adult basic skills Development programs. College-level their own using certificate or degree; and gained more points on the students are adults who occupational courses taught by a basic skills multivariate regression Comprehensive Adult Student Assessment System (CASAS) that lack high school-level instructor and a professional-technical faculty analysis, propensity score includes reading, listening, and math. No difference in persistence. skills). member. matching, and difference- Labor market outcomes (change in wages and hours worked): No in-differences. Study significant effect found on wages and average quarterly hours included data on 77,147 worked. basic skills students, including 1,390 I-BEST students who enrolled in I-BEST in 2006–07 and 2007–08. 45 Annex 2. Overview of Training Programs for Out-of-School Youth (without Impact Evaluations) China: Second Chance Education for Out-of-School Adolescents (1999-2001) Beneficiaries Education/Skills Training Evaluation Results Disadvantaged youth The project offered opportunities for This evaluation was of the Employment: Some adolescents used skills acquired to generate from poor families who obtaining basic literacy, numeracy, income- first phase of the project, income. had been forced to drop generation skills, and health information. The which ran from 1999- Education: Most of adolescents interviewed said they could read out of school. 2,250 design was largely flexible and informed by 2001. The evaluation was brochures, posters of scientific knowledge, and other training adolescents between the participants’ preferences. supposed to inform the materials without help. They also developed mathematical ability ages of 13-18, spread next phase of the project on multiplication and division and recording of expenditures and over 15 counties in 5 Lessons in Chinese and arithmetic aimed to (2001-2003). Evaluation incomes. Literacy and numeracy skills enabled the adolescents, provinces. enable participants to reach the level of comprised desk reviews especially girls to acquire new knowledge, feel confident, increase elementary school graduates. Most of the of project documents, possibilities of employment and reduce risk of being victimized by trainers were teachers from elementary interactions with staff, outsiders. schools. They held classes for out-of-school field visits to homes, Socio-emotional: Increased participants’ confidence. adolescents at weekends or in the evenings. structured interactions Other: Some participants encouraged other out-of-school through focus group adolescents to participate in the program or encouraged their Health education with emphasis on life skills discussions and siblings to continue in formal education. 30-50 of the participants and hygiene (such as HIV/AIDS prevention, structured interviews in each project county became peer assistants at the end of 2000. reduction of iodine deficiency disorders, and with randomly selected Girls constituted a significant number of these peer educators, so on). Training was provided mainly by the participant children, as who provided assistance to project staff in the management of medical staff of the local health bureau well as community the project. responsible for prevention/control of members, parents, and epidemic diseases. project staff from 2 counties in 2 provinces. Skill training focused on agricultural production and animal husbandry, usually carried out by trainers from local agricultural and animal husbandry departments. Apart from out-of-school girls, out-of-school boys and women in the village also attended the training to increase their earnings. 46 East Timor: Prepara Ami ba Serbisu (PAS)/Preparing Us for Work (2007-2010) Beneficiaries Education/Skills Training Evaluation Results Out-of-school men and Foundational training designed to prepare Regular program Completion: 1,597 graduated from the program (54% women) women aged 16-30 from school dropouts for entrance into further monitoring/tracking used Employment: 57.4% of graduates accessed employment, either rural communities (50% learning and earning activities, including to generate output data. through formal contract jobs or the initiation and/or improvement gender balance promoted return to formal education, transition to of a small business (53% women) at each learning center). further training, or employment. Further education: 151 graduates returned to formal schooling or 2,078 out-of-school youth transitioned to further training with government recognized received training. Phase I: Classroom instruction and work courses. experience simulations: • basic literacy and numeracy; • leadership and life skills; • work readiness; • introduction to business and entrepreneurship; • introduction to technical and income- generation skills. Phase II: guidance and mentoring while pursuing one of three livelihood pathways over four months: • internships; • non-formal education or training courses; • initiation/expansion of small business. Philippines: Alternative Learning System Beneficiaries Education/Skills Training Evaluation Results Youth and adults who New ALS enrollees take an initial placement Program evaluation Certification: 40% of ALS enrollees sat for the A&E exam, and 30% either did not enroll in test, called the “Functional Literacy Test,” to consisted of analyses and passed the exam and earned their high-school equivalency formal school or dropped assess their current education level. Enrollees regressions conducted on: credentials. Female participants consistently outperformed male out before completing the then develop Individual Learning Agreements counterparts, and urban participants passed the A&E exam at a basic education cycle. (ILAs) in consultation with their learning • official enrollee data and higher rate than rural participants. Majority are between 15 facilitators. routine program- Further education: About 60 percent of ALS participants who and 24. monitoring data available passed the A&E exam went on to enroll in tertiary education or Basic Literacy Program (BLP) and Accreditation with the Department of vocational training. In comparison, only 30 percent of those who & Equivalency (A&E) Programs. The former Education and did not pass the A&E exam pursued any kind of further education, aims to eradicate illiteracy among out-of-school 47 youth and adults by teaching basic literacy and • data from surveys (2013, and a negligible fraction attended colleges or vocational training numeracy. The latter targets people who are 2015, and 2017) institutions, functionally literate but did not complete basic conducted by the World Employment: Over 70 percent of ALS participants who passed the education, and it offers programs at both the Bank on a randomly A&E exam reported being employed or self-employed after primary- and secondary-school levels. The goal selected sample of completing the program, compared to just half of those who did of the A&E Programs, which span 10 months, is facilitators and current, not pass the A&E exam. to equip participants with the knowledge and prospective, and former Earnings: Passing the A&E exam was associated with a PHP 2,400 skills necessary to pass the national A&E exam, enrollees, from per month increase in earnings, or US$ 640 per year. Workers which provides an academic credential representative range of who had passed the A&E exam earned more than the average for equivalent to formal school diplomas for geographic areas. workers who did not complete secondary education. elementary and junior high school education. Obtaining this credential enables ALS participants to apply to higher education and training institutions or to jobs that require a high school education. The Informal Education component offers short training programs focusing on livelihood and entrepreneurship skills. Content is based on interest of enrollees and available resources. DepEd offers no standard programs, curricula, or support. Philippines: Education Quality and Access to Learning and Livelihood Skills Project (EQuALLs2) (2006 – Not specified) Beneficiaries Education/Skills Training Evaluation Results Out-of-school youth in Alternative basic education programs for those Mix of internal Employment: 1,961 youth or 73% of graduates either three conflict-affected wishing to go back to the formal school system or programmatic tracking, employed or self-employed (1794 or 67% graduates found jobs and Muslim-majority take the Department of Education’s Accreditation small evaluations of sub- and 6% set up small businesses.). This job placement rate regions, ages 15-25. and Equivalency Exam. Life Skills courses to parts and evaluation by apparently exceeded national benchmarks. Nine graduates develop interpersonal, creative-thinking, problem- International Youth continuing their education. solving, decision-making and conflict-resolution Foundation (IYF) Other: Participants showed improved scores on the Youth Life skills. Workforce development programs for those Consultant. IYF Skills Outcomes Scale after completing their training. who want to seek employment or start a business, evaluation involved either through short-term livelihood skills training responses from 169 or technical-vocational certificate courses (e.g., respondents from 6 Civil Trades Training, Agriculture Production, Agri- stakeholder groups: (a) fisheries, Small Engine Repair, Seaweed Industry, alliance partners; (b) Metalworks) employers who hired youth or helped by 48 accommodating youth internships; (c) direct youth beneficiaries; (d) trainers, (e) friends and family members of youth; and (f) implementing partners, including the Secretariat. Timor Leste: A Second Chance at Education (2010-2016) Beneficiaries Education/Skills Training Evaluation Results 1,670 youth and adults Project aimed to support Ministry of Education in Outputs were tracked Certification: 197 students had graduated by 2017, with another who had not completed establishing a sustainable national equivalency through program 113 expected to graduate in 2018. In 2016, 106 out of 108 formal school programs program, which could increase the number of out monitoring. enrolled students passed the equivalency exam. (basic and/or secondary of-school youth and adults who completed a Other: 1,100 students completed the Level 1 program. education). 55% were recognized equivalency program. women and 10% were people living with a • Level I training equivalent to Grades 4-6, disability. consisting of Linguistic Development (Portuguese, Tetum), Scientific and Social Development (Mathematics, Natural Science, Social Science), and Personal Development (Vocational Course) OR • Level II training equivalent to Grades 7-9, consisting of Linguistic Development (Portuguese, Tetum, English), Scientific and Social Development, and Personal Development (Elective Subject & Vocational Course). Successful graduates are awarded a certificate which is equivalent to a standard junior secondary education certificate, allowing them to pursue further education or employment. 49 Annex 3. Funding Modalities and Incentives for Training Programs for Out-of-School Youth (with Impact Evaluations) Brazil: Galpão Aplauso (2009 - 2013) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Began in 2005 and originally co- NGO provided the initial None specified. Male and female youth under None specified. financed by public sector. training. Galpão’s job 29 from households in Rio de However, the partnership was placement strategy was Janeiro with monthly discontinued due to a government loosely structured around income below minimum change, leaving the program formal and informal wage, screened through administrators with the task of agreements with local interviews and math and finding a new model of operation private sector firms. It also language tests. that would not rely on public did not have a formal funding. In 2009, the Inter internship program. In some American Bank's MIF partnered cases, in situ vocational with Instituto Stimulu Brasil to training was sponsored by finance the project. Now, it has a partner firms. private sector-oriented strategy. Colombia: Jóvenes en Acción (Youth in Action) (2001-2005) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Government of Colombia social Private sector institutions Payment conditional on Unemployed male and female Stipend of about US$2.20 per program, financed with loan from (for-profit and nonprofit) completion of training by youth between 18-25 from day to male and female trainees World Bank and Inter-American offered the classroom-based participants. Training lowest socio-economic strata without young children to cover Development Bank. skills training and chose, institutions paid according (mostly high-school dropouts) transportation and lunch. designed, and marketed the to market prices. in urban areas of Colombia. About US$3.00 per day for courses. In 2005, there were women with children under 7 a total of 118 training years to help cover childcare institutions offering 441 expenses. different types of courses. These institutions were selected through a competitive bidding process. On-the-job training was provided by 1,009 legally registered companies. These 50 companies operated in manufacturing (textiles, food and beverages, pharmaceuticals, and electricity), retail and trade, and services (including security, transportation, restaurants, health, childcare, and recreation). Dominican Republic: Youth and Employment Program, Juventud y Empleo (JE) (2001 – Not specified) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Ministry of Labor The program offers a wide None specified. Male and female high school Stipends for transportation and range of job training courses dropouts between 16 and 29; meals. such as administrative living in poor neighborhoods; assistant, baker, hair stylist, unemployed, underemployed clerk, auto mechanic, or inactive; not holding an bartender, and so on. The identity card. Ministry of Labor outsources the provision of training services to private training institutions that are registered and approved by the national training institution. Liberia: Economic Empowerment of Adolescent Girls and Young Women (EPAG) (2010 – 2014) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Liberian Ministry of Gender and Four NGOs selected by Bonuses awarded to Women aged 16-27 with basic Small stipends and a Development supported by World Liberian Ministry of Gender training providers that literacy and numeracy skills, completion bonus contingent Bank Adolescent Girls Initiative and Development through successfully place their not enrolled in school, and upon attendance; free childcare Trust Fund. competitive bidding process. graduates in jobs or living in one of nine target at every training site; Two of these NGOs further microenterprises. Bonus communities. beneficiaries assisted to open 51 subcontracted to four was last payment received savings account at local bank to Liberian NGOs. under their contracts. save their stipend money. Nepal: Adolescent Girls Employment Initiative (AGEI) (2010 – 2012) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers The Employment Fund is operated Training and Employment Outcome based payment - More than 4000 young None specified. by Helvetas, a Swiss NGO, in service providers, like TVET 40% on completion of women (age 16 to 24) partnership with the Government institutions, public and training and remaining 60% of Nepal. The Employment Fund is private providers as well as on gainful employment. financed by the United Kingdom’s skilled artisans, are Highest incentive for Department for International competitively selected. (The training most Development, the Swiss Agency for Employment Fund weighs disadvantaged. Development and Cooperation, the capacity and experience and the World Bank. The AGEI was of each provider, the market launched in 2009 to expand the demand for the proposed program’s reach to an additional trades being offered, and the 4,410 Nepali women aged 16-24 proposed costs.) over a three-year period. Uganda: Youth Opportunities Program (YOP) (2006 – 2008) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Government of Uganda Government of Uganda. None specified. Poor underemployed male Successful proposals received Grant recipients selected and female youth, aged 16-35, one-time unsupervised grants their own trainers, typically a from Uganda's conflict- worth US$7,500 on average — local artisan or small affected North. Many were about US$382 per group institute. rural farmers and functionally member, roughly their average illiterate. annual income. 52 United Kingdom: Youth Training Scheme (1983 – 1989) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Government of the United Industrial and commercial Firms could employ trainees 16- to 17-year-old Trainees were paid allowances Kingdom organizations in both the without incurring any wage unemployed youth slightly above the public and private sectors cost. Firms were free to pay unemployment benefit level any supplement to trainees if they wished. United States: Year Up (2000 – Not specified) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Support from private foundations Year Up, a non-profit None specified. Young male and female adults Weekly stipend during and corporations, individual organization headquartered aged 18 to 24 from low- classroom and internship contributions, public funds, and in Boston with network of income urban communities. phases of program tied to contributions from internship program sites across the performance contract. partners. In future, the program country Students must maintain high aims to draw on more public attendance rates, be on time, revenue sources. and complete assignments. Students who repeatedly fail to meet these expectations end up “firing themselves” from the program. United States: Job Training Partnership Act (Title II-A) (1982)/Workforce Investment Act (1998) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers United States Government Local training centers called Federal government Unemployed individuals None specified. Service Delivery Areas. Most defined performance (above 16) who met income services contracted out to measures that included eligibility limits or had low private providers, nonprofit employment rates and family income in the six agencies, or other average wage rates among months preceding application government agencies (such participants who found to the program. Each as community colleges). employment. At end of beneficiary assigned one of each program year, states three services offered. calculated performance 53 measures for each center and determined the reward it would receive. United States: I-BEST (2004 – Not specified) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Funded by Washington State Colleges in the Washington Funding for I-BEST is 1.75 Adult basic education Over half (58 %) of students Board for Community and community and technical times the normal rate per students enrolling in career got financial aid under Technical Colleges (SBCTC). college system must apply to full-time-equivalent student and technical education Opportunity Grants, State SBCTC for I-BEST program to compensate for the cost offered by community and Need Grants, and other welfare approval; every proposal of two instructors and high technical colleges. (Adult basic reform programs. I-BEST must have a “career coordination and planning skills students are adults who provides a structured pathway pathway", that is, a course of (a basic-skills instructor and lack high school-level skills). to college credentials and study that leads to an occupational instructor employment so that students postsecondary credentials teach in a classroom at the don’t have to find way on their and jobs in a given field for same time). own. which colleges must document demand. 54 Annex 4. Funding Modalities and Incentives for Training Programs for Out-of-School Youth (without Impact Evaluations) China: Second Chance Education for Out-of-School Adolescents (1999 – 2001) Funding Modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Supported by a UNICEF planning China Association for Science None indicated Disadvantaged youth from poor None indicated grant worth USD 110,000 approx. and Technology (CAST) was families who had been forced to The continuation of the project the implementing agency, drop out of school. Between the beyond the start-up phase was both leading coordination and ages of 13-18, spread over 15 supposed to be based on the implementation at the counties in 5 provinces. 2,250 review of the first phase. There is national level and undertaking adolescents attended training no clarity on whether phase two implementation at the courses. Gender neutral was conducted or not. township level. It appears that approach. local government provided support in mobilizing beneficiaries where needed and in training on literacy, arithmetic, production skills, and hygiene education. Lessons in Chinese and arithmetic were mainly delivered by teachers from elementary schools. Health education training was provided mainly by the medical staff of the local health bureau responsible for prevention/control of epidemic diseases. Skill training was usually carried out by trainers from local agricultural and animal husbandry departments. 55 East Timor: Preparing Us for Work (2007-2010) Funding modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers USD 5 million from USAID. Unclear Education Development None indicated Out-of-school men and women None indicated whether grant or credit/loan. Center, Inc (EDC) was the aged 16-30 from rural coordinator. The program communities (50% gender partnered with 14 local NGOs balance promoted at each to provide training to over learning center). 2,078 out-of- 2000 youth in 9 districts in East school youth received training Timor. The program assisted six of these NGOs with initiating registration with SEFOPE as certified training providers. Philippines: Alternative Learning System (ALS) Funding modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers The ALS Program is funded by the There are three models: There are different types Youth and adults who either did There are no incentives for Department of Education (DepEd), (i) “DepEd-delivered” of incentives for the not enroll in formal school or beneficiaries. In fact, ALS and other sources such as non- programs, which are different implementing dropped out before completing participation entails both governmental organizations and coordinated and implemented partners in the different the basic education cycle. explicit costs and opportunity local governments. The financing directly by the DepEd through models: Majority are between 15 and 24. costs of earnings foregone while arrangement is ‘direct-to- its mobile teachers and ALS •Learning facilitators in attending to studies. While facilitators’. Facilitators may also district coordinators. DepEd-delivered there is no tuition fee for the ALS raise financing from other (ii) “DepEd-procured” programs, including program, enrollees often need stakeholders, but they often report programs, which are district ALS coordinators to pay for transportation, meals, covering financial gaps using their coordinated and implemented and mobile teachers, are stationery and other supplies, own money. through service providers trained and vetted DepEd and other costs of participation. contracted by DepEd based on employees. Their careers the ALS Unified Contracting at DepEd are linked to Scheme. DepEd sets guidelines their performance, and and hires service providers they have strong such as public organizations, incentives to deliver high- private and public universities, quality work even without local government agencies, or frequent monitoring. community groups to deliver 56 ALS programs. These • Facilitators in DepEd- institutions further deploy procured programs are facilitators. external contractors and (iii) Small-scale programs that are not subject to the are funded by partners such as same institutional local governments and non- incentive structure. It governmental organizations does not appear that but follow the ALS learning there are any incentives materials and DepEd built into the contracts guidelines. between the DepEd and the service providers. Philippines: Education Quality and Access to Learning and Livelihood Skills Project (EQuALLs2) (2006 – Not specified) Funding modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Funded intervention of the U.S. Implementation Agency - EEA, None indicated Out-of-school youth in three None indicated Agency for International with the secretariat led by conflict-affected and Muslim- Development (USAID) in Consuelo Foundation and majority regions, ages 15-25. partnership with the Philippine supported by International Department of Education, DepEd Youth Foundation. Local NGO ARMM and Technical Education Partners implemented 6 Skills Development Authority projects under this program, (TESDA), local government units, of which 2 were industry and other government/non- associations. These NGOs government organizations. USD hired local teachers to conduct 617,968 from USAID and USD trainings. A total of 110 1,802,681 from partners (USD institutions, agencies, and 778,723 of cash contribution and private corporations were $1,023,958 of in-kind tapped to help the project. The contribution). The funding was programs adapted routed through the Education and Government-approved Employment Alliance (EEA) to curriculum. The nodal implementing non-profits. Government bodies were Department of Education - Bureau of Alternative Learning Systems, and Technical Education and Skills 57 Development Authority. Education Development Center was Coordinator/Implementor for the whole program. Timor Leste: A Second Chance at Education (2010 – 2016) Funding modality Training Providers Incentives for Training Beneficiaries Incentives for Beneficiaries Providers Funding of US$ 4.5 million was National Directorate of None indicated Youth and adults who had not None indicated provided by the World Bank Recurrent Education (NDRE) completed formal school through the International of the Ministry of Education is programs (basic and/or Development Association. the main implementation secondary education). agency. Municipal Education Offices also appear to have a role in local implementation, particularly enrolment, class distribution, implementation, local exams, and national exams management. The project set up nine community learning centers and prepared a curriculum, learning materials, and a teacher training process appropriate for mature students. Project also aimed to increase community participation in education (creation of community learning centers), improve adult literacy programs, and help train staff and develop school curriculum. 58 Social Protection & Jobs Discussion Paper Series Titles 2019-2021 No. Title 2103 Review of the Evidence on Short-Term Education and Skills Training Programs for Out-of-School Youth with a Focus on the Use of Incentives by Marguerite Clarke, Meghna Sharma, and Pradyumna Bhattacharjee January 2021 2102 Welfare, Shocks, and Government Spending on Social Protection Programs in Lesotho by Joachim Boko, Dhushyanth Raju, and Stephen D. Younger January 2021 2101 Cash in the City: Emerging Lessons from Implementing Cash Transfers in Urban Africa by Ugo Gentilini, Saksham Khosla, and Mohamed Almenfi January 2021 2010 Safety nets, health crises and natural disasters: Lessons from Sierra Leone by Judith Sandford, Sumati Rajput, Sarah Coll-Black, and Abu Kargbo December 2020 2009 A Reforma do Bolsa Família: Avaliação das propostas de reforma debatidas em 2019 by Matteo Morgandi, Liliana D. Sousa, Alison Farias, e Fabio Cereda November 2020 2008 The Role of Social Protection in Building, Protecting, and Deploying Human Capital in the East Asia and Pacific Region by Harry Edmund Moroz October 2020 2007 Boosting the Benefits of Cash Transfer Programs During the Early Years: A Case Study Review of Accompanying Measures by Laura Rawlings, Julieta Trias, and Emma Willenborg October 2020 2006 Expansion of Djibouti’s National Family Solidarity Program: Understanding Targeting Performance of the Updated Proxy Means Test Formula by Vibhuti Mendiratta, Amr Moubarak, Gabriel Lara Ibarra, John van Dyck, and Marco Santacroce August 2020 2005 Assessing the Targeting System in Georgia: Proposed Reform Options by Maddalena Honorati, Roberto Claudio Sormani, and Ludovico Carraro July 2020 2004 Jobs at risk in Turkey: Identifying the impact of COVID-19 by Sirma Demir Şeker, Efşan Nas Özen, and Ayşenur Acar Erdoğan July 2020 2003 Assessing the Vulnerability of Armenian Temporary Labor Migrants during the COVID-19 pandemic by Maddalena Honorati, Soonhwa Yi, and Thelma Choi July 2020 2002 Getting it Right: Strengthening Gender Outcomes in South Sudan by Samantha de Silva, Abir Hasan, Aissatou Ouedraogo, and Eliana Rubiano-Matulevich July 2020 2001 The Science of Adult Literacy by Michael S. C. Thomas, Victoria C. P. Knowland, Cathy Rogers January 2020 1936 Moving forward with ALMPs: Active labor policy and the changing nature of labor markets by Jose Manuel Romero and Arvo Kuddo November 2019 1935 Unbundled: A framework for connecting safety nets and humanitarian assistance in refugee settings by Karin Seyfert, Valentina Barca, Ugo Gentilini, Manjula Luthria, and Shereen Abbady September 2019 1934 Decentralization’s effects on education and health: Evidence from Ethiopia by Jean-Paul Faguet, Qaiser Khan, and Devarakonda Priyanka Kanth September 2019 1933 Extending Pension Coverage to the Informal Sector in Africa by Melis Guven July 2019 1932 What Employers Actually Want - Skills in demand in online job vacancies in Ukraine by Noël Muller and Abla Safir May 2019 1931 Can Local Participatory Programs Enhance Public Confidence: Insights from the Local Initiatives Support Program in Russia by Ivan Shulga, Lev Shilov, Anna Sukhova, and Peter Pojarski May 2019 1930 Social Protection in an Era of Increasing Uncertainty and Disruption: Social Risk Management 2.0 by Steen Lau Jorgensen and Paul B. Siegel May 2019 1929 Developing Coherent Pension Systems: Design Issues for Private Pension Supplements to NDC Schemes by William Price April 2019 1928 Pensions in a Globalizing World: How Do (N)DC and (N)DB Schemes Fare and Compare on Portability and Taxation? by Bernd Genser and Robert Holzmann April 2019 1927 The Politics of NDC Pension Scheme Diffusion: Constraints and Drivers by Igor Guardiancich, R. Kent Weaver, Gustavo Demarco, and Mark C. Dorfman April 2019 1926 Setting Up a Communication Package for the Italian NDC by Tito Boeri, Maria Cozzolino, and Edoardo Di Porto April 2019 1925 Sweden’s Fifteen Years of Communication Efforts by María del Carmen Boado-Penas, Ole Settergren, Erland Ekheden, and Poontavika Naka April 2019 1924 Information and Financial Literacy for Socially Sustainable NDC Pension Schemes by Elsa Fornero, Noemi Oggero, and Riccardo Puglisi April 2019 1923 Communicating NEST Pensions for “New” DC Savers in the United Kingdom by Will Sandbrook and Ranila Ravi-Burslem April 2019 1922 Harnessing a Young Nation's Demographic Dividends through a Universal NDC Pension Scheme: A Case Study of Tanzania by Bo Larsson, Vincent Leyaro, and Edward Palmer April 2019 1921 The Notional and the Real in China’s Pension Reforms by Bei Lu, John Piggott, and Bingwen Zheng April 2019 1920 Administrative Requirements and Prospects for Universal NDCs in Emerging Economies by Robert Palacios April 2019 1919 Bridging Partner Lifecycle Earnings and Pension Gaps by Sharing NDC Accounts by Anna Klerby, Bo Larsson, and Edward Palmer April 2019 1918 The Impact of Lifetime Events on Pensions: NDC Schemes in Poland, Italy, and Sweden and the Point Scheme in Germany by Agnieszka Chłoń-Domińczak, Marek Góra, Irena E. Kotowska, Iga Magda, Anna Ruzik-Sierdzińska, and Paweł Strzelecki April 2019 1917 Drivers of the Gender Gap in Pensions: Evidence from EU-SILC and the OECD Pension Model by Maciej Lis and Boele Bonthuis April 2019 1916 Gender and Family: Conceptual Overview by Nicholas Barr April 2019 1915 Labor Market Participation and Postponed Retirement in Central and Eastern Europe by Robert I. Gal and Márta Radó April 2019 1914 NDC Schemes and the Labor Market: Issues and Options by Robert Holzmann, David Robalino, and Hernan Winkler April 2019 1913 NDC Schemes and Heterogeneity in Longevity: Proposals for Redesign by Robert Holzmann, Jennifer Alonso-García, Heloise Labit-Hardy, and Andrés M. Villegas April 2019 1912 Annuities in (N)DC Pension Schemes: Design, Heterogeneity, and Estimation Issues by Edward Palmer and Yuwei Zhao de Gosson de Varennes April 2019 1911 Overview on Heterogeneity in Longevity and Pension Schemes by Ron Lee and Miguel Sanchez-Romero April 2019 1910 Chile's Solidarity Pillar: A Benchmark for Adjoining Zero Pillar with DC Schemes by Eduardo Fajnzylber April 2019 1909 Sweden: Adjoining the Guarantee Pension with NDC by Kenneth Nelson, Rense Nieuwenhuis, and Susanne Alm April 2019 1908 The ABCs of NDCs by Robert Holzmann April 2019 1907 NDC: The Generic Old-Age Pension Scheme by Marek Góra and Edward Palmer April 2019 1906 The Greek Pension Reforms: Crises and NDC Attempts Awaiting Completion by Milton Nektarios and Platon Tinios April 2019 1905 The Norwegian NDC Scheme: Balancing Risk Sharing and Redistribution by Nils Martin Stølen, Dennis Fredriksen, Erik Hernæs, and Erling Holmøy April 2019 1904 The Polish NDC Scheme: Success in the Face of Adversity by Sonia Buchholtz, Agnieszka Chłoń-Domińczak, and Marek Góra April 2019 1903 The Italian NDC Scheme: Evolution and Remaining Potholes by Sandro Gronchi, Sergio Nisticò, and Mirko Bevilacqua April 2019 1902 The Latvian NDC Scheme: Success Under a Decreasing Labor Force by Edward Palmer and Sandra Stabina April 2019 1901 The Swedish NDC Scheme: Success on Track with Room for Reflection by Edward Palmer and Bo Könberg April 2019 To view Social Protection & Jobs Discussion Papers published prior to 2019, please visit www.worldbank.org/sp. ABSTRACT Short-term education and skills training programs are a popular way to meet the needs of unemployed, out-of- school youth by providing them with an opportunity to quickly acquire qualifications and skills that can lead to productive employment. This paper reviews the global evidence to identify which programs are most effective at delivering results. How incentives for stakeholders are incorporated into the program design is given particular attention. Based on the findings, recommendations are offered for the future design of these training programs. ABOUT THIS SERIES Social Protection & Jobs Discussion Papers are published to communicate the results of The World Bank’s work to the development community with the least possible delay. This paper therefore has not been prepared in accordance with the procedures appropriate for formally edited texts. For more information, please contact the Social Protection Advisory Service via e-mail: socialprotection@ worldbank.org or visit us on-line at www.worldbank.org/sp