DISCUSSION PAPER NO. 1304 78980 The Nuts and Bolts of Designing and Implementing Training Programs in Developing Countries Maddalena Honorati and Thomas P. McArdle June 2013 The Nuts and Bolts of Designing and Implementing Training Programs in Developing Countries 1 Maddalena Honorati & Thomas P. McArdle June 2013 Abstract Training programs mainly address market failures related to lack of skills (technical, cognitive, non-cognitive). This paper conducts a comprehensive review of training programs effectiveness in developing countries. Based on relevant international experiences, the paper highlights key design features associated with program success as well as implementation challenges and discusses their policy implication. Success of training programs is deeply related with the content of the skills provided and how well they serve the local labor demand (demand-driven design) and with the presence of a sound governance structure for training providers and beneficiaries. In particular, the effectiveness of training programs for youth tends to be higher when a “comprehensive� approach is taken by combining different types of training with complementary support services. The ultimate goal is to inform new program design and improve the performance of current training programs. JEL Codes: J22, J24, J32, J38, I25 Key Words: training, skills, cognitive, active labor market policy, program evaluation 1 Contact information: mhonorati@worldbank.org. We thank Rita Almeida and Cornelia Tesliuc for a thorough review and thoughtful suggestions. We also thank David Grubb and Maria Laura Sanchez Puerta for their valuable comments. The paper also benefited from inputs and material provided by Pablo Acosta, Andreas Blom, Carlos Asenjo Ruiz, Arvo Kuddo, Cornelia Tesliuc, Nadine Poupart, Jaime Ramirez, and Diego Angel Urdinola. The authors also want to thanks participants to the IZA-WB-OECD conference on Activation and Employment Support Policies, Istanbul, May 2012. 1 Table of Contents 1. Introduction .................................................................................................................................... 4 2. Economic Rationale for Training Programs and Systems ............................................................... 7 3. Evidence of Training Program Effectiveness .................................................................................. 9 4. Program Design ............................................................................................................................ 15 Identifying and Profiling Beneficiaries.............................................................................................. 16 Types of Training Programs .............................................................................................................. 19 Modalities of Training Delivery ........................................................................................................ 22 Training Plans ................................................................................................................................... 25 Training Program Duration ............................................................................................................... 26 Support Services to Training Programs ............................................................................................ 28 5. Program Implementation ............................................................................................................. 30 Marketing a New Training Program ................................................................................................. 30 Recruitment process and beneficiaries selection ............................................................................ 31 Enrolment, Registration, and Orientation of Beneficiaries .............................................................. 33 Qualification of Service Providers and Quality Assurance of Training ............................................. 34 Selection of Training Providers......................................................................................................... 35 Delivering a Training Program .......................................................................................................... 36 Institutional Arrangements .............................................................................................................. 39 Governance ...................................................................................................................................... 43 Incentives to Program Stakeholders ............................................................................................ 44 Payment and compensation procedures ..................................................................................... 46 Complaint Mechanisms ................................................................................................................ 47 Assessment of Learning .................................................................................................................... 47 Costs and Financing .......................................................................................................................... 49 Monitoring & Evaluation (M&E) ....................................................................................................... 51 What Goes Wrong in Implementing a Training Program? Main Implementation Challenges ........ 55 6. Conclusions ................................................................................................................................... 57 References ............................................................................................................................................ 60 2 Annexes Annex 1. Recent trends in designing training systems Annex 2a. Kenya Youth Empowerment Project - Life Skills Training format and content. Annex 2b. Passport to Success Life Skills Training – International Youth Foundation (IYF). Boxes Box 1: Comprehensive Training in Honduras ....................................................................................... 25 Box 2: Improving the MIS for a Scaled-up Training System at Turkey’s İŞKUR .................................... 42 Box 3: Improving the MIS for a Scaled-up Training System at Turkey’s İŞKUR .................................... 53 Tables Table 1: Possible Target Populations.................................................................................................... 17 Table 2: Some Eligibility Criteria Often Used to Target Vulnerable Beneficiaries of Training Programs 18 Table 3: Barriers to Employment and Types of Training ...................................................................... 19 Table 4: Program Duration by Training Modality ................................................................................. 26 Table 5: Calculating Unit Costs for a Training Program ....................................................................... 50 3 1. Introduction The objective of this paper is document factors of success in program design as well as implementation challenges based on relevant international experiences with training programs in developing countries,2 with the ultimate goal to inform new program design and to guide practitioners and policy makers setting up their interventions. We summarize evidence from the existing impact evaluation literature on training programs, although still few training programs in developing countries have been evaluated and still major knowledge gaps exist to understanding how programs work, and why they work. We also know little about which design and implementation features increase the likelihood of program success. In order to fill these gaps and better understand the nuts and bolts of training programs, we complement the review of impact evaluation results with a desk review of operational manuals, project documents and quantitative and qualitative monitoring and evaluation (M&E) studies when available. This paper relates closely to a large, ongoing analytical and operational agenda at the World Bank focused on “activating� the most vulnerable (Almeida et al., 2012b) in the context of the 2012-2022 Social Protection and Labor Strategy, the 2013 World Development Report on Jobs3 and to a review of youth employment programs led by the World Bank Independent Evaluation Group. We also complement and update several other review papers on the effectiveness of active labor market programs and youth employment interventions (among others, Kluve et al., 2010; Betcherman et al., 2004; Betcherman et al. 2007; Todd, 2010; Almeida et al., 2012b). The conceptual and policy framework we adopt in the paper is drawn and adapted from existing works on building integrated set of skills development programs and policies during the entire life cycle (Banerji et al. 2010), building and upgrading job relevant skills for individuals who are leaving the formal general schooling system or are already in the labor 2 Programs reviewed are not necessarily the ones supported by the World Bank. 3 The 2013 World Development Report was launched during the Annual Meetings in the fall of 2012. 4 market (Almeida et al., 2012a), guiding all active labor market (ALMP) interventions for youth (Cunningham et al. 2011) and guiding activation and graduation policies and programs for the most vulnerable (Almeida et al., 2012b). While Almeida, Behrman and Robalino focus on pre-employment technical and vocational education and training (TVET), on-the-job training and training-related ALMPs, this paper proposes a simple framework to diagnose skills constraints and to design mainly training-related active labor market programs that are evidence based and positioned in a national training system. The scope of training programs discussed in this paper includes publicly-supported programs targeted to vulnerable people, short in duration (six months, maximum a year) and voluntary in nature. Governments support training programs usually through either direct provision (e.g., through public training institutes) or financial support (e.g., funding training costs and/or subsidizing trainees). Currently, the trend in many countries is to fund training costs and leave the actual training provision to the private sector. Programs are voluntary in the sense that beneficiaries are those who apply to participate in the program and satisfy appropriate program eligibility criteria (e.g., meet the age criteria for a youth program or the family income criteria for an income-targeted program). Vulnerable groups generally include the long-term unemployed, low-income, and low-skilled workers; youth who drop out of school or are transitioning from school to work; and welfare recipients. Various types of training programs can be designed to address skills constraints depending on the diversity of skills that matter to get better jobs. Specifically, we refer to (i) literacy and numeracy, (ii) vocational and technical skills; (iii) life skills; (iv) basic job readiness skills; and (v) entrepreneurial skills. Basic skills such as literacy and numeracy are the foundation for further cognitive skills development. For instance, vocational training programs generally assume a minimum level of educational competencies to succeed in a training program and a program’s content is often pegged at a minimum of grade nine for language, literacy, and numeracy skills. 5 “Second chance� programs offer either young school dropouts or adults an opportunity to return to an alternative educational setting to acquire secondary education qualifications. Vocational training programs develop job-specific skills that are demanded by employers. Technical skills include analytical skills specific to the sector, manual skills, routine processes. Comprehensive� training programs like the Jovenes4 model in Latin America are increasingly popular to reach out to vulnerable youth. They combine a strong investment in technical or vocational skills with life skills training and followed by a period of on�the�job work experience promoted through private sector internships. Some programs also offered job search assistance from six to twelve weeks. Life skills training modules (also known as “non-cognitive� or “soft�) tend to increasingly complement traditional vocational training programs for youth. Life skills training intends to develop basic behavioral skills and personality traits on social interactions and work� related skills such as communication, leadership, self�esteem, conflict resolution, decision making/problem solving but also as occupational hazard, sexual education, hygiene, and basic first aid. Basic job readiness skills training prepares and helps people search and retain a job by teaching how to perform in job interviews, personal strategies for looking for jobs, interview techniques, and curricula preparation. Basic jobs readiness skills also include information and communication technology skills such as word processing, spreadsheets and email which are nowadays often required in all formal sector jobs. Finally, training programs promoting entrepreneurships are designed for those who want to start their own business (potential entrepreneurs) or existing micro-entrepreneurs. Business development training programs often define basic entrepreneurial competencies such as 4 The Jóvenes type of model was developed in several countries Argentina (Proyecto Joven), Chile (Chile Joven), Colombia (Jóvenes en Acción), Dominican Republic (Juventud y Empleo), Peru (Projoven), and Panama (PROCAJOVEN). More recent experiences include the Kenya Youth Employment program; the Argentina Youth Internship Subprojects where work experience is combined with education and training, with the possibility of assessment and certification of competencies; and Honduras “Mi Primer Empleo�. 6 the ability to take risks, to plan and set goals, to gather information for decision-making, to persuade and negotiate effectively.5 Financial literacy, often combined with business training modules, includes understanding and managing the firm’s finances, deeper concepts such as business investment and growth strategies, different sources of finance for small businesses, the ways banks work, and the benefits of opening a bank saving account among others. The paper is organized as follows. The next section discusses the economic rationale for training programs and recent trends in national training systems. Section three takes stock of existing impact evaluation evidence of training programs in developing countries mainly on labor market outcomes. Section four proposes a framework for diagnostic analysis. It highlights key elements practitioners need to look at when designing training programs like identifying beneficiaries, understanding their constraints and skills needs, selecting the type of training material and curriculum, program duration, teaching and delivery methods. Section five provides guidance and good practice examples on some implementation areas such as selecting beneficiaries, the process of ensuring quality of training providers, learning assessment and certification, and how to best set up a governance structure. Based on existing quantitative and qualitative evidence particular attention will be paid on trying to understand main difficulties encountered as well as factors that may affect program success. Finally, the last section summarizes the main messages and concludes with policy recommendations looking forward. 2. Economic Rationale for Training Programs and Systems Training programs mainly address market failures having to do with lack of skills (technical, cognitive, non-cognitive) supply which is one of the main barriers to work in developing countries. Evidence from the World Bank’s enterprise surveys in more than 100 countries 5 Business training introduce basic business concepts and accounting skills, such as separation of business and personal household accounts, how to calculate costs and prices, profits and loss, how to identify businesses ideas and select the most viable one, marketing. 7 shows that lack of job-relevant skills among workers is seen by employers as one of the most binding constraint to firm growth. Training programs may be designed to address also other types of constraints—information asymmetries in the labor market, mobility6 and credit constraints and in some contexts also social constraints—which prevent the supply of job relevant skills to match labor demand. However, when market failures are associated to lack of labor demand because of slow growth and low job creation as for many developing countries, training interventions alone are not effective but must be complemented by other active labor market programs or being part of a more comprehensive labor market strategy. Skills constraints imply that individuals lack the appropriate technical, cognitive, and/or non�cognitive skills to respond to employers’ demand. Training programs are potential solutions to the problem of lack of skills for individuals who have left the formal schooling system, unemployed job seekers or those who are already in the labor market. But it can also be an issue for youth transitioning from school to work holding diplomas for which there is not enough demand. Workers who are forced to change areas of specialization in mid�career as a result of economic restructuring can also benefit from training programs. Lack of business and entrepreneurial skills is often an important constraint especially for the poorest who want to become self�employed. While training programs can lead to increases in productivity and employability, they can be costly and of limited use when job opportunities for trained workers are scarce. Governments should be involved in supporting training programs to increase opportunities for vulnerable people to access better jobs (wage employment and self-employment), hence reducing the burden of public expenditures on social assistance and unemployment insurance. Publically financed vocational and technical trainings are in most middle income countries the most popular activation programs. In the early 2000s, training represented the largest category (or 36 percent of total expenditure in active labor market programs 6 Firms may under-invest in specific training because of workers mobility. 8 (ALMPs)) among countries of the Organisation for Economic Co-operation and Development (OECD). For instance, OECD countries spend on average almost 0.8 percent7 of their Gross Domestic Product (GDP) on ALMPs8 which is modest compared to the policy implications of not addressing the problem of unemployment, especially youth unemployment. It was estimated (Belfield et al., 2012) that each of the 6.7 million unemployed and out-of-school youth in the United States cost to taxpayers almost $14,000 per year. The rationale for training provision is to equip disadvantaged people with the right skills for jobs, which are the ones demanded by employers, hence reducing the mismatch between skills demand and supply. Another argument for supporting training interventions is that training is seen by employers as a signal of higher productivity. However, there are cases in which the optimal policy response for governments is not necessarily to publicly subsidize or to directly provide training. Governments may have an important role to play in promoting skill development policies by providing accreditation and quality assurance policies for training providers, offering counseling services on career development, improving labor market regulation, and providing better information on existing training programs and more broadly to set up “training systems� (see Annex 1). 3. Evidence of Training Program Effectiveness Evidence from impact evaluations of training programs in developing countries is thin. While rigorous evidence in OECD countries shows low impacts on the effectiveness of job training programs to foster employment,9 the few existing impact evaluation studies in low and middle income countries suggest mixed returns on employment and earnings overall, but generally positive returns on female. 7 Betcherman et al. (2010). Expenditures on ALMPs programs on GDP are around 0.6 percent for ECA countries (Kuddo). 8 Including job training programs, entrepreneurial training on promoting self-employment, wage subsidies, public works, and employment services. 9 Betcherman et al. (2007); Card et al. (2011). 9 Evidence shows that training program types work differently across target groups in terms of labor market outcomes.10 Basic skills and second chance programs are more successful when they provide some certification. A certified degree or completion diploma represents a signal to potential employer that the person has acquired basic skills reducing informational asymmetries. The most well�documented and analyzed basic skills training programs are for OECD countries .11 Among middle income countries, the “second chance� component of Chilecalifica proved to be successful in increasing wages and post-program schooling outcomes, 12 highlighting the importance of accreditation and certification as part of training programs. Success of vocational and technical training programs is deeply related with the content of the skills provided and how well they serve the local labor demand (demand-driven design). Combining class-based technical and life skills training with on-the-job training such as internships or apprenticeships may more effective than providing classed based vocational training only. “Comprehensive� training programs for youth implemented in several countries in Latin America (the “Jovenes�) proved to be effective in promoting employability.13 For example, in Dominican Republic, results show almost zero impacts on employment and positive - though insignificant - impact on individuals’ earnings mainly driven from highly educated workers (Card et al., 2011). A more recent impact evaluation (Ibarraran et al., 2012) found that the program increased the formality of work, but did not lead to any increase in overall employment in the short-run. The impact of vocational training and internships of the Jovenes en Accion in Colombia were positive on both earning and employment, especially for women. Estimated returns have been of the order of 23-33 10 Impact evaluation studies focused their attention on four main labor market outcomes: employment, earnings, job quality (measured by getting a formal job or receiving health insurance benefits), and job stability (proxied by duration/contract type) conditional on employment. Among other outcome indicators the following are common: investment in further skills development, decrease in risky behaviors, psycho- social well-being, and non-cognitive skills. 11 Mattero (2010) and Cave et al. (1993) on the impact evaluation of JOBSTART program in USA. 12 Chilecalifica has been designed within a lifelong learning and training framework and is linked to the national system for Certification of Labor Competences. See Santiago Consultores (2009). 13 Ibarraran and Rosas Shady (2009), Betcherman et al. (2004, 2007). 10 percent in formal wages with respect to the control group. Female youths had 7 percent higher probability to have a paid job, an increase of three hours of work per week and 20 percent higher earnings than men. Overall, they found the higher the benefits the larger the time spent with on�the�job training (Attanasio et al., 2011). Significant impacts only on female employment were also found in Argentina (Aedo and Nuñez, 2004). Strong collaboration with the private sector, both in the selection of the training contents and in the provision of workplace internships, seems to be a critical factor of success in the Jovenes programs. Finally, complementing training with a personalized counseling and customized job search assistance is also associated with program performance (Betcherman et al., 2007). Anecdotal evidence from employers surveys around the world suggest that life skills—or behavioral, non-cognitive skills—are of increasing importance to whether a person is viewed favorably or unfavorably in the job market. Preliminary evidence from a randomized experiment of the Joventud y Empleo in Dominican Republic suggests that life skills training coupled with internships is as effective as vocational training on improving earnings and employment probability among women.14 The evaluation of the Jordan New Opportunities for Women program represents the first attempt to disentangle the effect of life skills training alone in a developing country. Groh et al. (2012) find no impact of life skill training on employment in the short term and a weakly significant 6.1 percent impact effect in the medium term, a year after program completion. The success of basic job readiness skills programs has rarely been evaluated. Although there is no rigorous impact evaluation, evaluations of the Entra 21 programs in Chile, 14 Martinez et al. (2012). The impact evaluation identification strategy uses an experimental design by randomly assigning applicant to a control group, a full treatment group receiving life skills, vocational training and internship, and a third group receiving only vocational training (and no life skills) and internship. For most outcomes, impacts from life skills only course are similar to impacts from vocational and life skills course. It suggests that conditional on work experience, vocational training does not contribute much to outcomes for women. 11 Colombia, Ecuador15 show that beneficiaries of ICT training by acquiring ICT skills, youth felt more motivated, improved the their job search strategy and social capital (business relationships with employers and their peers), and ultimately have better chances of getting a job but also a higher quality job. Rigorous impact evaluation studies looking at the effectiveness of training programs promoting entrepreneurship show different results depending on target populations. Generally, cost-effectiveness still remains to be tested. While the impact of business training programs on self-employment/business creation is positive, the evidence of the impact of business training programs on business performance is mixed for existing business owners. Business training programs have been ineffective in improving business outcomes of existing entrepreneurs in Peru (Karlan and Valdivia, 2010) and Dominical Republic (Drexler et al., 2012). However, receiving business training increased both the probability of starting a new business for potential entrepreneurs and the probability of expanding an existing business for existing entrepreneurs in El Salvador, Guatemala and Nicaragua, especially among women (Klinger and Schündeln, 2011). A business training program16 for women in Sri Lanka sped up the process of opening a business for those who were interested in starting a business but business training did not contribute to improve business performance (profits, sales) for women already operating subsistence enterprises (De Mel et al., 2011). Combined business skills and financial literacy training for female microfinance clients in India appears to yield modest returns on business income (Field et al., 2010). Some improvements in business practices, but no increases in business profits or in business 15 International Youth Foundation (2011). The Entra 21 initiative was developed in 2002 by the International Youth Foundation to improve employability of disadvantaged youth by providing training in information and communication technology, life-skills training, and continuous tutoring. The program was implemented in several countries in Latin America: El Salvador, the Dominican Republic, Peru, Panama, Colombia, Paraguay, Bolivia, and Brazil. 16 Start-and-Improve Your Business (SIYB) program. 12 survival rates were found among microfinance clients in Bosnia-Herzegovina (Bruhn and Zia, 2011). Training programs supporting entrepreneurship for youth tend to produce significantly positive short-term effects on self-employment and business creation.17 A program providing cash grants to vulnerable unskilled youth (aged 15-35 years old) in Uganda for investment in technical skills training relevant to starting a new business and capital for self- employment had a positive impact on probability of being employed and profits (Blattman et al., 2011). The number of hours of skilled employment increased by nearly a third; profits increased by nearly 50 percent on average. By contrast, a program providing technical skill training apprenticeship combined with entrepreneurial support and life skills training to vulnerable youth in Malawi found no effects on business creation or labor earnings four months after the apprenticeship (Cho et al., 2012b). The intervention had positive effect on the business knowledge and technical skills level, and investment in further skills development. Business training for the development of business plans and business plans competitions among university graduates has become a popular way of promoting entrepreneurial education among OECD countries and are recently tested in developing countries. Winners of business plan competitions usually receive honors and some start�up credit to develop their own enterprises. Experimental evidence in Tunisia shows that business training and business plans competitions were effective in fostering self-employment among university students. It was estimated that the program contributed to a three percentage point increase in the likelihood of being self-employed, results being mainly driven by men. However, the intervention did not increase the overall employment rate among beneficiaries. 17 The Word Bank is currently supporting a number of initiatives targeted to out-of-schoolgirls for which impact evaluation results are yet not available. These include, for example, the Adolescent Girls Initiative in Afghanistan, Liberia, Nigeria, Rwanda, and South Africa. The projects provide demand driven technical skills coupled with business development and life skills in order to increase the economic well-being and socioeconomic behavior of young women. 13 Preliminary results of a meta-analysis of impact evaluations (Cho and Honorati, 2013) of entrepreneurship promotion programs shows that impacts estimated on intermediate outcomes such as business practice and knowledge are likely to be more significant and larger than impacts on final outcomes such as earnings and profits. This is probably related to the short- and medium- term measurements of the effects. In fact, effects on business performance outcomes such as sales, number of employees are larger when estimates in the medium-longer term (after 18 months) after the intervention. The two results together suggest that final outcomes may take longer time to be materialized. Still, much remains to be learned on the cost effectiveness of programs and how they can be best delivered. Are job training programs worth social expenditures? Which are the design and implementation features of training programs that work best generating gains for participants that are large enough to justify their costs? And which are less successful? Evidence is scant, in particular on the cost effectiveness of different job training typologies, but also on the relative cost effectiveness of job training programs with respect to other ALMP interventions. In cost effectiveness analysis, benefits (usually estimated through impact evaluations) are complemented by cost analysis to identify which program alternative yields the highest rate of return in term of outcomes of interest. Cost effectiveness analysis is crucial for the potential policy impact. It allows policy makers to make informed decision on how to efficiently allocate resources (to programs that are more cost-effective). Calculations for some Jovenes programs have shown that comprehensive training can be cost effective. For example, the internal rate of return for the Projoven program in Peru have been estimated to be positive and above 4 percent. The Colombian Jovenes en Accion has also shown large returns assuming that that some of the skills depreciate overtime but that individuals have a working life of another 40 years. In Dominican Republic, the investment on training is recuperated after two years. Interestingly, recent evidence on the same program suggests that offering life skills training is more cost-effective. 14 4. Program Design The identification of the target population and their skill needs, along with the assessment of the country institutional capacity are key steps in designing a new training program. For instance, the design of a training program changes according to the target population and their skills needs: training programs addressed to the low skilled in general aim at enhancing human capital, training aimed at youth often has the objective of supporting youth entry into the labor market, training for experienced displaced workers has the objective to assist them in upgrading skills and transitioning between jobs, while training for unemployed may combine both training to get a job as well as an income support mechanism. The design of a training program should also be driven by the outcomes desired, e.g. wage employment in the formal economy, improving business performance and earnings in the informal economy, self-employment, acquisition of a particular skill such as growing a non- traditional crop, income generating activities, etc. This relates to the structure of the labor market in terms of formal versus informal activity, and goods versus services, and in particular the proportion of employment and types of employment in main sectors such as agriculture, manufacturing, community and social services, and cross-sectoral occupational areas such as office services work, as well as which sectors and occupation areas are growing and which are declining. What industries and job categories are growing? What occupational areas experience high turnover? What do these jobs pay? How many openings are projected? Where are these potential opportunities located? At what level are these jobs: semi-skilled, skilled, technicians, and line management? What opportunities exist for income generating activities and entry-level entrepreneurship? There are also differences between lower income and middle income countries that influence program design. In general, middle income countries will have more training capacity, with some public and private training institutions and possibly a national training 15 agency or another coordinating body. In higher capacity countries where national qualification frameworks (NQFs) and quality assurance systems (QAS) for certification exist, attention should be paid to partnering with a national training agency and integrating the new training program within the national qualification framework. Lower income countries in general feature smaller firms in their economies, more economic informality, and less developed institutional capacity. Given somewhat lower education levels, more attention may need to be paid to basic skills. Skills development in these countries will also likely focus more on capacity development, both basic and mid-level technical skills, including instructor development with additional emphasis on the informal economy and entrepreneurship. Identifying and Profiling Beneficiaries In training program design, the identification of the beneficiaries, or target group, is usually the impetus to the development of a training program as an activation strategy. One means of identifying beneficiaries is to use any national resource that serves as a registry of potential beneficiaries. This may include a registry of individuals or families receiving social assistance of any sort such as cash transfers, unemployment insurance, educational assistance, pre-natal and post-natal assistance, and disability payments. This approach is being used in Jamaica to target individuals in a welfare-to-work program targeting households receiving cash transfers. Poverty mapping is used to identify where potential beneficiaries may be located and concentrated as in the Dominican Republic’s Youth Employment Program. There may also be a registry for the unemployed or those seeking employment. Civil society organizations may have membership lists for such groups as small farmers, craft producers, persons with disabilities, informal sector vendors or small- scale producers. The portfolio of training programs we reviewed are targeted toward groups which vary in their vulnerability level: unskilled/skilled youth who drop out of school or are transitioning 16 from school to work, unskilled/skilled adults who are unemployed, welfare recipients, displaced or active workers living respectively either in urban or rural areas (see Table 1). Table 1: Possible Target Populations  Disadvantaged youth, including school drops  Mass lay-off workers (Romania) outs (Jóvenes)  Street & working children, orphans, neglected  The long term unemployed, including children (Yemen) graduates who have difficulty entering the  Prisoners & offenders leaving the correctional labor market (Morocco TAHEIL) system (Morocco)  Vulnerable women and single parent females  Conflict Affected People (Sierra Leone) (AGI)  Migrant Workers (China)  The Poor (including social assistance  Displaced workers (Serbia) beneficiaries, (Jamaica)  People with disabilities (Turkey)  Workers in the informal sector (Sri Lanka) A target group can be defined generally, such as the unemployed, or more specifically, such as unemployed youth, or even more precisely, such as unemployed rural women, and further, perhaps within a certain age group or geographical locale, or having certain characteristics such as “no secondary education�, etc. Program eligibility criteria define the specific target population the program intends to reach (Table 2). Criteria must be clear and easy to understand and implement, low-cost to measure, verify and monitor. Eligibility requirements might also intend to exclude certain categories of participants. For example, in a program aimed at young school dropouts, there may be pressure from school graduates to obtain admission if opportunities for jobs or tertiary education are scarce. Clear eligibility requirements mitigate this. Sometimes eligibility rules look at how a group is selected with the idea of ensuring that a certain proportion represents the most (or more) disadvantaged part of the target group. This may be necessary when training providers or firms dominate an actual selection process to counteract the tendency to “skim the cream� from the upper end of the disadvantaged group. 17 Table 2: Some Eligibility Criteria Often Used to Target Vulnerable Beneficiaries of Training Programs Employment status Generally eligibility is restricted to unemployed individuals Age Youth often targets an age range of about 17-25, but many projects expand age eligibility to older individuals (35). Education Many programs target youth who dropped out of school, but many include school graduates who do not pass a sufficient number of subjects or earn grades sufficient to matriculate into tertiary education Income Difficult to measure with reliability, sometimes proxy measures are used, but these may alienate applicants, e.g. indoor plumbing Previous training Applicants who have already participated in a government sponsored training program may be excluded Gender Training programs may be offered to vulnerable women only or include some gender targeting such as a minimum percentage of female program beneficiaries Geographic area, A program may target displaced agricultural workers in a particular demographic or social geographic area, or unwed inner city mothers, or street children and groups youth, or persons with disabilities The more rigid the eligibility requirements, however, the more prepared the implementers must be to exercise flexibility or redefine the requirements in light of actual implementation experience. Omission and inclusion errors are common during implementation, M&E systems should try to keep track and identify these cases. One means of identifying beneficiaries is to use any national resource that serves as a registry of potential beneficiaries. This may include a registry of individuals or families receiving social assistance of any sort such as cash transfers, unemployment insurance, educational assistance, pre-natal and post-natal assistance, and disability payments. This approach is being used in Jamaica to target individuals in a welfare-to-work program targeting households receiving cash transfers. Poverty mapping is used to identify where potential beneficiaries may be located and concentrated as in the Dominican Republic’s Youth Employment Program. There may also be a registry for the unemployed or those seeking employment. Civil society organizations may have membership lists for such groups as small farmers, craft producers, persons with disabilities, informal sector vendors or small- 18 scale producers. However, most of the times there is no registry and people simply enroll to training programs on an ad doc basis. Types of Training Programs Once a target group is clearly defined, beneficiary profiling is usually an exercise in defining the barriers to employment this group typically faces and, as a consequence, their skill needs. Successful training programs emphasize assessment of beneficiaries’ constraints to employability, through beneficiary profiling and program customization. Several types of training programs can be designed to address different types of skills constraints as shown in Table 3. Table 3: Barriers to Employment and Types of Training Barriers/Constraints Type of Training  Insufficient basic cognitive skills  Basic skills, literacy and numeracy, “second chance�  Lack of job-specific skills, technical skills mismatch  Technical & Vocational skills  Lack of behavioral skills relevant for jobs  Life Skills  Job searching strategies and techniques  Basic job readiness, ICT skills  Insufficient entrepreneurial skills  Entrepreneurial skills Basic skills training: These training programs provide an opportunity for participants to improve their basic education competencies and include both remedial education designed to enable participation in a technical skills training program, and so called “second chance� education programs targeted to school dropouts. Remedial education programs18 are sometimes inserted at the front end of a training program where educationally disadvantaged beneficiaries bring up their skills in reading, language and basic math sufficient to succeed in a technical training program. 18 As the term “remedial� often carries a stigma, it is not uncommon to rename remedial programs as Developmental Education. A program in the Caribbean operated by the private provider The Job Bank uses the term Documentation Skills. 19 Effective and efficient remediation requires diagnostic testing to identify the current level of the learners. This means a diagnostic instrument to test basic skills that need to be embedded in program design. Remediation is a promising area for computer-assisted learning and some positive results have been documented, although evidence with adult learners is scarce. A study in India compared a traditional remedial education program and a computer-assisted program (Banerjee et al., 2007). The traditional program resulted in an average gain of 0.28 standard deviations, while the computer assisted program showed a gain of 0.47 standard deviations. Technical and vocational training addresses technical skills mismatches and in particular the lack of job-specific skills demanded by employers. Generally, technical and vocational training offers a combination of theory/knowledge instruction with hands-on practical instruction working with the equipment, tools and supplies found in the workplace aimed at acquiring the competencies to work in a job or occupation. Competency-based training is a preferred method of delivery of technical and vocational training. Based on standards developed in cooperation with employers, a curriculum is developed using modules or units of learning and skill development linked to the standards. In competency-based training all the learning and skill development objectives and outcomes are explicitly stated in advance, learners move through modules at their own pace, assessment is conducted when the learner believes he or she is ready (sometimes based on evidence such as completion of a project or set of assignments), and certification is conducted in the units which accumulate into a full qualification over time. Life skills or “soft� skills and sometimes “employability� skills are of increasing importance to whether a person is viewed favorably or unfavorably in the job market and are relevant for all disadvantaged groups. Across the world, employers tell TVET providers and policy makers that these skills are even more important than technical skills in determining who gets and keeps employment. Life skills training aims at improving readiness for employment by developing behavioral skills—non-cognitive—and personality traits in 20 communication, team work, managing conflict, health and lifestyle education including STD/AIDS prevention, birth control, reducing involvement in drug use, crime and violence. The life skills training is also aimed at improving motivation and self-esteem. This area is also perhaps the most challenging area of training as the range of soft skills is wide and the number of topics is large (see Annex 2 for an example of life skills training content and format from the Kenya Youth Empowerment Project and the IYF “Passport to Success Life Skills� implemented in Mexico, India and other developing countries). Life skills programs are highly variable in how they are delivered and in their overall level of quality. Life skills training can be delivered either as a stand-alone program (but as a connected component within a comprehensive training program) or it can be integrated into vocational or business training programs. Basic job-readiness training helps people search and retain a job by teaching how to perform in job interviews, personal strategies for looking for jobs, interview techniques, and curricula preparation. Since almost all formal sector workers are now expected to have basic ICT skills, job-readiness training is increasingly including but also basic Information technology skills such as word processing, spreadsheets, and email. Some vocational training programs include a basic ICT skills course component of a predetermined set of hours, say about 40, to enable beneficiaries to acquire basic computing skills such as word processing, spreadsheets, and email. The integration of basic job readiness component to vocational and life skills training may be particularly important for displaced workers and older workers lacing ICT skills. Training programs promoting entrepreneurships are designed to promote self-employment among potential entrepreneurs—especially in labor markets with low wage employment— and to increase the productivity of existing micro-enterprises. The central feature of these programs is their promotion of entrepreneurial skills and attitudes, the ability to create and manage sustainable and efficient businesses that are capable of offering permanent jobs. The level of teaching should be tailored to program beneficiaries and simplified to be 21 understandable to people with low educational levels (below secondary). Most training programs supporting entrepreneurship are complemented with other business services such as start-up grants, access to credit and advisory services such as mentoring and counseling. Modalities of Training Delivery There are several terms commonly used to refer to a training program conducted in the workplace. For the purpose of this paper three kinds of on-the-job training are defined: apprenticeship, work experience, and internships. Apprenticeship is generally understood as a firm providing training to a learner referred to as an apprentice, who exchanges his or her low cost labor (although receiving wages) in return for training in a particular trade from a highly skilled (master) practitioner. The apprenticeship firm or master practitioner imparts all the training: both theory and practice. If there is a classroom component, it is provided within the firm. Thus apprenticeship is entirely on-the-job. Although the term is frequently used loosely to describe a variety of on-the-job training programs, apprenticeship is also often a legal term and operates in a legal framework. Informal apprenticeships are common in developing countries and are not regulated, but function with the same arrangements of low wages in exchange for training from a skilled practitioner. The Benin Support Project provided support to the on-the-job training of the crafts-persons and their apprentices to strengthen productivity of micro- enterprises, including improvements in technology, encouragement to transition to the modern sector, and increasing workers’ revenues. Follow-up studies show a positive effect of training on earnings, living conditions, and employment of participants. The apprenticeship program for vulnerable youth in Malawi offered vocational training in the desired occupations in master craftsmen’s shops combined with entrepreneurial support and life skills training. The duration and quality of apprenticeship training varied by trades and master craft-men. 22 Work Experience programs are aimed at giving new entrants to the labor force both job experience and the opportunity to learn a particular job rather than a trade. Both Jamaica and Trinidad & Tobago operate Work Experience programs. The Jamaica School Leavers Program has about 5,000 participants. The learners are not paid a wage, per se; rather, an employer may pay a stipend, and in the Jamaica program may deduct amounts paid against amounts owed to Jamaica’s training levy. In Trinidad & Tobago, the National Training Agency of Trinidad & Tobago pays the learners a stipend. These programs are one to two years in duration, depending on the job area. For example, the TAEHIL program in Morocco, where the National Employment Promotion Agency (ANAPEC) arranges apprenticeship-style programs using a tax incentive for firms to relieve them of paying social security contributions, and a vocational integration contract subsidy for firms who agree to provide apprenticeship for the long-term unemployed after a six-to-nine month vocational integration contract ends. Internships, sometimes referred to as traineeships, are shorter in duration and follow completion of a technical training program, and in this sense are not purely on-the-job (in this paper we treat internships as part of “comprehensive� training programs). The internship is intended to provide opportunity to apply skills learned in a technical training; the technical training may be conducted in a classroom teaching the theory and knowledge component of a job, or in a training center that uses both classroom and workshop. Internships are widely used in higher education, and are increasing popular in the design of new training programs aimed at disadvantaged groups and young people entering the labor market. These internships are generally of three to four months duration. Institution-based Training An institution-based training program features both classroom instruction, often with educational and life skills components, along with the theory/knowledge components of the occupational area, as well as workshops and labs for the practical, hands-on aspect of the occupation training. Since the educational requirements of most jobs have changed, 23 institution based programs usually have basic skills components that complement the training. They may also have remedial education offerings to enable admission of individuals who don’t pass an admissions test often used in screening.19 Institution-based training is conducted in a training facility operated by government or by a private for-profit or not-for-profit training provider to provide technical and vocational training. Institutional training is sometimes residential as in the USA Job Corps and at four institutions of Jamaica’s HEART Trust-National Training Agency. Non-residential training programs can be found in Colombia at SENA centers, SENAI centers in Brazil, at Turkey’s İŞKUR centers,20 and in many other countries. Nearly all developing countries operate some government training institutions variously named training schools, training centers, institutes, academies, etc. Many of these government centers in middle income countries now use a competency-based approach, with job standards validated by employer groups, and operate within a national qualification framework. Sometimes government-owned training institutions have been turned over to industry and sectoral groups to operate, with mixed success. Private training institutions have also proliferated in some countries in areas of labor market demand such as auto mechanics, chefs, nurses, and information technology. 19 Life skills, job preparation and job placement components may also be included in the offerings and some training institutions have moved toward the comprehensive training model described below which includes an internship. 20 İŞKUR operates both institution-based and on-the-job training programs, with institution-based training being by far the larger offering. 24 “Comprehensive� Training One of the more advanced modalities of training Box 1: Comprehensive Training in Honduras 21 is the “comprehensive� model that is built upon The Honduras Mi Primer Empleo (My First Job) both institution-based and on-the-job training. uses a comprehensive training design. The Widely used in Europe for many decades, so called program targets out-of-school, unemployed young people aged 15-21 living in five urban “dual training� traces its origins to Germany, and communities. The training program the term “dual� connotes both shared training components include life skills - employability skills, technical training of three to four between public institutions and firms and shared months, and an internship linked to that responsibility for training between federal and training along with counseling and job search assistance. Public and private training state governments. providers are selected through competitive procurement to provide training in agro In developing countries, comprehensive training business, tourism, light manufacturing, and forestry. The program provides stipends for usually combines life skills training with institution- food and transport for participants. based technical training of three to four months Implementation arrangements include the Ministry of Labor and Social Security as the lead followed by an internship placement of three to agency with the Ministry of Finance Project Coordination Unit, training institutions, and four months. Good practice is to involve employers in designing training plans and employer representatives as partners. recruiting learners. The internship component can also be implemented in intervals, e.g. two weeks of exposure to the workplace after the first month or so of technical training, with a second interval a bit later, and then a lengthier interval after the technical training is complete. This model has been very popular in Latin America and the Caribbean in a group of training programs referred to as the Jóvenes programs, and more recently has been adopted in the Kenya Youth Empowerment Program. Training Plans Training plans take two different forms in program design. An individual training plan aims at an individual beneficiary and selects interventions from a menu of training offerings, 21 Also known as “blended� “cooperative� or “dual� training. 25 support services and complementary services customized to meet the labor market constraints and overall needs of a client. In comprehensive training programs and in demand-led training in general, preparation of a training plan is usually centered on an employer or group of similar employers. Here the training program design would use labor market information to identify growing sectors and occupation areas, recruit participating firms, identify specific job areas for training with the firms, review and validate available standards, and develop a course outline and curriculum. This approach has the benefit of assured relevance of the content to the needs of firms and should lead to better job placement outcomes for beneficiaries. This approach is practiced in both the Kenya KYEP and Morocco’s TAEHIL program. In the TAEHIL program, the training plan and program content is validated by at least three companies. Training Program Duration One of the main questions people ask about training is how long should a training program be. The length of training programs varies widely based on the skill/occupational area, the level of skill, the different components of the training program such as life skills or education components, information technology and exposure to entrepreneurship, and the number of hours per week participants spend in training. Table 4: Program Duration by Training Modality Country and Program On-the-Job Classroom-Institution Internship Total Life Skills Technical Duration Training Morocco TAEHIL 1 year max. 1 year Dominican Republic Joventud y 75 hours 8 weeks 12-14 weeks Empleo (PJE) Turkey İŞKUR Vocational Training 6 months 6 months “Jovenes� Programs for Youth 200 – 400 hours combining 2-3 months Average 6 Training in Latin America life skills and technical months training Kenya Youth Empowerment 80 hours 2-3 months 3 months 46 months Program (KYEP) Jamaica School Leavers Program 1 to 2 years 1 to 2 years Jordan New Opportunities for 45 hours Women 26 Most national qualification frameworks contain “nominal� (estimated) hours for qualifications at different levels, with “level one� programs ranging from 300 to 450 hours and “level two� programs ranging from 450 to 600 or even 700 hours. The total number of hours prescribed may include classroom and workshop time, and will often accommodate some education, life skills and information technology. The nominal hours are usually set against beneficiaries who meet some standard in terms of basic education achievement, with a fairly typical assumption that participants operate at a grade eight-to-nine achievement level for reading, language and numeracy. If the target group is less educated, this assumption must be carefully considered and remedial education hours should be built into the training design. The length of most comprehensive training programs is three to four months for technical training with an additional two to four months of internship. In comprehensive training programs like Joventud y Empleo in the Dominican Republic, the life skills teaching phase consumes 75 hours, delivered in about two or three hours a day for thirty days. The internship phase consists of an internship of eight weeks in a company during which the trainee is involved with productive work similar to an apprentice. From the existing evidence base we don’t know much about the minimum or optimal duration of training programs and its impact on employment rates. A study on dropouts in South Korea revealed that a minimum of 14-17 weeks in a technical training program was necessary to increase employment probability (Choe et al, 2011). In general, the duration of the modular training will depend on the needs reported by employers. In general, nominal hours should be set against beneficiaries meeting some standard in terms of basic education achievement, with a fairly typical assumption that participants operate at a grade eight-to-nine achievement level for reading, language and numeracy. If the target group is disadvantaged, this assumption must be carefully considered and remedial education hours should be built into the course in some manner. 27 Support Services to Training Programs Counseling Services: Counseling services are an important element in designing a new training program and may take different forms. Once an applicant has applied to a training program, a counseling session with the applicant can be set up to explore selection of an appropriate course of training, and sometimes to test vocational aptitude (ability) and interests, as well as academic achievement22. Lower cost alternatives include group sessions and “training fairs� in which applicants can explore the training in a hands -on fashion. For example, the comprehensive training program Joventud y Empleo in Dominican Republic includes the establishment of training fairs in which interested youth can learn about the different training options, thus providing them with choices to match their interests and skills. Job preparation and job placement. These services may be offered by counselors, job developers or individuals trained to provide these services, designed to share information about the local labor market, especially what jobs are available, where they are located and what they are likely to pay; these may be conducted individually, but are more often conducted in a group setting. There is evidence from developed and transition countries that these low cost services are effective, but perhaps more so for more educated workers, for women, and for private services rather than public ones (Betcherman et al, 2004). Training programs in developing countries that incorporate these services into the program design do seem to perform better and they have become part of the comprehensive training program designs experiencing success in the Latin American context (Jóvenes programs). Multilateral Investment Fund (IDB), Entra 21 and World Bank Adolescent Girls Initiative training programs in Afghanistan, Liberia and Nepal include strong job placement components. 22 While counseling is helpful in facilitating appropriate choice of a skill area to learn, it also may be cost prohibitive as this kind of service requires use of specially trained personnel. 28 Conventional wisdom among vocational training practitioners on job placement suggests that job placement activities need to be closely linked with the training provider team itself, and that training institutions should conduct job placement directly rather than through a centralized job placement function operated by a vocational training system. It is also possible to contract job placement services to third party providers who may specialize in this service, and use a pay for performance approach (although there is no literature evaluating this). In any case, the relationship between the job placement personnel and employers is a key to success, and it is important for the person doing the job placement activity to have real performance information about the beneficiary, and preferably first- hand knowledge. Mentorship. Effective on-the-job training requires mentorship. Workplace supervisors with responsibility for trainees need to be exposed to train-the-trainer and mentorship courses sponsored by the entity that places trainees in OJT. The ILO has been a leader in developing and delivering such courses and partners with other organizations frequently to arrange this kind of training. Mentorship training generally teaches how to bring structure to a learning process by following a guideline, training regulation, occupational standard, etc., how to demonstrate skills and model appropriate behaviors, how to motivate learners and provide feedback, and how to monitor and report progress in attaining desired each particular skill and the required skill levels. Services supporting self-employment. Other services that often accompany training to foster self-employment include small cash and in kind grants, business incubation services, assistance in accessing credit, links or preferential access to micro-credit programs23. There is evidence (Cho, 2012) that these services help beneficiaries start their own business or income generating entrepreneurial activity when integrated with business training. 23 Other complementary services include child care, specialized transportation services and disabilities learning assistance. 29 5. Program Implementation Marketing a New Training Program In order to recruit target beneficiaries it is critical to generate awareness of the program. There are many ways to advertise a new training program. First, free publicity and different media outlets (newspapers, radio and TV) do respond to news items sent to them. This can also lead to interviews of the organizers and potential participants and follow-up stories when the program begins, while it is in progress and when it graduates a group. Today social media can also serve as a low cost way to inform stakeholders about a new training program. Governments also often have an information service that gets donated access to media outlets and this mechanism should certainly be explored. For example, in the Middle East and North Africa Education for Employment (EFE)24 recruits students through local non-governmental organizations (NGOs), professional and community organizations, and advertisements and postings at university campuses. Many EFE graduates also act as role models and goodwill ambassadors, recruiting other youth into our programs through the inspiration offered by their own successful employment. Specifically, through its internet-based "EFE Online," EFE and Europe EFE manage an Alumni Network to provide ongoing mentoring and continuing education for graduates. A promotional campaign is a structured program of information dissemination that may involve various media. This may involve radio spots, newspaper articles, and television feature stories. Testimonial evidence from participating learners and employers is effective and desirable. Such a campaign should also consider placement of posters, pamphlets, and applications in places accessed by the public in the communities served. This might include 24 EFE is a network of local non-profit organizations that seek to empower young people in the Middle East and North Africa region with the skills and opportunities needed to create a better future for themselves, their families, and their communities through increased career opportunities. In both Jordan and Yemen, EFE affiliates are working with very vulnerable youth populations, such as high school drops outs. 30 schools, libraries, places of worship, community centers and recreation facilities, restaurants, retail stores, etc. Marketing efforts should not overlook the promotion of particular skill and occupational areas, especially if recruitment is a problem. Gather information on estimated job openings and earnings, and use this in the promotion effort. Recruitment process and beneficiaries selection Application forms need to be carefully developed, vetted with training providers and implementing agencies to ensure they capture all the information that is going to be required at the point of application, and need to be tested with participants so as to be user-friendly. In addition to demographic characteristics and contact information25 (with backup contact information), the application form needs to cover the eligibility criteria in a thorough, but simple, easy to follow format.26 Sometimes a birth certificate and social security or other identification is required and is attached to the application. It is also useful at this point to provide assistance to applicants who do not have an official birth certificate or evidence of enrolment in government programs like social security. Frequently a photograph is taken. These measures prevent fraud in both admission and for assessment. It may also be appropriate to gather information about schooling, sometimes including school records. It is now possible to complete an application on a computer, and this often has the advantage of eliminating data entry tasks, but often this is not practical. In any event, the application information needs to be checked by recruiters, and sometimes verified or corrected. It is then necessary to compile application data in some form, often using a 25 It is important to be able to track individuals who are not selected in the program or who are offered a place in the program, but, for some reason, do not take up the offer, as these applicants may be considered a control group later if an impact evaluation is implemented. 26 This will include date of birth, age at application, years of schooling completed, previous employment, previous training, secondary and skills training certifications, etc. 31 computerized database to allow program management to query this data and to generate reports. The recruitment and application process must also include a communication strategy with the applicants. If applicants feel ignored, they will complain: to political representatives and sometimes to the media; needless to say this is undesirable. Management of this step is crucial to favorable perception of a training program.27 Performance standards should be set for the amount of time to process an application, how long it will take to make a selection decision, and how long from selection to admission. The selection of beneficiaries is completed against the eligibility criteria. There may be several ways to select beneficiaries: the establishment of selection committees set by training providers, through interviews with employers in case of internships (Morocco TAHEIL, Kenya KYEP) or randomized procedures in case of program oversubscription. A selection committee can be incorporated into the design to include representation of the training provider management and an instructor, the implementing agency and someone from the community or a beneficiary representative. In a new program, however, it may be the case that qualified applicants exceed, sometimes greatly, available training spaces (over-subscription). In this situation some sort of a randomized selection procedure such as a lottery may be implemented to ensure each eligible applicant has the same probability to be selected. Public lotteries were used in the Kenya’s Youth Empowerment Program and the Dominican Republic’s Youth Employment Program. Second, it is good practice to make the randomized procedure as transparent as possible, for example by letting public and youth representatives attend lotteries to select beneficiaries. 27 In some jurisdictions it is possible for a national training authority to participate in the application process. Such an agency may already have an appropriate application form and the computer facilities to store and manage the data, along with locations where applicants can get information and submit an application. It is important to check whether that agency’s application form and information system can identify the participants in the specific program being designed. Such an agency may also have counseling services, and offer testing for aptitudes and interests of trainees, all of which help learners make good decisions about entering training. If available, this should definitely be considered. 32 Generally, trainers are trying to select those with the highest likelihood of succeeding in the training program, meaning those most ready to learn the material, and those with the resources to attend the program and complete it. In on-the-job training programs involving employers in the selection, these tendencies are even more pronounced. These tendencies work to screen out more disadvantaged participants, and the phenomen on of “creaming� is widespread; use of a committee and randomization may lessen this tendency. Inclusion of the more disadvantaged could be done, although it is rare in practice, with quotas, advocacy, and proportional approaches to selection, along with incentives, perhaps a differential unit cost and payment calculation to include a certain portion of disadvantaged participants. In Joventud y Empleo in the Dominican Republic training providers who operate in the most disadvantaged areas are awarded additional points in the procurement process. Finally, selection procedures must be clear, transparent, and easy to implement and should avoid stigmatizing the participants. Enrolment, Registration, and Orientation of Beneficiaries The actual admission of an applicant to a training program involves an enrolment and registration process, followed by an orientation to the participation in the training program. Enrolling an applicant may involve gathering additional information, such as health information, next-of-kin, and additional contact information to enable follow-up contact after completion of training (including relatives and friends). Some vocational training systems now use a Learning Management System (Jamaica, Mongolia). These web-based systems store and manage trainees’ records in a particular training program like milestone dates, grades and instructor comments, assignment tracking, assessment information like dates and results, and certification and placement information. Instructors use them to store learning materials such as curriculum, lesson plans, exercises, and the like, and to track learners’ progress through the course units or 33 modules. If one is available it is useful to see whether the new training program can utilize it. Orientation of trainees is conducted with the group and must cover the program schedule, the distribution of classroom, workshop and independent practice time, the assignments that will be made, the rules of the training venue, the expectations of the training provider, and the assessment and certification process, and it is useful to solicit the expectations of trainees in such a session. If information on the occupational outlook for the skill area has not been shared, now is also an appropriate time to do so. It is appropriate to follow-up the orientation session with an email (and an email address may need to be set up for each participant) stating the exact time and place the learner is expected to appear to begin the training program, and to attach such things as the schedule, the rules and other written orientation material. Qualification of Service Providers and Quality Assurance of Training A key step in implementing a training program is the approval process for training providers and training venues. This approval process evaluates the ability of a training provider to manage a training program using qualified personnel, whether the provider has a governance structure to hold the provider accountable, and will pay particular attention to the ability to manage assessment. The approval will also evaluate the facility in terms of the adequacy of available equipment, tools, materials, safety issues, classrooms and workshops, and should use a set of facility standards to do this. The qualifications of instructors are also evaluated, as well as provisions for support services such as guidance and counseling (which may be by arrangement with a partner), and administrative arrangements for record keeping and reporting. Meeting such standards should enable all training providers whether public or private, for-profit or not-for-profit, to compete to provide training services financed by a training authority. Countries with more advanced training systems apply more formal approval procedures to training providers and some actually conduct an accreditation process, usually aimed at 34 established training institutions. Others conduct an appraisal that leads to a designation as an approved training provider or approved training center. In the start-up phase of the Skills for Inclusive Growth projects in Saint Lucia and Grenada, a “pre -qualification� procedure was used to determine eligibility to bid on operating a training component. A different approach consisting in selecting highly skilled crafts-persons and training them to become instructors is used for programs targeting the informal sector, frequently in African countries with successful examples in Kenya, Mali, Malawi, Morocco, and Gambia. This may involve some upgrading of their technical skills and instruction in theory, but mainly focuses on acquiring instructional skills. The Kenya Youth Empowerment Project (KYEP) used a pre-qualification approach to select master crafts-persons for skill upgrading to provide training for informal sector workers. In this type of pre-qualification aimed at the informal sector crafts-persons, the designer would develop criteria and an inspection protocol for selection, but rely to some extent on an informal sector civil society organization like the Jua Kali organization in Kenya to identify candidates. In the Malawi apprenticeship program, the master craft-men were selected based on their expertise and business performance, not through a formal assessment of their skills. They were trained by the TVET authority and provided with the curriculum, training modules customized for main occupations and learning outcomes to achieve. Selection of Training Providers The contemporary trend for selection of training providers is competitive procurement from a range of diverse training providers approved to respond to calls for proposals from a financing agency. This in turn leads to a market for provision of training by NGOs, CBOs, and private specialized training providers who execute a contract to provide training leading to certification. In Lebanon, for example, the New Entrants to Work Program (NEW) selects competing NGOs and private providers on a competitive basis to provide life skills training and placement into traineeship. Competitive procurement for training providers is also implemented for the KYEP in Kenya and in several Latin American countries. 35 Contracts may be fixed price per participant enrolled/completing, or may be based on cost reimbursement of agreed and capped costs if there is no experience with a particular disadvantaged group. Performance-based contracts and pay-for-results contracts such as contracts that pay for certifications achieved, or job placements actually made - are highly attractive to financing entities, but carry more risk for providers. In general, performance based contracting does not work well in new training programs, but can be introduced after several rounds of training have clarified costs and risks for training providers, and thresholds for performance can be increased over time. Alternatively, a bonus may be added to fixed price contracts like in the Lebanon NEW program where a bonus/reward system based on successful completion of the training and the traineeship, monitored through a management information system, serves as vehicle to ensure the quality of the different components. Delivering a Training Program Instructional delivery of a training program involves assigning an instructor, developing a time table for the course, arranging classroom time, workshop time, field visits, and apprenticeship/internship time if delivered during the technical training period. The size of the class also needs to be considered, as well as the ways records will be kept, from the enrolment registry to attendance tracking to progress notes and to assessment results. Instructors. Qualified instructors are the key to effective training.28 A qualified technical skills instructor needs to be competent in all the units in which he/she delivers instruction and needs to be qualified to provide instruction. Unfortunately, many instructors employed in technical training are not really competent in all aspects of the skill area itself and often lack actual industry experience. Many are trained as teachers for the formal education system rather than as vocational instructors, and are weak in adult learning methodology. 28 Generally a qualified instructor has achieved certified mastery (not just competence) of all the different units in the course the program intends to conduct, and has a qualification to teach or instruct. 36 They often do not practice effective modeling of the professional behaviors and attitudes expected in the workplace, because they do not come from the industry. It is better to recruit industry personnel and teach them how to instruct than to prepare instructors and then try to get them industrial experience, as that experience takes several years of experience or training to acquire. Instructor training is intended to provide a comprehensive skill set to deliver training. It is possible to recruit highly skilled artisans and crafts persons and train them to become instructors, and this is a preferred approach for its cost-effectiveness in many jurisdictions. The ILO, among other organizations, has developed “train the trainer� courses aimed at this group in particular and often used to train on-the-job supervisors and master crafts persons to deliver training. National training institutions, national training agencies and other national apex bodies often offer diploma and degree programs to prepare instructors within their network of training institutions. Life skills instructors may have different backgrounds from technical instructors and may come from social work and counseling backgrounds, and may have worked in the NGO community. A qualification for the life skills instructors does not exist; therefore to evaluate a candidate the experience in similar programs and with the target group would be most appropriate. Time tabling. More art than science, a timetable seeks to optimize the scarce and costly resource of time. It is generally possible to use the time in a given day to conduct two sections of the same or a similar course of training in a given day, by devoting the morning to theory, and the afternoon to workshop for one section, and the opposite arrangement for the second section. It is also possible to blend theory and workshop time in a variety of other ways, and theory or knowledge aspects can be taught inside the workshop as well. Class size. As a general rule, practitioners advocate a class size of 16 to 20 individuals (for a skills training component, although a life skills class can be larger and programs for special 37 target groups may be smaller). Class groupings larger than this compromise effectiveness in reaching all the participants, and class sizes smaller than this may be too expensive. Record keeping. Usually an instructor is expected to maintain consistent records of attendance at sessions, to record whether learners complete and submit assignments, to keep notes about performance in the training course, and to maintain records of graded tests and assignments used as part of “formative assessment�. If the instructor is also the assessor, then of course clear and detailed records of the assessment must be kept and probably secured. Training materials for instructors. A curriculum should be built from the standards, but sometimes a curriculum exists without connection to a standard (where standards are not developed at all, or have not yet been developed), and is usually the principal guide for an instructor.29 While a curriculum will track very closely to the occupational standards, if these exist, it might combine several units into a coherent whole for the purpose of delivering instruction and assessment. A curriculum will define the learning outcomes, the content, how to organize the material in terms of scope and sequence, the methods recommended to impart knowledge and skills, appropriate settings and places, teaching media and materials, and information on assessment and certification, especially how to assess the skills. The competent instructor uses a variety of techniques and tools in providing instruction. This may involve simulations,30 role plays, use of guest presenters from the industry, using 29 An Instructor’s Guide is used in a vocational training system or NQF to guide the instructor generally in how to structure the learning process according to the standards, how to ensure quality, and how to best prepare learners for assessment. In jurisdictions that have an NQF, there will be occupational standards that outline the job tasks, performance standards for these tasks, the range of situations the tasks will involve, underlying knowledge and skills, and ways to assess competence in the skill described by the particular unit. 30 The best known simulation training is the use of a piece of equipment known as the flight simulator. Simulations is the use of equipment that mimics that used in industry, e.g. in auto mechanics or industrial maintenance. Simulation equipment allows learners to acquire the understanding of machines in a safe manner without touching an actual expensive and/or dangerous piece of machinery. Simulation may also involve the use of a training environment that mimics the features of the workplace or the use of a mock hotel room to train hotel housekeepers, or a simulated crime scene to train forensic investigators. 38 and maintaining the equipment and tools, using project-based learning, using knowledge of learning styles, using computer-based presentations, and incorporating instruction in underlying math and communication skills into lesson plans. Learning materials for beneficiaries. These may include textbooks, exercise books, handouts, articles, practice tests, and project journals.31 Generally, a curriculum references specific learning materials that will be useful in delivery of a training program. Institutional Arrangements Oftentimes, the impetus for a new training program emerges from a combination of stakeholder interests which coalesce around a social problem; these may include a variety of stakeholder groups in the public sector such as government agencies involved in social safety net provisions, political directorates, national training agencies or those dealing with technical and vocational training, ministries and specific units of ministries responsible for labor, education, social planning, finance, economic development, community development, youth, gender affairs, agriculture, etc. Civil society stakeholder groups may also focus attention that leads to developing a training program including civil society interest groups representing youth, women, the poor, etc., as well as employers and sectoral interest groups. Donor organizations are another impetus to training program development. The specific formal configuration of these actors in the training program comprises the implementation arrangements, and the project design produces a workable arrangement for how these different actors can work together to achieve the program’s aims in the implementation phase. As a general rule, one agency is designated as the implementing agency, with others assigned support or complementary roles. 31 Some jurisdictions using an NQF now use a Learner’s Guide. These are developed for each competency unit and are applicable in both on-the-job and institutional settings, and help the learner to identify resources and activities useful in learning and refining the specific skills related to the competency units comprising a qualification. 39 In a training program embedded in a much larger project, implementation arrangements and governance may involve many stakeholders. The Serbia Bor Regional Development Project is a project to assist the Bor region of Serbia to diversify away from mining, and includes divestment of mining operations via privatization, environmental remediation, and a retraining component aimed at the displaced workers. The implementation arrangements involve key stakeholders including the Ministry of Economy and Regional Development (MOERD) as the primary implementing agency, the Ministry of Environmental Protection (MEP), the Ministry of Energy and Mining (MEN), the Ministry of Finance (MOF), the National Bank of Serbia (NBS), the National Employment Service (NES) mainly responsible for the training program components, the SME Development Agency, and the Privatization Agency (PA). In addition, Local Steering Committees; comprised of representatives of municipalities, regional offices of NES, NGOs, and business associations; were formed to oversee the project implementation on the ground. Partnerships have become an important way of extending the reach and increasing the depth of social service provision and are increasingly used in education and training to provide stronger and more comprehensive services to clients. The challenge, however, is effective leadership and coordination of the partners. Partner organizations may play a cooperative, supportive or complementary role in the implementation of a training program, and sometimes there may be a partner who is in a competitive position with the new program. National training agencies and institutions (both apex agencies for vocational training and the institutional network these agencies support) may be part of the institutional arrangements for a new training program. A national training agency may be helpful in areas such as recruitment of beneficiaries, cost-sharing of stipends, and supporting assessment and certification, possibly within a NQF. There is possibility for supportive and complementary services, possibly intermediation for internships, involvement of one or more of the training institutions, assistance in the area of job placement, curriculum, use of workshop facilities, etc. This will depend on the policies and capacity of the training agency 40 and its institutions. There is also a potential that these agencies and institutions play a competing role, offering somewhat similar programs aimed at parts of the same target populations, or using some of the same training providers. The private sector is, in many countries, a core player in the implementation of training programs through NGOs or private providers of various services, or employers in private firms offering internship and being involved in the training plan preparation. The private sector should be involved in provision of training programs through different ways including coordination with employers’ associations. One advantage of using NGOs to implement new training programs is that they may already have working relationships with beneficiaries, and be able to recruit participants readily. A second is that the NGO community provides a diverse range of services that may be a good fit with the training program’s planned activities. A third is that these organizations are accustomed to contracted arrangements, and the financing body gains an advantage of flexibility in choice of service providers without long-term commitments, in a relationship that counts on performance as the determinant as to whether or not relationships continue. However, there may be competing aspect to consider in some cases. An NGO may have more than one project competing for its attention, and more than one funding customer. Calls for proposals will want to solicit information on current programs operated by bidders to assess this kind of competition for attention. Employers and sectoral interest groups may include chamber of commerce, or sectoral groups representing manufacturing, the hospitality industry, agriculture, craft producers, etc. There are also civil society group that represent the interests of special populations. This may include the local disabilities foundation, an arts cooperative, women’s groups, etc. Formal partnership agreements are increasingly used to clearly specify partner roles and responsibilities among parties in partnership arrangements for training programs. 41 The World Bank financed Skills for Inclusive Growth Project in Grenada developed a partnership between the National Training Agency (NTA) and the Ministry of Youth. Faced with a challenge of employer resistance to paying stipends during internship as called for in the program design, the partnership has the Ministry of Youth paying the cost of stipends from its Youth Apprenticeship Program Box 2: Improving the MIS for a Scaled-up Training System at Turkey’s İŞKUR As a response to the economic downturn of 2008, Turkey’s İŞKUR institution based training programs scaled up from 32,000 participants in 2008 to 200,000 participants in 2009. In response to this scaling up, it became apparent that the MIS and performance management systems did not support the kinds of user- friendly querying and report generation needed to examine the performance of programs. İŞKUR partnered with the World Bank and a contractor, Impaq International to establish a Performance Measurement and Management System (PMMS). The purpose of PMMS is to monitor and measure program implementation at various levels (e.g., national, provincial, local). This involved improvements to and development of the data entry system, measurement of performance and outcomes, development of performance targets, and a system for generating management report. Now the system can provide better information on participant demographics, on providers and course performance, localized information on program performance, comparative information on unit costs, generate better management reports, and integrate a survey tracking system for participant outcomes. It can also compare performance of various occupational training programs across regions, better measure and monitor costs, and use data to improve performance and outcomes. (YAP), while the NTA will work with the other participants in the YAP to place them on a path to certification by the NTA. This is a win-win situation. The project also features partnership between the NTA and National Skills Development Center to recruit learners, provide up front employability training, monitoring of providers selected through competitive procurement, and job placement services. A partnership between the NTA and the Ministry of Education provides counseling services from trained Ministry guidance and counseling personnel. Sustainability and scalability of training programs involves considerations as to how to continue what works and grow what has been successful. Sustainability and scalability will consider a variety of issues. Management of the costs of the training program, particularly assessment costs and stipend amounts and who pays stipends will need careful 42 consideration as successful programs are scaled up. Capacity of the implementing bodies is another area of concern: if a program starts small, scaling it up may challenge the existing implementation arrangements. In the Dominican Republic’s Youth Employment Program (PJE) arrangements have been agreed to gradually transition the program to INFOTEP, the Dominican Republic’s national training authority. It will also be important to maintain and increase the capabilities and skill sets needed for competitive procurement and oversight of training providers, and to plan for the improvement of the competitive features of contracts such as performance incentives. Governance A governance body for a training program designed initially as a project is comprised of main stakeholders and is formed to assist in directing the implementation. This may have various names such as steering committee, project management committee, or similar name. It is imperative that the program design carefully define the roles of all the actors involved in operating the training program and document these roles in written form. The governance of government sponsored institution based training networks is now widely under a national training authority, sometimes funded through a training levy, with stronger employer representation at the national and sometimes regional level. These national training agencies study the labor market and plan the training programs, and often select training providers through competitive bidding. In recent years these training systems have increasingly focused on marginalized groups. Because of these reforms, training institutions and their national governing bodies may be good candidates for partnerships for new programs aimed at the disadvantaged. Reforms instituted to institution-based training include separation of financing of training from operation of training programs by implementing national training agencies and TVET councils and better ties to industry and employers by involving them in validating standards and incorporating them into management committees at institutions. 43 Incentives to Program Stakeholders Governance structures determine the incentives facing program stakeholders. Incentives as described here are financial transfers that reduce the cost of participating in a training program by employers, government agencies, training providers, related service providers, and the beneficiaries. Employers. Incentives for employers generally involve reducing the costs of employer involvement. Potential employer costs will typically include providing supervision of trainees, the cost of materials and supplies, and sometimes a contribution to the stipend for the learners paid to the employer to distribute to the beneficiaries. Reducing any or all of these costs will increase the chances for employer participation. In some countries tax incentives are offered such as relief from social security contributions (Morocco TAHEIL), or from payment of training levies as in South Africa’s Sector Education and Training Authorities (SETAs), Cote d’Ivoire’s Vocational Training Development Fund (FDFP), and Chile’s National Training Fund (FONCAP).32 Wage subsidies are also used as an employer incentive, particularly in situations of low labor demand and possibly to integrate marginalized groups into employment. For example, the Apprentices Hiring Program at Chile’s SENCE promotes employment of young people 15 to 21 years (and to 24 if handicapped) with a subsidy of 40 percent of the minimum wage for up to 12 months, and pays for some training provided by specialists. In the Kyrgyz Republic workplace vouchers are given to young people by the Kyrgyz Employment Service, and a company creating a new job for two years redeems the voucher from the employment service; the voucher reduces the cost of this hiring by two-thirds. Training providers. Incentives for training providers would include performance-based contract provisions that would pay a premium for certifications, apprenticeship/internship placements or actual job placements achieved. This provides an incentive to achieve desired outcomes of the training program, e.g. achieving a job placement rate over a certain 32 Johanson, 2009. 44 threshold, or attaching part of the fee to each job placement made. In Pakistan, the Benazir Bhutto Shaheed Youth Development Program (BBSYDP) relies on performance based contracts in which an additional payment is made if a trainee is employed three months after the end of training. Beneficiaries. Stipends are used to subsidize (reduce) the cost of participating in a training program and generally their amount is set to cover transport and a meal expenses.33 Programs aimed at economically disadvantaged groups tend to offer stipends to incentivize attendance and completion. For instance, payment of stipends is often directly linked to attendance, as an incentive to ensure showing up. In Pakistan’s Benazir Bhutto Shaheed Youth Development Program (BBSYDP), full stipend is paid for 80 percent attendance, since a portion of the learners travel up to 75 kilometers per day. Late payment of stipends is frequently encountered in training programs that offer stipends, especially when these are handled by a government agency. In payment of stipends, it can be very useful to the participants if the financing agency works with a local financial institution to establish accounts for the learners. This gets them started in the banking system, considering that barriers to getting a bank account are not uncommon for the disadvantaged. This may then enable electronic transfer of stipend amounts into the accounts, eliminating risk associated with transporting and distributing cash payments. An innovative approach to payment of stipends, especially for rural groups for whom a financial institution may be inaccessible, is the use of cell phone credits, which are tradable by the participants for cash. In Kenya, where over 90 percent of homes now have mobile (cellular) phones, the use of “mobile money� has emerged with 73 percent of people using mobile money, mostly as a repository for funds (Demombynes et al., 2012). This phenomenon is facilitated within the M-PESA system launched by Safaricom in 2007. The 33 Setting too low stipend amounts may increase beneficiaries drop out. In some programs the distance participants’ travel may vary significantly and may be taken into account in defining the stipend amount. 45 Information Communication Technology for Development (ICT4D) Project there reported this as a new way for social programs to make cash transfers. Payment and compensation procedures Payment for a training program is determined by the contract which determines the payment provisions in advance. Contracts may be fixed fee, an agreed amount for the program or for each individual in the program, or based on cost reimbursement of agreed costs within predetermined levels back up by proof of payment. If a profit or surplus amount is possible, this is built into the fixed fee, or added to the cost reimbursement and termed cost plus fixed fee. As a general rule, compensation and payment policy and practice for these new demand- driven, results-oriented programs ties payments to performance milestones. An inception payment is made, followed by payments tied to the performance milestones. These payment milestones might include completion of training program components, placement into or completion of internships, award of certification, or job placement. Payments for award of certification and job placement for a specified period of time are sometimes awarded as bonus performance payments. Since payments may be made on a per-individual basis, i.e. tied to each individual who reaches a given milestone, the issue of dropouts will need to be considered, as dropouts may be outside of the control or influence of a training provider and may reflect the selection process. Sometimes unmotivated participants are selected; some disadvantaged individuals may dropout due to financial hardships, unexpected pregnancies, or other factors that the training provider cannot control. It is typical to cover all reasonable costs incurred by the training provider defined in the contract and to enable the contractor to manage risks. Compensation levels for the personnel involved in a training program are relative to other organizations competing for the same workers.34 34 Conducting a salary survey may be useful if there is no clear benchmark for the project being designed. 46 In a government operated training institution, the general principle of compensation for training instructors is an approximate parity with comparable industry personnel at different levels, with perhaps a small discount of about ten percent recognizing (perhaps, but not necessarily) greater job security and improved fringe benefits of working for the government. Private training providers will have compensation very much linked to the sector and industry of training, as well as the level of training. NGOs often pay their senior personnel at a much higher level than line personnel, and, again, it is useful to gather information about actual pay rates. If there is a national training entity already contracting training, they will have information on what training providers pay their personnel. There is no industry counterpart of the soft skills/employability instructor, so comparability with the teaching workforce and NGO services personnel is probably appropriate there. Complaint Mechanisms Things sometimes go wrong in training programs and participants are negatively affected. It is important to document a procedure for complaints and grievances. Late stipends, unsatisfactory instructors, unfair treatment, disputed assessment results, and staff misconduct can all occur. Learners need documented procedures for how to handle grievances with training programs, the certification awarding body, the national training entity, and appeal procedures for assessment decisions. In a training institution they must have a way to reach administrators and directors, and in a project, how to get an unresolved complaint or grievance up to the steering committee. This may also include defined dispute resolution mechanisms such as referral to a conflict resolution agency or formal arbitration. Assessment of Learning Assessment of competence plays a prominent role in implementation of national and regional qualification frameworks. Providing assessment services within a competency qualification framework is a means to recognize skills among citizenry to advance the 47 competitiveness of the workforce, regardless of where and how they were acquired (in a school or training center, in the workplace, or through independent study and self- employment). There are two types of assessment: first, “formative assessment� is used during training by instructors to monitor progress, address strengths and weaknesses of the trainees, and provide feedback and guidance. Second, “summative assessment� which is undertaken by qualified and often external assessors to assess skills learned. For the latter type of assessment, services will usually be provided by a trained and registered assessor with possible coordination by the certification awarding body. Good practice is that (1) assessments are conducted by independent assessors not associated with the training provider, and (2) that the assessors are drawn from the industry and not from the training sector. It is important to ensure that qualified assessors are trained and available, must plan and budget for the assessments to take place, and consider how to calculate this using integrated assessment in small groups to some extent.35 Assessment is conducted against the occupational standard and not the content of the training course per se. The assessment is supposed to include all the technical requirements of tasks, as well as the underlying knowledge and skills, inclusive of the soft skills required for the level of quality required by the standard. Results of an assessment are usually stated simply as “competent� or “not yet competent�. When all the units comprising a qualification have been assessed with a finding of “competent�, a recommendation is made for certification. The results of assessments, however, are reviewed within the quality assurance provisions to ensure the assessments conducted meet the standards set for assessment.36 35 Integrated assessment combines several elements or units for assessment with conducting assessment in small groups of three to five candidates. If this approach is not implemented, the cost of assessment is too high and poses problems for sustainability. 36 A mechanism for the appeal of assessment decisions needs to be in place and documented in an assessment guideline, and perhaps as well in a guide developed for the specific training program. 48 Costs and Financing Costs of a new training program will obviously include the costs of the skills training, but might also include costs related to capacity building such as developing a demand-led training system, developing standards, training assessors and internship mentors, administrative costs such as the ability to procure training through competitive bidding and improved ability to monitor and evaluate training provision. The calculation of unit costs for training programs should include consideration of start-up costs, the cost of implementing the training, costs related to assessment and certification, and costs related to program management and administration (overhead). The sum of these costs is divided by the planned number of participants to yield the unit cost. If the program has other components such as capacity building, these should be calculated separately for that component. Table 5 below lists four categories of likely costs (and how they might be computed) to calculate a planned unit cost that may be used to calculate the total cost of a new training program. Sources of financing might include government, through its consolidated fund, or through a training levy that many countries have that supports national training provisions and national training apex agencies.37 Funding from training levies has supported numerous new training initiatives. Jamaica’s National Training Agency, the HEART Trust, for example, implemented over 200 community-based training programs over the past ten years using its training levy. In most cases, development assistance partners provide bi-lateral assistance through country organizations like Canada’s CIDA or USAID, and include similar organizations in Britain, Japan and Germany. Multi-lateral assistance can come from the European Union, or from the development banks including the World Bank, Asian Development Bank, African Development Bank and Inter-American Development Bank. 37 A third source of financing is fees from learners. Since this paper is addressing mostly training for the disadvantaged, fees will not usually be a source of financing. 49 Table 5: Calculating Unit Costs for a Training Program Start-up Costs  Program launch: venue rental, flyers, posters, applications, exhibit  Recruiting learners: possibly testing them, data entry of applicants’ information  Recruiting training providers to submit proposals and employers to offer internships: cost of meetings and workshops Costs of Implementation of Training  Registration-data entry of registrants’ information  Soft skills/Life skills/Employability skills  Cost of venue  Cost of instruction (average hourly rate x hours of instruction)  Skills training: an instructor and administrator are needed to estimate these costs  Cost of venue: some pro-rated or rental estimate for facilities, equipment, supplies, not appropriate in all instances  Cost of instruction (average hourly rate x hours of instruction)  Cost of learning materials: Learner Guides, handouts, materials, textbooks and other resources in a library; these may vary considerably by skill area  Data entry of learner progress information  Optional Training: Basic or remedial education, Information Technology, Entrepreneurship (cost is calculated similar to soft skills training) Costs of Assessment & Certification  Assessment (Number of participants in a class) / (group assessment size using an average of 3.5 candidates) x (number of units in the qualification) / 2.5 units assessed simultaneously.  Data entry of assessment results  Internal verification of assessment: (number of group assessments) x (number of units / 2.5) x hourly rate for verifier (although this may not necessarily be an additional cost if a staff member is assigned as verifier)  External verification by training authority: (number of group assessments) x (number of units / 2.5) x (10% - 20% sampling rate) x (hourly rate for verifier) + 20% of cost to verify higher risk assessors and training providers  Certification costs: certificates, data entry  Graduation ceremonies (venue rental, rental of graduation gowns, gifts or honoraria for certain presenters, refreshments) Overhead costs  Administrative costs of the training provider for management services, payroll, HRD, etc., usually are a percentage of total direct cost, and should be justified and examined in detail by the contracting organization.  Insurance coverage such as liability insurance for the financing agency and training providers Source: Authors’ calculation. Training programs may be financed through payments to the organizations that operate the program, usually arranged in tranches related to milestones. Alternatively, vouchers given to qualifying learners have also been used as a financing mechanism for training programs. 50 These put power in the hands of those intended to benefit and introduce an additional competitive element into training provision. In Kenya, the World Bank supported a special voucher program for young adults to evaluate demand for public and private vocational training and the impact of training on job seekers. The evaluation showed that offering young adults vouchers that cover program costs does encourage young adults to enroll, and that those who can use the voucher for a private training program are more likely to sign up and stay in school.38 Vouchers for training are also widely used in the United States within the Workforce Improvement Act training program, where an Individual Training Account finances training for eligible individuals (those receiving unemployment insurance and others). Beneficiaries select a training course from a list of approved training providers. A recent study by Mathematica Policy Research Inc.39 showed cost-benefit advantages of higher training awards for these mostly experienced workers. Monitoring & Evaluation (M&E) A strong M&E system is crucial for program success and improvement of performance over time. It will ensure that problems can be identified and solved as they emerge and that lessons learned can be incorporated into project design and ultimately that project outcomes can be assessed. A good M&E system also provides management with a timely decision-making tool. Managing for results usually involves specifying expected results and aligning processes and resources behind them, as well as ongoing monitoring and evaluation, and continuous feedback to improve performance. Results frameworks are critical tools for results-based management and provide the basis for program monitoring and evaluation. They help in 38 Students who received vouchers that could only be used for public training programs also were 16 percentage points more likely to drop out than those who could use the voucher for both private and public institutions. 39 Perez-Johnson et al, 2011 51 clarifying and getting consensus on programs’ expected results and making explicit the theory of change behind them. Results chains depict the theory of change behind a program. In other words, results chains are graphic representations of a series of logical steps of cause and effect that link the activities of an intervention to the desired outcomes. At the top of the chain we will usually find the overall larger objectives a project or program is intending to achieve, then the activities, outputs, short term and intermediate outcomes that cumulatively lead to achieving the overall objective in a chain of cause and effect. For each link in the results chain, indicators are set to track program performance respectively in the area of inputs, outputs, activities and outcomes.40 A good results framework will also specify the data needed to measure each indicator, how frequently the indicator is measured and who is mainly responsible to achieve and monitor and update the indicator. Tracking the indicators, usually done monthly or quarterly, but certainly annually, helps us see if we are making progress toward achieving objectives. Ideally, performance monitoring systems should build upon reliable program records (generated by MIS) and be complemented with a combination of tailor-made data sources and national surveys. The latter should be considered not only for monitoring purposes but within a program evaluation agenda. An M&E System often includes the following elements: Management Information System (MIS):41 In order to effectively provide M&E the program MIS are set up to store and manage data and to generate reports. The MIS is usually a database system and stores administrative data such as data on beneficiaries. 40 Project planning literature routinely recommends using SMART indicators. The acronym SMART conveys the criteria for developing good quality indicators. SMART translates to Specific, Measurable, Attributable (or sometimes Achievable), Realistic, and Time-based (or sometimes Targeted). In setting indicators we look for concise and clear statements, and we set targets that “stretch� performance and thereby stimulate improvement and increased efficiency, but we never set unachievable targets, as people will give up if they view a target as unreasonable. 41 The same program performance data can be compiled for all programs in a training system. 52 Characteristics relevant in a training program will include bio data on participants (gender, age, geographic origin, education levels, employment history data, whether they have children etc.), data on learner progress and performance (sometimes grades), data on completion of units and progress toward certification, data on internship placement and job placement. MIS also collect the raw numerical data for the program’s capacity (the planned and budgeted enrolment), actual enrolment, completion, dropouts, certifications, internship and job placements, and then calculation of various rates: enrolment rate, completion rate, certification rate (and how exactly this is measured and stated),42 internship placement rate, job placement rate, total performance measures, and actual unit cost of programs. It is also useful to track measures of program performance over time. Box 3: Improving the MIS for a Scaled-up Training System at Turkey’s İŞKUR As a response to the economic downturn of 2008, Turkey’s İŞKUR institution based training programs scaled up from 32,000 participants in 2008 to 200,000 participants in 2009. In response to this scaling up, it became apparent that the MIS and performance management systems did not support the kinds of user- friendly querying and report generation needed to examine the performance of programs. İŞKUR partnered with the World Bank and a contractor, Impaq International to establish a Performance Measurement and Management System (PMMS). The purpose of PMMS is to monitor and measure program implementation at various levels (e.g., national, provincial, local). This involved improvements to and development of the data entry system, measurement of performance and outcomes, development of performance targets, and a system for generating management report. Now the system can provide better information on participant demographics, on providers and course performance, localized information on program performance, comparative information on unit costs, generate better management reports, and integrate a survey tracking system for participant outcomes. It can also compare performance of various occupational training programs across regions, better measure and monitor costs, and use data to improve performance and outcomes. Evaluation: There are several different kinds of evaluation studies that can help improve the performance of training programs. A program evaluation will examine a set of variables related to how well a program achieves what its design intends, often by examining inputs, processes and outputs, and assessing and making judgments about the instructors, facilities, equipment, curriculum, teaching practices, etc., and examining a program’s performance 42 Certification rates should generally be calculated as the total number certified divided by the initial enrolment, and should thereby take dropouts into account, and should not be calculated only in relation to completers. In any case, calculation of this rate needs to be clear. 53 over time in terms of trainee selection certification rates, job placement results and the like. A program evaluation should collect information from the trainees, instructors and administrators. Tracer studies are aimed at understanding outcomes of training and follows graduates of a training program to see what happens in terms of labor market absorption, the relationship between the jobs they get and what course they took, their earnings and the quality of jobs obtained. This may involve a sample if the group is large, but it is important to get a sufficient sample size to make generalizations about each particular training program. The survey might also ask for information about how they feel about the program and whether they want to obtain further education and training. Some tracer studies seek to learn about unintended effects of training such as effects on family size, labor mobility, various behaviors such as drug and alcohol use, etc. A tracer study provides a labor market indicator as to whether training in a particular skill area was absorbed by the market. Creating a tracking system for trainees who may not be hired in places where they did the internship, could be helpful in order to improve their employability—searching for jobs, providing information on open work positions, or improving their occupational qualifications. Beneficiary and employer satisfaction surveys. A survey can be administered to the learners at the conclusion of the course to rate learner satisfaction with the course and instructor. Employer surveys evaluate whether trainees actually have the knowledge, skills and attitudes employers desire and employers satisfaction with the training program. Return on Investment (ROI) studies look at the economic investment in the training and compare this to the earnings of the graduates, including consideration of income foregone as a result of attending the training. Finally, impact evaluations attempt to measure the impact of training programs with respect to a counterfactual (how beneficiaries would have been without the program). 54 What Goes Wrong in Implementing a Training Program? Main Implementation Challenges Not sufficiently involving employers, not listening to what employers say, and responding to employers’ needs too slowly. Employers generally can only forecast for the immediate future, like the next year, unless they are fairly certain of growth or contraction over the medium term in a particular industry. If they say they will not be in a position to accept trainees or trained individuals, it is not likely they can be persuaded to do so. When employers stress the importance of soft skills, the project better ensure that the learners acquire these and make sure these are really assessed properly. Choosing skill areas without adequate labor market information. Many countries lack sufficient labor market data to understand how the labor market is changing and what jobs are increasing and decreasing in number. A low growth economy may produce almost no new jobs at all, and thus no growing opportunities. The use of tracer studies, rapid appraisal techniques, focus groups, and small scale surveys may address this problem, but may also produce considerable uncertainty. Training programs do not create jobs and must follow actual and not imagined labor market demand. Putting too little into job placement activities. If trainees get internships they are much more likely to be offered a job with that employer, so training programs need to secure internships. Those not absorbed in this way need job preparation and job placement services. The best job placement services are organized by the training provider and not some centralized job placement activity where the job developers do not know the learner personally. Setting stipends too low. If stipend amounts are too low, there will be a high risk of trainee drop-outs due to financial hardships of attending. Communication between beneficiaries and implementing agency. Failure to deal effectively with complaints and grievances poses a great risk for negative publicity and negative political consequences. 55 The first cycle of a new demand driven training program is often plagued by problems. Pressure to start training on the agreed schedule, including political pressures to start, often compel the implementing agency to move before things are truly ready. Furthermore, implementation complexity is often underestimated. This often results in problems at every step: in the initial implementation it is common to have too few or too many applicants, and selecting applicants who are not ready to benefit, or who are placed in a skill area that does not actually interest them. In the proposal process it is not uncommon to receive poor quality proposals and proposals from training providers who are not the ones envisaged and desired. In conducting the training there are difficulties finding the right instructors, and coping with high dropout rates. With internships, we encounter difficulties recruiting employers and especially the more desirable employers. With assessment, there are challenges with weak assessment practices and too few assessors. Administratively, there may be problems with delays in making payments to providers and paying stipends to trainees. Finally, the planning may have underestimated the time and effort required to have the certification procedures and arrangements in place, and the trainees complete the program but don’t have the certification promised. These problems are common especially at the program start up or the first cycles. Generally, however, with proper management, M&E and governance, these problems are identified and solved during a second and third cycle. It may be a gradual process to improve the knowledge and skills of the main implementation actors, improve screening of beneficiaries, get better training providers involved and submitting proposals, identify gaps in the services and fill them, improve administration of payments, improve assessors’ performance and increase the number of assessors, adjust payment amounts by reviewing unit cost calculations and differentiating payment amounts according to the kind of training, and gradually get more and better employers involved. 56 6. Conclusions Many people lack the right skills to access formal wage employment and self-employment. Which are the types of training programs to invest in? How to design and implement them? This paper represents one of the first attempts to conduct a comprehensive review of the evidence of training programs in the developing world in order to draw lessons for policy design and implementation. We propose a conceptual framework to review and analyze programs drawing form both the impact evaluation literature and the operational evidence based on program documents and M&E reports. Designing and implementing a training program is a complex but manageable process that should lead to more productive and fulfilling lives for beneficiaries and their families, as well as benefits to society in terms of reduced dependency and increased tax revenues, and benefits to firms in terms of availability of skilled workers to enable improved productivity and competitiveness. In program design the profiling of beneficiaries, and matching their needs and barriers against types, modalities and duration of training is the main challenge, while striking a balance between the range of needs of the target group, the financial resources available, the skills needs of firms and the economy, and the number we intend to serve. In program implementation the task is to bring all the elements into a coherent whole that enables stakeholders to work together effectively while making their particular contributions, and to operate an efficient and effective program with all the necessary supports that leads to beneficiaries acquiring the skills intended and becoming employed, while being accountable for the desired results. The best design and implementation will probably be a comprehensive and integrated program that addresses both job skills and the life skills/employability skills that are currently seen as critical to success, while also providing actual work experience on-the-job. 57 How to achieve this comprehensive and integrated quality may be the major challenge going forward. And while comprehensive training programs have demonstrated notable success, this success is not as compelling as policy makers would like to see. One aspect that deserves further attention is building stronger life skills programs, including how to prepare instructors to deliver such programs. Since these show such potential as key contributors to outcomes, can they be made even more powerful? A second aspect worthy of further experimentation and study is the duration of training, especially for programs aimed at the youth population. It is probably true that the opportunity costs of training programs aimed at adults are a serious constraint on program duration, but this is likely not so true of youth. Given 16 or 18 years of disadvantage as a youth, including serious educational disadvantage, why should we expect to overcome this in five or six months through a comprehensive training program? And how did this duration of training become the norm? It does not appear that this typical duration is derived from empirical methods. Perhaps it is driven more by the availability of resources to finance these programs and a desire to reach more disadvantaged youth with limited funds. Given what is being learned in the United States about the high social cost of “opportunity youth� it will be useful to understand this cost in developing countries, and the effects on GDP created by out-of-school unemployed youth and young adults. This may in turn inform the levels of expenditure appropriate to address the problem of youth unemployment and enable experimentation to determine whether longer programs achieve even better. A third aspect is how to individualize training and the supporting and complementary services more effectively. There is not much individual customization reported in the literature. In a more comprehensive system it should be possible to create individual training plans from a menu of possible services that more closely match individual needs and preferences. 58 The issue of effective job placement services is another promising area for improved integration into training programs. The use of more trained specialist job developers, with effective ties to the training provision and with first-hand knowledge of candidates, appears to hold significant promise to yield better ultimate results. This is particularly applicable to institution-based training. While performance-based contracts and payment for results have great appeal to policy makers and financing bodies, many designs that attempt this in developing countries have to make adjustments and/or postpone implementation of these features as they simply present too much risk for training providers. More evidence needs to be gathered as to when and how these provisions can be introduced. Finally, training does not create jobs, although a trained workforce may attract job-creating investments. This raises the question of building effective training systems that enable further and higher level training to facilitate economic competitiveness, and pathways to higher skills and earnings for workers. 59 References Aedo, C., and S. Nuñez. 2004. “The Impact of Training Policies in Latin America and the Caribbean: The Case of Programa Joven,� Research Network Working Paper #R-483, Inter-American Development Bank. Almeida, R., M. Barouni, R. Grunn, P. Premand, and S. Broadman. 2011. “Entrepreneurship Training and Self�Employment among University Graduates: Evidence from a Randomized Evaluation in Tunisia.� World Bank, Washington, D.C. Almeida, R., J. Behrman, and D. Robalino. 2012a. “The Right Skills for the Job? Rethinking Effective Training Policies for Workers.� Social Protection, Human Development Network, World Bank, Washington, D.C. Almeida, R. Arbelaez, J., Honorati, M., Kuddo, A., Lohmann, T., Ovadiya, T.M. Pop, M.L. Sanchez Puerta, and M. Weber. 2012b. “Improving Access to Jobs and Earnings Opportunities: The Role of Activation and Graduation Policies In Developing Countries,� Social Protection and Labor Discussion Paper No. 1204, World Bank, Washington, D.C. ANAPEC (National Employment Promotion Agency). 2006. “Program TAEHIL Procedures Manual: Procedure Relating to the Improvement of Employability,� ANAPEC, Kingdom of Morocco. Attanasio, O. P., A. D Kugler., and C. Meghir. 2011. "Subsidizing Vocational Training for Disadvantaged Youth in Developing Countries: Evidence from a Randomized Trial," American Economic Journal: Applied economics 3:188-220. Banerji, A., W. Cunningham, A. Fiszbein, E. King, H. Patrinos, D. Robalino, J. Tan. 2010. Stepping Up Skills for More Jobs and Higher Productivity. Washington, D,C: World Bank. 60 Banerjee, A., S. Cole, E. Duflo and L. Linden. 2007. “Remedying Education: Evidence from Two Randomized Experiments in India,� Quarterly Journal of Economics 122 (3): 1235- 1264. Belfield, C., R., Levin, Henry M., and R. Rosen. 2012. “The Economic Value of Opportunity Youth,� Report prepared for the Corporation for National and Community Service and the White House Council for Community Solutions, Washington, D.C. Accessible at http://www.dol.gov/summerjobs/pdf/EconomicValue.pdf Berge, Lars Ivar Oppedal. 2011. “Measuring Spillover Effects from Business Training Evidence from a Field Experiment among Microentrepreneurs," mimeo, MIT. Betcherman, G., K. Olivas, and A. Dar. 2004. “Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries,� SP Discussion Paper 0402, World Bank, Washington, D.C. Betcherman, G., M. Godfrey, S. Puerto, F. Rother, and A. Stavreska. 2007. “A Review of Interventions to Support Young Workers: Findings of the Youth Employment Inventory,� SP Discussion Paper 0715, World Bank, Washington, D.C. Betcherman, G. 2010. “The Impact of ALMPs: What Have we Learned from Evaluations?� PowerPoint presentation, World Bank, Washington, D.C. Blattaman, C., N. Fiala, and S. Martinez. "Can Employment Programs Reduce Poverty and Social Stability? Experimental Evidence from Uganda Aid Program," mimeo, World Bank, Washington, D.C. Bruhn, Miriam, and Bilal Zia. 2011. “Stimulating Managerial Capital in Emerging Markets: the Impact of Business and Financial Literacy for Young Entrepreneurs,� World Bank Policy Research Working Paper No. 5642, World Bank, Washington, D.C. Bjorvatn, K., and B. Tungodden. 2010. “Teaching Business in Tanzania: Evaluating Participation and Performance.� Journal of the European Economic Association 8(2-3). 61 Card, David, Jochen Kluve, and Andrea Weber. 2010. “Active Labour Market Policy Evaluations: A Meta-Analysis,� Economic Journal 120: F452-F477. Cho, Y., and M. Honorati. 2013. “Entrepreneurship programs in Developing Countries: A Meta Regression Analysis�, World Bank Policy Research Working Paper No. 6402 and IZA Discussion Paper 7333. Cho, Y., D. Kalomba, M. Mobaraq, and V. Orozco. 2012. “The Effects of Apprenticeship Training for Vulnerable Youth in Malawi�, mimeo, World Bank, Washington, D.C. Choe, C., A. Flores-Lagunesm, and S.J. Lee. 2011. “Do Dropouts Benefit from Training Programs? Korean Evidence Employing Methods for Continuous Treatments, � Working Paper No. 2011-34, CEPS/INSTEAD, Luxembourg1. Cunningham, W., M.L. Sanchez-Puerta, and A. Wuermli. 2010. “Active Labor Market Programs for Youth: A Framework to Guide Youth Employment Interventions, � World Bank Employment Policy Primer 16 (November). De Mel, S., D. McKenzie, and C. Woodruf. 2011. Business Training and Female Enterprise Start-up and Growth in Sri Lanka. Washington, D.C.: World Bank. Demombynes, G., and A. Thegeya. 2012. “Kenya’s Mobile Revolution and the Promise of Mobile Savings,� Poverty Reduction and Economic Management Unit, Africa region, World Bank. Drexler A., G. Fisher, and A. Schoar. 2011. “Keeping it Simple: Financial Literacy and Rules of Thumb�, mimeo. Entra21. 2011. “Final Report on the Entra21 Program,� International Youth Foundation. Field, E., S. Jayachandran, and R. Pande. 2010. “Do Traditional Institutions Constrain Female Entrepreneurship? A Field Experiment on Business Training in India,� American Economic Review: Papers & Proceedings 100:125–129. 62 Groh M, N. Krishnan, D. McKenzie, and T. Vishwanath. 2012. “The Impact of Training and Wage Subsidy Programs on Female Youth Employment in Jordan,� Policy Research Working Paper No. 6141, World Bank, Washington, D.C. Karlan, D., and M. Valdivia. 2011. “Teaching Entrepreneurship: Impact of Business Training on Microfinance Clients and Institutions,� The Review of Economics and Statistics 93 (2): 510-527. Klinger, B., and M. Schündeln. 2007. “Can Entrepreneurial Activity Be Taught? Quasi- Experimental Evidence from Central America,� Center for International Development at Harvard University Working Papers 153. Kluve, J., F. Rother, and M.L. Sánchez-Puerta. 2010. “Training Programs for the Unemployed, Low Income and Low Skilled Workers.� In Skills Development Strategies to Improve Employability and Productivity: Taking Stock and Looking Ahead. World Bank, Washington, D.C. Kuddo, A. 2009. “Employment Services and Active Labor Market Programs in Eastern European and Central Asian Countries,� SP Discussion Paper 0918, World Bank, Washington, D.C. Ibarraran, P., and D. Rosas Shady. 2009. “Evaluating the Impact of Job Training Programs in Latin America: Evidence from IDB Funded Operations,� Journal of Development Effectiveness 1(2): 195–216. Ibarraran, P., L. Ripani, B. Tapoada, J. M. Villa, and B. Garcia. 2012. “Life Skills, Employability and Training for Disadvantaged Youth: Evidence from a Randomized Evaluation Design,� IZA Discussion Paper No. 6617. ILO (International Labour Office). 2004. “Recommendation 195, Recommendation concerning Human Resources Development: Education, Training and Lifelong Learning,� ILO, Geneva. 63 International Youth Foundation (IYF). 2006. “Implementing Life Skills for Employability�, Field Notes, n. 6. Johanson, R. 2009. “A Review of National Training Funds,� SP Discussion Paper 922, World Bank, Washington, D.C. Martinez, S. 2012. “Vocational and Life Skills in Youth Training.� Presentation at the IZA/WB/OECD Conference on Activation and Employment Support Policies, Istanbul, May. Mattero, M. 2010. “Second Chance Education: A Conceptual Framework and Review of Programs.� Human Development Network, World Bank, Washington, D.C. Perez-Johnson, Q. I. Moore, and R. Santillano. 2011. “Improving the Effectiveness of Individual Training Accounts: Long-Term Findings from an Experimental Evaluation of Three Service Delivery Models,� Mathematica Policy Research, Inc. Accessible at http://wdr.doleta.gov/research/FullText_Documents/ETAOP_2012_06.pdf Santiago, Consultores. 2009. “Minuta Ejecutiva: Evaluación en Profundidad Chilecalifica.� Ministerio de Educación, Ministerio del Trabajo y Previsión Social. Tesliuc, C., S. Martinez, P. Acevedo, C. Asenjo, and S. Vivo. 2010. “Nota Conceptual Evaluación de Impacto Programa Juventud y Empleo.� Banco Mundial, República Dominicana. Todd, P.E. 2010. “Effectiveness of Interventions Aimed at Improving Women’s Employability and Quality of Work: A Critical Review of Impact Evaluation Studies.� World Bank, Washington, D.C. Van Adams, A. 2008. “Skills Development in the Informal Sector of Sub-Saharan Africa.� Draft report, World Bank, Washington, D.C. 64 World Bank. 2011. “Program Document for Morocco First Skills and Employment Development Policy Loan.� Maghreb Country Management Unit, Middle East and North Africa Region, Human Development Department, World Bank. Annex 1. Recent trends in designing training systems Training systems are national models of how training is organized. Reforms intended to improve the relevance, effectiveness and efficiency of training programs support the separation of the governance43 and financing of training - often centralized in small countries and regionalized in large countries - from the provision of training programs which is decentralized> the objective of such reforms is to enable all kinds of training providers to operate and create a more competitive market for training. In this training system model, national training agencies administer revenues from a variety of sources (general taxation, training levies, fees, etc.) and procure training services through competitive procurement procedures for private and public institutions. The private sector is, in many countries, a core player in the implementation of activation programs through non�governmental organizations (NGO) or private providers of various services and through employer associations. These basic reforms also reinforced the impetus to adopt lifelong learning paradigm and national qualification frameworks (NQF) to rationalize training offers, allow articulation between levels of training, to some extent across both industry sectors and education sub- sectors (secondary, post-secondary, and higher education), and enable workers to acquire higher levels of recognized skills. Lifelong learning encompasses learning throughout the life cycle and includes all forms of learning whether acquired in traditional educational settings (formal), in educational 43 The governance of government sponsored training institution is widely under a national training authority, sometimes funded through a training levy, with stronger employer representation at the national and sometimes regional level. 65 experiences outside the school system (non-formal), and through independent means. Implicit in a lifelong learning paradigm is the idea of how to connect various stages, kinds and levels of education into a more coherent whole with greater occupational and geographic mobility. Lifelong learning has direct implications for training systems and training programs as it involves the connection between basic schooling, vocational training programs, informal learning, learning on-the-job, and higher education. The idea of bridging levels of education from secondary to post-secondary vocational skills training, to higher education in a lifelong learning framework has led to the emergence of NQFs in many countries around the world.44 In a NQF, the “level of employment�45 intended determines the level of training and certification. For most training programs/systems the “level of employment� is the basis for categorizing a training program rather than the type of certificate or credential such as diploma or a degree. Another feature of contemporary training systems using a NQF is the implementation of a quality assurance system (QAS). These systems aim to provide credibility to the certifications awarded in the eyes of the society and particularly in the eyes of employers46. Terms such as demand-driven, demand-led and demand-oriented have been used to describe the contrast between economically relevant training and the legacy of what can be called, conversely, supply-driven training programs. In supply-led training, the availability of 44 Beginning in the United Kingdom in the early to mid-1990s, these frameworks were subsequently introduced in many Commonwealth countries and spread from there. For instance, ILO Recommendation 195 (2004) calls for the establishment of NQFs in all countries. 45 Lower levels mean a more narrow range of job skills, routine and repetitive work processes, less responsibility, and lower autonomy/closer supervision of a worker. Typically lower levels include apprentices, semi-skilled and some skilled workers new to an occupation. Middle levels involve a wider range of skills, more responsibility and autonomy, more teamwork, application of technical knowledge, and an ability to work without close supervision and sometimes to supervise a small group of workers. Jobs in the middle category would usually be at Level 3 and would include “journeyman� skilled workers, technicians, line supervisors, and independent entry level entrepreneurs. Higher levels capture workers referred to as “master crafts-persons,� technologists, managers, independent entrepreneurs and contractors, and then higher still professionals such as engineers, teachers, architects etc. 46 Quality assurance is commonly defined as the systematic process of checking to see whether a product or service being developed is meeting specified requirements. The implementation of a QAS in a training system is the responsibility of the body that awards certifications. 66 training is related to the supply of available teachers and instructors in the skill areas they know, and the facilities, workshops, equipment, curriculum and learning material available. Training programs offer the same courses year-after-year and there is little or no labor market information, planning and evaluation to guide the training material. By contrast, in demand-led training systems the preparation of a training plan is usually centered on an employer or group of similar employers. Here the training program design would use labor market information to identify growing sectors and occupation areas, recruit participating firms, identify specific job areas for training with the firms, review and validate available standards, and develop a course outline and curriculum. In the Morocco’s TAEHIL program for example, the training plan and program content is validated by at least three companies. This approach has the benefit of assured relevance of the content to the needs of firms and should lead to better job placement outcomes for beneficiaries. From the trends described above it is important to position a new training program into any national framework that exists and to determine the jobs and occupational areas in demand in different sectors of the economy. 67 Annex 2a. Kenya Youth Empowerment Project - Life Skills Training format and content. TRAINING DOMAINS (80 hrs): The content to be covered is as follows; A. Communication, Leadership and Interpersonal Skills (24 hours) 1. Interpersonal communication skills a. Verbal/Nonverbal communication b. Active listening c. Expressing feelings; giving feedback (without blaming) and receiving feedback 2. Conflict management 3. Negotiation/refusal skills a. Negotiation, assertiveness, refusal, agreement skills 4. Empathy a. Ability to listen and understand another's needs and circumstances and express that understanding 5. Leadership, cooperation and teamwork a. Contemporary leadership trends and issues b. Desired leadership qualities c. Expressing respect for others' contributions and different styles d. Assessing self and others’ abilities and contributing/leveraging contributions to the group e. Leading and following f. Delegating effectively g. Leading change 6. Advocacy and lobbying skills a. Influencing skills & persuasion b. Networking and motivation skills B. Decision Making and Critical Thinking Skills (16 hours) 1. Decision making / problem solving skills a. Information gathering skills b. Evaluating future consequences of present actions for self and others c. Determining alternative solutions to problems 68 d. Ability to understand the source and motivations of other people’s attitudes and actions 2. Critical thinking skills a. Considering options and engaging in innovative thinking – creativity b. Analyzing peer and media influences concerning attitudes, national values, social norms and beliefs and factors affecting these c. Identifying relevant information and information sources (research) C. Personal Management and Coping Skills (24 hours) 1. Skills for increasing internal locus of control /positive personality a. Self esteem/confidence building skills b. Self awareness skills including awareness of rights, influences, personal values, attitudes, rights, strengths and weaknesses c. Goal setting skills d. Self motivation to pursue goals e. Self evaluation / Self assessment / Self-monitoring skills 2. Emotional intelligence and self control a. Skills for managing feelings b. Skills for managing stress 3. Whole person health issues a. Whole person sexuality and self respect b. Whole person relationships c. Physical health maintenance d. Lifestyle health and diseases D. Citizenship and social concern (8 hours) 1. Whole person health issues 2. Patriotism and nationhood 3. Social justice and responsible citizenship 4. Service learning 5. Sustainable environmental practices in Kenya E. Occupational Health and First Aid (8 hours) 1. General risk assessment and management 2. Basic first aid 3. Workplace safety management and injuries 69 4. Fire hazards a. Causes of fire b. Basic fire fighting courses TRAINING FORMAT: Training is designed to suit the following stipulations: 1. The training will be 10 days long. 2. Total number of training hours = 80 hours. 3. Each working day will have: a. 8 hours of instruction and supervised activities b. 1 hour for lunch CONTENT DELIVERY AND PEDAGOGY: Trainers developed the following as part of their tools;  A Trainer’s Manual  The Course/Module outline  Handouts where applicable  PowerPoint slides/presentations where applicable The trainer should used a combination of the following methods;  Lectures  Games  Brainstorming sessions  Video clips  Guest speakers  Role plays  Small group discussions  Relevant outdoor fun and learning activities  Other appropriate methods 70 Annex 2b. Passport to Success Life Skills Training – International Youth Foundation (IYF) The Passport to Success program was launched in 2004 in India and Mexico. The curriculum was originally intended to help at-risk youth stay in school and successfully transition to the workplace. It has been used in vocational training institutes in India with former child laborers and street children, with out-of-school youth and gang members in Mexico, in formal school systems in Poland and Hungary, in teacher training colleges, and by nongovernmental organizations. The expanded program has benefited 27,000 youth in India and Mexico, where 86% and 97% of program graduates respectively were in school or employed six months after program completion. The program has also been used in Morocco, Pakistan, the Philippines, Russia, and Tanzania. TRAINING DOMAINS (core curriculum): A key feature of the program is to offer a flexible curriculum made of 60 core lessons in the following areas:  Personal competencies: managing emotions, cooperation, personal responsibility, developing confidence, and respect for self and others (20 lessons)  Problem solving: managing conflicts and reducing bullying (4 lessons)  Effective work habits: teamwork, interviewing, workplace protocol, time management, and workers’ responsibilities and rights (18 lessons)  Healthy lifestyle: substance abuse prevention, nutrition, STD/HIV/AIDS prevention, healthy relationships, and decision making (7 lessons)  Service learning: civic responsibility, community service, volunteering (10 lessons)  Closure (1 lesson) Each country is then given the flexibility to complement this core curriculum with the development of additional lessons designed to meet local needs and address the local culture. For example, India has developed 15 country-specific lessons, which place a greater emphasis on such areas as contraception, entrepreneurship, environmental protection, nutrition, and sexually transmitted diseases such as HIV/AIDS. Mexico’s 12 country specific lessons include ones dealing with gender discrimination, domestic abuse, and tolerance47. 47 IYF, 2006. 71 TRAINING FORMAT: Youth educators and teachers worldwide trained in experiential learning to transform their classrooms into more engaging learning environments. The following four elements are found in every lesson: 1) Generating Interest in the Topic: A short introduction to the lesson is presented. The purpose of this section is to stimulate participants’ interest in the topic and connect what they know or have experienced. This could be through a quote, a game, a discussion, a riddle, a brief statement, or similar method. 2) Information to Share: Information, concepts, or skills are presented or demonstrated. This presentation can be made by the leader or through a variety of methods, such as short lectures (5-10 minutes), large- or small-group activities, role plays, demonstrations, and/or discussions. 3) Group Activity/Practice: Participants will practice using the concepts or skills presented in the lesson. Using and/or practicing the new information or skill is essential to mastery of the content. This could be accomplished through a game, a practice situation between pairs or small groups of participants, a role play, a skit, a discussion, or similar method. 4) Personal Application: Participants take what they have learned and practiced and consider how they might use it in their lives. This is usually a silent reflection, small-group or paired discussion, or large-group discussion. This is a critical aspect of the lesson. Without an opportunity to apply what they have learned, participants may not see the relevancy to their lives. 72 Social Protection & Labor Discussion Paper Series Titles 2011-2013 No. Title 1304 The Nuts and Bolts of Designing and Implementing Training Programs in Developing Countries by Maddalena Honorati and Thomas P. McArdle, June 2013 1303 Designing and Implementing Unemployment Benefit Systems in Middle and Low Income Countries: Key Choices between Insurance and Savings Accounts by David A. Robalino and Michael Weber, May 2013 1302 Entrepreneurship Programs in Developing Countries: A Meta Regression Analysis by Yoonyoung Cho and Maddalena Honorati, April 2013 1301 Skilled Labor Flows: Lessons from the European Union by Martin Kahanec, February 2013 1220 Evaluating the Efficacy of Mass Media and Social Marketing Campaigns in Changing Consumer Financial Behavior by Florentina Mulaj and William Jack, November 2012 1219 Do Social Benefits Respond to Crises? Evidence from Europe & Central Asia During the Global Crisis by Aylin Isik-Dikmelik, November 2012 1218 Building Results Frameworks for Safety Nets Projects by Gloria M. Rubio, October 2012 1217 Pension Coverage in Latin America: Trends and Determinants by Rafael Rofman and Maria Laura Oliveri, June 2012 1216 Cash for Work in Sierra Leone: A Case Study on the Design and Implementation of a Safety Net in Response to a Crisis by Colin Andrews, Mirey Ovadiya, Christophe Ribes Ros and Quentin Wodon, November 2012 1215 Public Employment Services, and Activation Policies by Arvo Kuddo, May 2012 1214 Private Pension Systems: Cross-Country Investment Performance by Alberto R. Musalem and Ricardo Pasquini, May 2012 1213 Global Pension Systems and Their Reform: Worldwide Drivers, Trends, and Challenges by Robert Holzmann, May 2012 1212 Towards Smarter Worker Protection Systems: Improving Labor Regulations and Social Insurance Systems while Creating (Good) Jobs by David A. Robalino, Michael Weber, Arvo Kuddo, Friederike Rother, Aleksandra Posarac and Kwabena Otoo 1211 International Patterns of Pension Provision II: A Worldwide Overview of Facts and Figures by Montserrat Pallares-Miralles, Carolina Romero and Edward Whitehouse, June 2012 1210 Climate-Responsive Social Protection by Anne T. Kuriakose, Rasmus Heltberg, William Wiseman, Cecilia Costella, Rachel Cipryk and Sabine Cornelius, March 2012 1209 Social Protection in Low Income Countries and Fragile Situations: Challenges and Future Directions by Colin Andrews, Maitreyi Das, John Elder, Mirey Ovadiya and Giuseppe Zampaglione, March 2012 1208 World Bank Support for Pensions and Social Security by Mark Dorfman and Robert Palacios, March 2012 1207 Labor Markets in Middle and Low Income Countries: Trends and Implications for Social Protection and Labor Policies by Yoonyoung Cho, David Margolis, David Newhouse and David Robalino, March 2012 1206 Rules, Roles and Controls: Governance in Social Protection with an Application to Social Assistance by Lucy Bassett, Sara Giannozzi, Lucian Pop and Dena Ringold, March 2012 1205 Crisis Response in Social Protection by Federica Marzo and Hideki Mori, March 2012 1204 Improving Access to Jobs and Earnings Opportunities: The Role of Activation and Graduation Policies in Developing Countries by Rita Almeida, Juliana Arbelaez, Maddalena Honorati, Arvo Kuddo, Tanja Lohmann, Mirey Ovadiya, Lucian Pop, Maria Laura Sanchez Puerta and Michael Weber, March 2012 1203 Productive Role of Safety Nets by Harold Alderman and Ruslan Yemtsov, March 2012 1202 Building Social Protection and Labor Systems: Concepts and Operational Implications by David A. Robalino, Laura Rawlings and Ian Walker, March 2012 1201 MicroDeterminants of Informal Employment in the Middle East and North Africa Region by Diego F. Angel-Urdinola and Kimie Tanabe, January 2012 1120 Employment Generation in Rural Africa: Mid-term Results from an Experimental Evaluation of the Youth Opportunities Program in Northern Uganda by Christopher Blattman, Nathan Fiala and Sebastian Martinez, December 2011 1119 Measuring Governance and Service Delivery in Safety Net Programs by Gloria M. Rubio, September 2011 1118 Assessing Safety Net Readiness in Response to Food Price Volatility by Margaret Grosh, Colin Andrews, Rodrigo Quintana, Claudia Rodriguez-Alas, September 2011 1117 Social Safety Nets in Fragile States: A Community-Based School Feeding Program in Togo, August 2011 (also available in French) by Colin Andrews, Elena Galliano, Carolyn Turk and Giuseppe Zampaglione, August 2011 1116 Strengthening Governance of Social Safety Nets in East Asia by Sara Giannozzi and Asmeen Khan, August 2011 1115 International Portability of Health-Cost Coverage: Concepts and Experience by Martin Werding and Stuart McLennan, July 2011 1114 Liberia’s Cash For Work Temporary Employment Project: Responding to Crisis in Low Income, Fragile Countries by Colin Andrews, Prospère Backiny-Yetna, Emily Garin, Emily Weedon, Quentin Wodon and Giuseppe Zampaglione, July 2011 1113 Employability and Productivity among Older Workers: A Policy Framework and Evidence from Latin America by Edmundo Murrugarra, July 2011 1112 Cash Transfers, Children and the Crisis: Protecting Current and Future Investments by Ariel Fiszbein, Dena Ringold, Santhosh Srinivasan, June 2011 1111 Severance Pay Programs around the World: History, Rationale, Status, and Reforms by Robert Holzmann, Yann Pouget, Milan Vodopivec and Michael Weber, May 2011 1110 Portability of Pension, Health, and other Social Benefits: Facts, Concepts, Issues by Robert Holzmann and Johannes Koettl, May 2011 1109 Disability and Poverty in Developing Countries: A Snapshot from the World Health Survey by Sophie Mitra, Aleksandra Posarac and Brandon Vick, April 2011 1108 Advancing Adult Learning in Eastern Europe and Central Asia by Christian Bodewig and Sarojini Hirshleifer, April 2011 1107 Results Readiness in Social Protection & Labor Operations by Laura Rawlings, Maddalena Honorati, Gloria Rubio and Julie Van Domelen, February 2011 1106 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Service Delivery Projects by Julie Van Domelen, February 2011 1105 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Safety Nets Task Teams by Gloria Rubio, February 2011 1104 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Funds Task Teams by Julie Van Domelen, February 2011 1103 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Labor Markets Task Teams by Maddalena Honorati, February 2011 1102 Natural Disasters: What is the Role for Social Safety Nets? by Larissa Pelham, Edward Clay and Tim Braunholz, February 2011 1101 North-South Knowledge Sharing on Incentive-based Conditional Cash Transfer Programs by Lawrence Aber and Laura B. Rawlings, January 2011 To view Social Protection Discussion papers published prior to 2011, please visit www.worldbank.org/sp Abstract Training programs mainly address market failures related to lack of skills (technical, cognitive, non-cognitive). This paper conducts a comprehensive review of training programs effectiveness in developing countries. Based on relevant international experiences, the paper highlights key design features associated with program success as well as implementation challenges and discusses their policy implication. Success of training programs is deeply related with the content of the skills provided and how well they serve the local labor demand (demand-driven design) and with the presence of a sound governance structure for training providers and beneficiaries. In particular, the effectiveness of training programs for youth tends to be higher when a “comprehensive� approach is taken by combining different types of training with complementary support services. The ultimate goal is to inform new program design and improve the performance of current training programs. About this series... Social Protection & Labor Discussion Papers are published to communicate the results of The World Bank’s work to the development community with the least possible delay. The typescript manuscript of this paper therefore has not been prepared in accordance with the procedures appropriate to formally edited texts. The findings, interpretations, and conclusions expressed herein are those of the author(s), and do not necessarily reflect the views of the International Bank for Reconstruction and Development / The World Bank and its affiliated organizations, or those of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The author(s) attest(s) that the paper represents original work. It fully references and describes all relevant prior work on the same subject. For more information, please contact the Social Protection Advisory Service, The World Bank, 1818 H Street, N.W., Room G7-803, Washington, DC 20433 USA. Telephone: (202) 458-5267, Fax: (202) 614-0471, E-mail: socialprotection@worldbank.org or visit us on-line at www.worldbank.org/sp. Printed on recycled paper