Report no. 94682-IN Labor Market Impacts and Effectiveness of Skills Development Programs in 5 States in India: Assam, Andhra Pradesh, Madhya Pradesh, Odisha and Rajasthan The World Bank Education Global Practice April 2015 1 Contents Executive Summary....................................................................................................................................... 4 1. Introduction. ......................................................................................................................................... 9 2. The Skills Development Programs ...................................................................................................... 10 3. Employment Outcomes of SD Programs............................................................................................. 14 4. Wage and Earnings Effects of SD Programs ........................................................................................ 23 5. Cost-Effectiveness of SD programs. .................................................................................................... 24 6. India vs. International Experience....................................................................................................... 24 7. Findings from the Qualitative Study. .................................................................................................. 26 8. Conclusions and Policy Recommendations......................................................................................... 31 Annex 1 – Socio-Economic Characteristics of Male and Female Trainees and non-Trainees by Program . 36 Annex 2. Socio-Economic Characteristics of Current Trainees by SD Program .......................................... 38 References .................................................................................................................................................. 39 List of Tables Table 1 – Skills Development Programs in India: Objectives and Number Trained, 2011-2014 ................ 11 Table 2. BPL (Below Poverty Line) Card holders among respondent beneficiaries .................................... 12 Table 3. Five most popular courses within each SD program ..................................................................... 13 Table 4. Characteristics of Firms/Employers of Trainees............................................................................ 14 Table 5. Aspirations of Current Trainees after Completing the Training by SD Programs......................... 15 Table 6. Placement rate at completion of training and Job Status 1-2 years later.................................... 15 Table 7 – Initial Placement Rate and Employment Rate 1-2 years after Training by Gender .................... 16 Table 8 – Initial Placement Rate and Employment rate 1-2 years after Training in any of the 5 SD programs by State ....................................................................................................................................... 16 Table 9. Proportion of Trainees placed at end of training found not working 1 or 2 years later by Programs ..................................................................................................................................................... 17 Table 10- Labor Market Impacts of Skills Development Programs ............................................................. 18 Table 11 - Labor Market Impacts of other factors ...................................................................................... 20 Table 12. Selected Economic Indicators in the 5 States under study. ........................................................ 22 Table 13. Funding Mechanisms of SD Programs and Selection of Training Providers ............................... 27 Table 14-- Summary of Policy Recommendations ...................................................................................... 35 2 List of Boxes Box 1. Survey Instruments. ........................................................................................................................... 9 Box 2. Impact of Short-Term Training Programs: International Evidence. ................................................. 25 Box 3. The Qualitative Study. ...................................................................................................................... 26 Box 4. Some Examples of Good Practices to enable Placement and Retention ......................................... 30 3 Executive Summary Committed to increase human capital investments to foster growth, and aware that youth come out of the education system lacking critical technical and soft skills, the Government of India (GOI) has set an ambitious target of skilling 500 million workers in India by 2022. A little over half of this target (285 million) is to be met by five large national skills development (SD) programs which provide short-term training, most of it lasting from 6 days to 6 months, in a wide range of trades. Since 2011, around 6 million youth have already been trained through these programs and a huge program expansion is required to reach the set target. Before embarking on this expansion, GOI requested than an evaluation of the labor market impacts and effectiveness of these schemes be made. This study attempts to respond to this request, evaluating those 5 schemes with data from a set of surveys of trainees, non-trainees (comparison group), employers and training providers. The quantitative analysis is complemented by a qualitative study based on interviews and focus groups discussions. In agreement with the GOI, five states have been taken into consideration for this evaluation: Assam, Andhra Pradesh (AP), Madhya Pradesh (MP), Odisha and Rajasthan. The five SD programs under study (NSDC, SDIS, STEP-UP, ASDP and RSETI)1, although run by different ministries and with different administrative structures, share a common goal, that of skilling youth. However, they target somewhat different population groups: two of them focus on rural youth, one on urban youth, and the remaining ones cover both. They can all prepare for wage or self-employment but emphasize one or the other. Three schemes target the poor while the others do not discriminate according to socio-economic status. As shown by the analysis of survey data, the main characteristics of SD programs and of their beneficiaries are as follows:  Most program beneficiaries are young (mean age is 21-23) and 90 percent of them are first time labor market entrants. Their education level of trainees is fairly high as only 10-20 percent of them have not completed secondary education. Women represent close to half of trainees in 4 out of the 5 schemes  Programs which target the poor only partly achieve this goal. However, in all programs, the representation of disadvantaged social groups seems in line with their representation at the national level.  In all programs, the financing of training is well separated from the provision of training which can be done by public or private providers (TPs), with a large majority being private.  Although programs offer a wide range of trades, over 50 percent of trainees concentrate over a limited set of trades which appears similar across all programs.  In the case of wage employment following training, hiring firms are mostly private and of very small size. 1 Full names of these programs are: NSDC (National Skills Development Corporation), SDIS (Skills Development Initiative Scheme, STEP UP (Skill Training for Employment Promotion amongst Urban Poor). ASDP (Aajeevika Skills Development Program) and RSETI(Rural Self-Employment Training Institute) 4 Regarding the labor market impacts of SD programs, the main findings of the analysis are:  The placement rate of trainees (shortly after the end of the training period) is modest: 27 percent overall, ranging from 23 to 36 percent across programs, but with substantial state-wide variations. One or two years after training, the employment rate of trainees is not much different: 28 percent. However, significant labor market churning has occurred over the period of one or two years after training: trainees initially placed have left their jobs, looking for other opportunities (or simply quit); others have entered the labor market later. This strongly suggests that, although initial placement facilitates insertion in the labor market, it does not ensure a good job match and sustainability of employment.  Although all programs aim at providing gainful employment, a large proportion of trainees (28 percent) indicate that they do not intend to work after training, but rather to pursue further studies. This lack of motivation raises questions about the adequacy of the selection process of participants.  Although the employment rates of trainees are found to be fairly low, the analysis of survey data, including employment and earnings data of non-trainees (comparison group), shows that SD programs make a difference. They contribute to increase the employability of trainees by 7 percentage points overall, with a stronger effect on women. They give access to better quality jobs (as measured by proxies, such as a job contract, shorter work hours, and access to pension plan) and provide a positive wage premium, of the order of 21 percent.  Not all programs perform equally well. When a distinction is made between programs, the positive impact on employment and wages disappears for 2 of the 5 programs (ASDP and RSETI).  Employment and wage impacts vary, depending on personal characteristics, such as age, education level, marital status, prior work experience, as well as on training characteristics. Worth noting are the stronger positive impact of training when it combines classroom teaching with practical training in industry (or apprenticeship) as well as the much higher probability of employment for those with prior work experience. Another finding worth underlying is that training has no employment effect for very young trainees aged 15-20, and only starts to be effective for those aged 21 or more.  Location matters. Differences across states appear strongly with higher probability of employment in AP and Assam than in MP, and lower probability in Odisha and Rajasthan than in MP. The second part of the study – qualitative – based on focus group discussions and interviews, points out that several aspects in the process of selection of training providers and courses, such as insufficient attention to the quality of the offerings and to labor market demand are not conducive to ensuring high quality and relevance of programs. It also detects weaknesses in the quality assurance mechanisms, monitoring and evaluation, and placement/post-placement support. The main policy recommendations derived from these findings are: 5 Program outcomes could be improved by taking more into account the factors that increase the chances of success. 1. The most obvious factor is the quality and relevance of the training. A more extensive use of labor market information and closer links with employers for the identification of trades and skills in demand, as well as a more careful selection of training providers would improve the quality and responsiveness of programs to the needs of the labor market. An increasing use of apprenticeships, or programs combining classroom with practical training in enterprises (which were shown to have a clear positive impact on employment probability) would also bring closer links with employers and help ensuring that training offerings respond quickly to changes in production processes. 2. A more careful targeting of programs to the population groups for whom those programs are particularly effective could lead to a more efficient use of public resources and a higher employment rate:  Selecting a greater proportion of applicants aged 21 or more would increase program effectiveness. For those aged 15-20, for whom the programs were found to have no impact, efforts should be made to identify the causes of failure and on this basis, to reconsider the type of programs that could be beneficial for them  A more careful screening of applicants to reduce the large number of seats attributed to youth who do not intend to look for a job after training and thus, the selection of a larger pool of motivated candidates would also improve program effectiveness. 3. Promoting independent assessments and standardization of certificates would enhance the certificate�value�and the marketability of beneficiaries. The absence of independent evaluation gives scope for masking quality compromise. Furthermore, the lack of standardization of certificates reduces the value of the information that certificates are supposed to give to potential employers regarding the acquired skills of job seekers. Practices followed by some TPs such as adopting a course-specific standard approach to assessments or initiating joint-certification with industry partners are examples of ways to address this issue and worth considering. Ensuring the availability of a pool of qualified assessors would also facilitate the recourse to independent evaluation. 4. Strengthening post-placement support would make SD programs more effective. Given the significant labor churning which occurs after training, an effective post-placement support during at least a period of 6 months to a year would help achieve a more stable employment as it would facilitate information about job availability, lower the costs of search and thus, reduce the length of unemployment spans between jobs. This again supposes an active engagement with the local industry and service sector. Enlarging the scope of SD programs to cover not only unemployed youth but also people already in the labor force (whether or not former trainees), would provide them with opportunities to reskilling between jobs, change occupations and become more productive. It would generate productivity increases by facilitating the adjustment of the labor force to the changing needs of 6 a modernizing economy. It would open the way to life-long learning and address a weakness highlighted by enterprise surveys, namely, the very low level of in-service training provided by enterprises to their workers in India compared to competitors internationally. 5. Linking funding to desired outcomes improves efficiency, provided incentives are well designed. This is already been used (to various degrees) by SD programs with the use of placement targets as conditions for funding. This practice should be maintained as it obliges TPs and program managers to shift focus from inputs towards outcomes. It incites TPs to establish links with potential employers and to offer courses that match labor market demand and brings some competition into the sector. Placement also gives opportunities to youth to gain some useful work experience, even if of short duration. However, given the difficulty to achieve a high target, this practice should be accompanied by a more stringent selection of applicants to ensure that beneficiaries are those most motivated. False reporting might be encouraged otherwise. As placement does not ensure sustainability of employment, this practice should also include incentives to strengthen monitoring and post-placement support. 6. Pursuing efforts to strengthen M&E is critically needed across all programs. Despite progress, the stage of development of the M&E system still varies across programs and states. The results of this study also suggest that a more in-depth evaluation of ASDP and RSETI programs for which no labor market impact could be detected is needed. This evaluation should be conducted before envisaging any large expansion of those schemes. An additional recommendation would be to include, as part of these evaluation efforts, a study to assess the long-term effects of SD programs to measure the full return on the investments made in skill development and bring additional useful insight for policy makers as one limitation of this study (common to most evaluation studies world-wide) is the limited time period during which the impact of training is being observed. 7. Learning from the experience of States that have been more successful could generate greater efficiency. The study identifies differences in practices, and organizational features across states which have contributed to better coordination and better outcomes of SD programs in AP. Other states could gain by learning from the AP experience and examining whether those practices could be adapted to their own environment. Some good practices may also exist in specific areas in States other than AP. Sharing lessons from experience would benefit all of them. 8. One final recommendation is not to consider the SD programs analyzed in this study in isolation but rather as part of a diversified package of human capital investments which includes education and longer-term training programs. Even if most SD programs are found to have positive labor market effects, expectations of their overall impact on the Indian economy need to remain modest. Courses are of short duration and cost is low. The amount of human capital that can be provided through those programs is likely to remain modest. While these programs can be a useful policy instrument to facilitate insertion in the labor market, they cannot be expected to yield huge 7 productivity increases and need to be complemented by other types of human capital investments. Their targets should be adjusted accordingly. 8 1. Introduction. Over the past few years, the Government of India (GOI) has been implementing five large national skills development (SD) programs2 to improve the employment and earnings prospects of urban and rural youth. The training is of short duration, most of it lasting from 6 days up to 6 months, and covers a wide range of trades. Since 2011, around 6 million youth have been trained and more than US$500 million have been spent under these programs. Committed to increase investments in human capital to foster growth, and aware that youth come out of the education system lacking critical technical and soft skills, the GOI has set in the Twelfth and Thirteenth five-year Plans (2012-17 & 2017-2022) an ambitious target of skilling 500 million workers in India by 2022 and gave a combined target of 285 million to the five SD programs considered for this study. This would mean increasing the current annual intake of trainees about ten-fold during each of the next seven years. Before embarking on the huge increase in public spending that reaching such a target would require, the GOI has requested that an evaluation of these schemes be made. The critical questions to be addressed are: What have been the employment outcomes of those programs? What earnings premium do they give to trainees? Do program benefits justify the significant public investments made into them? What organizational aspects affect delivery and reach? and finally, based on these findings, what should be the course of action? This paper tries to address those questions, using data from a set of surveys of trainees (past and current), non-trainees (comparison group), employers and training providers (see Box 1). The quantitative analysis has been complemented by a qualitative study based on interviews and focus group discussions, with focus on business processes, program management and M&E. Five states have been taken into consideration for this analysis: Assam, Andhra Pradesh (AP), Madhya Pradesh (MP), Odisha and Rajasthan. This paper comprises several sections. The first section will provide a brief description of the five programs and of their operating modes. An analysis of their employment outcomes and wage and earnings effects, as well as the measurement of their cost effectiveness will be given in the following three sections. This will be accompanied by a review of, and comparison with results from evaluations of similar programs conducted in other parts of the world. Following will be a summary of findings from the qualitative survey. A concluding section will present policy recommendations derived from the analysis of both the quantitative and qualitative data sources. Box 1. Survey Instruments. To evaluate the performance of five national skills development (SD) programs, surveys of training providers, trainees and employers were conducted in five states: Andhra Pradesh, Assam, Madhya 2 Those 5 SD programs are: NSDC (National Skills Development Corporation), SDIS (Skills Development Initiative Scheme), STEP UP (Skills Training for Employment Promotion amongst Urban Poor), ASDP (Aajeevika Skills Development Program) and RSETI (Rural Self-Employment Training Institute) 9 Pradesh, Odisha, and Rajasthan. 1) Survey of Training Providers (TPs). A random sample of 110 training institutions in 3-4 districts in each of the five states was surveyed. The sampling frame data of TPs was made available by the program managers or Ministries overseeing each program. The questionnaires sought information about the infrastructure, training program, trainee and trainer characteristics and services provided under the program. 2) Survey of Past Trainees, Current Trainees and Non-Trainees (comparison group). A representative sample of trainees having completed the training (6 per TP), trainees currently enrolled (5 per TP) and non-trainees (5 per TP) was selected from the files of participating training providers. Past trainees and non-trainees (comparison group) are from early 2012 batches to allow them to be in the labor market 1 or 2 years after training and/or before being interviewed under the study. Current trainees were undergoing the training at the time of the survey. The questionnaire provides information about personal and socio-economic background, employment or self-employment status and corresponding earnings, training and placement services received from the TPs. 3) Survey of Employers. Local employers who recruit trainees from the participating training institutes (5 per TP) or who offer jobs requiring similar skills were interviewed. Data was collected on sector of employment, size and employer type, recruitment of trainees and wage offers. Due to difficulties in finding training providers and past trainees in some states, and to the reluctance of some respondents to provide information, the data sample finally analyzed is a bit smaller than originally intended. The final dataset comprised of 400 TPs, 2620 past trainees, 1995 current trainees, 2017 non-trainees (comparison group), and 669 employers. 2. The Skills Development Programs Providing skills to the labor force and facilitating employment has been one key mandate within India’s national development agenda for several decades. While its original focus was on poverty alleviation, it became part of an agenda of “building a competitive labor force� after the establishment of the National Skill Development Policy in 2009. The five programs under study were established or realigned to the National Skill Development Policy after its formulation, and an autonomous body called the National Skill Development Agency (NSDA) was setup in 2013 with the task of coordinating and harmonizing the skill development efforts of the government and the private sector. What are the objectives of these programs and how do they operate? Characteristics of the programs are diverse and outcomes could as well be different (see Table 1). These five programs are run by different ministries and with different administrative structures. One program (NSDC) is fee-based while others are subsidy-based. Two programs (ASDP and RSETI) are focused on rural youth, another (STEP-UP) on urban youth and the remaining ones have a wider focus. They can all prepare for wage employment or self-employment but emphasize one or the other. Three schemes (STEP-UP, ASDP and RSETI) target the poor while the others do not discriminate according to socio- economic status. 10 Table 1 – Skills Development Programs in India: Objectives and Number Trained, 2011-2014 Name of Starting Year Ministry/Agency Objective/Target Group Nos. Trained Program 2011-14 National Skills 2010 National Skills Skilling urban & rural youth for 1,900,000 Development Development wage employment Corporation Corporation (NSDC) (NSDC) Support to private sector initiatives through public- private partnerships (PPPs) Skills 2007 Ministry of Labor Skilling urban & rural youth for 1,500,000 Development & Employment wage employment Initiative (MoLE) Scheme Testing & certification of people (SDIS) Since May with informally acquired skills 2014,Ministry of Skill Development, Building capacity for Entrepreneurship, development of competency & Youth Affairs & assessment standards, course Sports curriculum & learning materials Rural Self- 2009 Ministry of Rural Training of poor rural youth for 950,000 Employment Development self-employment training (MoRD) statute (RSETI) Aajeevika SGSY started in Ministry of Rural Training of poor rural youth for 700,000 Skills 2007, revamped Development wage employment Development into ASDP in (MoRD) Program 2013 (ASDP) Skill training 2009 Ministry of Training of poor urban youth 850,000 for Housing and for wage/self- employment Employment Urban Poverty Protection Alleviation amongst (MoHUPA) Urban Poor (STEP UP) Data Source: NSDA While sponsored by the Central Government through specific ministries, these programs, apart from NSDC, are implemented with a decentralized implementation structure. It can be through a partnership between central and state governments (ASDP), or through some state level committee/organization (SDIS, RSETI), or directly through a municipality or district administration (STEP-UP). Due to this decentralized administration system, there can be significant variation in implementation from state to state. Although modalities for selection of training providers differ across programs, one particular feature, common to all, is the separation of the financing from the provision of training which can be done by 11 public or private institutions. NSDC calls for proposals from training providers and adopts a multi-stage process with the help of professional organizations to select proposals for funding. The two most important criteria for selection are sustainability of the business model and linkages with employers. Under STEP-UP, most States adopt a L1 based (lowest cost based) bidding for selection of training providers while a few states adopt a quality-cum-cost based bidding. Project implementation agencies under ASDP are selected through an open tender and competition is based on a scorecard that measures their financial robustness, experience, operational preparedness, trainees placement and certification arrangement. For RSETI, state-level committees delegate to the Lead Commercial bank the responsibility to set up a rural self-employment training institute (RSETI) in each district. SDIS relies on training providers registered by the respective States/UTs and which are eligible to apply after fulfilling certain conditions, such as being affiliated to universities, or approved by relevant councils (for example, All India Council of Technical Education, Indian Nursing Council, etc.) Who are the beneficiaries? The youth population is a priority for all the programs, and data confirm they are the main beneficiaries. Most trainees are young (mean age is 21-23), still unmarried and live in households headed by their father. Their education level is fairly high. Around 80-90 percent have at least 10 years of schooling (29 percent have attended higher education). Thus, the proportion of beneficiaries who have not completed secondary education is small, ranging from 5 to 9 percent depending on programs, except in RSETI where it reaches 20 percent. Having worked in the 24 months preceding the training seldom happens: about 90 percent of trainees are first time labor market entrants. The participation of women is high: they represent close to half or more than half of trainees in all programs, except NSDC (where they are only 1/3 of trainees) (see Annex 1 for more detailed statistics). “Poor� rural or urban youth are the target of three of the programs: ASDP, RSETI and STEP-UP (NSDC and SDIS do not discriminate according to socio-economic status). Data indicate that this goal is only partly achieved and that only around one third of beneficiaries in those three programs belong to very poor families (with BPL card or family income less than 6,000 Rs. per month) {see Table 2). A slightly higher percentage of poor is only observed for ASDP in recent batches, perhaps due to a more extensive use of the Participatory Identification of Poor (PIP) process. Nevertheless, disadvantaged and low income social groups are well represented across all programs, in line with their representation at the national level: 20/30 percent of trainees are SC/STs, and 40-50 percent, OBC; and the family income of at least 51 percent of trainees is less than 9,000 Rs. per month (see Annex 1). Table 2. BPL (Below Poverty Line) Card holders among respondent beneficiaries Programs not discriminating by Programs Targeting the Poor socio-economic status BPL holders (%) NSDC SDIS STEP-UP ASDP RSETI Past Trainees 17 29 33 33 25 Current Trainees 16 21 30 42 26 Source: Surveys of Past and Current Trainees 12 Who delivers the training and what type of training? Training is delivered by training providers (TPs), which can be public or private. In the case of NSDC, ASDP and STEP-UP, 90 percent of TPs within the sample are private. For SDIS, 75 percent of TPs are private, and RSETI relies on a particular delivery mechanism which involves either banks or private TPs. At least two-thirds of TPs are located in urban areas and the remaining in rural areas. TPs can run training for one or several schemes. They usually select trainees through a mobilization process which mostly rests on pamphlets, newspapers announcements, and is followed by an admission process comprising interviews and/or tests. Basically, participation is based on self-selection. To various degrees, depending on programs and states, TPs interact with the government to have instructors trained, or to receive equipment/toolkits. They also establish links with the industry but here also, there seems to be wide variation across states and programs, in the manner in which the industries are engaged. Around half of TPs claim to get feedback from industry on curriculum and about 40 percent of them to get instructors from the industry or arrange for apprenticeships with the local industry. Although the survey of 1,995 current trainees reports participation in a wide range of trades, ranging from 18 different offerings in RSETI to 51 in SDIS, there seems to be a dominant set of trades which attracts over 50 percent of trainees and appears to be quite similar across programs (see Table 3). Table 3. Five most popular courses within each SD program NSDC SDIS ASDP RSETI STEP-UP Computer Tailoring Tailoring Tailoring Information technology Data entry Tally Beautician Beautician Tailoring operator Tally Electrical Machine operator Driving Beautician Retail sales & Beautician Tally Basic computer Hospitality marketing management Hardware & Computer Domestic business Mobile repairing Banking & networking process accounting outsourcing Note: within each program, course names are given according to the number of participants, starting with the course that attracts the largest number of participants. The number of trainees enrolled in those 5 top courses exceeds 50 percent in the sample of current trainees. Who hires the trainees? Although some programs (RSETI and STEP-UP) are more directly designed to help beneficiaries become self-employed, all programs can prepare for wage employment. Hiring firms are mostly private and their size is small: 60 to 80 percent of them (depending on programs) have less than 10 employees (see Table 4). Most employers of RSETI and ASDP trainees operate in the service sector. For the other schemes, more then 2/3 are in the service sector and 1/3 in manufacturing. Most hiring firms being 13 small, employers are not subject to labor regulations mandating paid leave, pension rights and other non-wage benefits. Table 4. Characteristics of Firms/Employers of Trainees Employers/Firms NSDC SDIS STEP-UP ASDP RSETI % hiring from Scheme 42 36 18 14 13 % Private 91 95 93 93 92 Employment Sector Agriculture 2 2 3 2 1 Manufacturing 23 22 31 10 15 Services 75 76 66 88 84 Size Less than 10 employees 60 81 65 76 80 10-19 employees 14 10 18 13 16 More than 20 26 9 17 11 4 No. of hiring firms 283 239 122 95 86 3. Employment Outcomes of SD Programs. Job placement at the end of the training All programs put emphasis on the need for TPs to provide support for job placement at the end of the training period and in some cases, provide financial incentives to do so. This support is provided through placement cells and/or job fairs, faculty connections and/or direct contact with local employers. Trainees confirm having received support and having benefited from counseling, information about vacancies, help in filling applications and help for interviews. Data indicate that the placement rate of trainees is 27 percent overall, ranging from 23 to 36 percent across programs, and slightly higher for males than females, except in SDIS and STEP-UP (see Tables 5 and 6). However, this average hides substantial state-wide variations: Andhra Pradesh (AP) stands out with the highest placement rate (50 percent) while Rajasthan and Odisha show the lowest rates (20 and 12 percent respectively) (see Table 8). The placement rate is fairly low, suggesting that TP support has not been very effective in the past, except in Andhra Pradesh3 where the funding of TPs has been linked to performance on placement for already some time and where performance targets were set quite high. This assessment of overall low placement performance, however, needs to be qualified by the fact that, although all programs aim at providing gainful employment, not all participants have this motivation. In the survey of trainees interviewed while still participating in the training program, a question was asked about their plans after 3 The relative performance of AP derived from the analysis of data from the survey of trainees is corroborated by the information provided by TPs. Those even claim reaching a 100 percent placement rate in AP. 14 completing the training: 28 percent of trainees indicated that they did not intend to look a job but rather to pursue further studies (technical education such as ITIs, polytechnics or formal education) (see Table 5). Interestingly, the proportion of trainees who intend to go for further studies is significantly lower for all programs in AP which, as discussed above, is the state achieving the highest placement rate. Table 5. Aspirations of Current Trainees after Completing the Training by SD Programs NSDC SDIS STEP-UP ASDP RSETI Total % of 31% 33% 32% 15% 11% 28% beneficiaries intending to [8% in AP ] [almost [2% in AP ] [none in AP] [3% in AP] go for none in AP] further studies Source: calculations from survey of 1,995 current trainees. The lack of intention to work after training of a significant proportion of trainees obviously makes placement more difficult and raises a question about the adequacy of the selection process of participants. Unless those programs provide long-term benefits which can be captured by beneficiaries several years later, the admission of these students could represent a waste of public resources and a more stringent screening of applicants could increase the performance of programs. Job Status 1-2 years after training The employment rate 1-2 years after training is 28 percent overall. This indicator may be more relevant than initial placement to measure the impact of training as it is observed after a period of job search and adjustment has taken place. More trainees may have had the time to look for a suitable job and a good match. Evidence, however, shows that, after a period of one or two years, the employment rate is not markedly different from the initial placement rate for all programs, except for RSETI trainees whose employment rate drops significantly (see Table 6). Trainees who have successfully entered the labor market have either opted for wage employment or self-employment, in proportions that reflect the focus of the program they participated in. Table 6. Placement rate at completion of training and Job Status 1-2 years later Program Initial Placement rate % working 1-2 years of which (o/w) wage o/w self-employment After training employment NSDC 31 32 22 10 SDIS 23 26 18 8 STEP-UP 30 29 13 16 ASDP 24 25 10 15 RSETI 36 23 8 15 15 Significant labor churning has occurred over this period of one or two years after training. Although placement and employment rates are fairly similar, many changes have occurred. Trainees initially placed have left their jobs, looking for other opportunities (or simply quit for family or other reasons); others have needed more time to find a job and have entered the labor market later. Labor market churning affects both males and females (see Table 7) and also occurs within and across States. One program, NSDC, shows a clear increase in the employment rate of male trainees and an opposite trend for females. The reverse is observed for STEP-UP and ASDP. The employment rate of both male and female SDIS trainees increases while it falls in the case of RSETI trainees. Similarly, for a given program, while the employment rate may increase in one state, it may decrease in another. Again, overall, AP stands out in achieving the highest employment rate of trainees, followed by Assam. In MP and Rajasthan, the level of the initial placement rate is not even sustained (see Table 8). Table 7 – Initial Placement Rate and Employment Rate 1-2 years after Training by Gender Program Males Females Initial placement % employed Initial % employed 1-2 Rate 1-2 years later placement rate years later NSDC 33 49 27 21 SDIS 23 25 23 29 STEP-UP 27 23 32 34 ASDP 29 22 17 30 RSETI 39 25 33 21 Overall 28 28 26 28 Table 8 – Initial Placement Rate and Employment rate 1-2 years after Training in any of the 5 SD programs by State Assam Madhya Rajasthan Odisha Andhra Pradesh Pradesh Initial Placement Rate 26 32 20 12 50 Employment Rate 42 28 13 18 55 1-2 years later While placement may play an important role in providing some work experience and in facilitating insertion in the labor market, it does not ensure a good job match and sustainability of employment. In fact, a large proportion of those placed earlier, right at the outset of training (ranging from about 40 percent for NSDC to 70 percent for RSETI) are found not working one or two years later, and looking for other opportunities. A small percentage (ranging from 2 to 7 percent) has even withdrawn from the labor market and is no longer looking for a job (Table 9). More than half of those indicate they are studying. 16 Table 9. Proportion of Trainees placed at end of training found not working 1 or 2 years later by Programs Programs % not Working 1or 2 o/w Looking for Work o/w not Looking years after training NSDC 42 36 6 SDIS 52 45 7 STEP-UP 56 51 5 ASDP 61 59 2 RSETI 70 65 5 Evidence of short job tenure is also provided by employers who report that many newly hired leave their jobs after 2/3 months of employment. Around one-third does so because of other job opportunities but others, according to employers, leave because of low pay, distance from home or for family reasons. Have the training programs made a difference in terms of employability? Is there a positive treatment effect? The purpose of SD programs is to provide skills or some form of social capital that increases the employability of youth and facilitates their insertion in the labor market. Even if the probability of employment of trainees is not found to be high, policy makers need to know what would have been the employment rate in the absence of training and be able to compare both rates. This amounts to estimating a treatment effect, defined as the difference between the employment status, depending on whether the individual was trained or not. Given that an experimental design (with trainees and non-trainees randomly selected) was not possible, data from a survey of youth of similar age, socio-economic background4, who did not participate in the training, were used to estimate the counterfactual for each program. As in other non-experimental evaluations, estimates are obtained controlling for individual characteristics of both treatment and comparison groups, such as age, gender, marital status, education, family income, and state of residence. Estimates indicate a modest but positive treatment effect (see table 10). The participation in SD programs is found to increase the employment rate by 7 percentage points overall, with a stronger effect for women than for men (12 vs. 4.5 percentage points). Given the different population groups targeted by each program and their different operating modes, estimates were also obtained for each program, separately5. In this case, results indicate that participation in NSDC training increases the employment rate by 11 percentage points, SDIS training by 7 percentage points, and STEP-UP by 5 4 The comparison group comprised youth who had been admitted to any of the reference schemes but did not complete the training and youth of same age and socio-economic background, such as friends or neighbors of trainees, who could have been eligible to undertake the training. 5 In this case, separate regressions are run, with a matched control group for each program 17 percentage points. No impact of training on employment rate could be found for the other two programs (RSETI and ASDP) While these estimates have been obtained by comparing the employment outcomes of trainees with those of a group of non-trainees that have the same observable characteristics ( in terms of age, marital status, education level, and state of residence), it has been impossible to control for other unobservable characteristics such as, for example, motivation and willingness to work. If such differences do exist and explain why some have been trained and others not, the estimates of the employment effect may be biased upward. In such case, the estimates of the treatment effect would be an upper estimate. Participation in SD programs also seems to give access to better quality jobs, as measured by proxies such as length of the work day (not more than 8 hours worked per day), and whether individuals were given a job contract, were provided with a pension plan and paid-leave by employer. Although the proportion of workers in the sample with “quality “jobs is small6, the analysis suggests that those who participated in SD programs were 7.6 percentage points more likely to have job contracts, 14.7 percentage points more likely to have access to a pension plan and 9.5 percentage points less likely to work more than 8 hours a day than controls. No difference between trainees and non-trainees (comparison group) could be detected with respect to paid-leave. Differentiating by types of programs suggests that the higher probability to get access to better quality jobs is associated with participation in STEP-UP, SDIS or NSDC. Among the three programs, STEP- UP seems to have the highest impact. In the case of the other two programs (ASDP and RSETI), differences between trainees and non-trainees (comparison group) in the probability to access better quality jobs were not found statistically significant. The absence of impact on employment and quality of jobs of the two schemes (ASDP and RSETI) which target rural youth and are more pro-poor than others may reflect program design and implementation issues as well the greater difficulty faced in rural areas to find work opportunities. A smaller sample size7 may have also contributed to it. Table 10- Labor Market Impacts of Skills Development Programs Employment Rate Wages/Earnings Job quality All Trainees 7.3 percentage points 21%*** Formal contract: 7.6 p.p.** (p.p.) *** Workday more than 8 hours: minus 9.5 p.p.** Pension: 14.6 p.p*** Paid-leave: not stat significant 6 Only 15% of workers (trainees and non-trainees) have a contract, 37% have paid leave, 24% are given access to a pension plan and 21% work more than 8 hours a day. 7 The small number of training providers under RSETI has led to a sample of only 268 observations (trainees and controls). The sample size for ASDP is larger (682 observations) but most data come only from 2 states, MP and Rajasthan. In other states, interviewers found that most of the selected TPs were closed or no longer running the scheme. 18 Men 4.5 p.p. *** Women 12 p.p. *** NSDC 11 p.p. *** 43%*** Workday more than 8 hours: minus 8.5 p.p.* Pension: 19.2 p.p*** Contract and paid-leave not stat significant SDIS 6.9 p.p. *** 43%*** Formal contract: 12.9 p.p.** Workday more than 8 hours: - 7.7 p.p.* Pension and paid-leave not stat significant STEP-UP 5.2 p.p. * 23%*** Formal contract: 21.1 p.p.*** Workday more than 8 hours: - 12.1 p.p.** Pension: 52.5 p.p*** Paid-leave not stat significant ASDP 5.1 p.p. not statistically 12% not statistically None of the variables significant significant statistically significant RSETI -3.8 p.p. not statistically -9% not statistically Paid-leave: -18.9 p.p.** significant significant Other variables are not stat significant *** - statistically significant at 1%, **- statistically significant at 5%, *- statistically significant at 10% Note: In the case of all trainees, regressions were run with a pooled sample of all trainees and controls, with no distinction by program. In all other cases, separate regressions were run for each population group (male/female) and program with a matched control group. Do personal characteristics and location matter for employment outcomes? Does the impact of training vary depending on characteristics of trainees? Personal characteristics such as age, education, marital status and location do matter for employment outcomes (see Table 11). In India, as in other parts of the world, men who are married have a higher probability of being employed than men who are single, widowed or separated. This most likely, reflects the effect of having family responsibilities and a more pressing need for earnings, and translates in a more active job search. The positive impact of being married on employment probability is also observed for women. 19 Table 11 - Labor Market Impacts of other factors Employment Rate Wages/Earnings Job quality Age  An additional year of  Not statistically  Has positive effect on age increases significant pension, probability of  Not statistically significant employment by 1.2 for contract, paid-leave, p.p. overall (2.0 p.p for and long working hours males, and 0.8 p.p. for females). Moreover, when comparing age groups, 20-25 years old are 3.8 p.p. and more than 25 years old are 15 p.p. more likely to be employed than those less than 20 years old  The employment effect of training is not homogeneous with respect to age: zero for age 15-20, 6pp for age 21-25 and 14pp for age>25 Education  Higher secondary  Those with primary  Less educated are less educated are 7-10 p.p. and below, and likely to have paid-leave less likely to be secondary education and pension than higher employed than those receive 27% and 17% educated with higher education lower wages,  Not statistically significant or lower education respectively, than for contract and long level those with higher working hours education Family  Belonging to a low income income family makes no difference in terms of probability of being employed State  Individuals in AP and  Wages in Rajasthan Some state variation in Assam are more likely are 55% and in AP contract, long working hours to be employed than 45% higher than in and paid-leave those in MP, MP. Wages in Odisha individuals in Odisha not statistically 20 Employment Rate Wages/Earnings Job quality and Rajasthan are less different from those likely to be employed in MP. than those in MP Others8  Married are 14-21 p.p.  Field of training more likely to be related to the job is employed associated with 17%  Knowing English higher wages increases probability of  Marital status is not employment by 7p.p. statistically significant  Part of the training conducted in industry increases prob. of employment by 8.8 p.p.  Work experience prior to training increases prob. of employment by 60 p.p. *** - statistically significant at 1%, **- statistically significant at 5%, *- statistically significant at 10%  Education levels also matter for employment outcomes. Individuals with less than 10 years of schooling are more likely to be employed than those with higher education, perhaps because of more pressing needs for income. One somewhat surprising finding, however, is that individuals with higher secondary education (12 years of schooling) are less likely to work than those with both lower and higher education levels, possibly because a large proportion of them still intend to pursue further studies.  The employment impact of training increases with the age of trainees. Both the effect of education and that of marital status on employment are observed for both trainees and non- trainees and training makes no difference. This is not the case when considering the effect of age on employment. As can be expected from what is the usual life-cycle profile of labor force participation of the population, the probability of employment increases with age for men and women, trainees and non-trainees. Being at least 20 years of age (and even more being more than 25 years old) raises significantly chances of employment relative to being much younger. However, distinguishing trainees and non-trainees shows that training reinforces the age effect on the probability of employment. While the participation in SD programs was found to increase the employment rate by 7 percentage points overall (table 10), it is found to have no employment impact on trainees aged 15-20, and only to start being effective for those aged 21 or more (increasing by 6 percentage points the employment rate of trainees aged 21-25, and by 14 percentage points the employment rate of older trainees) (Table 11). 8 Information on prior work experience, knowledge of English, and type of training is only available for trainees. 21  Location matters for employment. Differences across states already mentioned earlier, appear clearly and strongly, even after controlling for other trainee and control characteristics. The probability of employment is much higher in AP than in all other states. Being in AP increases the employment rate by 20 percentage points relative to Assam, by nearly 40 percentage points relative to MP, and about 45 percentage points relative to both Odisha and Rajasthan. From the point of view of employment, AP and Assam appear to be “better� states and Odisha and Rajasthan “worse�states with MP standing in the middle. An interesting question is how much of those differences between states can be imputed to the programs (and how effective their implementation is) or to other factors that equally affect trainees and non-trainees, such as growth or a more formal or more fluid labor market. The econometric analysis was not able to disentangle9 between the two hypotheses but the fact that AP, among the 5 states, has a significantly higher per capita GDP, a larger proportion of the population living in urban areas and the smallest share of self-employed (see Table 12) gives weight to the hypothesis that external economic and social conditions contribute to the explanation of higher impact in AP. However, evidence from Assam would lead to an opposite conclusion as none of the indicators for this state could explain why Assam performs better than others. Its GDP per capita is the lowest; it has the smallest share of the population living in urban areas and the highest proportion of self- employed. Differences in youth unemployment rates across states (with Assam having the highest, and MP the lowest) do not either support the hypothesis that a more formal or fluid labor market is a strong explanatory factor. This suggests that, although a better economic and social environment may facilitate labor market insertions of trainees, at least some of the differences observed across states are due to factors related to the way programs are designed and implemented. Table 12. Selected Economic Indicators in the 5 States under study. States GDP per capita Labor Force Share of Urban Unemployment (rupees base Participation Self- Population. Rate year Rate Employed % of Total Age 15-29) 2004-05) Population Andhra 48,142 66.4 43.5 32.8 .09 Pradesh Assam 24,970 52.8 67.4 10.4 .19 Madhya 26,505 58.9 56.1 25.5 .05 Pradesh Odisha 30,623 61.6 59.4 15.1 .16 Rajasthan 32,134 61.9 61.9 23.7 .08 Source: NSS 9 Interaction variables were introduced in the regressions to test the hypothesis of positive links between program performance and state but coefficients were not found statistically significant. 22  Survey data also indicate that some other characteristics10 make employment of trainees more likely. The most significant is the existence of some work experience prior to the training period. It increases the probability of employment by 60 percentage points. Other factors with positive impact are the knowledge of English and the combination of classroom training with apprenticeship (or some training in enterprise). Both increase the probability of employment by 7 and 8.8 percentage points, respectively. 4. Wage and Earnings Effects of SD Programs The analysis of wage and earnings effects can only be done with a smaller sample. Information about wage/earnings and work history can only be available for those (trainees and non-trainees) who are (or) have been working. This reduces the original sample size to about one-third. In addition, a number of respondents have refused to give full information about their wages and work history, and/or could not be reached a second time after information gaps were detected. Despite the smaller sample11, a number of findings can be drawn:  Participation in a SD program appears to provide a positive wage premium. The premium is of the order of 21 percent and is statistically significant.  There is an additional wage gain when the content of the training program is directly relevant for the job being held.  Female wages/earnings are systematically about 20 percent lower than those of males, after controlling for the education level, participation (or no-participation) in a particular SD program, and the state of residence.  As could be expected, earnings increase with the level of education for both trainees and non- trainees.  As already observed in the case of employment, wage differences across states are significant. Wages/earnings are highest in Rajasthan and AP and lowest in Odisha and MP, with Assam in middle ground. Those differences appear after controlling for age, marital status, gender, education, and participation in an SD scheme.  However not all programs seem to be equally effective. When a distinction is made by program type12, the positive wage premium persists for NSDC, SDIS and STEP-UP trainees but disappears for those who participated in RSETI or ASDP. Estimates seem robust to different specifications. Alternative methods considered were: (i) cross- sectional propensity score matching to estimate the average treatment effect instead of a simple 10 Information about past work experience, knowledge of English, and type of training is only available in the sample of past trainees. There is no such information in the sample of individuals in the comparison group. 11 The sample with wage/earnings information has 302 observations. 12 In the case of wage regressions, only a pooled sample was used because of the limited number of observations. However, results are very consistent with those found previously with regressions run separately for each program. 23 regression controlling for observable characteristics; (ii) the second method relied on a different comparison group, drawn from data from the labor force module of national sampling survey (NSS). The first method (propensity score matching) led to an estimate of an average treatment effect of about 30 percent, i.e. not very different from what was previously found. With the use of a different comparison group (drawn from the national labor force survey), estimates of the wage effect also remained of similar magnitude (21-26 percent) and coefficients of all other variables (age, gender, education, state of residence) led to similar conclusions. 5. Cost-Effectiveness of SD programs. A precise measure of the cost-effectiveness of the different schemes would require estimates of costs and benefits, differentiating by types of programs, course duration, location and other factors. Such a precise estimate cannot be obtained. However, a rough calculation can be done, using the average estimates of the employment and wage effects of training indicated earlier, that is, assuming that participation in SD programs increases the employment rate by 7 percentage points, and the wage of those employed after training by 21 percent. The cost of training per participant will obviously vary depending on content and duration of training. Taking into consideration an average cost of 18,000 Rs. per participant 13 and previous estimates of the employment and wage effects, the investment cost will be recouped if those working after training remain employed at the higher wage rate for at least 21 months. On the basis of those rough estimates, programs run by NSDC, SDIS and STEP-UP would generally be cost–effective. The absence of employment and wage effects found for ASDP and RSETI trainees does not allow a similar conclusion for those two programs. 6. India vs. International Experience There are a number of similarities between findings in India and in other parts of the world (see Box 2). The heterogeneity in impacts depending on the characteristics of participants, the benefit from training greater for women than for men, and the positive effect of part of the training conducted in industry are findings common to many programs world-wide. Indian results also seem consistent with the conclusion reached by earlier reviews that training impacts are more positive in developing countries than in the US or Europe (Betcherman et al. 2004). A comparison with Latin American programs seems particularly relevant as programs implemented in Latin America have a number of features common with those implemented in India. Programs are of similar short duration. For most of them, there is a separation between financing and the provision of training and on average, the investment cost is fairly low. The cost per participant in Latin American programs is in the range of US$400-US$750 (see Gonzalez-Velosa, et al., 2012); the average cost of Indian programs as communicated by NSDA is around 18,000 INR, i.e. around US$300. 13 This figure is provided in NSDA Dovetailing/Rationalization Report as being the average cost of training across GOI funded schemes. The average duration is approximately 600 hours and the average hourly cost, Rs 30 per hour. 24 In both Latin America and India, the employment effect of training is modest but positive, and roughly in the same range. The wage effect seems somewhat higher in India but there, estimates may be overestimated due to the small sample size and some selectivity bias. Box 2. Impact of Short-Term Training Programs: International Evidence. Short-term training programs have been widely used in many parts of the world as a policy instrument to increase the employability of particular population groups. Focus groups have mostly been unemployed and/or youth, with particular attention to low skilled or disadvantaged youth. These programs have been evaluated, albeit with various degrees of rigor, in the United States and Europe, as well as in some developing countries. The majority of the existing empirical evidence related to the US case suggests that there is substantial heterogeneity in impacts depending on the characteristics of the participants and the type of training. Many studies conclude that women benefit more from training than men and that on-the-job training is more effective than classroom training. With respect to youth, evaluations have not yielded robust results. Firm conclusions have been difficult to draw from the European evidence (Heckman et al, 1999), in part because of the lack of experimental studies and the wide variation in evaluation methods. One key finding, however, to be drawn from the European experience, is that programs targeted to youth are less likely to show positive effects than programs for adults. In developing countries, evidence of the impact of training is limited but in recent years, a number of evaluations were conducted, notably in Latin America, following the wide promotion of short training programs across the region since the early 90s. Programs that have been evaluated are implemented in 7 countries: Dominican Republic, Chile, Colombia, Argentina, Panama, Peru and Mexico. Program modalities differ across countries but common features are: (i) the separation of financing from the provision of training and a competitive selection of training providers and courses: (ii) the nature of the training is labor market demand-driven, not set by the government; (iii) classroom training is often followed by an internship in the firm. Some of these evaluations have been carried out with an experimental design relying on randomization, others with a non-experimental design. Results suggest that the employment effect range from null (or non- statistically significant) in Argentina and Dominican Republic to positive and significant (around 5 percentage points) in the other countries, and with impact of 6 to 12 percentage points for some groups such as women in Colombia and Panama. As in the US, results show substantial heterogeneity: the effect of job training on employment rates varies by age, gender and region. Higher rates are obtained, in general, among women and younger people. In terms of earnings, the analysis suggests a small positive impact. In addition, in most cases, there is significant impact on job quality, measured by getting a formal job, having a contract and/or receiving health insurance. Given the low cost of these programs (ranging from $300 to $750), they are generally found cost-effective. In Sub-Saharan Africa, where informal, purely practical, apprenticeships remain a key mechanism to help youth find employment, a number of programs are now run along lines similar to those of Latin America. Programs implemented in Ethiopia, Uganda, and Kenya, are of short duration, usually combine some training in basic skills and a specific trade with some on-the –job experience. Some attempt to link training with access to credit. Compared to Indian programs, Sub-Saharan programs tend to include a 25 larger proportion of applicants who have dropped out of school before completing secondary education. Most evaluation reports indicate some positive impact on motivation/self-perception and qualifications but so far, do not include sufficient data to determine whether programs were successful in improving employment. In the Uganda “Youth Opportunities Program�, however, an evaluation conducted through a randomized trial, showed that groups which received grants to enroll in training and practice the trade after graduation, were twice as likely to engage in a skilled work and improved their profits by 50 percent compared to the control group. Sources: Ibarrarán, P. & Rosas-Shady, D. (2009), Gonzalez- Velosa, C., Ripani, L. & Rosas-Shady, D. (2012) and Inoue, K.,E. Di Gropello, Y/ Sayin Taylor & J. Gresham (2014). 7. Findings from the Qualitative Study. The first objective of the qualitative study (see Box 3) was to examine in detail the operating modalities of the five skill development programs. All five programs were found to have comprehensive guidelines, outlining their mandates. Most are financed through grants, except for NSDC which usually provides concessional loans to training providers (see Table 13) and occasionally, offers grant or equity options. This part of the study also confirms the earlier finding that providers can be public, private or a civil society organization, and that, in all cases (except for RSETI) some form of competition is used to select training providers. Competition, however, is based on criteria which can vary across programs and can be more or less stringent. Focus Group Discussions and interviews further confirm that the mobilization of beneficiaries is done through various means (publicity vans, newspaper ads, pamphlets, awareness camp, road shows as well as word of mouth). For programs targeted towards the poor, mobilization efforts are carried out through the BPL lists available at the Village/ Block level and/ or through participatory process with the support of the community and block level structures. The RSETI program is reported to make special efforts in ensuring that at least 70 percent of candidates enrolled are from the BPL category as training fees are only recouped for BPL candidates. However, this finding is in contradiction with what data from the quantitative surveys showed (see Table 2): i.e. only 25 percent of RSETI beneficiaries being BPL Card holders14. Box 3. The Qualitative Study. The objective of the qualitative study was to review the organization, system and processes of the five National Skill Development programs and help understand how these affect the reach, delivery, and impact of those programs. The study covered the same five states as the quantitative study, i.e. Andhra Pradesh, Assam, Madhya Pradesh, Odisha and Rajasthan (and two districts in each state in addition to the state capital). 14 The true percentage of poor beneficiaries may be somewhat higher as RSETI accepts alternate modes of identification (such as a certificate issued by the Gram Panchayat certifying the BPL status) when BPL cards are not available. 26 Each SD program was evaluated over five thematic areas through one to one discussions with key stakeholders and through focus group discussions. The thematic areas that were explored are: Program guidelines & Management, Functions and inter-relationships, Fund Flow mechanism, Program delivery, Monitoring & Evaluation, Program efficiency and effectiveness. For each program, discussions took place at the national, state and district levels. At the national level, respondents were officials of the relevant ministry, and at the state level, officials entrusted with the management of the program. At the district level, interviews were held with project implementation agencies, training providers, trainers and trainees. Source: PwC report (2014) Table 13. Funding Mechanisms of SD Programs and Selection of Training Providers Programs ASDP RSETI STEP-UP SDIS NSDC Funding Grant: 75% Grant for Grant -75% Grant – 100% Loans to TPs Mechanisms from the reimbursement from Central from Central on basis of Central Govt & of training cost Govt &25% Govt. business plan, 25% from the of rural BPL from State to be repaid State Govt. For youth. Govt. back to NSDC North East For North East (with States and J&K, Training cost of States and J&K, concessional the Central non-BPL youth the Central interest). Govt. share is paid by Govt. share is 90% and the respective lead 90% and the In some cases, State Govt. banks State Govt. equity share is 10%. share is 10%. financing and grant funding Training Private, Public Lead Bank of IITs, NITs, Vocational Private training Providers and civil the district Industry Training providers society Associations, providers organizations Engineering registered colleges, under Central Management or State Govts, Institutes, or Industrial Foundations, institutes, reputed NGOs professional councils Selection Competitive, Lead Competitive Online Multi-stage Process based on commercial bidding (least application diligence scorecard with bank in district cost-based) in subject to process, with weights given sets up a RSETI most states inspection professional to certain process organizations parameters In few states, as partners (financial quality-based robustness, or quality-cum- experience, cost based 27 Programs ASDP RSETI STEP-UP SDIS NSDC etc.) Some of the key findings that emerge from this part of the study refer to critical steps in program implementation and are the following: (i) Quality and Relevance of Programs While a competitive process (or the assurance of meeting certain criteria) is widely used to select training providers, selection criteria vary, which most likely affect the relevance and quality of program delivery (see Table 13). In several instances, the lack of a well- laid process to validate the training proposals on qualitative grounds was noted. Competition mostly based on cost, or mainly considering the infrastructure set-up or the availability of trainers as eligibility criteria is less likely to ensure quality than competition that takes into account several dimensions, including quality of the offerings and labor market demand. NSDC seems to be the only program which, through its multi-phase selection process, contrasts proposals with results from skills gap analysis, an important factor likely to ensure better program relevance. However, even in the case of NSDC evaluation, there is no mechanism to cross- check whether the identified skills needs are already covered by other projects or programs. The insufficient use of labor market analysis and attention to market demand lead to the selection of providers and course offerings based mostly on availability of trainers and existing infrastructure, rather than on job availability. This also explains why there is a concentration of offerings over a limited number of trades across all programs15 (see also Table 3). Furthermore, even within this dominant set of trades, there could be high variance in quality. Across states, recruitment process and qualifications of trainers may vary. In Assam, faculty members are recruited through word of mouth; in other states, through advertisements. In AP, trainers are expected to have a qualification equivalent to a B. Ed. In other states, this is not required. In addition, compliance with the National Occupational Standards developed by the Sector Skills Councils is only verified for courses administered under NSDC and select courses under ASDP. Another factor that may affect quality is ‘franchising’. It seems widely used by all programs, except STEP-UP and RSETI, in the hard to reach areas, to help reaching training targets. However, while there is a competitive process in place (even if imperfect) to select providers, there is no such mechanism for franchised institutions. The review process does not look at the capacity and quality of their offerings and monitoring systems do not capture their performance or lack thereof. 15 Interviewed trainees confirm that the choice of courses offered by training providers is limited and that there is often little demand for those skills in the local market. 28 (ii) Assessment and Certification Assessment and certification are important quality assurance mechanisms. They are found to differ across, and even within programs, and to be deficient in many cases. While ASDP and SDIS guidelines mandate third-party assessments and certification, there is no such mandate for STEP-UP. In the latter program, assessments are done by the training providers and followed by certification with no standardization across providers. In RSETI, certificates are issued by RSETI director but, while considered as a necessary requirement, those are not sufficient conditions to obtain a loan. In NSDC, assessment is also carried out by TPs with no external independent evaluation. While enabling TPs to carry out assessments has the advantage of a lower cost and a more rapid process, the absence of independent evaluation gives scope for masking quality deficiencies and the lack of standardization gives uncertain value to the information that certificates are supposed to give to potential employers about the skills of trainees. (iii) Monitoring, Reporting & Evaluation. All programs have a monitoring system in place, some even mandate a fairly sophisticated one but weaknesses are significant. The development of web-based systems which would allow quick capturing and analysis of information, is mandated by several schemes (ASDP, STEP-UP) but implementation varies as this requires significant investment and capacity building. Many states are still unable to capture data online and TPs often lack the capacity (servers, internet connectivity) to meet reporting requirements. Monitoring reports are often mainly aimed at claiming reimbursement and cases of duplication in accounting of beneficiaries by providers which operate across schemes have been reported. When used to evaluate TPs, monitoring reports mostly inform about placement metrics, geographical spread, revenue and targets. How much verification is done is uncertain. For example, the placement rate of RSETI graduates is reported to be higher than 60 percent while the quantitative surveys indicates a much lower rate. There are, however, cases of more in-depth evaluation being done, such as tracer studies or mid-term evaluation, as for example done by the DGET for SDIS, but it is not clear whether there is any follow up. Another limiting factor is the absence of a labor market information system (LMIS) operational at a state level which prevents customizing training courses to local needs. Only NSDC is reported to have developed a State Skill Gap Assessment with demand mapped at the district level, but there is no indication that program implementers are using this information to rationalize course offerings. (iv) Post-Training and Post-Placement Support Sustainability of employment is the objective of all SD programs and a critical element in program guidelines. Investing in hand-holding, post-placement skill upgrading is encouraged as well as follow-up for several months. However, implementation in most cases has just been initiated although some cases of good practices have already been observed (Box 4). 29 Box 4. Some Examples of Good Practices to enable Placement and Retention Through the various interviews of training providers (TPs), some good practices have been identified and are worth highlighting:  One TP in Rajasthan reported to have voluntarily initiated the post-placement tracking and support to the beneficiaries for a period of 3-6 months after initial placements  TPs in Andhra Pradesh reported having appointed state-level nodal persons to be responsible for providing placement and post-placement support and guidance  In Odisha, a TP indicated to have established a team of professionals which actively connect to the local industries, enable demand-based placements, and extend post-placement support to employers as well as beneficiaries.  One of the TPs in Madhya Pradesh reported to have implemented on-the-job internship for the beneficiaries, for a period of 15 days within the local labor market to enable the beneficiaries get hands-on experience with the technical equipment, working environment and learn about the dynamics of teamwork.  One of the major TPs in Madhya Pradesh indicated to have forged linkages with over 1,400 industry partners across 6 states to facilitate placements of trainees. Post-placement support, when it exists, remains limited to a short period (3, 6 months, 1 year maximum). Since most trainees are young, in their early 20s, this implies support focused on labor market entry. By targeting youth and focusing on the beginning of working life, SD programs leave unattended the needs for upgrading skills/reskilling over the several decades of professional life and therefore, forego a large potential for productivity increases. Some additional findings specific to particular states and programs are worth noting: At different stages of the review, Andhra Pradesh (AP) stood out as better performer, a finding which is consistent with results discussed in the previous sections. Evidence of clear demarcation of roles and responsibilities and better harmonization and coordination among stakeholders with, in particular, the creation of umbrella organizations16 overseeing SD programs was reported in AP. Similarly, the adoption early on, for several programs, of a mechanism linking the reimbursement of 75 percent of the training cost to successful placement and post-placement support to beneficiaries was found to differentiate AP from the other states and to affect quality. AP was also reported to have an effective e-governance system that greatly facilitates monitoring, reporting and evaluation, to hire more qualified trainers and (as said before) to have state-level program-specific nodal persons responsible for providing placement and post-placement support and guidance. While the quantitative analysis had 16 Such as EGMM (Employment Generation and Marketing Mission) which aims at better coordination across programs and at avoiding duplication in terms of targeting beneficiaries and accounting. 30 systematically found higher employment outcomes of SD programs in AP, it could not detect the causing factors. Those described above could well be some of them. Several weaknesses of the RSETI program were pointed out, such as the absence of clear-cut guidelines for the selection of trainers leading to different methods and qualification criteria across states, a curriculum that has not been updated since 2010, and the limited acceptability of RSETI certificates for loan provision (despite the program focus on self-employment and implementation through the Banking network). Those weaknesses, jointly with the lack of positive impact on employment found by the quantitative analysis, suggest the need for a thorough review and detailed evaluation of this particular program. 8. Conclusions and Policy Recommendations. The analysis of survey data on employment and earnings of participants and non-participants in five Indian SD programs has shown that these programs have, on average, contributed to increase the employability and earnings of participants. They have also contributed to give them access to better quality jobs, even if those are still predominantly in the informal sector. While positive, the employment effect remains modest and significant labor market churning occurs during the period of one to two years following the training. The wage effect seems substantial, considering international evidence but may have been somewhat overestimated. The comparison of outcomes of trainees and non-trainees (comparison group) was done without controlling for non- observable characteristics such as willingness to work or motivation and may have led to upward biased estimates of the employment and wage effects. The second part of the study, based on interviews and focus groups discussions, pointed out that several aspects in the process of selection of training providers and courses were not conducive to ensuring high quality and relevance of programs. It also detected weaknesses in the quality assurance mechanisms, monitoring and evaluation, and placement/post-placement support. Both parts of the study suggest that all five SD programs may not be equally effective. When a distinction is made between them, no positive impact on employment and wages could be identified for two of the programs, ASDP and RSETI which target rural youth. Some weaknesses in the operating processes of the RSETI scheme were also identified through the qualitative study. All five states under study do not either seem equally effective. AP stands out followed by Assam in achieving the highest employment rate. It also stands out as having more efficient operating modalities and causes of greater success may have to be found both in differences in program implementation capacity and processes and in differences in the economic and social environment specific to each state. The following policy recommendations emerge from those findings: Program design and implementation could be improved by taking more into account the factors that increase the chances of success: 31 The most obvious factor is the quality and relevance of the training. It is well established that close links with employers and a good knowledge of labor market conditions are effective to improve the quality and responsiveness of programs to the needs of the labor market and the employability of trainees. This could be achieved by: (i) developing and using more extensively labor market information systems, sectoral skills gap analyses, tracer studies and feedback from sector skill councils/industry associations, for the identification of trades and skills in demand; (ii) improving the selection process of training providers (and franchisees) with consideration of quality (in particular, compliance with the National Occupational Standards and integration of soft skills into the course content) and market relevance of course offerings as selection criteria; (iii) making increasing use of apprenticeships, or programs combining classroom with practical training (which were shown to have a clear positive impact on employment probability). All these factors would help ensuring that training offerings match the demand and respond quickly to changes in production processes. A more careful targeting of programs to the population groups for whom those programs are particularly effective could lead to a more efficient use of public resources and a higher employment rate:  Selecting a greater proportion of applicants aged 21 or more would increase program effectiveness. The impact of training was found to be heterogeneous with respect to age and no employment impact was found for the very young applicants, aged 15-20. For those, efforts should be made to identify the causes of failure and on this basis, to reconsider the type of programs that could be beneficial for them17  Screening applicants to attribute a larger proportion of seats to youth motivated to seek employment after training. One factor that affects the probability of employment is the motivation of applicants, and their interest in following the training as an entry point into the labor market. In programs focusing on self-employment, one important factor of success is the entrepreneurial trait of the applicant. As indicated earlier, around 28 percent of applicants enroll in SD programs with no intention to work after completion. Unless programs yield long- term benefits that can be captured several years later, enrolling those applicants represent a waste of public resources. Promoting independent assessments and standardization of certificates would enhance the certificate�value�and the marketability of beneficiaries. As noted before, while enabling TPs to carry out assessments has the advantage of lower cost and greater timeliness, the absence of independent evaluation gives scope for masking quality compromise. Furthermore, the lack of standardization of certificates reduces the value of the information that certificates are supposed to give to potential employers regarding the acquired skills of job seekers. Practices followed by some TPs such as adopting a course-specific standard approach to assessments to enable standardization or initiating joint- certification with industry partners are examples of ways to address this issue and worth considering. Training qualified assessors and ensuring the availability of a pool of qualified assessors would also 17 Perhaps longer-term courses offering different levels of specialization and a wider range of skills. 32 facilitate the recourse to independent evaluation (several TPs in SDIS indicated that the lack of expert assessors had constrained timely assessments). Strengthening post-placement support would make SD programs more effective. The quantitative analysis highlighted that initial placement does not ensure sustained employment and that significant labor churning occurs during the early years of working life. An effective post-placement support during at least a period of 6 months to a year would help achieve a more stable employment as it would facilitate information about job availability, lower the costs of search and thus, reduce the length of unemployment spans between jobs. This again supposes an active engagement with the local industry and service sector. In states in which local job opportunities are limited, the placement support should include information about job openings outside the state and help in handling migration related challenges. Enlarging the scope of SD programs and promoting life-long learning would generate large productivity increases by facilitating the adjustment of the Indian labor force to the changing needs of a modernizing economy As the need to acquire new skills, both cognitive and non-cognitive, is clearly not limited to the early stages of working like, enlarging the scope of SD programs to cover not only unemployed youth but also people already in the labor force (whether or not former trainees), would provide them with opportunities to reskilling between jobs, change occupations and become more productive. It would open the way to life-long learning and address a weakness highlighted by enterprise surveys, namely, the very low level of in-service training provided by enterprises to their workers in India compared to competitors internationally18. Linking funding to desired outcomes improves efficiency, provided incentives are well designed. It is indeed well established that incentives can be powerful tools, but international evidence has also shown that incentives need to be carefully designed to achieve the desired effect. Given the objective of ensuring employment for trainees, placement rate has been the main indicator of success and placement targets have been used (to various degrees) as conditions for funding in SD programs. Recent modifications of guidelines have even tended to make these conditions more stringent. Encouraging placement through appropriate incentives should be maintained as it obliges TPs and program managers to shift focus from inputs towards outcomes. It incites TPs to establish links with potential employers and to offer courses that match labor market demand and brings some competition into the sector. Placement also gives opportunities to youth to gain some useful work experience, even if of short duration. However, a high placement rate may be very difficult to achieve if programs enroll trainees who do not have a firm intention to enter the labor market. In the overall population, the labor force participation of youth in the 15-24 age group is much lower than the mandated 70 or 75 percent placement rate of some programs. Achieving a high placement rate thus requires a more stringent screening of applicants to ensure that programs are targeted towards those most motivated. False reporting might be encouraged otherwise. 18 Investment Climate surveys showed that about 17 percent of manufacturing firms in India provide in-service training to their employees, compared to an average of 30 percent in Europe and Central Asia, and to about 55 percent in East Asia (World Bank 2007). 33 An additional important observation is that focusing on placement is not sufficient to ensure a positive employment effect. As shown earlier, placement does not ensure sustainability of employment. Measuring the performance of SD programs requires a longer monitoring period, capacity for follow up during at least a couple of years. Therefore, performance-based payment mechanisms should include incentives to strengthen monitoring and tracking mechanisms and post-placement support in each program. Pursuing efforts to strengthen M&E is critically needed across all programs. This means, first of all, improving data collection and analysis, increasing the use of independent evaluations and tracer studies, and what is most critical, making use of the information generated for corrective action. There has already been progress in this direction but the stage of development of the M&E system still varies across programs and states. The results of this study also suggest that a more in-depth evaluation of ASDP and RSETI programs for which no employment or wage effect could be detected is needed The difficulties encountered in finding training providers running the scheme in some states for ASDP and the small sample size for RSETI prevent reaching definite conclusions but results do suggest that a more extensive evaluation should be conducted before envisaging any expansion of those schemes and that significant changes in design and implementation could be required. An additional recommendation would be to include, as part of these evaluation efforts, a study to assess the long-term effects of SD programs. Such assessment does not need to be done on a routine basis, but should at least, be conducted once. This would bring additional useful insight for policy makers as one important limitation of this study (common to most evaluation studies world-wide) is the time period during which the impact of training is being observed. The analysis covers a period of one or two years after training and cannot assess the long-term effects of training. Knowing the long-term effects would be needed to measure the full return on the investments made in skill development. It would also help ascertain whether training youth who enroll with no intention to enter the labor market after completing training brings lasting benefits or represents a waste of public resources. Learning from the experience of States that have been more successful could also generate greater efficiency. The study has identified a number of practices, organizational features such as the use of umbrella organizations, which have contributed to better coordination and much better outcomes of SD programs in AP. Other states might gain by learning from the AP experience and examining whether those practices could be adapted to their own environment. Some good practices may also exist in specific areas in States other than AP. Sharing lessons from experience would benefit all of them. One final recommendation is not to consider the SD programs analyzed in this study in isolation but rather as part of a diversified package of human capital investments which includes education and longer-term training programs. Even if most SD programs are found to have positive labor market effects, expectations of their overall impact on the Indian economy need to remain modest. Courses are of short duration and cost is low. The amount of human capital that can be provided through those programs is likely to remain modest. While these programs can be a useful policy instrument to facilitate insertion in the labor market, they cannot be expected to yield huge productivity increases and 34 need to be complemented by other types of human capital investments. Their targets should be adjusted accordingly. Table 14-- Summary of Policy Recommendations Objective Policy Recommendation Increase the quality and labor  Develop & use LMIS, skills gap analysis, tracer studies, market relevance feedback from employers or SSCs for identification of of programs relevant trades and skills  Improve the selection process of TPs ( including franchisees) with consideration of quality (not only cost) and relevance for the labor market  Strengthen links with employers through greater proportion of courses including apprenticeships, or combination of classroom/practical training Improve targeting of  Increase the proportion of trainees aged 21 or more beneficiaries (revise/reconsider training programs for youth aged 15-20)  Improve the mobilization and selection process of beneficiaries, giving preference to those highly motivated to enter the labor market Strengthen assessment and  Encourage third-party assessments (or joint-certification with enhance the certificate “value� industry partners) and standardization of certificates  Train qualified assessors Strengthen post-placement  Provide post-placement support for 6 months to one year support  Include migration support in states with limited local job opportunities Enlarge the scope of SD  Provide access to SD programs not only to youth but also to programs and consider life-long adult workers (former trainees or others) learning Link funding to outcomes to  Maintain funding conditions linked to placement targets improve efficiency  Combine with financial incentives for effective monitoring & tracking mechanisms and post-placement support Strengthen M&E  Pursue the development of an effective M&E system, including systematic data collection, tracer and evaluation studies  Strengthen the capacity for analysis  Carry out an in-depth evaluation of RSETI and ASDP programs before any program expansion  Consider a one-time analysis of long-term impacts of SD programs. Learn from successful practices  Examine the practices and organizational features which & share lessons from experience have made AP more effective and their possible introduction in other states. Develop a comprehensive skill  Consider short-term SD programs as part of a package of development strategy diversified human capital investments and adjust targets accordingly 35 Annex 1 – Socio-Economic Characteristics of Male and Female Trainees and non-Trainees by Program Panel A: Past Trainees Trainees NSDC SDIS STEP-UP ASDP RSETI Male Female Male Female Male Female Male Female Male Female Mean Age 21 22 22 22 22 23 22 23 22 23 Social group (%) SC/ST 21 18 24 17 30 28 29 23 25 20 OBC 46 46 41 39 39 53 37 52 45 55 Other 33 36 35 44 31 19 34 25 30 25 % never married 92 87 90 83 93 75 84 73 77 66 Religion (%) Hindus 94 93 91 80 90 86 91 92 97 82 Muslims 5 6 9 19 8 13 8 6 2 17 Education (%) Illiterate 0 0 0 0 0 1 0 0 0 1 1-9 yrs school 5 6 1 9 1 13 2 16 11 29 10-12 60 66 63 64 67 69 69 68 59 48 Higher educ. 35 28 36 27 32 17 29 16 30 22 Sample size 446 199 475 373 226 356 209 156 87 93 Panel B: Non-Trainees (Comparison Group) Non- Trainees- NSDC SDIS STEP-UP ASDP RSETI /Controls Male Female Male Female Male Female Male Female Male Female Mean Age 22 22 22 22 21 22 22 22 21 22 Social group (%) SC/ST 21 21 23 15 29 24 31 22 23 28 OBC 48 36 45 44 43 51 47 48 46 40 Other 32 43 32 41 28 26 22 30 31 32 % never married 93 88 89 84 97 81 81 67 92 76 Religion (%) Hindus 88 91 93 82 96 89 95 97 86 85 Muslims 11 8 6 17 4 11 5 3 10 13 Education (%) Illiterate 1 0 0 0 0 2 0 0 0 0 36 1-9 yrs school 7 13 5 9 2 11 14 25 7 14 10-12 yrs school 74 73 68 65 75 74 67 57 85 74 Higher educ. 18 14 27 25 23 13 20 19 9 13 Sample size 354 177 383 265 176 195 207 110 81 69 37 Annex 2. Socio-Economic Characteristics of Current Trainees by SD Program Characteristics of 1,995 trainees who were still in training at the time of interview were very similar to those of previous cohorts as shown by the table below. NSDC SDIS STEP-UP ASDP RSETI Mean Age 23 21 21 23 23 % <21 52 57 54 43 41 % Female 33 51 59 47 57 Social Group (%) SC/ST 22 25 22 21 39 OBC 44 40 46 49 37 Other 34 35 32 30 24 Religion (%) Hindu 91 86 86 91 92 Muslim 6 13 13 8 8 % never married 91 90 86 82 82 Education level %) <10 yrs 8 4 7 13 21 10-12 68 74 75 67 64 Higher Education 24 22 18 20 15 % having worked in 24 1 3 1 2 3 months prior to training % with prior vocational training 4 8 3 4 3 Number of household members 5 5 5 5 5 Father as head of household (%) 89 89 79 88 80 Sample size 531 649 413 252 150 38 References Betcherman, Gordon, K. Olivas and A. Dar (2004) “Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries�. World Bank Social Protection Discussion Paper 0402, Washington DC. Ibarrarán, Pablo, and David Rosas-Shady (2009), “Evaluating the Impact of Job Training Programs in Latin America: Evidence from IDB funded operations�, Inter-American Development Bank, Washington DC. Inoue,K., E. Di Gropello, Y. Sayin Taylor and J. Gresham, (2014) “Out-of-School Youth in Sub-Saharan Africa: A Policy Perspective�, The World Bank, Education Global Practice, Washington DC, mimeo (to be published in the World Bank Directions in Development) Gonzalez-Velosa,Carolina., L.Ripani and D.Rosas-Shady (2012) “How can Job Opportunities for Young People in Latin America be Improved? , Inter-American Development Bank, Labor Markets and Social Security Unit, Technical Notes, No. IDB-TN-345 Heckman, J., R. Lalonde and J. Smith (1999) “The Economics and Econometrics of Active Labor Market Programs�in O. Ashenfelter and D. Card (eds) Handbook of Labor Economics, 3 (Chapter 31), Amsterdam: NH Elsevier. PriceWaterHouse (2014) “National Skill Development Programs: System and Processes Study� 39