Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) Report Number: ICRR0022195 1. Project Data Project ID Project Name P132698 Skills Development Country Practice Area(Lead) Sri Lanka Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-55160,IDA-55170 31-Dec-2019 47,812,018.85 Bank Approval Date Closing Date (Actual) 20-Jun-2014 31-Dec-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 101,500,000.00 0.00 Revised Commitment 58,504,065.39 0.00 Actual 47,812,018.85 0.00 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD, p. 6) and the Financing Agreement of September 12, 2014 (p. 5), the objective of the project was “to expand the supply of skilled and employable workers by increasing access to quality and labor market relevant training programs." b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) No c. Will a split evaluation be undertaken? No d. Components The project included two components: Component 1: Program Support to the Skills Sector Development Program (appraisal estimate US$93.6 million, actual US$47.81 million. This component was to use a Performance–Based Financing (PBF) modality. Disbursements up to a capped amount were to be made against specific line items in the annual budgets of the Ministry of Youth Affairs and Skills Development (MYASD) budget. Disbursement against these eligible expenditure programs (EEPs) were to be conditioned on the achievement of specified results, as measured by disbursement– linked indicators (DLIs). Three closely related types of interventions were to be supported by this component, linked to the five Skills Sector Development Program (SSDP) policy areas: (a) strengthening governance and management (policy area 1 of the SSDP); (b) enhancing the quality and relevance of training (policy areas 2 and 3); and (c) increasing access to training opportunities (policy areas 4 and 5). This component was to finance the following pillars: Pillar 1: Strengthening Sector Governance and Management: The main objective was to strengthen governance and management of the sector through a tighter and more intensive focus on results and more efficient use of resources. This pillar was to support reforms to accomplish the following objectives: a) ensuring adequate and predictable financing of the sector; b) improving analysis of labor market supply and demand; c) improving monitoring and evaluation (M&E) of training provider performance; d) increasing coordination within MYASD and across ministries; e) rationalizing utilization of existing Technical and Vocational Education and Training (TVET) resources; and f) implementing performance–based financing to support institutions that perform well. For this pillar, the release of IDA funds was to be linked to achievement of three DLIs: a) MYASD’s annual budget approved and executed in line with the SSDP and its fiscal and budgetary framework (DLI1); b) timely availability of reliable institution and agency data and periodic analysis of courses, centers, and teacher performance (DLI2); and c) performance–based funding of public training providers designed, piloted, evaluated, and expanded (DLI3). Pillar 2: Improving the Quality and Relevance of Skills Development: The objective was to improve the quality and relevance of the skills development programs offered by all public and private institutions and non-governmental organizations. This pillar was to support: a) simplifying, updating and developing the National Vocational Qualification (NVQ) system to respond to employer needs, including increasing demand for soft skills; b) addressing the shortage of qualified instructors by designing and putting in place a revised teacher management policy (recruitment, retention, and career progression) with special allowances linked to performance; c) designing and implementing a teacher professional development program to upgrade instructor skills; d) establishing effective mechanisms and incentives to involve employers in training design and management of training institutions; and e) unifying the Sri Lanka qualification framework and establishing pathways between general and technical education. For this pillar, release of IDA funds was to be linked to the achievement of the following DLIs: a) more relevant training programs for students through increased employer participation (DLI4); b) improved quality assurance mechanisms (DLI5); c) Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) implementation of a revised human resource policy and plan (DLI6); and iv) upgrading of teaching staff skills in priority areas (DLI7). Pillar 3: Expanding Access to Quality Skills Training: Under this pillar, the government was to increase equitable access to quality skills development programs through priority interventions that the project was to support and monitor: a) designing and conducting a social awareness strategy to inform the public of the benefits of skills training and reduce attached stigma; b) adopting a mechanism (the employment–linked training agreement (ETA)) to increase the number of trainees in key growth sectors by purchasing training services from private or public providers; c) designing and launching a targeted stipend program to remove financial obstacles for meritorious low–income students and workers; and d) expansion of market–driven middle–level technician training programs by setting up university colleges. For this pillar, the release of IDA funds was to be linked to the following DLIs: i) implementation of the ETA model (DLI8) and ii) improved use and dissemination of system information (DLI9). Component 2: Innovation, Results Monitoring, and Capacity Building (appraisal estimate US$7.9 million, actual disbursement was zero): This component was to strengthen the capacity of the MYASD and its participating agencies and help them achieve the SSDP objectives. This component was to provide financing for M&E, coordination, planning, communication, financial management, procurement, and safeguards. It was to also support the MYASD to design and realize policy reforms, such as piloting and evaluating innovative approaches and facilitating regular dialogue between the public and private sectors on skills. This component was to finance training, recruitment of short- and long-term technical experts, studies, surveys and evaluations, and procurement of goods for specific activities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$93.6 million. Actual cost was US$47.81 million due to implementation delays that resulted in the cancellation of US$43.4 million (differences in Dollar amounts are due to exchange rate changes). Financing: The project was part of the Skills Development Program (SDP), financed by a Bank credit in the amount of US$21.7 million (of which zero disbursed), a Bank credit in the amount of US$79.8 million (of which US$47.81 million disbursed), and planned financing in the amount of US$100 million by the Asian Development Bank (ADB), US$26 million by the Korean Government, and US$17 million by the German Technical Assistance Corporation, for which no actual financing data was provided in the ICR. Borrower Contribution: The Borrower was to contribute US$417 million. The Bank team (July 2, 2020) stated that the ICR was not able to obtain any actual financing data from the Borrower. Dates: The project was approved on June 20, 2014. On September 23, 2019, the project was restructured to: i) shift responsibilities among implementing agencies; ii) amend components and costs; iii) cancel US$43.3 million in undisbursed funds due to unlikely achievement of the PDO, following lengthy discussion between the Bank and the government that did not reach an agreement on project restructuring; iv) reallocate between disbursement categories; v) change institutional arrangements; and vi) change the implementation schedule. The project closed as scheduled on December 31, 2019. Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) 3. Relevance of Objectives Rationale According to the PAD (p. 1), in 2012, two years before project appraisal, Sri Lanka was a lower-middle- income country with a Gross Domestic Product (GDP) per capita of US$2,923. While the GDP was above the South Asia region as a whole, it was considerably below the average for middle-income countries. The economy was transitioning from a predominantly rural, agriculture-based economy to an urban economy centered on services and manufacturing. Sri Lanka had been making significant progress on its human development indicators. At the time of project appraisal, Sri Lanka’s workforce was the most educated in South Asia with about 96 percent of its citizens completing primary school and 87 percent completing secondary school. The primary net enrollment rate was 99 percent and the country had achieved gender parity in both primary and secondary education. However, the ICR (p. 5) stated that out of a cohort of approximately 450,000 students who finished the grade 11 terminal exam from the general education system, only approximately 20 percent were expected to obtain higher education and only a third of the remaining students were likely to pursue a TVET program. This resulted in only a small fraction of the labor force having any vocational education and training. According to the ICR (p. 6), the government tried to address the skills gaps and mismatches of the workforce through TVET. However, there were several challenges: i) low quality and relevance of skills development programs, not meeting the needs of the labor market; ii) low access to TVET, especially for women and economically disadvantaged youth; iii) limited public financing for skills development, resulting in underinvestment; and iv) centralized structures and procedures that negatively impacted demand- oriented skills development opportunities. In order to address these challenges, the government adopted the SSDP for the period 2014-2020. The project’s objective supported the SSDP. Also, according to the ICR (p. 12), the project’s objective was in line with the 2019 National Policy on Technical and Vocational Education, which aimed to: i) improve the quality and relevance of TVET programs, increasing employer and industry collaboration in TVET provision, and enhancing human resources in TVET; ii) improving access to training; and iii) fostering more efficient use of resources. Furthermore, the project was in line with Sri Lanka’s Vision 2025, which aims to drive economic transformation by increasing annual export income, attracting foreign direct investment, and creating employment opportunities for at least one million people by 2025. At the time of appraisal, the project’s objective was in line with the Bank’s Country Partnership Strategy (FY12-16), which aimed to aimed to support Sri Lanka in achieving its middle-income country agenda, especially through investments in education, skills, and research, and innovation to support structural shifts in the economy. At the time of project closure, the project’s objective was in line with the Bank’s most recent Country Partnership Framework (FY17-20), which aims to “strengthen education and training systems to deliver the skills needed for the advanced industrial and service sector activities of a globally competitive middle-income country" (p. 24). Rating Relevance TBL Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Expand the supply of skilled workers by increasing access to quality and labor market relevant training programs Rationale The project’s theory of change envisioned that project outputs such as spending recurrent and capital budget releases in accordance with an imprest plan, implementation of M&E plans, and production of analytical reports were to result in the intermediate outcome of improved governance and management of the skill sector. Also, the project’s theory of change envisioned that project outputs such as establishing Industry Sector Skills Council (ISSCs) for priority sectors, training registered providers, accrediting training programs, setting up Quality Management Systems (QMS) in training institutions, recruiting TVET teachers, and training TVET teachers and other staff were to result in the intermediate outcome of improving quality of training programs. These two intermediate outcomes were to result in the project’s objective of increasing the supply of skilled workers. Outputs to strengthen skills sector governance and management:  104 public training institutions developed and implemented Center Management Plans (CMPs) that were used to inform the government’s regular funding allocation to centers based on the needs identified. These plans were also an incentive for the centers to increase the number of students, utilization of resources, introduction of new courses, improvement of industry and community involvement in skills training, etc. The PBF model that was envisioned for the project included two aspects: first, the development and implementation of CMPs by each institution, and second, additional funds against several defined performance criteria following the implementation of CMPs. However, government policy was not amended as planned to support application of the PBF model. Therefore, the target of 80 public institutions being covered by PBF was not achieved (DLI3).  ISSCs in four economic priority areas such as information and communication technology (ICT), construction manufacturing, and tourism were established. The aim of the ISSCs was to foster industry expertise for sector skills gap analysis, development and updating of competency standards and training programs, teacher training, industry orientation and career fairs, and awareness building activities on TVET, in order to improve the market relevance of training programs and alignment of the skills of workers to job market demand.  A quality assurance system to register 1,290 public and private institutions under the Tertiary and Vocational Education Commission was established, not achieving the DLI5 target of 2,481 institutions. 2,691 courses were accredited, not achieving the target of 3,601 courses. Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698)  The number of training providers with NVQ levels 1 to 6 and with a QMS increased from 8 in 2013 to 234 in 2019, not achieving the target of 340 providers.  The percentage of capital releases spent in accordance with the imprest plan increased from 75 percent in 2013 to 100 percent in 2019, exceeding the target of 85 percent. The percentage of recurrent releases spent in accordance with the imprest plan remained steady at 98 percent, meeting the target.  In 2013 only two out of nine entities submitted audit reports in a timely manner. This increased to 39 entity audit reports of MYASD and participating agencies being submitted on a timely basis (by June 30th) in 2019, not achieving the target of 45 entities.  Four ISSCs or equivalent were functional in priority sectors by project closure, achieving the target of four ISSCs. According to the Bank team (July 2, 2020), the aim of the ISSCs was to provide industry expertise for sector skills gap analysis, development and updating of competency standards and training programs, etc.  Five institutions provided timely reliable data on institutional and sector performance, achieving the target of five institutions (DLI2). Data provided included information on enrollment, completion rates, and certifications by courses, centers, and teachers, as well as data on employment outcomes of TVET graduates.  The implementation of the MYASD annual budget was approved in line with the government’s SSDP and medium-term budgetary framework, achieving the target of DLI1.  The proposed restructuring included a new activity, the establishment of Centers of Excellence (CoEs). According to the ICR (p. 25), implementation of this new concept required substantial capacity building within the implementing agency. Also, it took a long time for the MYASD's Skills Development Division (SDD) and the Bank to agree on the activity's design, and several issues came up. First, there was disagreement on the autonomy of CoEs, which, according to the government, would have required a lengthy approval process including the Department of Management Services, the National Budgets, and other agencies. These consultations would have required an extension of the implementation period. Second, several actions that had to be implemented by the SDD in preparation for the restructuring, such as selecting the COEs, were not completed during the set time period. Third, after ministerial leadership changes, the Bank was asked to include the construction of housing facilities for school principals under the proposed restructuring. The Bank asked for further clarification on this request, which was not received. And fourth, the National Agency for Public Private Partnerships under the Ministry of Finance was to implement a sub-component on private sector training delivery. However, the Ministry of Skills did not agree. All these bottlenecks prevented agreement on pathways forward, resulting in the cancellation of remaining undisbursed financing. Outputs to improve the quality of teaching:  To improve teaching strength, a human resources (HR) policy and professional development plan was developed, followed by implementation of staff performance allowances in five implementing agencies under the new HR plan. These reforms were seen as important to attract and keep talent in the skills sector. However, the target for DLI6 - teacher recruitment (at least five major TVET entities having filled 90% of needed teacher positions) was not achieved.  832 teacher posts were sanctioned and 596 posts were filled between 2014 and 2019. Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698)  The teacher vacancy ratio decreased from 34.46 percent in 2013 to 33.40 percent in 2019, not achieving the target of 10 percent.  6,049 TVET teachers were trained in subject-based training, updated curricular activity delivery, leadership and soft skills, induction training for new staff, and diploma and postgraduate programs, surpassing the DLI7 target of 2,640 teachers. Also, 1,492 assessors were trained in ICT, assessment practices, leadership, and soft skills to improve the quality of assessments, surpassing the DLI7 target of 1,000 assessors. Outcomes:  The number of trainees enrolled in public and private training institutions increased from 148,131 trainees (of whom 81,047 were male and 67,084 were female) in 2012 to 940,848 trainees (of whom 538,781 were male and 402,067 were female) in 2018, surpassing the target of 827,241 trainees (452,609 male and 374,632 female). Of these trainees, 291,397 received competency-based training and certification (NVQ), exceeding the target of 180,000.  However, according to the ICR (p. 13), the provision of free TVET in public institutions in 2016 resulted in a surge in enrolled trainees, but many were not committed to completing their programs. The completion rate of trainees enrolled in public institutions decreased from 77 percent (74 percent male and 81 percent female) in 2012 to 67 percent (64 percent male and 72 percent female) in 2018, not achieving the target of 81 percent (78 percent male and 84 percent female). In addition, a "chronic shortage" of assessors resulted in a longer waiting period for assessments, which then led to some students completing their trainings and starting jobs even though they had not undertaken a final assessment (ICR, p. 13; the ICR did not discuss the reasons for the shortage of assessors even with the target for assessor training having been exceeded). Overall, while the number of students recruited to public TVET institutions rose as a result of the project, the number of students completing these programs, while remaining marginally above baseline for most years of the project, did not increase commensurately (ICR, Annex 11). While some of the targeted outcomes were achieved, completion rates declined, and the absolute number of program completers remained relative stagnant. Several outputs related to the improvement of quality of teaching and strengthening of governance and management were not achieved. Overall, the achievement of the first objective is rated Modest. Rating Modest OBJECTIVE 2 Objective Expand the supply of employable workers by increasing access to quality and labor market relevant training programs Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) Rationale The project’s theory of change envisioned that project outputs such as the ETA model being piloted, implemented, and scaled up in priority sectors, placing students in jobs through the ETA program, conducting awareness programs, and disseminating labor market information were to result in the intermediate project outcomes of enhanced labor market relevance of TVET programs and increased access to quality and relevant programs and improved social perception of TVET. Neither the PAD nor the ICR provided a definition of “employable." Outputs:  291,397 trainees (of whom 118,383 were female) received competency-based training and certification aligned with industry needs, surpassing the target of 180,000 trainees. Of these NVQ graduates, 7,247 trainees (of whom were 2,601 female) completed the highest two levels of accreditation (NVQ5-6).  An ETA model was piloted and scaled up to promote public-private partnership in skills training and encourage private providers to raise the quality of training and ensure employment of graduates through an incentive-based scheme, achieving DLI8 (implementing the ETA model). The ETA model was piloted in the construction sector in 2015, and in 2019 was expanded to four additional economic priority sectors: manufacturing, hospitality, ICT, and health care). A total of 5,092 students (of whom 2,601 were female) benefitted from the ETA program, surpassing the target of 3,500 students. The ICR (p. 19) stated that, according to the SSDP, the average employment rate of NVQ-certified ETA beneficiaries six months after completing the program was 69 percent in 2016 and 50 percent in 2017 across the supported sectors and occupations. The average employment rate for ETA graduates in high-demand sectors was higher in 2018: for construction, 95 percent, and health care, 88 percent. The ICR also stated that when the program expanded, the tracking of graduates became more challenging due to the lack of a Management Information System (MIS). Outcomes:  The average earnings of TVET graduates relative to earnings of General Certificate of Education (GCE)-O level graduates increased from a ratio of 1.08 in 2013 to 1.15 in 2018, surpassing the target of 1.12. The average earning of male TVET graduates relative to earnings of GCE-O level graduates increased from a ratio of 1 in 2013 to 1.18 in 2018, surpassing the target of 1.04.  In 2017, a Skills Demand Survey found that 43.2 percent of employers were satisfied with TVET graduates, not achieving the target of 55 percent. According to the ICR (p. 17), the baseline and midline used different questions to capture employer satisfaction. Therefore, these two data points do not provide a reliable comparative measure.  The ICR (p. 18) used National Labor Force Survey (LFS) rounds to calculate employment rates among youth with and without TVET. The employment rate among youth between the ages of 15 and 25 with TVET increased from 70 percent in 2014 to 74 percent in 2017. Among youth without TVET, there was no change in the average employment rate during the same time period. Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698)  Two rounds of an ADB tracer study of graduates from training institutes under SSDP support showed that the employment rate among TVET certificate graduates increased from 47.5 percent in 2011 to 54.5 percent in 2016, but then declined to 51 percent in 2019. The ICR (p. 18) stated that the main reason for the decline was an increase in the number of TVET graduates pursuing higher education (approximately 31 percent more in 2019 than reported in the 2016 tracer survey). According to the tracer study, TVET graduates who received a NVQ accreditation had a consistently higher employment rate than non-NVQ qualified graduates. Furthermore, using the national employment rate definition (employment rate = number employed within six months of course completion / total surveyed sample – voluntarily unemployed), the employment rate among TVET graduates increased from 56.2 percent in 2016 to 64.8 percent in 2019. For NVQ certified graduates, the rate increased from 57.8 percent in 2016 to 65 percent in 2019.  The average earnings of female TVET graduates relative to earnings of GCE-O level graduates decreased from a ratio of 1.14 in 2012 to a ratio of 1.04 in 2018. According to the ICR (p. 17), the reason for the decline in female TVET graduates’ income might be related to the programs they chose. The ICR stated that females might have chosen courses that did not respond to labor market demand and hence lowered the chances of employment and income increases compared to male TVET graduates. Although employment and earnings rates of TVET graduates increased, the ICR presented insufficient evidence of a direct correlation between the project's activities and the improved employability of workers. The selected indicators (“average earning of TVET graduates” and “employer satisfaction index”) to measure achievement of the objective were not adequate, as they were affected by exogenous factors; it is not clear whether gains were due to project-supported training or to other factors related to economic growth and the labor market. Furthermore, the indicator “employment rate of TVET graduates” was not measured by the project, as the baseline was set based on a Skills Toward Employment and Productivity (STEP) survey, and a planned follow-up round of this STEP survey was not conducted. Achievement of this objective is therefore rated Modest. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The achievement of both objectives is rated Modest. While enrollment in public TVET programs increased, the number of skilled graduates remained relatively stagnant due to program non-completion and student entry into the job market without completing assessment. Various measures show improved employment performance of TVET graduates, but there is inadequate information to demonstrate that this was due to project-supported training rather than general labor market trends. In addition, the ICR provided a useful Annex 7 with detailed information about contributions in the sector from other partners, but it did not include Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) an analysis of how this project fit into that landscape. Taking everything together, the project’s overall Efficacy rating is Modest. Overall Efficacy Rating Primary Reason Modest Insufficient evidence 5. Efficiency The PAD (p. 90) included an economic analysis of the project. The PAD identified the SSDP's expected benefits, to which the project was expected to contribute, as increasing the employability and earnings of TVET graduates by improving the quality and relevance of the skills they acquire. At the sectoral level, project- supported interventions were expected to: a) improve TVET system management and sector financing efficiency through organizational and managerial reforms and better M&E; b) enhance the quality and relevance of TVET programs by increasing employer involvement in program design and delivery; and c) expand private participation in the delivery of training programs by creating incentives and a more favorable environment. For society, project benefits were expected to include: a) a more productive labor force of workers with quality labor market-relevant skills; b) strengthened firm productivity and competitiveness through the more productive labor force; c) heightened performance of the economy because workers and firms are being more productive; and d) potential reduction of school-work transition times for TVET graduates, which could help to address Sri Lanka’s youth unemployment problem. The PAD calculated for the SSDP (2014-20) an Economic Internal Rate of Return (EIRR) of 3 percent and a Net Present Value (NPV) of US$ 40.1 million (using a discount rate of 12 percent). The ICR (p. 20) recalculated the NPV and EIRR at appraisal, as the PAD used 2012 STEP data, but labor market outcome indicators have since been updated using the annual LFS (2013 – 2017), which is not comparable to the STEP 2012 data. The recalculated NPV at appraisal was US$131.9 million and the EIRR was 5 percent. According to the ICR (p. 20), at project closure, based on the actual number of TVET graduates and the cost of the SSDP, the NPV was calculated at US$107.8 million due to lower benefits arising from fewer people completing TVET compared to the target. The IRR was estimated at 16 percent due to the reduction in project costs. The ICR conducted an additional cost-benefit analysis on the investments made through the government’s SSDP. The analysis was based on cohort-based-age-specific earning profiles between 2014 and 2019. The ICR (p. 20) stated that, since earnings increase with age, using average earnings differences between TVET and non-TVET graduates of all age groups from the beginning of their careers distorts the discounted value of the benefit at present. The analysis used target beneficiaries with base scenarios and actual beneficiaries with base and high-impact scenarios, resulting in a NPV of US$170.3 million and an EIRR of 10 percent. These analyses suggest that the project was a worthwhile investment. However, there were significant shortcomings in operational efficiency. The project experienced significant implementation delays, especially during the first year, due to national elections and resulting changes in ministries. Across the implementation period, frequent institutional changes, high staff turnover, and lack of continuous leadership as well as skills sector coordination and engagement issues all resulted in implementation delays. According to the ICR (p. 23), due to limited planning, communication, and coordination of technical Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) assistance between the Bank, the government and the ADB, the government was reluctant to use the technical assistance credit under component 2 because of a grant-based technical assistance that supported similar activities under an operation implemented by the ADB. Also, all DLIs were of equal weight independent from their significance for achieving policy reforms, resulting in the implementation of easier to achieve DLIs first. Furthermore, discussions between the Bank and the government regarding project restructuring lasted 16 months and did not result in an agreement. These challenges resulted in the cancellation of approximately US$43.4 million. Based on these significant shortcomings, the project’s efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objectives was High, given alignment with the Bank’s most recent Country Partnership Framework (FY17-20), which aims to “strengthen education and training systems to deliver the skills needed for the advanced industrial and service sector activities of globally competitive middle-income country." Efficacy was Modest. While enrollment in public TVET programs increased, the number of skilled graduates remained relatively stagnant due to program non-completion and student entry into the job market without completing assessment. Various measures show improved employment performance of TVET graduates, but there is inadequate information to demonstrate that this was due to project-supported training rather than general labor market trends. Efficiency was Modest due to operational inefficiencies such as significant implementation delays and the cancellation of US$43.4 million. Therefore, the project experienced significant shortcomings, and the overall Outcome rating is Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) 7. Risk to Development Outcome The potential risks to development outcome can be classified into the following broad categories: Political: According to the ICR (p. 31), the project’s objectives continue to remain relevant for economic development. Some of the project activities, such as quality assurance under TVEC and the expansion of the ETA program, have already been taken up as regular activities with a dedicated budget under the government program. Technical: The ICR (p. 21) stated that the project was able to improve budget planning and built management capacity at the ministerial and implementation levels. Also, TVET sector planning capacity was improved and the HR system of the MYASD strengthened by introducing an HR policy and professional development plan aiming to support need-based teacher recruitment, incentivize good performance to keep talented staff, and build teaching capacity in priority areas through relevant professional development opportunities. Financing: Changes to the economic environment or rapid technological advances will require continuous financing to developing TVET training that incorporates newly introduced technology. According to the ICR (p. 31), the new government is continuing to work with development partners such as the Bank and the ADB in these areas. 8. Assessment of Bank Performance a. Quality-at-Entry According to the PAD (p. 10), the project design took into account lessons learned from previous IDA- financed projects in Sri Lanka and skills operations in other countries financed by the Bank and the ADB. These lessons included: i) sustained leadership and commitment are critical in raising awareness of skills development and making it a priority in the country’s economic development plan; ii) coordination and monitoring are essential for reducing inefficiencies in the delivery of skills programs and meeting labor market skills demand; iii) a market-driven, responsive skills development system is critical; iv) policy interventions to incentivize training providers are essential for enhancing diversity and excellence in training and improving accountability for training results; v) adequate and predictable short- and medium- term financing is essential for rational planning; vi) a robust M&E system is critical for achieving results; vii) donor activities need to be coordinated. According to the ICR (p. 23), the Bank did not sufficiently coordinate with other development partners such as the ADB, which prepared a project to support the SSDP at the same time. While the ADB also used DLIs, they were significantly different for similar interventions and project results targets. Also, the ICR (p. 29) stated that the design of component 2 (technical assistance) was not sufficiently coordinated with analogous activities to be implemented by the ADB. Therefore, the government did not want to use component 2 to finance technical assistance, since the ADB supported similar activities. Implementation risks in the PAD (p. 13) were identified. The risk of limited implementation and absorptive capacity was rated as High. The risks of fragmentation of responsibilities between ministries and Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) agencies, lack of coordination (making accurate identification of development needs difficult), budget shortfall and unavailability of resources on a timely basis, and limited involvement of the private sector were rated as Substantial. However, mitigation strategies were inadequate. The project design was complex and included the involvement of nine implementing agencies and university colleges. Also, the project was to be implemented by a new ministry and implementing agency using the PBF mechanism. Furthermore, it was the first project for the Bank in the sector. The ICR (p. 23) stated that mitigation measures included DLIs on adequate and timely budget availability, fostering links with the private sector, and providing technical assistance to strengthen implementation capacity. However, according to the ICR (p. 29), institutional changes and frequent turnover in leadership had a negative impact on the adequacy of mitigation measures. It was not adequately anticipated that the government would not develop policy to permit full implementation of the PBF mechanism (see Section 4). The results framework was comprehensive but had several significant shortcomings (see Section 9a for more details). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision According to the ICR (p. 30), the project had four different Task Team Leaders over a five-year implementation period, which did not allow for continuous dialogue and follow-up with the project management unit and other stakeholders. Also, the Bank did not conduct regular supervision missions. The Region's comments (September 28, 2020) on the draft of this ICR Review stated that the Bank could not conduct regular supervision missions due to a highly volatile political environment. The comments further stated that due to frequent changes in counterparts, issues had to be re-debated and agreements reached several times over a period of 18 months. Also, supervision was hampered due to government requirements to hold off on supervision missions (between 2014 and 2016 only four supervision missions were conducted) until new authorities came on board, resulting in delays. According to the Bank team (July 2, 2020) a total of 11 supervision missions were conducted, but the ICR stressed that a single consultant in the field provided daily project support. Furthermore, the lack of supervision resulted in reporting delays of untracked PDO indicators, and lack of progress on actions decided during the 2017 mid-term review (MTR). The lack of regular supervision activities resulted in delays. The Bank team did not note until the MTR that three out of four PDO indicators and several intermediate outcome indicators were not being tracked. The ICR (p. 30) stated that until the MTR, the project’s focus was mainly on achieving the DLIs rather than the project’s objectives. Even though the government and other development partners adjusted program activities when bottlenecks had been identified, the Bank did not do so. According to the ICR (p. 30), the project would have benefited, in the case of DLI5 - quality assurance, from integrating the QMS activity into a rebranded Quality Improvement System. Also, since PBF could not be completely implemented, the project could have benefited from tracking CMP development and implementation in public training institutions (DLI3). Furthermore, the ICR Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) (p. 30) stated that, while the project’s Implementation Status Reports and Aides Memoire were generally candid and submitted on a timely basis, they lacked focus on results. Given several implementation challenges, the Bank and the government aimed to restructure the project. However, due to overly lengthy discussions that did not reach agreement and the project being in an Unsatisfactory status, it was decided to cancel the remaining funds instead. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s results framework was comprehensive and included four PDO indicators and 14 intermediate outcome indicators. The majority of the indicators had a baseline and all included a target. The project’s theory of change and linkages between key activities, outputs, and intended outcomes were sound and reflected in the results framework. While the first part of the objective (“increase the supply of skilled workers”) was clearly defined, the second part of the objective (“increase the supply of employable workers”) was not sufficiently defined. Therefore, it was not clear how “employable” should be measured. The selected PDO indicator “average earnings of TVET graduates relative to earnings of GCE-O level graduates" was not a good measure for this objective, since earnings of TVET graduates were influenced by external factors beyond the project’s scope. Also, the PDO indicator “employer satisfaction index” could not be directly linked to the objective. The third PDO indicator measured the completion rate for trainees in public institutions, but there was no measure of outcomes related to the project's efforts in raising the quality of private training providers. Disbursements up to a capped amount were to be made against specific line items in annual MYASD budgets. In addition, disbursement against these EEPs was conditioned on the achievement of specified results, as measured by the DLIs stated above. The SDD of the MYASD was responsible for the project’s M&E activities, which were to include periodic review of program and project progress, preparing and disseminating periodic progress review reports on monitoring indicators, preparing and communicating progress on DLIs, and preparing and disseminating Annual Performance Reports on the National SSDP and any other report requested by the MYASD. b. M&E Implementation According to the ICR (p. 11), the MTR in November 2017 found that out of the four PDO indicators, only the first PDO indicator (number of trainees enrolled in public and private training institutions) was on track to achieve its target. The second PDO indicator (completion rate of trainees enrolled in public institutions) Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) was significantly below target, and the third PDO indicator (average earnings of TVET graduates relative to earnings of GCE-O level graduates) and fourth PDO indicator (employer satisfaction index) were not being measured. The ICR (p. 25) stated that the project’s M&E shortcomings were due to the lack of a dedicated M&E expert within the SDD, poor understanding of the third and fourth PDO indicators, and gaps in data collection. Also, M&E arrangements were complex and required coordination with partnering agencies and training institutions. Furthermore, an MIS was not implemented as planned. The ICR (p. 25) stated that most of the intermediate outcome indicators were tracked on a regular basis based on administrative data and published statistics from national surveys. Also, according to the ICR (p. 27), some important studies such as a performance analysis report of four major TVET agencies (Ceylon German Technical Training Institute, Department of Technical Education and Training, National Apprentice and Industrial Training Authority, and Vocational Training Authority) and four skills gap analysis for ICT, construction, tourism, and light engineering sectors were conducted. c. M&E Utilization According to the ICR (p. 27), progress reports on project activities informed decision-making on follow- on activities. Also, the skills gap analysis provided information for training programs and other activities, which were implemented by four ISSCs for the ICT, construction, manufacturing, and tourism sectors. However, the ICR also stated that the tracking of intermediate outcome indicators was insufficient. Also, the lack of an integrated MIS and analysis resulted in a low utilization of M&E data for decision making such as adjusting activities to address declining completion rates of TVET students and low employer satisfaction. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as category B and triggered the Bank’s safeguard policy OP/BP 4.01 (Environmental Assessment) due to the financing of rehabilitation/renovation/refurbishment of existing training institutions. According to the ICR (p. 28), an environmental due diligence guideline was developed that focused on environmental standards for planned rehabilitation and construction in existing TVET institutions. The project also adopted the Environmental and Social Safeguard Management Framework developed by the ADB, which was accepted by the Bank as the environmental assessment. The ICR stated that the project management unit had an environmental safeguard expert to ensure compliance with the Bank’s safeguards. The project did not experience any significant safeguard-related issues. However, due to the slow handling of one complaint, the project’s safeguard rating was downgraded to Moderately Unsatisfactory in August 2019. However, the complaint was addressed appropriately, and the complainant received compensation for damages from the project. Overall, the project complied with OP/BP 4.01 (ICR, p. 28). Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) b. Fiduciary Compliance Procurement: According to the ICR (p. 29), the project’s procurement was rated Satisfactory throughout project implementation. The project did not implement any procurement activities under component 2. However, the project management unit's procurement staff was trained on preparing procurement plans and use of IDA’s electronic Systematic Tracking of Exchanges in Procurement system. Financial Management: According to the ICR (p. 28), the project’s Financial Management (FM) performance was rated Satisfactory throughout most of the implementation period. During two supervision missions it was downgraded to Moderately Satisfactory due to delays in external audit, weaknesses in financial reporting, and inadequate financial management staff. The ICR stated that the project management unit experienced FM staffing issues but addressed these issues by speeding up the process for identifying qualified FM staff. According to the ICR (p. 29), interim unaudited reports were submitted in a timely manner, and the quality of the annual consolidated audited financial statements was found satisfactory. However, the audit opinion was qualified in some of the consolidated annual financial statements. The Bank team (July 2, 2020) stated that this was due to delays in the submission of audit reports (in some instances due to elections, a terrorist attack, and changes in government personnel). c. Unintended impacts (Positive or Negative) None noted. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Moderately Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) The ICR (pp. 31-33) included several key lessons learned, adapted here by IEG:  Identifying a stable location for basing the project management unit so that it is not affected by ministerial and leadership changes has a positive impact on project implementation. In this project, the project management unit was moved among eight different ministries and was subject to continuous changes in leadership at various levels resulting in implementation delays and significant loss of institutional memory. Also, due to these changes, the dialogue with the government stopped, resulting in a two-year stalemate and the cancellation of IDA resources.  Tying appropriate incentives to targets to ensure timely achievement of DLIs is critical in a results-based operation. In this project, all DLIs were of equal weight independent of their significance for achieving policy reforms. Process-linked DLIs were achieved much more easily than output- or outcome-linked DLIs. Linking the output-outcome results to higher value amounts during the final years of implementation might have helped to place greater emphasis and incentives among different actors to achieve the desired results.  Coordinating and cooperating with other development partners positively affects strengthening project outcomes and avoiding duplications. This project and a project by the ADB were prepared at the same time. While both entities agreed to coordinate, there were significant differences in the DLI design and project result targets. This resulted in inefficiencies and ultimately a government not to use the Bank’s financing for the project's technical assistance component. Also, the government having to work with two different DLI designs might have added an extra administrative burden. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided an adequate overview of project preparation and implementation, complied with guidelines, and included an appropriate economic analysis. Also, the ICR was internally consistent and relatively concise, providing additional data where needed outside the results framework. Especially, the ICR included a useful analysis of DP contributions, and provided additional data to support the efficacy discussion. Furthermore, the ICR provided useful lessons learned that were sufficiently detailed to allow for a learning experience for other Bank projects. However, the explanation for the project's cancellation of funds did not become clear until relatively late in the document, and the impacts of frequent government changes on project implementation were not fully explained. Overall, the quality of the ICR was Substantial. a. Quality of ICR Rating Substantial Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Skills Development (P132698) Page 18 of 18