Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) Report Number: ICRR0021655 1. Project Data Project ID Project Name P095673 JM Early Childhood Develop Proj -- SWAP Country Practice Area(Lead) Jamaica Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-75540,IBRD-83340 30-Sep-2013 26,470,581.08 Bank Approval Date Closing Date (Actual) 13-May-2008 28-Sep-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 15,000,000.00 0.00 Revised Commitment 26,470,581.08 0.00 Actual 26,470,581.08 0.00 Prepared by Reviewed by ICR Review Coordinator Group Dale M. Hill Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Original Project Development Objective (PDO): ”The Project endorses the objectives of the National Strategic Plan (NSP). Its Development Objective is to: (i) improve the monitoring of children’s development, the screening of household-level risks affecting such development, and early intervention systems of the Borrower to promote such development; (ii) enhance the quality of early childhood institutions (ECIs); and (iii) strengthen Early Childhood organizations and institutions” (Loan Agreement (LA), Schedule 1). Page 1 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) Revised PDO: “The Project endorses the objectives of the NSP. Its development objectives are to: (i) improve parenting education and support programs; (ii) improve the monitoring of children’s development, the screening of household-level risks affecting such development, and early intervention systems of the Borrower to promote such development; (iii) enhance the quality of ECIs; and (iv) strengthen Early Childhood organizations and institutions” (LA, Schedule 1 and Fourth Restructuring Paper, January 10, 2014, page v). This Review will assess achievement of three objectives that were in effect across the project's entire lifetime: (i) improve the monitoring of children’s development, the screening of household-level risks affecting such development, and early intervention systems; (ii) enhance the quality of ECIs; and (iii) strengthen Early Childhood organizations and institutions. It will also assess achievement of one objective that was added at the January 2014 restructuring and Additional Financing (AF): improve parenting education and support programs. Because the objective on parenting support--added in 2014--was already implicitly a part of the NSP supported by the project (and the original 4 PDO indicators included one on parenting education), it is considered to have been in effect across the project's entire lifetime, and therefore a split rating is not performed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 10-Jan-2014 c. Will a split evaluation be undertaken? No d. Components The original project contained two components. Part 1: Supporting the implementation of the early childhood development (ECD) NSP (2008- 2013) (appraisal: USD 14.1 million; AF: USD 11 million; actual: USD 24.65 million). Part 1 was designed as a Sector Wide Approach (SWAp) that would reimburse Government budget lines when disbursement linked targets (DLTs) were met. The budget lines were: grants to community schools and to the Early Childhood Commission (ECC, under the Ministry of Education, Youth and Information (MoEYI)). Various agencies were responsible for meeting the DLTs, but the largest number were under the responsibility of the ECC. The areas the NSP supported included:  Parenting Education and Support (targeted ECC institutional development; sub-strategy development and implementation; ECD strategy; public awareness campaigns; accreditation system; and grants to help service providers meet accreditation standards). Page 2 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673)  Preventive Health Care for Children (reorganization and strengthening of well-child clinics, including an accreditation system; transformation and scaling up of nutritional programs; and introduction of child health passports to clinics).  Screening, Diagnosis and Early Intervention (national policy for screening, referral, and early intervention; application of screening to households enrolled in the Borrower’s income support program (Program of Advancement through Health and Education, PATH); and training of child development therapists who would work with children with disabilities.  Safe, Learner-Centered Early Childhood Facilities (strengthening inspection system for ECIs at ECC; recruitment of development officers to technically support ECIs; and training of ECI caregivers).  Trained Early Childhood Practitioners (human resources (HR) strategy; improved ECD curriculum; increased number of trained practitioners and improvement of their skills; and development of continuing education and licensing systems).  ECD Sector Governance (reorganization and staffing of the ECC; national ECD policy including local governance; ECI management models; and identification of specific budget lines of the Borrower’s national budget for monitoring ECD expenditures).  Evidence-Based Decision Making (strengthening ECD management and information system (MIS) and ECD statistics, including coverage of the HR situation and an instrument to monitor early child status at age four). Restructurings: Support for these NSP strategy areas did not substantially change with the AF. The revised PDO added explicitly a new objective of parenting support (which had been part of the NSP all along). The AF allowed for implementation of parts of the original ECD NSP (2008–2013) that had been delayed, and commencement of support for the 2013-2018 ECD NSP (developed with technical assistance (TA) from the first phase of the project, hereafter called “parent project”). Part 2. Technical Assistance, much of which was to directly support the strategy areas above, through the following activities (appraisal: USD 0.86 million; AF: USD 0.97 million; actual: USD 1.67 million).  Development and support of implementation of a national policy on screening, referral, and early intervention, including: design of risk screening systems; public education strategy; service delivery models for nutrition; strategy for well-child clinics; mapping of existing parenting programs; and development of accreditation systems for parenting support institutions.  Redesign of the Borrower’s grant facility supporting ECIs, after an analysis of needs and gaps in service provision by ECIs.  Development of a comprehensive ECD HR strategy, including compensation and training.  Studies to: strengthen the legal framework governing ECD; develop national ECD policy; and elaborate proposals for local governance and ECI management models.  Monitoring and evaluation (M&E) support: evaluation of the NSP at mid-term and near project-end; development of a national ECD M&E system; and development of an instrument to measure child development status at age 4.  Development of the Borrower’s ECD NSP for the period 2013-2018. Restructurings: TA support was restructured periodically as new needs were identified. In the first restructuring dated June 15, 2011 (not level 1), the changes consisted of more training, support for ECI inspections, and the purchase of laptops and software for education officers. The fourth restructuring, which brought the AF, reallocated funds in favor of the TA component, since some contracts had been Page 3 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) financed in-house or by donors, allowing new activities such as: development of an ECI management guide; training of stakeholders on parenting education and support strategy; development of nutrition guidance and training of ECI cooks and principals on the new Nutrition Strategy; temporary hiring of fire, health, and/or police inspectors to carry out ECI inspections; and purchase of computers and software to support Development Officers in their field work, which fed into ECI inspections. In other restructurings, minor changes in the TA were agreed and approved through the Bank’s periodic review of the procurement plan. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: For the parent project, total project cost was estimated at USD 495.75 million. Actual total costs were not reported in the ICR. The project team later provided the estimated total costs at the AF as USD 522.87 million; and actual total costs as USD 509.42 million. Financing: The project was financed as a Specific Investment Loan that used a SWAp disbursing against DLTs for Part 1 and standard financing modalities for Part 2 (PAD, para. 12). Of the original total costs given above, the IBRD loan was to finance USD 15 million for the parent project, and USD 12 million through the AF. At project end, of the total planned IBRD financing of USD 27 million, the loan had disbursed USD 24.65 million for Part 1 and USD 1.67 million for Part 2, for total actual IBRD financing of USD 26.32 million. Borrower Contribution: The Borrower contribution was originally planned at USD 493.87 million (97% of total project costs), and USD 495.87 million (94% of total costs) after the AF. The actual contribution was USD 484.28 million (information provided later by the Project Team). The 2008 global financial crisis reportedly led to some shortages of counterpart funds that restricted some needed staff increases. Other fiscal constraints also arose from time to time during the project. Dates: The original/parent project was approved May 13, 2008 and became effective February 9, 2009. The mid-term review was conducted as planned in May 2010. There were five restructurings.  The first restructuring (June 15, 2011) changed disbursement arrangements against DLTs for Part 1 to alleviate bottlenecks, amended the Part II TA as noted above, and changed the Results Framework (RF) indicators and targets to better align with the pace of implementation.  The second restructuring (February 27, 2013) further changed the RF, reducing the number of indicators from 71 to 67, prioritizing those most closely supporting the PDO.  The third restructuring (September 10, 2013) extended the closing date to September 30, 2013, to allow for preparation of the AF.  The fourth restructuring (Level 1, January 10, 2014) added the AF of USD 12 million and revised the PDO to explicitly add the improvement of parenting education and support as an objective. The approved AF project closing date was September 28, 2018. Both PDO and intermediate results indicators (IRIs) were revised to accord with the new ECD NSP for 2013-2018.  The fifth restructuring (June, 2018) followed Government reprioritization of activities that considered some sub-components as redundant and changed the RF indicators further (ICR, paras. 11–16). The project closed on time, on September 28, 2018. Page 4 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) 3. Relevance of Objectives Rationale The project objectives were substantially relevant to both Government and IBRD strategy at appraisal and remained so at closing. The Government showed its support for ECD by establishing the ECC by an act of Parliament in 2003 to oversee and coordinate different agencies and sectors involved in ECD (e.g. education, health, labor, national training institute). The project aligned well with the Government’s development strategy for 2004-2007, in that it directly supported the results-based NSPs developed by the ECC to improve the system performance of the ECIs. (The parent project supported the plan for 2008-2012; the AF support the updated plan for 2013-2018.) More recently, the Country Partnership Strategy (CPS) for 2014-2017 included references to “building human capital, starting from early childhood, [to provide] the foundations for a more resilient, stronger, and more prosperous society“ (page 20). Likewise, the project supported the CPS's outcome 6, “strengthened social protection programs and improved institutional capacity for their management” and outcome 7, “increased opportunities for the poor and vulnerable.” The 2017 CPS Performance and Learning Review extended the Framework's validity for two more years (ICR para. 20). An area of misalignment, however, was that the objective was pitched at a level that did not fully match the development status and capacity in the country, as described in the CPS (regardless of the Government's commitment to the goals), especially given: a) ECC’s status as a relatively new agency; b) plans for the ECC to take on more functions during the life of the project; c) ECC's sometimes limited authority to obtain commitment from other agencies to fulfill their DLTs; and d) the overall complexity of the project. A less complex alternative to supporting the whole NSP (with its seven areas of emphasis, and the project's multiple objectives and 71 indicators) was presented in the PAD, but was not chosen due to "need" and a reference to compensating capacity in other agencies. (See Section 8a on Quality at Entry). The ICR review manual calls for two criteria other than alignment to be considered in relevance: country context, including capacity, alluded to above, and previous sector experience. For a first project in the ECD sector, the objectives set were overambitious given this context and experience. Relevance of objectives is therefore rated substantial rather than high because the objectives of the project were pitched at a level that would have required a higher level of capacity among government institutions than existed. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 5 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) OBJECTIVE 1 Objective Improve the monitoring of children’s development, the screening of household-level risks affecting such development, and early intervention systems of the Borrower to promote such development Rationale This ICR Review relies on the theory of change implicit in the NSP rather than the theory of change presented in the ICR, as the NSP's theory of change is more comprehensive and causal relationships are clearer. For this objective, desired outcomes were plausibly to be achieved through (a) development of a policy and strategy covering child screening, monitoring, and early intervention; (b) development and implementation of specific screening tools; (c) adjustments to interventions with at-risk children based on data collected through those screenings; and (d) public information campaigns. Four types of activities/institutions were to be involved in the screening: ECIs (private sector, aided by grants to undertake “improvement plans”), health centers with well-child clinics (not known if eligible for grants); the Ministry of Labor and Social Security (MLSS)’s at-risk child intervention systems, and the PATH income support program. The envisioned outcome was more children at the household level screened for risks to health and delayed development. Note: all baselines were zero unless otherwise stated. Outputs Foundational elements achieved: a) development of an ECD policy that includes screening, referral, and early intervention approaches; b) a public education strategy on risks for children; and c) an Infant and Young Child Feeding Policy approved by the Cabinet (covering both under and over-nutrition). Screening models developed and implemented: development and approval by the Ministry of Health (MoH), MLSS, and ECC of screening and documentation models for high-risk households:  A child health development passport (CHDP) was developed under the project, approved in 2009 and rolled out nationally in September 2010 to replace immunization records. The CHDP added information on the child’s education and family health, plus information to make the parent aware of common risks. Since 2013, CHDPs have been issued to 98% of babies born, exceeding the original target of 50%. All health centers now use the CHDP as a risk screening device. Its use facilitated achievement of 92% national coverage for the third dose of diphtheria-tetanus-pertussis vaccine vs. a baseline of 85% in 2008. In comments on the draft of this ICR Review, the GP noted that "the CHDP captures information that enables parents, health care workers and teachers to better monitor children's growth and development and to take early corrective action as needed. Interviews with health care workers and early childhood education teachers and principals at the various sites visited confirmed that the CHDPs were being used appropriately at clinics and schools, and that although completing them increased staff reporting duties, parents appreciated the CHDPs." (Evidence to support this note was not provided in the ICR or in the GP comments.)  A simple first-line screening tool (“Eleven Question Screen”) for developmental delays was developed by the MLSS in 2012 and improved in 2013. Using this tool (part of the CHDP), the number of children aged 3-6 annually accessing MLSS programs reached 763 against a revised target of 550. The cumulative number of children accessing the programs over the life of the project was 5,068. As context for interpreting this result, it is important to note that the total number of children attending ECIs with permits was almost 92,250 and that the ICR does not indicate what percentage of children Page 6 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) with a need for intervention were identified or reached by this at-risk screening process; thus, an important caveat is that this indicator may not provide an accurate basis on which to judge achievement of the objective.  The "Eleven Question Screen" was used to identify high-risk children among households with children age 0 – 6 that accessed the PATH cash transfer program. The number identified with high risk was 764, vs. a 2013 target of 300 and a revised 2014 target of 470. The same caveat on judging this achievement relative to need applies as that cited in the point above. Monitoring data used and reported: The ICR indicates there was complete transfer of information from the CHDPs to the MOH’s MIS. In addition, the ECC Annual Review Publication reported comprehensively on performance of the early childhood sector and ECD outcomes, including school readiness, nutrition status, and growth ratios. Planned outputs (noted in the PAD) on which information was lacking in the ICR: a) DLTs related to “a national service delivery model for screening diagnosis and early intervention for child development risk” as applied to health centers were deleted; the GPs comments on the draft of this ICR review clarified that this was due to a change in Government priorities given other project activities; and b) information was lacking on the number/percentage of PATH workers trained to use the screening tool. Outcomes Because of the development and use of improved screening tools, behavioral and institutional change took place, resulting in improved ECD systems for screening, identifying at-risk children, and referring those children for intervention. Targets for children reached were met or exceeded, as noted above. Rating High OBJECTIVE 2 Objective Enhance the quality of early childhood institutions Rationale For this objective, desired outcomes were plausibly to be achieved through: (a) enhancing the inspection system; and (b) recruiting development officers and child development therapists to provide technical support for ECIs. These activities were to lead to improved quality of facilities. Outputs Foundational elements for improving ECI facilities and staffing: a) standards and inspection system developed, with increased TA support for the latter in the first restructuring; b) a comprehensive HR strategy Page 7 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) developed, including compensation and training (extent of training increased in second restructuring); c) ECI management guide developed; d) parish-level community development model for delivery of ECD services developed; and e) nutrition policy approved by the Cabinet, and a service delivery model for nutrition support for children aged 4-6 approved by MoH. Accreditation system for ECIs approved by the relevant authorities (MoH, ECC): In application, the standards initially set for accreditation were found to be too high, and in subsequent restructurings, lowered thresholds for quality were used: for ECIs, the indicator was changed to “issuing permits“; and for health centers and parent support organizations (covered under the third objective), “certification." For ECIs, permits to operate were issued when no significant health and safety breaches were identified; areas for improvement were listed as part of the permit process. The decisions to lower standards were justified after piloting revealed the difficulty in meeting the standards, and there were concerns about cost-effectiveness and ability to reach as broad a group of children and parents as possible. The discovery of inability to meet standards came quite late for the MoH clinics, causing them to miss targets while piloting revised standards. Later, the ECC established criteria for ECI “certification” (a standard above issuing permits) that required ECIs to meet all 12 of the ECC's operating standards. After inspections, the average quality of ECIs improved:  At least 75% of ECIs received one initial inspection, and 40% a complete inspection, by an ECC- trained inspector.  These inspections resulted in closure of about 500 undocumented ECIs deemed to be unsafe for children. 1977 out of 2626, or 75%, passed inspection and were issued permits, exceeding targets.  129 ECIs met the higher standards for “certification." A DLT calling for information on these certified ECIs to be accessible to the public was achieved.  1,461 out of 2,626, or 52%, implemented their improvement plans, exceeding the original target of 45%. Presumably these plans were financed by the revamped grant system; no information was given in the ICR on the criteria for making grants, or the content, quality, or distribution of grants awarded. Examples of improvement plans were not provided in the ICR. Planned outputs that were only partially achieved, or for which targets were revised downward or information was not provided in the ICR:  Health centers offering well child clinics: A strategy for well-child clinics was to have been developed; the ICR does not state whether this was achieved. The percent of health centers “assessed” using the well-child assessment system was 12%, against an original target of 5%. The assessment, however, did not result in many certifications. Only 6 out of 317 centers offering well- child clinics were certified (an original target of 30%, later revised to 10%). The program was started late, and there were delays deciding institutional arrangements (information from Project Team). The MoH stated that more time was needed to train the child health teams and provide printed protocols and guidelines for the centers (ICR, Annex 5). There is no information in the ICR on whether the health centers were eligible for grants for improvement plans (as the ECIs were) and if any took advantage of such programs. Nonetheless, the foundations have been set and next steps laid out, and MoH can benefit from the experience with certification of ECIs and parenting support programs (the latter covered under Objective 3). Page 8 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673)  Piloting of nutrition support: A nutrition support strategy and service delivery model for children aged 0–3 was developed, and the fourth restructuring document (para 8) stated that the next step was piloting. The pilot of the service delivery model however, was dropped, however, due to multiple/duplicative nutrition initiatives taking place in Jamaica at the same time. One example was the Jamaica Brain Builders Program, an initiative launched by the ECC in July 2018 as a framework for action in the first 1,000 days of a child's life, including nutrition interventions. Outcomes The number of ECIs delivering the standard curriculum effectively (based on COT scores) was 1,000 out of 2,626 (36.5%, exceeding the original target of 25%). Based on the shortcomings related to well child clinics, achievement of this objective is rated modest, in agreement with the ICR. Rating Modest OBJECTIVE 3 Objective Strengthen early childhood organizations and institutions Rationale The theory of change for this objective plausibly held that strategy, governance, and data improvement outputs, as well as training and incentivizing ECI caregivers through improved continuing education and a licensing system, would strengthen early childhood organizations and institutions at the macro level. Outputs Foundational elements on overall policy, ECC strengthening, strategy development, and monitoring: a) an ECD Policy including local governance approved by 2018, a 2014 target; b) a resource mobilization strategy approved (including from the private sector); c) reorganization of ECC with increased staffing, and successful transfer of functions from MoEYI to help assure sustainability; d) a spatial analysis of ECIs conducted by ECC; e) ECD MIS and statistics strengthened, through data collection included in the Survey of Living Conditions, and two rounds of the age 4 School Readiness Assessment; f) 2013-2018 NSP prepared, and preparation of 2019-2023 financed (the ICR did not note if the 2019--2023 strategy was completed); g) performance of the ECD sector and the NSP for 2013-2018 reported in the Government's annual review publications, and ECD expenditures reported by the Ministry of Finance and Public Service; and h) improved financial and accountability arrangements for grant support to ECIs approved, and subsidy manual produced. Curriculum improvement, continuing education programs, and licensing developed: The National Training Agency helped the project develop advanced ECD vocational qualification standards (Level III and IV) and a licensing system for ECD practitioners. The ECC Board approved a curriculum for child development therapists, a new cadre of workers whose role was to address vision, hearing, speech Page 9 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) impairments, and behavioral development challenges. An online course was developed for ECD teachers and students in the pipeline (a 2014 target). Outcomes  The percentage of ECD practitioners at Level 1 proficiency was lowered from 60% in 2007 to 6% by project end. The number qualified at Level III and above was 5,158 out of 11,381 (78%, against a 53% original target and a baseline of 7%).  A classroom observation tool (COT) was developed for use by ECD development officers to assess quality of ECI teaching. While the ICR referred to only 16 development officers having been trained, 100% of them are using the COT tool.  The number of parishes that have at least one full-time child development therapist on staff in the public sector was 7, achieving the target of 7. All used the COT tool.  90% of ECIs have at least one early childhood practitioner trained in seven areas of ECI safety. Community-based integrated ECD service delivery model implemented at parish level: Six parishes (target met) implemented the model. Implementation of the model also involved piloting an ECD Community Council to coordinate service delivery for children aged 0 to 8. Based on overachievement of targets for proficiency of practitioners and meeting of all other targets, achievement of this objective is rated High. Rating High OBJECTIVE 4 Objective Improve parenting education and support programs (added at 2014 Additional Financing; was implicit in National Strategic Plan) Rationale Parenting education and support programs are called “parent places” in Jamaica. Project documents did not make clear whether they are a subset of ECIs or fall in a separate category. The ICR’s examples of services provided by parent places included helping parents teach nutrition to young children and discussing parents’ challenges with discipline. The NSP theory of change called for this objective to be achieved by: development and implementation of a sub-strategy for these programs; an accreditation system; grants to help providers meet accreditation standards; and a public awareness campaign. (Note: all baselines are zero unless otherwise given.) Page 10 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) Outputs Foundations: A national parenting strategy, parenting standards, and an “accreditation system” for early childhood parenting education and support programs were approved in 2010. The National Parenting Support Commission was established in 2013 to support implementation. As in the case of ECIs and health centers, a pilot showed that a lowered accreditation standard of “certification” was more practical and cost-effective than full accreditation, and could reach more children. Certification covered the physical environment, program design, program administration, human resources, program materials, and program M&E. (Not all of the ECC's 12 operating standards for ECIs were on this list, e.g. child rights.) Parent places identified: The Borrower’s Annual Review publication included identification and mapping by location of early childhood parenting programs. The resulting total count was not provided in the ICR. Assessment and certification of parent places:  The percentage assessed was 66% against a revised (lowered) target of 60%. It is not known if any facilities were closed due to unsafe conditions (as with the ECIs). The percentage certified was 48 of 92 (52%, against an original target of 30%, revised downward to 20% in 2011, raised again to 60% at the 2014 restructuring, which added the parenting support objective explicitly, and revised again downward to 45% in 2018).  A Communication Strategy (2014 target) was developed and implemented. It covered early childhood stimulation, parenting, and nutrition practices, and used media campaigns, public service announcements, and newspaper articles.  The ICR provided no information on grants to help parent places achieve accreditation or certification. Outcomes The ICR reports that project-financed improvements resulted in circulation of parenting information. The percentage of parents/guardians of children ages 0-6 who have ever received any information on parenting (excluding that received from family members and friends) reached 45% by 2011; after 2011, this indicator was dropped. No information on survey methods was provided, and therefore the quality of this evidence cannot be assessed. Rating Substantial OVERALL EFF TBL OLD Rationale Objectives 1 and 3 were achieved at a High level. Objective 4 was Substantially achieved. Objective 2 was Modestly achieved. Overall Efficacy is therefore rated Substantial. Page 11 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) Overall Efficacy Rating Substantial 5. Efficiency Economic Analysis at Appraisal The objectives of the analysis were to estimate internal rate of return (IRR) of the investments under NSP; assess the distribution of access to ECIs according to socio-economic characteristics; and assess the impact of NSP investments on recurrent costs after project closing (PAD, para 56). The private IRR to the NSP investment plan was calculated by estimating the effect of NSP interventions on educational outcome variables using research evidence. Benefits assumed some decreased rates of dropout and repetition, as well as the effect of changes in educational outcome variables on wages using current income data. It was assumed that benefits would extend beyond the project period (at that time 2013), but with some depreciation over time – assumed to be 20% after two years, and for each of the four following years an additional 20% depreciation from the original investments. A sensitivity analysis of scenarios decreasing the depreciation assumptions to 15% changed the IRR estimate only by 0.1%. The projected private IRR varied between 9.0%, 8.5% and 7.5% under high, middle, and low scenarios for changes in assumed educational attainment of beneficiaries. The societal IRR would likely be higher (PAD, para 187). Access to ECIs for 4-6 year olds was found to be equitable; enrollment rates had increased for the lowest income quintiles from 1996 to 2004, resulting in rates above 93% in all by 2006 (PAD, para 59). This analysis justified the NSP's focus on improving quality. The implementation of the NSP was estimated to cost USD 495.74 million, out of which USD 83.76 were incremental; parent fees were not included. Most (90%) incremental costs were recurrent costs, which were projected to increase by 27.9% during project implementation. The IRR estimate included all costs, both non- incremental and incremental, and the Bank loan financed a portion of both (PAD, para 17). Ex-Post Economic Analysis The ICR's approach in calculating the IRR was “as close as possible” to that used for appraisal (ICR, para. 6). The ICR stated that the “resulting IRR is higher than at project appraisal, possibly due to scale up of activities,” but such an argument would need to show that scale/benefits/beneficiary numbers increased at a higher rate than cost, especially since the ex-post IRR only used data from 2013–2018 (see below). The ICR's method of IRR analysis differed from the ex-ante analysis in several key ways:  Possible improvements in education efficiency, e.g. a reduction in dropout and repetition rates, were not included (information provided by Project Team).  The IRR was calculated using data from 2013–2018, which corresponds to the second half of the project. The reason given was that the MoEYI provided detailed information on participants of each intervention during those years (ICR, para 34), allowing a comprehensive calculation of benefits (Project Team). IEG considers it likely that this methodology would introduce a positive bias to the IRR. Considering implementation delays at the beginning of the project (see below), it is likely that the benefit/cost ratio during the first phase was lower than the second. (The Project Team confirmed that it would require additional data and time to test that hypothesis.) Page 12 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673)  The ICR did not provide information on cost analysis, but the Project Team confirmed that total costs were used. The ICR did not, however, repeat the PAD’s analysis of the impact of the project on the Government's recurrent costs, a factor important for sustainability. The IRR was estimated at 10.86% with a benefit-cost ratio of 2.32. The calculations assumed a work life of 38 years starting at age 22 and a discount rate of 8% and used 2017 gross domestic product per capita as a proxy for average wage. The employment rate assumed was 54.3%. At end-project, the impact was assumed to last for five more years, reported to be a conservative assumption according to the literature. As in the ex-ante analysis, benefits having to do with improved parenting, screening and early intervention, health, and strengthened institutions were not captured by the calculations. “Different tiers of impact” were assumed according to how many components children participated in and the duration they were exposed to the program. No account was taken of health interventions, since data was from MoEYI. The benefit-cost ratio stayed positive even when the discount ratio was varied from a low of 5% to a high of 10%. The sensitivity analysis also found that even after lowering the assumed impact to a level below that recorded in impact evaluation studies and the literature, the IRR range and benefit/cost ratio remained positive (ICR, Annex 4). Implementation Efficiency The initial project was planned for five years. At restructuring, additional financing was added, along with an additional five years. The ICR does not present detailed information to underlie an assessment of whether the scaleup in results/benefits made possible by the additional financing was proportional to the amount of additional financing. The ICR and project documents describe early delays due to: a) inexperience of the ECC, staffing constraints, and inexperience with international procurement procedures (ICR, paras. 35, 45); b) flaws in the reimbursement/DLT synchronization (later fixed at the first restructuring) (First Restructuring Paper, para. 24), and c) limited authority of ECC to mobilize all partner agencies to action (ICR, para. 47). The 2008 financial crisis apparently also limited counterpart funds, causing staffing constraints. Detailed analysis of delays was difficult due to the complexity inherent in the presence of 71 DLTs and indicators, many of which changed with each restructuring. Also, the ICR contained scant information on the timing of DLT achievements. Later delays resulted from a transfer of responsibilities from MoEYI to ECC, and the difficulty of finding the right consultants in certain technical areas (ICR, para 47). Exogenous factors (epidemics, regional aid responsibilities) were also cited as contributing to delays. Because the project had many interdependent parts, delays in some aspects caused ripple effects on progress in others, resulting in inefficiency; this point was made both in the ICR and in the Government’s end-project review (ICR, Annex 5, pp. 57-58), although the project documents provide no critical path framework to demonstrate this definitively. The ICR lacks full information on which DLTs and indicators were dropped because they were achieved, and which were dropped for other reasons--for example, because there were better indicators, or because they were not realistic. The ICR gave little perspective on how NSP fared in implementation progress compared to other education sector quality assurance efforts. An example of a Mauritius project was shared in response to the circulation of the draft ICRR. No unit costs were reported. Project efficiency is rated modest, based on: (a) multiple delays (without full quantitative evidence on them, e.g. DLTs achieved on time or not); (b) limitation in the ICR's IRR analysis which arose from unavailability of relevant Page 13 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) data for the first half of the project; and (c) lack of full data on scaleup. Taken together, the assessment of efficiency suggested that the implementation of the project was likely not least cost. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 63.00 Appraisal  7.50  Not Applicable 54.00 ICR Estimate  10.90  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance is rated Substantial, as the project's objective was pitched at a level that was not well aligned with development status and capacity in the country. Achievement of objectives is rated Substantial, based on High achievement of two objectives, Substantial achievement of one objective, and Modest achievement of one objective. Efficiency is rated modest due to evidence of delays, lack of full data on scaleup, and shortcomings in the IRR calculation (resulting from unavailability of relevant data for the first half of the project). Overall, these factors are indicative of moderate shortcomings in the project's preparation and implementation, consistent with an Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Most of the risks rightly identified at appraisal, with regard to fiduciary aspects and administration of grant funds, show signs of having been resolved, although the ICR did not treat the latter in much detail. The ECC’s capacity and authority has been strengthened and a Steering Committee formed to further improve coordination between sectors. The Government’s commitment remains high, as evidenced by the third installment of the NSP covering 2018-2023. The foundations for both standards setting and the inspections systems underpinning its application are largely in place. The one subsector that lagged behind targets – well-child clinics -- is now benefitting from a piloting of revised standards that have proven attainable in some Page 14 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) clinics, boding well for a continuation of momentum. While only 2% of the clinics have been certified, 12% had been assessed by project end. The main risk for sustainability is financial constraint. The Government funded the bulk of the expenditures of this project, but the project did increase recurrent costs, and, at various times during implementation, fiscal strains constrained staffing and other key implementation steps. To mitigate this risk, the project financed a resource mobilization strategy that included the possibility of raising funds from the private sector. The NSP successfully raised some funds from foundations, and official donors other than the Bank continue to show interest in supporting the sector as well. 8. Assessment of Bank Performance a. Quality-at-Entry Overall project design (parent project stage) drew on key elements of the Bank’s Country Assistance Strategy of 2005, particularly pillar 11: improving human development and opportunity, and also supported other complementary projects (ICR, paras. 9, 85). While Jamaica had achieved both high enrollment rates for ECD programs and high vaccination rates (PAD, paras. 43, 66), the quality and standards applied in the mostly private-sector ECD centers had been uneven, and there was a need to add parenting education. Thus, the project’s focus on quality assurance (QA) was well-founded. Detailed project design was based on sound sector analysis and a literature review including more than fifty studies on the impact of early childhood development on later productivity and earnings (and possible other societal benefits such as reducing crime) (PAD, para. 62). Preparation was led by a multi-sector team of high-level Jamaican ECD experts with deep knowledge and experience in the sector (ICR, para. 42). The relevance to the Government's development objectives was clearly established. The design of the ECD NSP had been collaborative and was comprehensive, including all aspects potentially affecting ECD, reflecting international best practice. Regionally, Jamaica's approach to the sector has been appreciated as innovative, as other countries seek to emulate the approach in the NSP. The project's anchoring in the NSP, bolstered by previous supportive legal actions, meant that its strategic directions were endorsed at the highest levels, ensuring strong client ownership. There were prior assessments of financial and procurement capacity, as well as the operation of the grants system (for ECIs), and the PAD drew on these effectively to design conditions of negotiations, effectiveness, and disbursement. The risk assessment and design of mitigation measures were sound. The economic and financial analysis at appraisal was well done, as it went beyond the usual IRR and benefit-cost analysis (as covered in Section 5). The project's design, a SWAp with DLTs to be fulfilled, was meant to “mimic an incentive framework similar to performance-based budgeting," which the Government sought to pilot (PAD, para 117). The inclusion of design of a resource mobilization strategy as a component was an added plus. The social safeguard study was particularly participatory and thorough (see Section 10a), confirming the focus on quality assurance. The emphasis on multiple indicators of quality of the ECI services in the results framework and provision for support of a survey to measure ECD outcomes was laudable (see Section 9). However, there were some significant shortcomings. Page 15 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673)  The project was overly complex given the ECC's status as a relatively new agency. As designed, the project supported seven areas of emphasis of the NSP, with multiple compound objectives, 71 indicators, and DLTs that were under the responsibility of seven different agencies. While the PAD addressed alternatives considered (para. 36), one of which was “a phased approach … whereby phase I focused on providing technical assistance to the ECC, and phase II focused on the implementation of the NSP,” alternatives were rejected based on “urgent need” and an assessment of other Government capacity that proved to be an overestimate. The ICR did not indicate whether the project team considered a SWAp design that was less comprehensive, supporting only a part of the Government's ECD NSP in a first phase.  The thresholds for standards were overly optimistic. In all three types of institutions targeted for improved quality (parenting centers, ECIs, and health clinics), the initial standards laid out for “accreditation” were later lowered, due to both prohibitive costs and the desire to reach more beneficiaries.  Although grants constituted the largest proportion of project costs, provisions for assessing the effectiveness of the selection and distribution process were scant. The PAD did not mention the criteria used in the past for the targeting of grants to ECIs, and whether or how the targeting would change under the project. This aspect was also not captured in the project's indicators, which were all focused on aggregate performance (See Section 9a). As a result, the project did not achieve its aim to create incentive structures similar to performance based budgeting.  In comments on the draft ICR Review, the GP stated that the lack of design focus on grants was intentional, and consistent with the SWAp approach adopted. IEG acknowledges this point, while considering the lack of attention to the effectiveness and distribution of grants as, at most, a design flaw, and at least, a missed opportunity to achieve the government’s stated goal of piloting incentive structures similar to performance-based budgeting.  The design of monitoring and evaluation also did not include sufficient provision for measuring benefits, a significant gap in a project focused on quality assurance. This gap also ultimately compromised the precision of the economic analysis. The shortcomings noted are considered significant, consistent with a rating of Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision In general, the Bank team proactively identified issues with achievement of development outcomes and assisted the client in addressing them (ICR, para 60). Examples of this proactivity include the revamping of disbursement processes and reporting at the first restructuring (Restructuring Paper 1, para 24); procurement of an HR expert to assist in resolving recruiting and staff retention problems (ICR, para. 46); and eventual streamlining of the burdensome number of indicators (see Section 9b). The Borrower’s comments (Annex 5) described Bank supervision as prompt and “highly satisfactory." The ICR stated that reporting in the Implementation Status Reports was transparent, with risks flagged appropriately. Fiduciary Page 16 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) and procurement aspects were intensively supervised, assessed and reported on (para 61). Project M&E data were used regularly for decision-making by the ECC and the Bank, particularly for adjusting objectives and targets for key indicators. The mid-term review was used to justify and begin design of the AF. The restructuring papers were well prepared, covering completely implementation status at the time and the state of risks. There were minor shortcomings. There was not regular monitoring of recurrent costs, or of the fiscal impact of the increased expenditures incurred by the Government for the project (information later added by the Project Team). The supervision team did not recognize--and therefore could not address--a lack of focus on the “micro-effectiveness of grant distribution” (Project Team). At one point, however, due to safeguards supervision, a pattern was noted of neglecting some poor districts, and this was pointed out to the Government. Although laudable surveys of beneficiaries were carried out in the design stage, these were not repeated in the mid-term review or later in the project implementation period, to inform the AF design. Similarly, there was no mid-term review planned for the second-phase AF. These were considered missed opportunities (confirmed by the Project Team). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design It is noteworthy that the NSP included evidence-based decision-making as an area of emphasis. Measurement and monitoring were therefore built into the foundations of the project. The NSP had a component to improve the national system for monitoring early childhood outcomes, including the introduction of impact indicators: a) a questionnaire module to measure child development in the Survey of Living Conditions; and b) a regular assessment of child development at age 4, to track effectiveness and provide feedback to ECD practitioners. The project also financed tools for monitoring HR capacity, a mid- term review (of the NSP) to feed into design of the NSP for 2013–2018, and a near-end project review. The project monitoring system design laid out in the appraisal documents consisted of: a) DLTs, which were mostly discrete qualitative steps to monitor progress on the NSP; and b) PDO and intermediate results indicators that were mostly quantitative (PAD, para 45). There were 71 of these indicators in total, with overlap between them. This complex design posed challenges for both Government and the supervision team. Although broader data collected by the borrower filled some gaps, shortcomings of the project indicators included: a) an over-reliance on indicators stated in terms of percentages without including absolute numbers, which hindered measurement of reach or coverage (the total number of beneficiaries in terms of children, households, and ECIs were notable by their absence among key indicators); and b) lack of disaggregation by gender. There was no theory of change or critical path Page 17 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) description in the appraisal document, making it difficult to judge the consequences of delays or the importance of dropped indicators that were not achieved. Overall, the project’s targets and monitoring had an aggregate focus. Less attention was paid to the content, distribution, and results of grants to ECIs, despite the fact that the grants comprised the highest proportion of project expenditures. There is no evidence that the question “Are the right ECIs receiving the grants, and by what criteria?” was addressed in the M&E design. A full-time MIS officer was recruited for ECC. Monitoring DLTs was the responsibility of the ECC, but delivering on some of the DLTs involved six other agencies, requiring memos of understanding and collaboration, and carrying the potential to create bottlenecks in data collection and monitoring. b. M&E Implementation Data collection measures were appropriate, and the data were, in general, found to be reliable and of good quality. The web-based MIS was effective in capturing progress on indicators monitoring the NSP. ECC hired the planned dedicated Research/MIS Officer responsible for maintaining the data and troubleshooting any quality issues. The updated results framework was discussed during each supervision mission, leading to corrective actions. The targets for indicator achievement (in terms of both time frame and level) changed many times. At each restructuring, some indicators were dropped, for varied reasons: a) they were achieved (usually DLTs); b) they were judged redundant; or c) they were no longer realistic. At the fourth restructuring, new indicators and targets were also introduced. The cumulative effect of these changes during restructurings was a reduction, by end-project, to five PDO indicators and 21 intermediate results indicators. The ICR reported some absolute figures on children reached by project interventions, disaggregated by gender, showing that the ECC MIS collected some data beyond that required for Bank reporting on the project. The MoEYI and MoH also had MIS systems that contributed to some results reporting; the ICR does not discuss whether the project strengthened those systems. The midterm review was completed and served its purpose of assisting in design of the AF. Information presented in the ICR on the end-project review conducted by the Borrower (ICR, Annex 5) listed useful basic data on documentation of achievement of DLTs, but it also contained inconsistencies and was limited in coverage. c. M&E Utilization Project M&E data were used regularly for decision-making by the ECC and the Bank, providing the basis for adjusting targets that could realistically be met to support reimbursement of key expenditures. For example, M&E data detected a slippage in the percentage of children enrolled in ECIs with permits to operate. The teams determined that one contributing factor was increasing demands on ECC inspection staff, and corrective actions were agreed (ICR, para 54). As noted, the mid-term review led to agreement between the Borrower and the Bank on the need to restructure the project and add AF. Page 18 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project did not trigger environmental safeguards and was rated Environmental Assessment category C. No buildings were planned, although allowance was made for possible minor works financed with counterpart funding, for which construction standards were to be in the Operational Manual and all Requests for Proposals. In addition, the Borrower agreed not to provide health services that produced medical waste. A comprehensive social assessment identified risks of exclusion of children with disabilities and from minority communities (Maroon and Rastafarian) (ICR, para 56). The assessment included not only desk review, but also a survey of communities and interviews with key stakeholders, including those working with disabled children (PAD, para 192). The design mitigated this risk by: a) ensuring that the curriculum included age-appropriate and culturally-sensitive references to diverse groups; and b) financing specialized training of ECI staff on identification and support for children with disabilities. Also, the PAD recommended that the ECC expand its presence in minority communities and provide parent education programs that were sensitive to diverse social groups (PAD, paras 56, 64). The development of parish-based ECD delivery models, and the formation of a Community Council and communication activities, supported these efforts. Safeguards (social factors) were monitored periodically by specialists, and at one point a potential inequity was identified and reported to Government. In general, however, the distribution of grants from an equity perspective was not monitored (information later provided by the Project Team). b. Fiduciary Compliance Financial Management (FM): The PAD used FM and procurement assessments (2006, 2007) at appraisal, identifying weaknesses and factoring these into the risk assessment, but set out to use country systems while addressing weaknesses through loan conditions. Challenges were discussed at the highest political level, and it was agreed that action plans would be carried out to strengthen FM and internal auditing (PAD, para 63). The project supported establishment of an FM system at ECC, following a transfer of authority from the MoEYI, and extra staffing and the establishment of an internal audit unit at ECC (ICR, para 57). FM status at the time of the AF was reported to be “good," as the accounting and audit systems (the latter “highly effective”) had been strengthened under the parent project. Still, close supervision was planned to continue (Restructuring Paper, Annex 2). The ICR reported that FM arrangements were satisfactory and in compliance with guidelines throughout most of the implementation period (ICR, para 57). Procurement: Similar to the case of FM, prior assessments in 2006 showed weaknesses in the procurement systems that affected transparency (PAD, para 158). The procurement system was multilayered, involving a National Contracts Commission, various committees, and a contact point at the Ministry of Finance and Public Service (PAD, para 160). Corrective measures built into design included: a) recruitment of a full-time procurement specialist for ECC with experience in international aid procurement Page 19 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) procedures, whose terms of reference the Bank would approve; b) submission to the Bank of a preliminary procurement plan for the first 18 months of the project (condition of negotiations) (PAD, para 165); and c) supervision missions every six months and post review of procurement once a year (PAD, para 167). The ICR reported frequent procurement delays (paras 35, 45, 47). However, in the fiduciary section, the ICR stated that most of the time procurement arrangements were satisfactory, with problems mainly surfacing toward the end of the project, specifically persistent delays on a few key contracts (ICR, para 58). Ex-post reviews confirmed compliance of procurement procedures by the ECC. The ICR and the Borrower’s end-project review report cited other factors beyond the control of the ECC causing procurement delays (difficulty in finding technical experts; one case of a conflict of interest discovered late) (ICR, Annex 5, p. 57, and additional information provided later by the Project Team). c. Unintended impacts (Positive or Negative) The ICR reported that the project leveraged financing from the United Kingdom aid agency (through the Early Learning Partnership Trust Fund) and United Nations Children's Fund for research. Already, ECC representatives have spoken at international conferences, participated in a South-South knowledge exchange, and are planning to open a regional ECD professional development institute scheduled for May 2019 (ICR, para 41). The added support for research should allow this regional leadership to continue. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Efficiency is rated modest due to shortcomings in implementation efficiency and an IRR calculation Outcome Satisfactory Moderately Satisfactory affected by unavailable data. Achievement of one objective is rated Modest. Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial There were minor to moderate shortcomings in following ICR Quality of ICR --- Substantial guidelines, reporting on DLTs, and presenting information on efficacy and efficiency. Page 20 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) 12. Lessons A Sector Wide Approach (SWAp) using disbursement-linked targets (DLTs) can align incentives within Government to achieve a project’s targets and aid intersectoral coordination. In this case, central government authorities were incentivized to engage relevant line ministries, and the implementing entities outside the main implementing agency (the Early Childhood Commission, ECC) were encouraged to collaborate to meet project targets. A conventional input-focused project may not have mobilized all agencies in a unified action plan in the same way. Nonetheless, a more streamlined model with fewer DLTs and a simpler results framework may well have reaped the same relative benefits, with less need for changed schedules and restructurings. Whether a project uses a SWAp or traditional investment project design, it is necessary to ensure that adequate incentives, authority, and communication mechanisms are in place to sustain commitment and accountability from all partners, and that cross-sectoral coordination bodies have leverage to drive results. In this case, the ECC initially lacked full authority to hold other executing agencies accountable and to sanction non-compliant early childhood institutions. Project targets should be set realistically, accounting for the implementing organization’s experience. In this case, ECC faced a steep learning curve as it began project implementation. A phased approach to build capacity initially, or setting more explicit objectives of strengthening overall project management, might have increased effectiveness and reduced the need to adjust targets. Building and sustaining effective quality assurance (QA) systems starts with careful setting of initial standards. In this case, for the certification of well-child clinics, the Ministry of Health created standards that almost no facilities were prepared to meet, contributing to the project’s missing the certification target, and leaving a temporary void in standards. Where project disbursements are against grants to a large number of decentralized institutions with improvement action plans, effects on equity need to be monitored. In this project, disbursements were linked to aggregate targets, for example, the percentage of ECIs and health centers with well- child clinics certified. Although grants to these institutions formed the largest share of disbursements, there was little evidence in the project documents that the criteria against which grants were made were examined, revised, and/or monitored throughout the project with an eye toward effects on results and equity. The desire to meet targets related to standards could result in disproportionate resources allocated to facilities that are close to higher standards at the outset, thus serving more advantaged areas. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain The ICR stated that this is one of the first SWAp projects in the Bank. Given that this was the case, the project's complexity was a design shortcoming, and an assessment may shed light on alternative designs, Page 21 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) although a later SWAp project may be a better-practice case for assessment. In addition, the ICR did not treat comprehensively: (i) the consequences of all delays and changes in indicators for efficacy or implementation efficiency; (ii) the consequences of using only latter-year data for calculations of IRR and benefit-cost ratio (2013–2018, when the project started in 2008); and (iii) the criteria for allocation of grants to ECIs, their rationale, the possible need for change, and monitoring for equity concerns. Thus, an assessment could shed further light on these issues not well covered in the ICR, if judged consequential for either ratings or lessons. 14. Comments on Quality of ICR The ICR presented many valuable facts about project implementation, usually well organized to support conclusions. The discussions on context, relevance, and project restructurings were well written. The inclusion of an Annex 7 attempting to describe all changes in indicators for each restructuring was laudable, although the ICR author stated it did not cover all DLTs. The analysis of factors affecting implementation was thorough, although it was sometimes difficult to place them in time, e.g. link them to supervision report ratings. Writing was clear and concise. In many cases, useful examples were provided as evidence, e.g. for utilization of M&E and quality of supervision. Presentation of a Theory of Change diagram was a welcome addition, since the PAD did not present one; however, it had shortcomings (see below.) The ICR went out of its way to cite some absolute numbers in its efficacy discussion and annexes on achievements, as opposed to just the percentages in the formal project results framework, including gender disaggregation. The ICR was candid in presenting a key case where project outputs were not achieved. Lessons were well identified and described. The ICR effectively covered the way in which the project addressed gender issues. However, there were minor to moderate shortcomings. Most notably, Annex 3 contained inaccuracies in several respects. First, it did not list total costs, by component, with all sources of financing identified, including Borrower contributions. Second, the stated amount of the IBRD loan “at approval” was inconsistent with the amount noted in the PAD (para 92) and the LA (p. 11). Likewise, the efficiency section and Annex 3 lacked information on total costs. Annex 5, “Government, Cofinancier, and Other Partner/Stakeholder Comments,” was not presented in full (or at least in summary form that contained key information such as title and location of the original document or information on the ministry or agency that prepared it; instead, a note at the front said only that the introduction had been omitted deliberately “for brevity"). In commenting on the draft ICR Review, the GP provided information on the parts omitted. The theory of change presented in the ICR did not clearly link NSP areas of emphasis/outcomes to the project's activities (and outputs and outcomes). Only some of the last two areas of NSP – improving governance and evidence based decision-making – were covered. The efficacy section did not refer to the theory of change, focused only on selected indicators, and did not mention some key indicators that were dropped or some cases where standards were lowered. Better presentation of a “critical path” to outcomes, specifying which steps were achieved, which dropped, and which not achieved, and the consequences, was needed. The information was not well organized in a way that would enable the reader to place the succinct efficacy narrative in the context of a more comprehensive examination of outputs and outcomes achieved or not achieved (using Annex 1, Parts A and B and Annex 7, each of which contained different degrees of detail and some inconsistent terms). Information on the original and revised DLTs--including whether and when they were achieved, whether and when they were dropped, and why they were dropped--was insufficient. DLTs were the basis for disbursement releases under the SWAp (by far the largest part of the project compared to TA), and some were a critical-path Page 22 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JM Early Childhood Develop Proj -- SWAP (P095673) step to achievement of PDO indicators. Lack of full information on them had consequences for assessment of efficacy and efficiency. The analysis of efficiency was not concise and was incomplete. The narrative sometimes mixed concepts of efficacy and efficiency. The reasons given for limiting use of data to the second half of the project (2013–2018) were not sufficient, and consequences were not covered. The summary statement “Overall, efficiency was satisfactory and in line with expectations in the sector” (ICR, para 35) was not supported by evidence. The ICR did not report unit costs or comparisons. a. Quality of ICR Rating Substantial Page 23 of 23