Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) Report Number : ICRR0021136 1. Project Data Project ID Project Name MG-Emerg Supp to Critical Ed, Health, P131945 Nu Country Practice Area(Lead) Additional Financing Madagascar Health, Nutrition & Population P148749,P160666 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51860,IDA-53820 31-Jul-2016 75,000,000.00 Bank Approval Date Closing Date (Actual) 29-Nov-2012 30-Jul-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 65,000,000.00 0.00 Revised Commitment 72,917,077.14 0.00 Actual 68,112,651.76 0.00 Prepared by Reviewed by ICR Review Coordinator Group Gisela M. Garcia Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The project development objective (PDO) as stated in the Legal Agreement (p.5) was to preserve critical education, health and nutrition service delivery in targeted vulnerable areas in the Recipient's territory. The PDO stated in the Project Paper was identical. The PDO did not change during implementation. Target values for two key outcome indicators associated with nutrition were revised upwards (increased) under the Additional Financing in March 2014. The targeted number of project beneficiaries was also increased and subsequently decreased through a third restructuring in July 2016. Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) The project originally targeted five regions with the highest poverty rates as per the 2011 Poverty Map. Twenty-four percent of the total population of Madagascar lives in these regions. The Additional Financing targeted two of these five regions and three additional regions. The project was extended once and had three restructurings. Although these changes impacted some of the targets of key outcome indicators, there is no need for a split rating because all targets were met or exceeded. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Component 1: Preserving Critical Education Services (original cost of $ 23.50 million, actual cost of $22.02 million), was to include four subcomponents/activities: • Subsidies of community teacher salaries; • School grants; • School health and nutrition interventions, i.e. the distribution of treatment against neglected tropical diseases (NTDs); and • Capacity strengthening, project management, and monitoring and evaluation. Component 2: Preserving Critical Health Services (original cost of $ 25 million, actual cost of $24.81 million), was to include two subcomponents: • Support to the delivery of a critical package for pregnant women and children under five at the health facility level. This sub-component was to include the following activities: (i) rehabilitation of and procurement of basic equipment for primary health care facilities; (ii) procurement of essential obstetric and neonatal equipment; (iii) training on obstetric and neonatal care for medical staff at the facility, district and regional levels; (iv) procurement of safe delivery kits; (v) procurement of transport and medical equipment for community outreach; (vi) recapitalization/restocking of public pharmacies at the facility level; (vii) establishment of a fee exemption system for pregnant women and children under 5 years old; and (viii) distribution of treatment against NTDs through participation in school health and nutrition interventions, including the procurement of drugs, and through mass drug administration at the community level; and • Project management and monitoring and evaluation. Component 3: Preserving Critical Nutrition Services (original cost of $ 10.50 million, received Additional Financing of $ 10 million; actual cost of $21.28 million), was to include two sub- components: • Support to basic community nutrition services. This sub-component was to include the following key activities: (i) establishment of new community nutrition sites (identification of vulnerable Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) communities,awareness campaign, and selection of community nutrition workers); (ii) support to existing and new community nutrition sites, including the development and distribution of manuals, posters, and other communication materials to support awareness presentations delivered by community nutrition workers, as well as the procurement of cooking kits for culinary demonstrations, scales, and so forth; (iii) the recruitment of non-governmental organizations to supervise and support community nutrition sites; (iv) the testing of different approaches, namely intensive counselling through household visits, lipid-based nutrient supplementation for children, lipid-based nutrient supplementation for pregnant and breast-feeding women, and early childhood stimulation, which were to be evaluated through a randomized controlled trial; and (v) participation in the distribution of treatment against NTDs, including provision of small meals prior to administering NTD medication and sensitization of communities on the NTD campaigns. In addition, a new set of food security activities at the community and household levels, including home gardens, short- cycle farming and agriculture, and food conservation, were introduced under the Additional Financing; and • Project management and monitoring and evaluation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The original project was to be financed by a US$ 65 million International Development Association (IDA) Credit, of which US$ 59.1 million was disbursed. The Additional Financing was in the form of a US$ 10 million IDA Credit, of which US$ 9.0 million was disbursed. Total actual financing was therefore US$ 68.1 million. No Recipient contribution was planned or made. Project costs were similar to estimates at appraisal, when considering the Additional Financing for component 3. The March 2014 Additional Financing expanded Component 3 to introduce a new set of food security activities at community and household levels, and to add geographic coverage to regions that had been severely affected by a locust infestation. Consequently, the targets for three nutrition-related outcome indicators were revised upward (increased). The project was approved on November 29, 2012 and became effective on April 25, 2013. An Additional Financing in February 2014 was approved to address rising risks of food insecurity by adding food security activities in five regions (three new regions, and two of the five that were part of the original project) and financing an additional 837 community nutrition sites and related interventions at the community and household levels in three new regions. Targets were increased for: • The number of direct project beneficiaries • Number of people with access to a basic package of nutrition services • Number of children under 2 years old benefiting from improved infant and young child feeding practice. A first restructuring was approved in July 2015 to introduce activities to improve quality and expand coverage of Component 3. The Additional Financing closing date was extended by one year (from July 31, 2016 to July 30, 2017). Key changes in components included: • Component 1: provision of school manuals and trainings for community teachers; Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) • Component 2: continuation of a results-based financing pilot initiated under an STI/HIV/AIDS Prevention Project (P090615); • Component 3: introduction of behavior change communication to improve utilization of nutrition messages at the household level, and expansion of the geographical scope of nutrition activities to one more region. A second restructuring was approved in June 2016 to reassign unallocated resources to Component 3 to intensify efforts in the South through the identification and treatment of children under five years old who had acute malnutrition (in partnership with the World Food Program). A third restructuring was approved in July 2016 to allow for the completion of activities in Components 1 & 2. Target values for the indicators on direct project beneficiaries and for two intermediate results indicators were decreased to reflect changes in population projections. 3. Relevance of Objectives Rationale The PDO was highly relevant to the detailed description of the country context at appraisal, as reflected in the Project Paper (pp. 2-6) and in the ICR (p. 5-6). The PDO was highly relevant to the FY12-13 Interim Strategy Note (ISN). As per the ISN, Madagascar's health/nutrition and education sector were in a "state of affairs already close to an emergency situation... with the poorest bearing the brunt of the political crisis and suspension of aid...The severity of the situation was denounced by the UN Special Rapporteur on the right to food in July 2011, who alerted that 68 percent of the population in the South of the country is food insecure and that Madagascar has one of the highest levels of child malnutrition in the world" (ISN pp. 9-10). As per the ICR, country development challenges did not significantly change during implementation (ICR p.14). The PDO was highly relevant to the priorities and recommendations identified in the 2015 Systematic Country Diagnostic that included a renewed focus on nutrition and improving the quality and equity of education (ICR p. 6, Country Partnership Framework, CPF p. 29). Moreover, the PDO was aligned with the 2015-19 National Development Plan, which included “adequate human capital for the development process” as one of five strategic areas (ICR p. 14; CPF p. 24). The PDO remained highly relevant to the CPF for FY17- FY21, and in particular to the CPF's objective to "Increase resilience and reduce fragility." Given that this project funded a combination of health, nutrition, and education investments in children’s “early years,” it also contributed to the CPF objective of strengthening children’s human development. (CPF pp. 25, 31-32). Rating High 4. Achievement of Objectives (Efficacy) Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) PHEFFICACYTBL Objective 1 Objective Preserve critical education service delivery in targeted vulnerable areas Rationale As envisioned in the project's theory of change, supply-side activities financed by the project (such as the subsidies of community teacher salaries, and school grants to meet basic operational needs and minor infrastructure repairs) helped ensure that primary school services were available and functional. In addition, health and nutrition interventions delivered in schools helped stimulate demand for primary school services. These interventions helped preserve critical education services as measured by the number of functional schools and the number of students enrolled, to be maintained at 2012 levels. This objective exceeded its targets as measured by the following outcome indicators: • 1,131,353 students were enrolled in primary schools (surpassing the target of 974,300); and • 6,682 schools received grants (surpassing the target of 6,050). Outputs that contributed to the achievement of the objective included: (i) 100% of school grants were paid on time; (ii) 16,999 community teacher salaries were subsidized; (iii) 6,688 parents' associations/school management committees were trained on teacher accountability processes and school grants; (iv) 626,583 school manuals were purchased; and (v) 19,852 community teachers were trained. Rating High PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Preserve critical health service delivery in targeted vulnerable areas Rationale As envisioned in the project's theory of change, supply-side activities financed by the project included the rehabilitation and equipment of primary health care facilities; training of medical staff; the recapitalization/restocking of public pharmacies; and capacity building of district services to ensure that primary health care services were available and functional. Demand-side activities included the establishment of a fee exemption for maternal and child health interventions; the distribution of free safe delivery kits; and the provision of transportation and equipment to expand the radius of community outreach services. These interventions helped preserve critical health services, as measured by the number of births attended by skilled health personnel and the number of children immunized, which increased and reached pre-crisis levels. Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) This objective exceeded its targets as measured by the following outcome indicators: • 131,434 deliveries were attended by skilled health personnel (surpassing a low target of 12,600); • 286,194 children under 12 months were immunized against DTP3 (surpassing a low target of 18,300); and • 113,131 pregnant women received antenatal care during a visit to a health provider (surpassing a low target of 18,283). Forty-eight percent of women attending antenatal clinics were tested for syphilis (half of the revised target of 90%, but surpassing the original target of 36%). The ICR (p. 17) explained that syphilis tests and treatments were procured as an emergency measure under the project, but that these inputs were not delivered in time to compensate for shortages caused by a delay in Global Fund resources. Outputs that contributed to the achievement of the objective included: (i) 347 primary health care facilities were rehabilitated and provided with basic equipment; (ii) heads of 347 primary health care facilities were trained on obstetric and neonatal care, the fee exemption system, medical waste management, and administrative data collection; (iii) 157,251 safe delivery kits were distributed; (iv) 347 primary health care facilities were provided with transport and medical equipment for community outreach; and (v) 296 solar- powered refrigerators were procured for primary health care facilities located in remote areas. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Preserve critical nutrition service delivery in targeted vulnerable areas Rationale As envisioned in the project's theory of change, supply-side activities financed by the project included the functionalization of new and existing local community nutrition sites to deliver nutrition services (i.e. weighing, nutritional counseling, and cooking demonstrations), and the recruitment of non-governmental organizations (NGOs) to support and supervise community nutrition agents at community nutrition sites. Demand-side activities included targeted household visits. Activities also included support for the design and evaluation of various interventions to improve the quality of nutritional counseling services. These interventions helped preserved critical nutrition services as measured by the number of people with access to a basic package of health, nutrition, or reproductive health services, and the number of children benefiting from improved child feeding practices, all of which increased and reached pre-crisis levels. This objective was exceeded its targets as measured by the following outcome indicators: Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) • 2,268,854 people with were provided with access to a basic package of health, nutrition, or reproductive health services (surpassing the revised and initial targets of 1,619,303 and 932,426 respectively); and • 425,360 children under the age of 24 months benefited from improved infant and young child feeding practices (surpassing the revised and initial targets of 289,340 and 164,220 respectively). Outputs that contributed to the achievement of the objective included: (i) 462,315 children under two years old were enrolled in a growth monitoring program; (ii) 515,764 children between two and five years old were enrolled in a mid-upper arm circumference program; (iii) 3,582 community nutrition agents were trained to provide health and nutrition education; (iv) 92 percent of nutrition sites complied with the requirement of submitting monthly reports; (v) 6,587 schools were supported by community nutrition agents during deworming sessions; (vi) 23,114 households received support kits for short-cycle agriculture and livestock activities; and, (v) 1,005 new community nutrition sites were established and equipped with scales, manuals, cooking kits, and work clothes. Rating High PHREVDELTBL PHOVRLEFFRATTBL Rationale The project maintained social service delivery in targeted areas, using and strengthening existing country systems and contributing to removing geographical and financial barriers to access through supply- and demand- side activities in health, education and nutrition. The project exceeded or fully achieved the target values for all outcome indicators. The project contributed to increased access and utilization of services at the primary health center level, and increased access to cost-effective nutrition interventions for pregnant women and children under five in the poorest areas of the country. In total, more than 2 million children and mothers benefited from essential maternal and child health and nutrition and education services (around 10 percent of the country's population). Achievement of the education and nutrition objectives is rated High. Achievement of the health objective is rated Substantial due to shortcomings in the syphilis testing and treatment activity, and because outcome targets were set relatively low. Overall Efficacy is therefore rated High. The ICR noted that there were no other interventions in the areas targeted by the project that could have impacted the observed outcomes (ICR p.14). Yet, the efficacy discussion in the ICR might have benefited from presenting data on outcomes in non-targeted areas and compared them to the ones observed in project areas to make the attribution discussion stronger. Overall Efficacy Rating High PHREVISEDTBL Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) 5. Efficiency The ICR discussed both economic and implementation efficiency in detail, devoting 17 pages in an annex (ICR Annex 4). Economic efficiency. The ICR used a cost-benefit analysis (CBA) to assess project economic efficiency, comparing the situation before project implementation (2013) with the situation at project closure (2016/17). No CBA was conducted at appraisal due to the emergency conditions under which the project was prepared; rather, only benefit-cost ratios of education, nutrition, and health services were presented at appraisal (Project Paper pp. 117-126). The ICR followed current guidelines and described the assumptions and methods used in the CBA, its limitations, and the sensitivity analysis performed. The CBA used actual population, service utilization, and cost data from project implementation and surveys. The CBA followed the logic presented in the project's theory of change between project components, intermediate outcomes, and objectives. There was a good discussion of the potential impacts based on analysis of a wide scope of benefits and relevant literature. In addition to using available information from the broader literature to estimate benefits, the CBA used information available from the project's impact evaluation in one region that, given its characteristics, was considered representative of the project-targeted areas. Expected benefits were estimated using long-term returns on investment based on the literature and information on student performance in Madagascar. The analysis found a benefit-to-cost ratio for the project of 3.1. The investment was found to be justified on economic grounds, even when the CBA likely underestimated the benefits of the project as it captured only the impact of nutrition interventions on under 5 mortality (and not on maternal mortality), it did not count the impact of nutrition interventions on future productivity, and it did not capture potential changes in morbidity/disability due to project interventions. The investment's rate of return varied from 14-18% based on different scenarios, significantly higher than the cost of capital in Madagascar as reported by the ICR (ICR p. 62, table A.13). Implementation/operational efficiency. The ICR highlighted that the project leveraged existing interventions and institutional arrangements for project implementation, which included a cross-sectoral steering committee that created synergies between sectors, provided overall guidance, and ensured service delivery, which in turn contributed to smooth implementation and a fast disbursement rate (ICR p. 23). The also stressed that little time elapsed between approval and effectiveness (less than 5 months) and between effectiveness and first disbursement (1.3 months), well below other projects in Madagascar and in Africa in general (ICR p. 63). The ICR also pointed to the fast disbursement rate of the project (3.5 years) and low staff turnover (on both the government and Bank side). Procurement of vehicles and information technology was performed through the United Nations Office for Project Service, which allowed for timely completion of procurement processes and reduced costs (ICR p .27). Some procurement challenges emerged during implementation that may have contributed to delays, yet these procurement challenges were handled following the Bank’s procurement guidelines (ICR p.28). Operating costs amounted to 12% of total project costs. In addition, the project benefited from changes in the exchange rate that lead to a devaluation of the local currency (most costs were paid in local currency) (ICR p. 29). Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) As discussed in more detail in Section 10, some challenges emerged at the end of the implementation period that may have reduced operational efficiency: (i) shortcomings in budget monitoring by the Education Project Implementation Unit that resulted in unexpected undisbursed resources at the end of the project; (ii) weaknesses in contract management related to the food security activities in Component 3; and (iii) delays in the collection of required documentation on expenditures. The ICR also noted challenges related to Project Implementation Unit staffing that contributed to delays that, in turn, led to the extension of the closing date (ICR p. 27). Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Considering that the project exceeded or fully achieved the target values for all the outcome indicators, the high project relevance given the country context and national development program, the strong project design as reflected in the project's theory of change/results logic, and the substantial efficiency rating, the overall outcome rating is Highly Satisfactory. a. Outcome Rating Highly Satisfactory 7. Risk to Development Outcome As per the ICR, despite the risks associated with country fragility, the risk to development outcome is low due to the many activities that have continued after project completion (e.g. subsidies to community teacher salaries, transfer of school grants, and stationary and mobile maternal and child care consultations), strengthened government capacity thanks to project efforts, and existing demand from the population for the Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) interventions supported by the project. In addition, the ICR mentioned that two new Bank projects, one in education and another one in health and nutrition (the Basic Education Support Project, approved in March 2018 for $55 million; and the Multiphase Programmatic Approach to Improving Nutrition Outcomes, approved in December 2017 for $90 million) will guarantee continuation of some of the activities supported by the project (ICR p. 35). 8. Assessment of Bank Performance a. Quality-at-Entry Quality at entry is satisfactory, given the clarity of project objectives; the straightforward design and associated theory of change; the attention to M&E and use of existing administrative data to select the indicators to be included in the results framework; the attention to ensuring inter-sectoral coordination and buy-in from local stakeholders; and the reliance on existing evidence and lessons learned to inform project design. The project paper included a comprehensive risk assessment that discussed potential stakeholder, social, and environmental risks; risks associated with local capacity, policy, and governance; risks associated with the multi-sectoral nature of the operation; and risks associated with monitoring and sustainability (Project Paper, Annex 4). The operation's overall risk was rated Substantial, and detailed mitigation measures were included in project design. The Bank team followed OP/BP 8.00 (Rapid Response to Crises and Emergencies), as the project aimed at preserving essential services and not addressing long-term economic issues. Yet the project team integrated lessons from analytical work in the education and health sectors in the country as well as from other projects in the sectors in Madagascar and elsewhere. Lessons included the importance of specifying indicators with reliable data and then using that data for decision making; being flexible in project design to respond to changes in urgent needs; community involvement and capacity building to local stakeholders to improve accountability and governance; support to decentralized levels of health and nutrition; and collaboration with NGOs, among others (Project Paper, Annex 11). Minor shortcomings in quality at entry had to do with the issues mentioned in M&E design (Section 9a): lack of definitions or formulae for some indicators, as well as low targets for the indicators on number of assisted deliveries and number of children immunized . Quality-at-Entry Rating Satisfactory b. Quality of supervision Evidence presented in the ICR suggested adequate supervision by Bank teams, including supervision visits held every six months; monthly videoconferences held after the mid-term review; continuity in the project team (co-task team leaders in the health and education sectors ensured this continuity); and use of aides-memoire discussing implementation progress and issues, fiduciary aspects, environmental Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) safeguards, an update about the issues discussed in the previous aide-memoire, and pending issues with responsibilities and deadlines (ICR p. 33-34). Moreover, procurement challenges that emerged during implementation were handled following the Bank’s procurement guidelines (ICR p. 28). Additionally, the Bank team showed proactivity in adjusting target values to reflect changes in project scale. During the February 2014 additional financing, target values of indicators were increased to reflect the revised scale of the project, and then lowered during the third restructuring of July 2016 to reflect changes in population projections (ICR p. 13). However, the ICR highlighted some areas where the Bank could have been more responsive or provided more timely support and clear guidance to avoid delays. These included more flexible procurement policies, and better guidance on contracting with NGOs and on the disclosure of the updated medical waste management plan and the implementation of mitigation measures (ICR p. 29). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s theory of change was clearly articulated in the results framework defined at appraisal, capturing expected outcomes as well as the outputs and activities contributing to their achievement. Indicators selected were chosen based on available administrative data, and thus included baselines and targets. As per the ICR, considering the emergency nature and short-term focus of the operation and the fragile context, the PDO indicators were selected based on a principle of being simple, realistically achievable, and measurable using existing credible sources (ICR p. 13). However, the ICR pointed to some shortcomings in that a few indicators lacked definitions or a formula (ICR p. 30), and the targets at appraisal for some of the indicators were very low (in particular, for the number of deliveries assisted by health professionals and number of children immunized). Despite the emergency conditions under which the project was designed, the importance of M&E was embedded in project design, and the project included innovative features both in the instruments used for data collection and on the institutional arrangements for M&E. The project included M&E activities under all components to ensure regular supervision, along with timely, regular data collection to measure progress and results and to inform sector policy dialogue with the goal of facilitating the capture and integration of lessons learned during implementation (Project Paper p. 13). Existing Project Implementation Units (PIUs), one per sector, were used to facilitate implementation and monitoring. Each PIU was responsible for monitoring activities and results related to its sector (i.e the Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) education PIU monitored activities and results under component 1, the health PIU did the same for component 2, and the nutrition PIU for component 3). A cross-sectoral technical steering committee and a project coordination cell ensured cross-sectoral coordination and approved all progress reports. Besides activities to strengthen the existing monitoring and evaluation system that would allow for relying on existing administrative data, M&E activities included rapid data collection via mobile phones, periodic surveys and assessments, and community involvement and third-party verification by NGOs to ensure accountability. b. M&E Implementation Until the Mid-Term Review, there were some issues associated with high turnover of the M&E specialist in the Project Management Unit, as well as some shortcomings (mentioned above) in terms of indicators that lacked definitions or a formula (ICR p. 30). However, data was collected as planned using administrative databases. As per the ICR, evidence of achievement of targets was regularly monitored in supervision missions that happened every six months (ICR p. 30). Administrative data used to report on indicators was found reliable in the independent verifications conducted by the Bank (ICR p. 30). As per the ICR, the cross-sectoral Technical Steering Committee met biannually, in accordance with the Financing Agreement, including to approve annual work plans and review project implementation progress (ICR p. 27). The project benefited from an impact evaluation (Garchitorena et al, 2017) funded by the Strategic Impact Evaluation Fund, the Early Learning Partnership Program, a World Bank Innovation Grant, Grand Challenges Canada, the World Bank Research Committee, the Japan Nutrition Trust Fund, and the Power of Nutrition Trust Fund. The impact evaluation reported positive changes in service utilization in the region of Vatovavy- Fitovinany that the project team used to extrapolate the benefits of the whole project in its efficiency analysis (ICR p. 11, 57). c. M&E Utilization As per the ICR, data on performance and results was used to inform project management and decision- making, and even other operations (ICR p. 31). The changes implemented with the additional financing, first restructuring, and after the Mid-Term Review are examples of the regular use of monitoring data to inform project management and decision making. There was no reference in the ICR to the use of the data from the impact evaluation, except for the CBA as noted above. M&E Quality Rating Substantial Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) 10. Other Issues a. Safeguards The project was classified as “Category B, Partial Assessment” at appraisal (Project Paper pp. 81-82). OP/BP 4.01 (Environmental Assessment) was triggered (Project Paper p. 82, ICR p. 28), requiring a revision of the existing medical waste management plan. As per the ICR, the Government complied with safeguards requirements, although at a slower pace than expected. The ICR cited a 22-month delay in the public disclosure of the revised waste management plan and, consequently, in the implementation of mitigation measures (ICR pp. 28, 31). b. Fiduciary Compliance The ICR suggested project compliance with Bank policies on financial management (OP/BP 10.02), procurement (OP/BP 11.00), and disbursement (OP/BP 12.00). The government complied with regular financial management reporting: reports were prepared and transmitted to the Bank in a timely manner, and audit reports had unqualified opinion (ICR p. 32). Yet, some challenges emerged at the end of the implementation period: (i) shortcomings in budget monitoring by the Education PIU that resulted in unexpected undisbursed resources at the end of the project; (ii) weaknesses in contract management related to food security activities in Component 3; and (iii) delays in the collection of required documentation on expenditures. These shortcomings led to an increase in the project's financial management risk in April 2016 (from "Moderate" to "Substantial") and, consequently, to a downgrade of the financial management rating in April 2017 (from "Satisfactory" to "Moderately Satisfactory") (ICR p. 32). Procurement of vehicles and information technology was performed through the United Nations Office for Project Service, following the financing agreement, which allowed for timely completion of procurement processes and reduced costs (ICR p. 27). Some procurement challenges emerged during implementation, that, as per the ICR, were under the government's control. Despite having contributed to delays, these procurement challenges were handled following the Bank’s procurement guidelines (ICR p. 28). c. Unintended impacts (Positive or Negative) None reported. d. Other --- Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment High relevance and high achievement of two out of the project's three objectives, Outcome Satisfactory Highly Satisfactory combined with a rating of substantial on efficiency, produce an Outcome rating of Highly Satisfactory. Bank Performance Satisfactory Satisfactory --- Quality of M&E Substantial Substantial --- Quality of ICR High --- 12. Lessons Useful lessons are derived from the ICR: It is important for an emergency operation to have a realistic objective and a simple and flexible design. As per the ICR, and given the emergency context of the operation, setting a realistic objective and maintaining flexibility, by keeping a significant unallocated amount of resources, were key to the successful implementation and outcomes of this operation (ICR p. 35). This project incorporated lessons learned from another emergency operation in Madagascar, and changes during implementation proved the usefulness of having the flexibility of unallocated resources to respond to a locus infestation. In addition, a simple design for a multi-sectoral operation not only benefited from previous experience but also tapped into government ownership and willingness to experiment with innovative interventions (ICR p. 35). Procurement rules are prudently adapted to country characteristics. Small countries like Madagascar may not have the scale to follow standard procurement rules that require international and competitive bidding. While transparency and due diligence in procurement has to be assured, the Bank could show more flexibility and understanding in such cases and provide government teams with realistic solutions to avoid unnecessary delays. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was candid in its analysis of the evidence and very comprehensive in the quality and discussion of the evidence, yet was concise and outcome-driven. The ICR followed guidance and was internally Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG-Emerg Supp to Critical Ed, Health, Nu (P131945) consistent. The theory of change visual derived by the ICR clearly indicated how activities led to outputs, and how outputs led to intermediate outcomes and outcomes, through the use of specific arrows leading from one element to the next. The efficiency discussion was of particularly high quality: it followed current guidelines and described the assumptions and methods used in the CBA, its limitations, and the sensitivity analysis performed in great detail; such detail is rarely seen in ICRs, thus this analysis could be considered best practice. Derived lessons were based on the analysis and discussion of the evidence. The ICR discussed attribution by noting that there were no other interventions in the areas targeted by the project that could have impacted the observed outcomes (ICR p. 14). Yet, the discussion might have benefited from presenting data on outcomes in non-targeted areas and comparing them to the results observed in project areas to make the attribution discussion stronger. a. Quality of ICR Rating High Page 15 of 15