ICRR 14315 Report Number : ICRR14315 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/30/2013 Country : Niger Project ID : P096198 Appraisal Actual Project Name : Multi-sector US$M ): Project Costs (US$M): 12.5 12.5 Demographic Project L/C Number : CH309 Loan/ US$M ): Loan /Credit (US$M): 10.0 10.0 Sector Board : Health, Nutrition and US$M): Cofinancing (US$M ): 2.5 2.5 Population Cofinanciers : UNFPA Board Approval Date : 06/19/2007 Closing Date : 03/31/2013 03/31/2013 Sector (s): Central government administration (57%); Other social services (27%); Sub-national government administration (11%); General education sector (5%) Theme (s): Population and reproductive health (29% - P); Gender (29% - P); Education for all (14% - S); Personal and property rights (14% - S); Economic statistics; modeling and forecasting (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Hjalte S. A. Sederlof Judyth L. Twigg Christopher D. Gerrard IEGPS2 2. Project Objectives and Components: a. Objectives: The Project Development Objective (PDO), as set out in Schedule 1 of the Financing Agreement, was “to strengthen the Recipient’s capacity to address its demographic challenges through : (i) enabling the Ministry of Population (MOP) to design and implement a nationwide multi -sector population program; and (ii) increasing general awareness on population and reproductive health issues .� The substantively same PDO was included in the Project Appraisal Document (PAD). b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The original project components were the following (estimated project costs are indicated, while actual costs are indicated in the paragraphs describing the revised components ). Component 1: Advocacy and Communication (estimated cost at appraisal US$ 4.3 million). This component aimed at keeping population issues high on the national development agenda and building a strong national consensus around them; modifying traditional pro -natalist attitudes; promoting birth spacing and reproductive health services; and strengthening links between family planning and HIV /AIDS mitigation efforts. A population and development sub -component aimed at establishing an advocacy and communications strategy, encompassing all aspects of population and development relationships, to be implemented at the central and decentralized levels. A second sub-component aimed at developing and implementing information, education and communications (IEC) activities and behavior change communication (BCC) activities in reproductive health, family planning, and breastfeeding . Target groups were to include rural populations, traditional and religious leaders, and NGOs active in rural areas, as well as decision makers and stakeholders at the national level . Component 2: Women ’s Autonomy and Couples ’ Empowerment (estimated cost at appraisal US$ 1.2 million). This component was to help strengthen the autonomy of women and consequently foster higher demand for birth spacing, family planning, and reproductive health services . It was to include activities in three areas alongside the IEC and BCC outreach work supported in Component 1: reinforce efforts to improve girls ’ school enrollment and performance through community outreach and sensitization on the importance of female schooling, focusing on areas where the gender gap in enrollment was greatest (this activity was to supplement a successful outreach program already supported under a Bank-financed education project ); strengthen women’s economic opportunities, including better economic and statistical analysis and carrying out a study of obstacles to women ’s economic empowerment; skills training at the community level to help women manage income-generating activities; and provision of local women ’s associations in key disadvantaged areas with equipment and materials specifically aimed at alleviating domestic task burdens; and help trigger legal reforms, including measures needed to increase the minimum marriage age; study customs and practices relating to marriage, divorce, and other family issues; and conduct outreach and sensitization activities on gender and legal issues . Component 3: Harmonization and Coordination of Multi -Sector Interventions (estimated cost at appraisal US$ 0.9 million). The component was to help MOP coordinate, monitor, and evaluate all population -related activities in Niger in accordance with the Government ’s population policy, its strategic framework, and work plans . Annual work plans (AWPs) were also to be prepared under this component . These AWPs were to assign specific tasks and goals to stakeholders and development partners and provide monitoring and evaluation indicators . A national annual review was to be organized as soon as Project Year 2 to harmonize all population and reproductive health contributions from key sectors, Component 4: Capacity Building and Monitoring & Evaluation (estimated cost at appraisal US$ 6.0 million). The component was to focus on two major areas : collection, analysis, dissemination, and utilization of data for monitoring and evaluation of the project; and institutional strengthening and capacity building in the MOP to allow it to influence broader decision-making on the allocation of public resources, including drawing on the monitoring and evaluation mechanisms set up under the project (see Section 10). The component also was to strengthen sub-national public institutions affecting population and reproductive health, as part of the process of decentralization. Project restructuring 1 Following a mid-term review in June 2010, the project's design was simplified in a restructuring effective in May 2011. The main reasons, as stated in the Restructuring Paper, included (i) slow progress on project implementation, notably in advocacy at the sub -national level; (ii) an imbalance in the design of the project between demand-generating activities and activities meant to ensure an increased supply of reproductive health services; and (iii) excessive centralization in project management by the MOP . The restructured project included the following components : Component 1: Advocacy, Communication and Coordination (estimated cost US$ 4.4 million; actual cost US$ 4.4 million). This component remained the same, but now also included all the main themes of former Component 2, as well as the harmonization and coordination of multi -sector interventions, an activity that previously had been under Component 3. However, implementation was to be organized differently, with a stronger definition of responsibility between participating agencies . IEC as well as BCC activities were now clearly separated by implementing agency : (i) activities that were to be implemented at the community level by NGOs contracted by the MOP; (ii) activities that were multi-sectoral, implemented by MOP in accordance with its defined role in multi -sectoral advocacy and communication (advocacy towards other ministries, political parties, mayors, members of the Parliament, regional authorities, national organizations, and the media and journalists; preparation of Annual Work Plans; conducting of Annual Reviews); (iii) the strengthening of women’s economic opportunities that would complement advocacy in helping bring fertility decline in Niger, by MOP; (iv) the promotion of girls’ schooling, by the Ministry of Education; and (v) the preparation of legal documents for the protection of women and children, by the Ministry of Justice, contracted by the MOP. This Component was also to help align project activities with those of other IDA -funded projects in Niger. Component 2: Strengthening the Supply of Reproductive Health Services (estimated cost US$ 0.9 million; actual cost US$ 0.9 million). This essentially new component was to address supply -side constraints to reducing demographic pressures at the community level. It was to train nurses and midwives to replace retiring staff, as this was leaving a gap in the capacity of many regions to provide family planning services . It was to train community health workers in promoting family planning, to complement services provided in health centers and health posts . It was also to finance contraceptives to satisfy increased demand and smoothen distribution at the community level . This activity was to complement interventions delivered by a concurrent IDA health project . Component 3: Capacity Building and Monitoring and Evaluation (estimated cost US$ 7.1 million; actual cost US$ 7.1 million). This component included the same activities as the former component 4, plus training of NGOs in project and financial management, and the supervision of projects implemented by NGOs . The component also was to support the fiduciary management of the project, which remained under the responsibility of the MOP . Project restructuring 2 The project underwent a second restructuring in March 2012. It involved the reallocation of IDA proceeds to ensure that key activities would have sufficient financing to allow the project to attain its development objectives . The enhanced activities included : (i) the production of communications toolkits for increased awareness; (ii) awareness campaigns in villages by NGOs; and (iii) intensified supervision by the Ministry of Health (MOH) and its regional offices to ensure quality implementation of the IEC /BCC campaigns and better monitoring of project results . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost and financing. Project cost at appraisal was estimated at US$ 12.4 million. This was to be financed by an IDA Grant of US$ 10.0 million and a United Nations Population Fund (UNFPA) contribution of US$ 2.4 million. Total project costs remained the same after the two restructurings, and so did actual spending totals and financing structure. Borrower contribution: There was no planned or actual Borrower contribution . Dates: The project was approved on June 19, 2007. The project closed as scheduled on March 31, 2013. The project was restructured twice, on May 12, 2011 and again on March 6, 2012. The first restructuring focused on increasing efficiency in project implementation, mainly through a better definition of institutional responsibilities and a commensurate regrouping of project elements . The second restructuring involved a reallocation of funds to emphasize PDO-supportive activities. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High . Around the middle of the past decade, Niger ’s population was still growing at a rate of 3.3 percent per annum, High. reflecting a combination of declining mortality and sustained high fertility levels . The total fertility rate was estimated at 7.1 children per woman in the 2006 Demographic and Health Survey. It was among the highest in the world and a severe obstacle to Niger’s ability to reach its poverty alleviation or Millennium Development Goals . Among the main causes of high fertility was the poor status of women . Women’s knowledge of reproductive health was poor, female education levels were dismal, and discrimination hampered women ’s participation in economic and community life . Population and reproductive health issues have received increasing attention in the Government . A declaration of population policy and the establishment of a Ministry of Population and Social Reform (MOP) have served as a catalyst for the development of a strategic framework and an overall work plan for population policy . They continue to serve as driving forces for coming to terms with the population issue . The Bank's Country Assistance Strategy (CAS) has consistently included the issue of population growth as an important cross -cutting issue for increasing broad awareness of and access to family planning; the most recent CAS (FY13-FY16) sees that agenda as only partly achieved. Under its pillar for reducing vulnerability, it continues an emphasis on cross -sectoral interventions as a means to support higher national awareness and action on family planning and health issues through education and training. Thus, for both government and Bank policy and in relation to country conditions, the project objectives remain highly relevant. b. Relevance of Design: Modest . While planned project activities in the original configuration supported project objectives, the design did not sufficiently recognize potential limitations to (i) the capacity of reproductive health services to respond to increased demand generated by implementation of the project; (ii) a recently established MOP in building its capacity to coordinate a multi-sector strategy; and (iii) launching women’s empowerment activities in the absence of an appropriate strategy. These shortcomings would subsequently contribute to a mid -term restructuring of the project . The results framework did not make a clear distinction regarding which activities contributed to which objective . In part, this may have been caused by the two project objectives being interdependent, with achievement of the first objective likely to influence results on the second . 4. Achievement of Objectives (Efficacy): The project included an overarching objective : “to strengthen the Recipient ’s capacity to address its demographic challenges, � and two intermediate objectives. Efficacy is assessed on achievement of the overarching objective, Efficacy is rated modest . Outputs Annual work plans for a five-year period were developed to implement Niger ’s strategic framework for coordinating, monitoring, and evaluating all population -related activities, including performance Indicators . Sub-national coordination units to address population and reproductive health issues were established . An M&E system including indicators was developed . A database was established for its use and is available for all sub-sectors and regions. Capacity building in the form of training in the coordination of multi -sector interventions and implementation of annual work plans was provided to MOP staff . Semi-annual national reviews have been introduced to monitor the harmonization of population and reproductive health activities in key sectors . A steering committee has been set up to facilitate coordination between sectors and with donors . 95 percent of the population over the age of 15 was sensitized about population and reproductive issues through IEC and BCC campaigns facilitated by NGOs, provided to the general population and, in particular, to politicians, the media, religious leaders, women’s groups, and youth organizations at central, regional and local levels . 682 health personnel, compared to a target of 1000, and 270 community health workers, compared to a target of 1000, received training on family planning and reproductive health . Two legal texts and measures to increase the minimum age of marriage were prepared but not passed by Parliament. A national gender policy was adopted and is under implementation . 600 women’s groups received training in business management . 15 sensitization campaigns on women ’s rights and birth spacing were implemented (three per year). Outcomes, based on DHS data: The total fertility rate increased during the project period from 7.1 in 2006 to 7.6 in 2012. Births per 1000 women age 15-49 over the preceding 36 months increased from 199 in 2006 to 206 in 2012, against a target of reduction to 180. Outcomes based on the ICR (including information from the DHS): The percentage of women expressing a desire to wait for two years before having their next child increased from 49% in 2006 to 51% in 2012, not meeting the project target of 55%. The proportion of women ages 15 to 49 who know at least one contraceptive prevalence method increased from 72 percent to 90 percent, meeting the target. Prevalence of modern contraceptive use among women in union age 20-24 years increased from 4.4 percent to 8.2 percent, compared to a target of 7 percent. The prevalence of modern contraceptive use among women ages 15 to 49 increased from 5 percent to 8.3 percent. The percentage of children 0-5 months exclusively breastfed rose from 13.5 percent to 23 percent, compared to a target of 20 percent. While women/girls were targeted for education /awareness campaigns, and women for income -generating activities, there is no information provided in the ICR beyond the fact that programs were completed . Still, girls’ enrollment in the higher basic cycle increased from 11 percent at the outset of the project to 18 percent at its completion. The median age of marriage among women rose slightly, from 15.5 years to 15.7 years. In terms of "capacity", the ICR (page 15) notes that it remained weak at the end of the project period . In terms of the objective of addressing demographic challenges, results for key data points - birth and fertility rates - deteriorated. Progress along some knowledge and behavioral indicators does not appear to have translated into effective addressing of the demographic challenge of high fertility and birth rates . 5. Efficiency: No estimates of rates of return for the project were made at appraisal or in the ICR . Instead, the PAD contains qualitative analysis based on the economic implications of population growth for poverty and economic well -being. The ICR did undertake a cost -effectiveness analysis of the direct health benefits arising from Component 1 (awareness-raising about reproductive health behavior ) and revised Component 2 (increased provision of reproductive health services ). The analysis took a multi-dimensional approach to efficiency, looking at allocative, productive, implementation, and technical efficiency . Allocative efficiency reflected the use of 44 percent of project resources to raise awareness of population and reproductive health issues . Productive efficiency was reflected in the focus on cost-effective ways of delivering key messages . Implementation efficiency focused on cost -effective means of maintaining institutional functioning . The analysis concludes that the cost per disability -adjusted life year (DALY) gained is similar to that for Sub-Saharan Africa as a whole. Efficiency is rated substantial . ERR)/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: high as the objectives addressed a central constraint to improving the well -being The relevance of the PDO is rated high, of the Niger population, one that is addressed both in Government and Bank country strategies . The relevance of design is rated modest , reflecting a confusing results framework and the need for adjustments to the project structure during implementation. Efficacy is rated modest , reflecting weak outcomes in institution building as well as demographic indicators. Efficiency is rated substantial , reflecting cost-effectiveness in the use of Bank funds . Taken together, these amount to significant shortcomings, resulting in an Outcome rating of Moderately Unsatisfactory. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to achieving sustainable development outcomes from the project hinges on government commitment and in particular the ability of the MOP to continue to coordinate a nationwide multi -sector population program, and on the Government maintaining behavior change programs . In the absence of strong indications of good outcomes of institution building in the MOP, its role as a driving force for the implementation of population policies seem uncertain, although this may be facilitated by the stakeholder - and donor-inclusive approach adopted by and reinforced during implementation of the project. There are indications that behavior change programs may be working, judging from the results achieved under the second project objective, but their sustainability may be in question; the ICR (page 22) signals declining Government budget allocations to the MOP . The ICR also draws attention to institutional weaknesses in failing to link women ’s activities to the behavior change programs, and to a loss of fiduciary and technical staff at the end of the project . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project supported initiatives that were highly relevant to the Government ’s development priorities and the Bank's Country Assistance Strategy . The design was informed by prior sectoral analysis, best practices in the design of population interventions, and the experience of an earlier population project . Preparation included a high degree of stakeholder and donor involvement . Critical risks were defined where mitigating measures leaned heavily on buy-in by stakeholders. In retrospect, some of the mitigating measures appear insufficient and buy -in by stakeholders more challenging than anticipated . The design had other weaknesses, notably in the unclear articulation of responsibilities between participating agencies, including centralization of management in a recently established ministry, MOP. While the intention may have been to build up capacity in the MOP and establish its role as a coordinator of population activities, this may have been premature, and would in practice underlie the slow progress in project implementation . Imbalance in the demand and supply of maternal and child health services, notably bottlenecks on the supply side, would also affect project implementation . Finally, the results framework that set out the project ’s logic was weak. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Supervision missions were carried out on a regular basis and intensified with the introduction of a field -based task team leader in July of 2009. Detailed review of financial management was carried out during supervision . Implementation issues arose early on, partly influenced by political turmoil and difficult communications with the Minister of Population. This may explain why major design and implementation issues were tackled successfully only in 2011, once a more amenable government authority was in place . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: While the Government appears to have been committed to project development objectives – they formed part of a national strategy – some of the problems that arose during implementation could have been mitigated by more forceful interventions by the Government . The role of the MOP as a coordinator of population initiatives appears to have been left somewhat ambiguous, and its profile might have been heightened by a more forceful government definition of its responsibilities . Only with the belated introduction of a steering committee and a re-clarification of ministerial roles was this better concretized . Likewise, the lagging implementation performance of women’s empowerment initiatives might have been avoided with stronger Borrower commitment . At the same time, it should be noted that the political situation was volatile, including a year -long political crisis in 2009 and an attempted coup in 2010, making timely interventions difficult . Still, the project was implemented without extensions, and revisions to project design were supported . Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: MOP was the main implementing agency. It was a new ministry, recently established . Its insistence on managing the project centrally with little delegation of responsibilities to technical ministries, together with its reluctance to contract with NGOs, contributed to delays in project implementation and, ultimately, to both project restructurings . Once these issues were addressed following the mid -term review and the two restructurings, performance improved. A national multi-sectoral M&E framework was developed and put into use . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Monitoring and evaluation design for the project included outcome and process indicators to monitor project implementation. The project included the following key outcome indicators to have been achieved at the end of the project: 1. Annual Work Plans (AWPs) would be designed, including Monitoring and Evaluation (M&E) indicators, adopted through consultations (annual reviews) with stakeholders and donors, and implemented under monthly supervision of the Ministry of Population and Social Reform (MOP). 2. 85 percent of the population over the age of 15 has been sensitized on population and reproductive health issues. 3. The median age at marriage among women aged 25-49 has increased from 15.5 years to 16.5 years. 4. The percentage of children ages 0-5 months exclusively breast-fed has increased from 13.5 percent to 20 percent. 5. Prevalence of modern contraceptive use among women in union aged 20-24 has increased from 4.4 percent to 7 percent. Intermediate outcome indicators were specified to allow monitoring of project implementation . Baselines for outcome indicators drew on data extracted from the results of a 2006 Demographic and Health Survey (DHS) and focus group discussions. Project elements (under Component 3) would support the development of a national multi -sector M&E system, with a population database centralizing M&E data and spatial data from sectors and regions . An action plan, guides for indicators, and training tools and manuals were to be developed . b. M&E Implementation: Baseline data collection was carried out, and the national M&E framework was implemented with information collected and disseminated on a quarterly basis . These data formed the basis for annual work programs, including indicators, for implementing the population program . At the decentralized level, implementation was affected by constraints on skilled technical staff and working tools . With the May 2011 project restructuring that revamped the components, key outcome indicators and intermediate indicators were adjusted. The new set of key outcome indicators included the following : 1. Number of annual work programs (including indicators) prepared, adopted and implemented each year . 2. Proportion of women ages 15 to 49 who know at least one contraceptive prevalence method . 3. Contraceptive prevalence rate (modern methods for women ages 15 to 49). c. M&E Utilization: As noted above, data were used to develop annual work programs for the population program, and to monitor progress and assess outcomes of the project . That said, it seems that the information that was being collected for M&E purposes was not being sufficiently drawn on to provide a more comprehensive assessment of the achievement of the objectives. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: No safeguard policies were triggered . b. Fiduciary Compliance: Financial management. The project implementation team in MOP was strengthened with the appointment of an accountant, and all fiduciary staff were trained in Bank financial management and disbursement procedures . No issues arose in financial management during project implementation . Project audits were performed on time and were unqualified. Procurement. The first few years of project implementation included challenges related to getting procurement arrangements underway: the implementation team’s two procurement specialists resigned, one because of conflicts with the MOP, and the other because of a conflict of interest (he was working for another project ). Arrangements concerning an intended UNFPA-funded communications specialist were only finalized in 2010. As a consequence, the MOP had periods when needed technical expertise was not in place . Moreover, minor irregularities were found in the purchase of office supplies and cleaning goods, leading to the Bank sending a letter to the authorities in June 2009 requesting full procurement compliance . Consequently, no further procurement issues arose . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The relevance of the PDO is rated high Satisfactory Unsatisfactory , as the objectives addressed a central constraint to improving the well-being of the Niger population, one that is addressed both in Government and Bank country strategies. The relevance of design is rated modest , reflecting a confusing results framework and the need for adjustments to the project structure during implementation. Efficacy is rated modest , reflecting weak outcomes in institution building as well as demographic indicators. Efficiency is rated substantial , reflecting cost-effectiveness in the use of Bank funds. Taken together, these amount to significant shortcomings, resulting in an Outcome rating of Moderately Unsatisfactory. Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Risk mitigation measures appear Satisfactory Unsatisfactory insufficient and buy-in by stakeholders more challenging than anticipated . There was unclear articulation of responsibilities between participating agencies, including centralization of management in a recently established ministry, MOP. Imbalance in the demand and supply of maternal and child health services, notably bottlenecks on the supply side, would also affect project implementation. Finally, the results framework that set out the project’s logic was weak. Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: These lessons are drawn from the ICR (pp. 24-25): Evaluator observations are included in bullet points two and three. Bullet point 4 is introduced by the evaluator and the panel reviewer . Addressing the population agenda needs to be anchored in the highest level of government .A multi-sectoral population agenda cannot be the sole responsibility of a single sector ministry . Rather, it should be managed through a high-level representative multi-sectoral body or through an entity such as a Prime Minister’s office. IEC /BCC campaigns may be more successful in While there are no data to confirm it, the ICR notes that IEC/ raising awareness when they are linked to women ’s empowerment programs that engage women more directly in transmitting behavioral change messages . Empowerment programs raise social status and may create economic opportunities, strengthening the demonstration effect that empowered women have on their surroundings To be effective, IEC /BCC design and implementation need to take into account local culture, needs, and issues, and follow -up and M&E structured to detect locally sensitive issues . In this project, a particular effort was made to tailor messages to Islamic sensibilities on raising the age of marriage for girls and boys, and on girls’ schooling. Again, there are no data to measure the impact of such culturally sensitive messages . There can be considerable value in drawing on outside data sources for setting baselines, developing milestones, and measuring outcomes, as a complement to information that is being collected under the project . This project was framed by demographic and health surveys that provided baseline data and data for 2012, towards the closing of the project . Neither appears to have been directly generated by the project intervention. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of the evidence provided in most key areas provided the necessary basis for developing the ICR Review . Notable exceptions were formed by the discussion on Borrower performance and Fiduciary Compliance; these are short and non-committal, especially on procurement, where nonetheless problems were encountered .. The analysis in the ICR is sufficient to underpin the evidence and understand its ratings . Results orientation could have been stronger, but still, a commendable effort is made considering the weaknesses in the project ’s results framework. The document is internally consistent, and it follows OPCS guidelines . The lessons draw on the evidence and analysis in the text. At 25 pages, the main text could be more concise . a.Quality of ICR Rating : Satisfactory