IEG Report Number: ICRR14749 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 05/11/2015 Country: Yemen, Republic of Project ID: P104946 Appraisal Actual Project Name: Yemen Safe Project Costs (US$M): 7.57 4.34 Motherhood Voucher Program L/C Number: Loan/Credit (US$M): 6.23 3.56 Sector Board: Health, Nutrition and Cofinancing (US$M): 0 0 Population Cofinanciers: Board Approval Date : 06/02/2008 Closing Date: 06/30/2012 02/28/2014 Sector(s): Health (100%) Theme(s): Population and reproductive health (50%); Social safety nets (25%); Other human development (25%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Judith Hahn Gaubatz Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Grant Agreement (page 6), the project objectives were as follows:  To provide quality maternal care to about 40,000 eligible women in targeted districts in Sana 'a; and  To design and implement a model of maternal care which demonstrates how Yemeni public policy on maternal and child care can be effectively integrated with private health provision . The project objectives were articulated similarly in the Project Commitment Document (page 13), although a specific target was not included:  To provide quality maternal care to eligible women in targeted districts in Sana 'a; and  To design and implement a model of care that provides key lessons and serves as a demonstration effect of use of innovative models of care that integrate Yemeni public policy on maternal and child care and private health provision in Yemen . In March 2011, the target for the first objective was formally revised from 40,000 women to 30,000 women. This was due to a re-assessment of unit costs for maternal and neonatal health services, such that the higher costs led to reduced coverage of services. In May 2012, the target was again revised from 30,000 women to 15,000 women. This was due to the withdrawal of one of the two health service providers from the project, such that the remaining service provider would only be able to cover services for 15,000 women. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval: 03/22/2011 c. Components: The project was a performance/output-based grant financed by the multi-donor Global Partnership on Output-Based Aid (GPOBA) and supervised by the Bank. The project had several aims: to explore a role for the private sector in health services provision; to implement a subsidy program in a social sector; and to implement an innovative model for the delivery of maternal health services. The Bank served as the project supervisor (including fiduciary management), while the grant funds were disbursed to two private sector hospitals (Saudi German Hospital (SGH) and University of Science and Technology Hospital (USTH)) and one national non-governmental organization (SOUL). 1. Project preparation and start-up (Appraisal: US$ 0.04 million; Actual: US$ 0.04 million): This component was to finance the preparatory work for project implementation. Activities included: baseline study by SOUL to document socio-demographic and service-related factors in the targeted districts, as well as perceptions of safe motherhood among potential beneficiaries; and provision of equipment to create a database of beneficiaries (including biometric server and eye/fingerprint equipment). 2. Establishment of satellite clinics (Appraisal: US$ 0.46 million; Actual: US$ 0.17 million): This component was to provide financing to the two private sector hospitals to set up twelve satellite clinics in the targeted districts. The satellite clinics would provide basic maternal health services through midwives, while emergency or complicated obstetric cases would be referred to the two hospitals. 3. Service delivery (Appraisal: US$ 6.1 million; Actual: US$ 2.6 million): This component was to finance subsidies to the two private hospitals (SGH and USTH) for the provision of a defined "Mother-Baby package," including antenatal care, skilled birth attendance, postnatal care, complicated care services, and family planning. The package price was estimated at US$ 150/beneficiary, taking into account higher costs for approximately 15 percent of cases requiring Caesarean sections/emergency care. A user fee/co-payment of US$ 15 was to be required from the beneficiaries, based on the current practice of Yemeni families paying US$ 10-20 (out of pocket) to a birth attendant on the day of delivery. The hospitals were to be reimbursed on a quarterly basis through the project's designated Fiduciary Agent, according to the number and level of services provided (i.e achievement of targets for % of births assisted by skilled worker, neonatal mortality rate, accuracy and accessibility of patient medical records, % of patients that complete at least four antenatal visits). 4. Community outreach administrative cost (Appraisal: US$ 0.35 million; Actual: US$ 0.55 million): This component was to finance the activities of the local NGO, SOUL, to identify, evaluate and document eligible beneficiaries. Activities included maintaining a database of beneficiaries (including digital eye/finger prints) that would be matched with the recorded data at each service visit. 5. Education and awareness campaigns (Appraisal: US$ 0.19 million; Actual: US$ 0.24 million)): This component was to finance outreach activities of SOUL to raise public awareness of and demand for quality maternal care. The project cost also included fees for the Independent Verification Expert (US$ 90,000) , which would verify service delivery outputs by the two hospitals, and a Fiduciary Agent (US$ 80,000), which would manage the flow of funds in the project. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost  The actual project cost was US$ 4.34 million, compared to the appraised cost of US$ 7.57 million. The shortfall was due to the withdrawal of one of the service providers (SGH) from the project, and subsequent downward revision of project outputs.  Component 2 (Satellite clinics) disbursed significantly less than expected due to shortfalls in setting up clinics (seven out of twelve).  Component 3 (Service delivery) disbursed significantly less than expected due to the withdrawal of SGH as a service provider. Financing  The project was financed by a grant from the GPOBA Trust Fund. The actual amount disbursed was US$ 3.56 million, compared to the appraised amount of US$ 6.23 million.  The two private hospitals were expected to contribute US$ 0.28 million towards the establishment of satellite clinics. The actual amount provided was US$ 0.17 million. The shortfall was primarily due to the withdrawal of one of the private hospitals (SGH) from the project.  User fees of US$ 0.61 million were expected to be generated from the beneficiaries utilizing services. The actual amount generated was US$ 0.25 million. The shortfall was due to the significantly lower number of beneficiaries served by the project.  US$ 2.3 million of the grant was cancelled in May 2012, as SGH withdrew from the project. Borrower contribution  There was no planned Borrower contribution, as the grant recipients were two private sector hospitals and one non-governmental organization. Dates  March 2011: Following up on findings from the Mid-Term Review (2010), which determined that the unit cost per beneficiary was higher than expected, a revised price list for the defined package of services was drawn up and it was concluded that the allocated grant amount would only be able to cover 30,000 women. Therefore, the project was formally restructured to revise the target number of beneficiaries from 40,000 to 30,000.  July 2011: Following a period of political turmoil and a deteriorating security situation, the Bank officially suspended disbursements to all Bank projects in the country. The suspension was lifted in January 2012.  May 2012: One of the two private hospitals (SGH) participating in the project withdrew, necessitating a further downward revision of targets. The project was again restructured to revise the target number of beneficiaries from 30,000 to 15,000. The project closing date was extended from June 2012 to February 2014 to allow completion of activities. US$ 2.3 million of the grant was cancelled due to withdrawal of SGH. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of the objectives is rated High. At the time of project appraisal, the maternal mortality rate was considered high at 570/100,000 live births (2000) and the proportion of births attended by skilled birth attendants was considered low at 27% (2003). Although maternal mortality had steadily declined over the past decade, it remained high towards project closing, at 148/100,000 live births (2013). Thus, the project objective to provide quality maternal care is considered highly relevant both at appraisal and at closing. In addition, given the low quality of maternal services available in the public sector, exploring innovative partnership arrangements with the private health sector is also a substantially relevant objective. These objectives were also consistent with the Bank's Country Assistance Strategy (FY2010-13), which identifies the fostering of human and social development, including through improved delivery of reproductive health services, as a key strategic objective. The country's development agenda (as articulated through a series of Development Plans for Poverty Reduction) also prioritizes strengthening the quality of health services. b. Relevance of Design: Relevance of the design is rated Substantial. Project interventions were likely to lead to the intended outcomes. The defined "Mother-Baby package" was technically sound and based on recommendations from the World Health Organization. The project focused on nine underserved districts in Sana'a, in order to reach poor women whose access to quality maternal care was constrained due to costs and lack of knowledge. Although income criteria (household income of less than US$ 2/day) was established to ensure that the poorest women were targeted, the ICR (page 16) reports that planned verification of income levels was not sufficiently rigorous and that the required co-payment (US$ 15) still prohibited some women from accessing the services. The project design was initially conceived as a voucher scheme; however, this was subsequently rejected as adding unnecessary complexity and costs to the overall project. The project design also included information and education campaigns about the importance of quality maternal care to increase demand for maternal health services, given the prevailing health-seeking behavior and attitudes. Finally, the project featured a performance/output-based approach, and there was corresponding emphasis on accountability for results, including clear mechanisms for verifying outputs and monitoring results. 4. Achievement of Objectives (Efficacy): To provide quality maternal care to eligible women in targeted districts Achievement of original target of 40,000 women: Modest Achievement of first revised target of 30,000 women: Modest Achievement of second revised target of 15,000 women: Substantial Outputs  Conducting of baseline study to assess health-seeking behavior among targeted women.  Conducting of health education and public awareness campaigns to increase demand for quality maternal care. A total of 23,429 women attended the outreach sessions.  Enrollment of 16,878 targeted women in the program, including maintenance of a database of the eligible beneficiaries. This fell short of the target of 19,053.  Establishment of 7 satellite clinics providing basic maternal health care services. This fell short of the target of 12 clinics.  Provision of subsidies to two private hospitals for delivering a defined "Mother-Baby package" to targeted poor women. Outcomes Intermediate Outcomes  10,316 of the project beneficiaries, or 67%, completed at least four antenatal care visits. This fell short of the target of 100%.  12,072 births, or 78%, were assisted by skilled attendants. This fell short of the target of 100%.  10,407 of the project beneficiaries, or 67%, received at least one post-natal care visit. This fell short of the target of 100%.  1,172 project beneficiaries with potential or acute obstetric complications were referred to the hospital. Maternal Health Outcomes  16,141 safe deliveries were achieved. This fell short of the original and (first) revised targets, but surpassed the (second) revised target.  Only one case of maternal death was recorded amongst the deliveries reported for the project. To design and implement a model of maternal care which demonstrates how Yemeni public policy on maternal and child care can be effectively integrated with private health provision Achievement is rated Substantial. Although there were no measurable indicators defined for this objective, the ICR (page 19) provides the following as evidence of achievement: Outputs  Evaluation report (by an Independent Verification Expert) on "all aspects of the project including an analysis of the service delivery model."  Review (by an external consulting firm) on options for a suitable service delivery model for maternal health services in Yemen.  Workshop presenting the performance-based financing experience of the project. According to the ICR (page 14), the workshop included discussions with the public and private sectors to explore service delivery options based on this approach. A second workshop, which included the participation of the Ministries of Health and Finance, discussed the concept of performance-based financing. Outcomes  The Bank has prepared a Maternal and Newborn Voucher Project (effective in August 2014), based on lessons from this Safe Motherhood Project as well as an ongoing KfW Reproductive Health Voucher project. The ICR (page 14) states that this project demonstrated that "it was important to work with a national quasi-governmental organization to be the payer for the scheme contracting public and private (profit and non-profit) health facilities." The ICR (page 19) also suggests that the newly effective Bank project reflects the government's "buy-in" to a results-focused approach involving partnerships with private service providers. However, it is notable that the financial sustainability of such a model was not clearly demonstrated through this project. 5. Efficiency: Efficiency is rated Modest due to lack of sufficient information to verify efficient use of project resources. The Project Commitment Document (page 17) references well-established research on the cost-effectiveness of antenatal and skilled delivery interventions. It also states that the unit cost/price package agreed upon with the two private hospitals was "deemed reasonable based on the comprehensiveness of the services to be provided in the Program and the quality as compared to existing services and pricing." However, a detailed economic analysis of the project was not undertaken due to absence of detailed pricing and household spending data. Similarly, the ICR cites the lack of available data to conduct an economic analysis at project closing. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Project under original targets : Moderately Unsatisfactory Relevance of the project objective is rated High and relevance of the project design is rated Substantial. Achievement of the objective to provide quality maternal care to eligible women in the targeted districts is rated Modest, due to significant shortfalls in achieving targets. Achievement of the objective to demonstrate a model of maternal care that integrated public policy and private sector provision is rated Substantial. Efficiency is rated Modest due to lack of sufficient information to verify efficient use of project resources. Project under first revised target : Moderately Unsatisfactory Ratings on relevance and efficiency are the same as above. Achievement of the objective to provide quality maternal care to eligible women in the targeted districts, under the first revised target, remains Modest, due to significant shortfalls in achieving targets. Project under second revised target : Moderately Satisfactory Ratings are the same as above, except that achievement of the objective to provide quality maternal care to eligible women in the targeted districts is rated Substantial, due to evidence of achieved outcomes and targets. According to OPCS/IEG harmonized guidelines, the overall outcome rating of a restructured project is determined by weighting the proportion of the grant that disbursed during each restructured period. Under the original target, the project grant disbursed US$ 0.57 million out of US$ 3.56 million, or 16%. Under the first revised target, the project grant disbursed US$ 1.1 million out of US$ 3.56 million, or 31%. Under the second revised target, the project grant disbursed US$ 1.9 million out of US$ 3.56 million, or 53%. Therefore, the weighted outcome rating is Moderately Satisfactory . a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR (page 21) reports that as of project closing, "no part of the [project] intervention is being continued after project closing (no enrollment of poor women or provision of subsidized maternal care to them by private service providers) because the grant subsidy is no longer available and the financial sustainability model envisioned was not realized." The Bank has started implementing a new Maternal and Newborn Voucher Project although, as noted in the ICR (page 15), the time lag and lack of direct linkages between the two projects (i.e. the satellite clinics) make it unlikely that it will help sustain project achievements. The continued volatility in the security and political situation also poses high risk to sustaining service delivery. a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: a. Quality at entry: The project was prepared by the GPOBA, with support on fiduciary aspects from the IFC and IDA. It was intended to be an innovative partnership between IDA and the IFC, although there was lack of clarity at times on documentation and reporting requirements, and the disallowance of pre-financing also led to significant cash flow problems for implementing agencies. Project preparation included a detailed appraisal and costing analysis (included a market study for average treatment costs) and institutional assessments of the three main implementing agencies. However, although the two private hospitals were determined to have adequate capacity at the hospital level, capacity to deliver services at the clinic level was unclear due to lack of prior experience. This risk of inadequate staff capacity was underestimated, which, alongside the lack of clear understanding of the project concept by the two hospitals (including the handling of complicated obstetric cases), led to significant implementation challenges. As reported in the ICR (page 21), "the service providers did not comprehend fully the nature of the innovative model and the capacity requirements of implementing the project, particularly setting up and operating the satellite clinics." Given the performance-based nature of the project, the M&E framework was clearly delineated to carefully monitor outputs. However, while the project was initially designed as a smaller-scale pilot during the first year, followed by expanded operations during subsequent years, it is not clear why this approach was not taken. Quality-at-Entry Rating: Moderately Unsatisfactory b. Quality of supervision: The supervision team (IFC and IDA) made critical adjustments to procedures and fiduciary requirements to address slow uptake of services and cash flow problems; these adjustments included allowing advances, broadening performance indicators for SOUL, and expanding eligibility criteria to include all poor pregnant women (not just those in their first trimester). The supervision team repeatedly communicated the service model parameters to the service providers and responded to their request to revisit the package pricing costs, which led to formal project restructurings and modification of targets. The ICR (page 22) reports that the second restructuring took place later than anticipated due to difficulties in finalizing the withdrawal of SGH from the project and subsequent legal issues (i.e. the three grant recipients bound by one legal agreement). Implementation Supervision Reports provided realistic ratings on project progress, although according to the ICR (page 23) there was limited discussion or justification of ratings, no issues were brought to management attention, and "there is no documented evidence of the Bank team following up on issues." There were no problems reported in fiduciary management or safeguards compliance, although full documentation of the latter could not be located. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government was not a grant recipient; however, a formal endorsement was sought in support of the project concept. The Project Commitment Document (page 5) reports that endorsement from the Ministry of Public Health and Population was "pending" and expected the week of June 4, 2007. The ICR does not indicate whether this endorsement was formally issued. Government Performance Rating Not Applicable b. Implementing Agency Performance: There were three main implementing agencies for this project: Saudi German Hospital (SGH), University of Science and Technology Hospital (USTH), and SOUL. As mentioned previously, while the capacity and commitment of the private hospitals were appraised during project preparation, their understanding of the service model was incomplete and their capacity at the clinical level was unknown. Capacity risks materialized almost at the outset of project implementation; the ICR (page 7) cites unsatisfactory performance of SGH including "lack of business standards and practices and poor understanding of the design and implementation of the project." The hospitals also lacked experience with the Bank's fiduciary management requirements, and thus there were disbursement delays that affected cash flow. The performance of SGH is considered unsatisfactory. The ICR (page 8) reports that SGH had problems such as irregular availability of staff (including female staff) at clinics, and alleged instances of malpractice "resulting in rumors that discouraged beneficiaries from participating in the project." Additionally, while SGH was responsible for preparing training content to identify and enroll beneficiaries, there were "a number of wrong messages in the training package that focused on informing the beneficiaries that they would be charged for complicated deliveries and Caesarean sections." Finally, according to the Restructuring Paper (2012, page 3-4), the escalating unrest in the country led to the suspension of Bank disbursements from July 2011 - January 2012, and contributed to significant cash flow constraints for SGH. The ICR (page 7) also notes that SGH was already experiencing the negative impact of the global financial crisis on its cash flow; due to these difficulties, SGH's Board "decided to freeze their participation in the project. The performance of USTH is considered satisfactory. Although USTH had initial difficulties in implementation due to lack of experience with the output-based disbursement model, it was overall effective in delivering services as planned. The performance of SOUL is also considered satisfactory. In response to initial low enrollment rates (due to cultural reluctance and mistrust of the "free" services being offered in private clinics), SOUL implemented a focused marketing strategy and enrollment substantially increased. The ICR (page 24) notes that SOUL effectively implemented its activities, sometimes going beyond its terms of reference, and met its performance targets regularly. The implementing agencies faced particular difficulties during the political crisis in 2011, during which all disbursements were suspended by the Bank. The ICR (page 10) reports that although SOUL and USTH continued to offer services, utilization rates were low due to the volatile security situation, continuous disruption of electricity, high prices for fuel and supplies, attacks on the satellite clinics and ambulances, disruptions in cash flow, and difficulties in conducting verification. However, although monitoring reports from the Independent Verification Expert state that waste management was carried out according to standards, the ICR team could not access detailed information to confirm satisfactory compliance with environmental safeguards. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The project design centered on a performance/output-based approach in which disbursements were triggered by project indicators; therefore, the M&E design featured a detailed, specific monitoring process and data collection/verification arrangements. This included an Independent Verification Expert, who would conduct a quality performance review for each six-month period. In addition to the indicators that were used to trigger disbursements (i.e. patient volume, number of services delivered), the project M&E framework also included indicators to assess broader impact of the project activities, such as proportion of targeted beneficiaries utilizing services and proportion of births assisted by skilled attendant. Evaluative activities were to include assessments of the quality of clinic and hospital service (based on random visits and patient and family satisfaction surveys) and monitoring for possible incidences of fraud. b. M&E Implementation: SOUL effectively maintained a comprehensive database on project beneficiaries, although verification through eye/fingerprinting was never fully implemented due to issues of cost, complexity, and technological difficulties in transferring data to hospitals. Instead, physical photo-identification cards were used for verification. The Independent Verification Expert was overall effective in monitoring and verifying outputs; however, the ICR (page 12) notes that in its final report, "there appear to be several inconsistencies in data, including misinterpretation of definitions, e.g. the number of women enrolled has been interpreted as meeting the final project target." The required project completion report from USTH had not yet been received as of the writing of the ICR. It is also not clear whether the planned evaluative activities were carried out. c. M&E Utilization: As embedded in the project approach, the monitoring indicators were used to trigger disbursements. Other uses of the M&E information are not reported. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: The project was classified as a Category "B" project and therefore an environment and social assessment was prepared. No other social safeguard policies were triggered. The ICR (page 13) reports that compliance with environmental guidelines was to be monitored by the Independent Verification Expert and reported in annual monitoring reports from each hospital. While the reports apparently state that medical and non-medical waste management was maintained according to standards, the ICR team reports that it did not have access to these reports and therefore could not confirm satisfactory compliance. b. Fiduciary Compliance: Financial management: The ICR (page 14) reports that the implementing agencies had acceptable financial management arrangements, financial reports were submitted regularly and were satisfactory, and audit reports were satisfactory. Although there were some initial disbursement delays due to lack of experience with the output-based disbursement model and also the disallowance of pre-financing which contributed to cash flow problems, these issues were largely resolved by project closing. Procurement: The ICR (page 14) reports that procurement carried out by SOUL and USTH was generally satisfactory, and there were no major problems reported. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Moderately Satisfactory Satisfactory Risk to Development High High Outcome: Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons drawn from the ICR, adapted by IEG:  Strong project preparation includes clear and effective communication with the implementation parties, particularly if it is an innovative project approach. In the case of this project, there was a lack of clear understanding by the two hospitals about the overall project concept, the output-based disbursement approach, and the handling of complicated obstetric cases, all of which contributed to significant shortcomings in implementation.  Government engagement is critical for achieving an objective that aims to have a demonstration effect for future public-private approaches. In the case of this project, the government was not a formal participant, and minimal efforts to engage the government, Ministry of Health, or Ministry of Finance prevented a greater impact on public policy. However, post-project workshops included the participation of the government, and there are indication of increased government buy-in to the performance-based approach using public-private partnerships. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of data and analysis is overall satisfactory, providing clear evidence of project outcomes. Lessons are well articulated and clearly draw from the project experience. There is limited information on safeguards compliance, although the ICR acknowledges this shortcoming and attributes it to lack of proper documentation/access to information. a.Quality of ICR Rating : Satisfactory