ICRR 13926 Report Number : ICRR13926 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/19/2012 Country : India Project ID : P050655 Appraisal Actual Project Name : India: Rajasthan US$M ): Project Costs (US$M): 105.98 95.32 Health Systems Development Project L/C Number : C3867 Loan/ US$M): Loan /Credit (US$M): 89.0 75.7 Sector Board : Health, Nutrition and Cofinancing (US$M): US$M ): 0 0 Population Cofinanciers : Board Approval Date : 03/11/2004 Closing Date : 09/30/2009 09/30/2011 Sector (s): Health (60%); Other social services (10%); Compulsory health finance (10%); Sub-national government administration (10%); Non-compulsory health finance (10%) Theme (s): Health system performance (40% - P); Population and reproductive health (20% - S); Other communicable diseases (20% - S); Child health (20% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judith Hahn Gaubatz Judyth L. Twigg Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: According to the Development Credit Agreement (DCA, page 17), "the objective of the Project is to assist Rajasthan in improving the health status of its population, in particular the poor and underserved population, through (i) providing such populations with equitable and greater access to health care; and ii) improving the (ii) effectiveness of health care through institutional development and increasing the quality of health care ." According to the Project Appraisal Document (PAD, page 2), "the project would assist Rajasthan in improving the status of its population, in particular the poor and underserved population . Specifically, the project would have the following two project development objectives : (i) to increase access of the poor [i.e. below the poverty line (BPL] and underserved populations to health care; and (ii) to improve the effectiveness of health care through institutional development and increase the quality of health care . As the DCA, which is a legally binding agreement, includes an additional explicit objective "to improve the health status of its population, in particular the poor and underserved population, " it is used as the basis for this review . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components: 1: Project Management, Policy Development, and Capacity -Building (Appraisal: US$19.32 million; Actual: US$18.44 million): This component aimed to establish a Strategic Planning Cell to build institutional capacity for health policy development and planning, as well as to support overall project implementation by the Project Management Unit (PMU). Activities included: building capacity for public-private partnerships; developing a regulatory framework for quality assurance; building health management information systems; and providing clinical and management training. 2: Development of Primary and Secondary Health Care Services in the Public Sector (Appraisal: US$50.41 million; Actual: US$47.67 million): This component aimed to strengthen health care facilities, specifically : 28 district hospitals, 23 sub-district hospitals, 185 community health centers, and 2 primary health centers. These facilities were selected on the basis of a scoring system that took account of the strategic location, access by disadvantaged groups, utilization rates, remoteness from district hospitals, and norms for beds at the first referral level . Activities included: renovation and upgrading of facilities; provision of equipment and supplies (including information and education campaign (IEC) materials); and development of clinical protocols, M&E systems, and referral mechanisms . 3: Health Care Innovations for the Disadvantaged (Appraisal: US$31.92 million; Actual: US$29.21 million): This component aimed to improve access (geographic, financial and social ) of disadvantaged populations, specifically : scheduled tribe (ST) and below-the-poverty line (BPL) groups. Activities included: implementation of IEC and outreach activities; support for traditional systems of care; and pilot initiatives to improve access . The project was extended for two years, with the following activities added : drug supply and logistics management system, special purpose fund for heavy duty repairs of infrastructure, pilot initiatives for improving preventive care and health promotion among disadvantaged populations, equipment for secondary level hospitals, and a hospital management information system. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: The actual project cost was US$ 95.3 million, compared to the estimated cost of US$ 105.98 million. This was due in part to favorable exchange rate fluctuations, as well as a partial credit cancellation (see below). Financing: Exchange rate fluctuations during the project period (strengthening of the US$) led to increased availability of local funds. During the project extension period, 13.8% of the credit was reallocated across categories . Allocations were increased for civil works (mainly due to increases in cost of steel, cement, and other raw materials ) and decreased for training and workshops (due to the government's decision to discontinue financing of international training). US$11 million of the credit was cancelled at project closing, due to incomplete implementation of activities (those introduced in the extension period ) as well as savings from exchange rate fluctuations . The IDA credit disbursed 97% of the revised amount. Borrower Contribution: The Borrower contributed US$19.6 million, exceeding the estimated US$17.0 million due to exchange rate fluctuations. Dates: April 2006: The results framework was revised to reflect the following changes : the overall number of indicators was reduced; the remaining indicators were revised to more measurable terms; and baseline and target values were identified. September 2009: The project closing date was extended from September 2009 to September 2011, in order to utilize an unspent credit of US$ 29 million. These funds were used to implement originally planned as well as new activities (see above). The unspent amount was due to delayed implementation of activities (especially under Component 3), changes in the exchange rate, and savings on certain procurement categories . December 2009: Target values for indicators for the first objective (increased access) were revised downward to what the Bank team considered more realistic levels (% of outpatients from BPL households : target revised from 50% to 17%; % of inpatients from BPL households : target revised from 50% to 13%; % of inpatients from ST households: target revised from 50% to 15%). The covenant on developing a regulatory framework for the private sector was also removed due to the drafting of similar legislation at the national level (which would eventually have the intended impact at the state level ). September 2011: US$11 million of the credit was cancelled, due to incomplete implementation of activities (those introduced in the extension period ) as well as savings from exchange rate fluctuations . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial . Access to and quality of health care in the state of Rajasthan continue to be critical issues, as reflected by health indicators (total fertility rate, infant mortality rate, antenatal coverage, institutional birth deliveries, immunization coverage) that lag behind the national average, especially among scheduled tribe households and poor (lowest income quintile) populations. In 2001, the national Ministry of Health and Family Welfare (MOHFW) identified Rajasthan as one of eight "Empowered Action Group" states that would benefit from targeted reforms and programs in health (on the basis of its large population and its poor health indicators ). The state's "Health Vision for Rajasthan 2025" articulated goals to improve the above -mentioned health indicators, with an accompanying health sector policy matrix identifying intermediate goals to increase financing and improve resource allocation, strengthen capacity, and increase access and equity of access . The Bank's Country Assistance Strategy (CAS) for FY2009-12, identifies increasing effectiveness of service delivery (including health) as a key pillar (with a focus on low-income states), along with "inclusive" growth (not specifically defined in the CAS ), and improvements in the sectors relating to the Millennium Development Goals. b. Relevance of Design: Substantial . Planned outputs were likely to contribute to intended outcomes for both the general and targeted populations. Activities under Component 3 (Health Care Innovations for the Disadvantaged ) specifically targeted the poor (15.3% of households, according to the 2001 census) and tribal households (12.6% of the population, according to 2001 official government data), particularly aiming to increase demand through outreach initiatives . These activities would complement supply side interventions on a broader scale, including upgrading health facilities and developing quality of care guidelines . Institutional capacity at the state level (improving policy and management of health issues) would also be strengthened to improve effectiveness of care . 4. Achievement of Objectives (Efficacy): After the project became effective, the Government of India launched several large health programs with similar objectives, which provided substantial additional resources for the health sector . These included the National Rural Health Mission (introduced in 2005), the Reproductive and Child Health II program (2006), and the Jonani Surakshi Yojana program (2005; provision of financial incentives for institutional birth deliveries ). Data reported for the two specific objectives reflect outcomes from project-supported facilities only; however, the results may be attributable in part to the interventions of the other ongoing health programs . Specific Objective : To provide equitable and greater access of the poor and underserved populations to health care . care. Substantial , based on evidence of increased utilization contained in the ICR, along with additional information provided by the project team to more strongly substantiate the outcomes . Outputs All 238 targeted facilities were upgraded /renovated (including 28 district hospitals, 23 sub-divisional hospitals, 179 CHCs and 2 BPHCs). According to the project team, 49 of the 238 facilities (located in 6 priority districts due to tribal and desert populations ) were identified for additional interventions in terms of IEC and BCC activities, health camps, pilots, and training in order to reach resident disadvantaged groups . Following the mid-term review, health camps were extended to three additional semi -desert/tribal districts based on perceived need. IEC activities, based on the development of an IEC strategy, were implemented in community health centers (CHCs), as well as in mass media. 4814 health personnel were trained in behavior change communication . Village contact drives were implemented in 100 villages surrounding newly renovated health centers in 4 districts. Antenatal care campaigns were conducted in six tribal districts to encourage ANC registration and uptake of emergency obstetric care. 1474 basic health camps were conducted in 9 districts with significant tribal and BPL populations . 131 counselors were assigned to project facilities to improve access of vulnerable groups to health care . Equity funds were provided to all project facilities to finance drugs for BPL patients . Pilot schemes were conducted for the following : health communicators to provide information about basic diseases to villages; and exemptions for the poor . Reviews were undertaken of the following : existing and pilot schemes for exemptions for the poor; IEC; and BPL medical care schemes. The results of the reviews are not reported in the ICR, although, according to the project team, routine data and final assessment reports are available on file . Outcomes The ICR provides evidence of trends in the share of inpatients and outpatients in the target groups, though achieving these targets is not necessarily indicative that the objective of increased access has been achieved . Other factors that could affect attribution– such as trends in the share of households that are BPL or changes in the poverty line itself – are not discussed. The proportion of outpatients seen at project facilities who were from BPL households increased from 8.7% in 2006 to 16.6% in 2011, falling slightly short of the target of 17.0%. Figures reported in the ICR (page 16) indicate consistent annual increases from 2006 to 2011. According to additional information provided by the project team, the proportion of the BPL population in Rajasthan fell by 9.6% between 2004-5 and 2009-10 (source: Planning Commission data). The proportion of inpatients seen at project facilities who were from BPL households increased from 8.5% in 2006 to 17.0% in 2011, surpassing the target of 13.0%. (See data on the decrease in the proportion of the BPL population in previous bullet.) The proportion of inpatients seen at project facilities who were from ST households increased from 8.3% in 2006 to 24.9% in 2011, surpassing the target of 15.0%. According to additional information provided by the project team, the proportion of the ST population in Rajasthan fell from 12.6% in 2001 to 8.2% in 2011 (source: Census data). The ICR (Annex 10, Table 13) also cites increased utilization of maternal and child health services among poor and scheduled tribe households as evidence of increased access . However, these data are drawn from the 1998/9 and 2005/6 National Family Health Surveys; while the project period was 2004-2011. The proportion of fully immunized children (age 12-23 months) in ST households increased from 13.3% to 35.0% and in the lowest quintile households from 4.8% to 14.0%. Antenatal coverage for pregnant women (at least 4 antenal care visits) increased in ST households from 9.4% to 11.6% and in the lowest quintile households from 3.7% to 7.5%. The proportion of births attended by a skilled worker increased in ST households from 23.4% to 29.8% and in the lowest income quintile households from 15.5% to 18.9%. Specific Objective : To improve the effectiveness of health care through institutional development and increase the quality of health care . Substantial , due to evidence of improvements in the quality of health care . Outputs Clinical training was conducted for 3,062 health personnel. The PAD (page 33) indicates the intended output was clinical training for 8,837 staff. Training in management, quality improvement, referrals, drugs, health care waste management and data management was conducted. All 238 targeted facilities were upgraded /renovated as planned. The Strategic Planning Cell in the state -level MOHFW was established. Hospital Systems Improvement Teams were established in all project facilities, in order to provide monthly reviews of quality of services . Guidelines were developed for the following : public-private partnership contracts; hospital management information system; personnel information system; quality of care improvements; and standard treatment protocols. All project facilities completed implementation of health care waste management plans . Pilot schemes were undertaken for consumer feedback and redress . Reviews were undertaken for the following : public expenditures in the health sector; and private sector role in health care provision. The results of the reviews are not reported in the ICR, although, according to the project team, routine data and final assessment reports are available on file . Outcomes The average number of maternal deaths per project facility (per month in 42 days of post-partum) decreased from 0.82 in 2005 to 0.27 in 2011. The average number of neonatal deaths per facility per month increased from 2.5 in 2005 to 3.57 in 2011. The ICR (page 19) suggests that this may be due to the sharp increase in institutional deliveries over the same time period. (Neonatal deaths were recorded for children born at the facilities, whether the deaths occurred at the facility or at home.) The percentage of institutional deliveries that were Caesarean sections decreased from 9.2 in 2005 to 4.23 in 2011. The ICR (page 19) suggests that this may be due to the declining share of high -risk pregnancies. The proportion of project community health centers (CHCs) conducting more than ten birth deliveries per month increased from 60% in 2006 to 96.6% in 2011. The percentage of 15 essential drugs that were in stock in project facilities, within a given quarter, increased from 69% in 2008 to 86% in 2011, surpassing the target of 80%. The proportion of staff positions filled at project facilities increased for doctors from 59.8% in 2006 to 64.3% in 2011, falling short of the target of 90%. The proportion of nurse positions filled increased from 89.5% to 117% (actual staff hired exceeded number of position openings ), achieving the target of 90% (although the target was essentially at the same level as at baseline ). 100% of social workers positions were filled, surpassing the target of 90%. The proportion of project facilities in which health systems improvement teams were meeting once a month was 85.3%, surpassing the target of 70%. The proportion of clients (patients and nonpatients) satisfied with the services received from a doctor at project facilities increased from 92% in 2008 to 94% in 2011. The proportion satisfied with services received from a nurse decreased from 85% to 72%. The average number of referrals from lower -level facilities increased from 29.4 per month in 2005 to 40.15 in 2011. The average monthly number of feedback responses about referred patients increased from 10.87 to 15.92. The ICR (page 19) suggests these may indicate access to specialized care . The number of laboratory tests conducted in project facilities increased from 874.7 in 2005 to 1761.6 in 2011; the number of radiography screens conducted increased from 252.0 to 890.3. The ICR (page 19) suggests these may indicate an improvement in the availability of diagnostic tests . Overall Objective : To improve the health status of the population, in particular the poor and underserved population . Modest . While there were substantial improvements in intermediate health outcomes in the general population, no evidence is provided of improvements in the targeted poor and underserved population that is consistent with the time frame of the project. Outputs See outputs reported above. Outcomes General population (Note: These data are drawn from two different types of surveys : 2005/6 National Family Health Survey and 2009 UNICEF Coverage Survey. There were no target figures, as these indicators were not part of the original M&E framework.) The proportion of fully immunized children (age 12-23 months) increased from 26.5% in 2005/6 to 53.8% in 2009. Antenatal coverage for pregnant women (at least 3 antenal care visits) increased from 41.2% in 2005/6 to 55.2% in 2009. The proportion of institutional birth deliveries increased from 32.2% in 2005/6 to 70.5% in 2009. Targeted population No evidence is presented on the health status of the targeted poor and underserved populations that closely parallels the project period. 5. Efficiency: Substantial . The project design included aspects that would likely contribute to efficient use of project resources, such as activities that explicitly targeted vulnerable populations (i.e. outreach activities). As these activities achieved substantial outcomes in increasing access to health services, and comprised approximately 30.0% of project disbursements, efficiency in the use of project resources in this regard is likely substantial . In addition, a number of outcomes were substantially achieved at an actual project cost that as lower than the appraised project cost, which also indicates efficiency. The ICR reports that there was frequent turnover in PMU leadership, project directors had multiple responsibilities outside the project, and vacancies were frequent; these factors led to increased procurement processing time and delays in critical decision-making. At the district level, however, project coordinator positions had little turnover . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the project objectives and design are rated Substantial . Achievement of the objectives to increase access to health care for poor and underserved groups and to improve effectiveness and quality of health care are rated Substantial. However, achievement of the objective to improve health status of the population, particularly for the poor and underserved, is rated Modest due to insufficient evidence of improvements for the targeted populations. Efficiency is rated Substantial . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Although a follow up operation is not planned, a number of project activities will continue to be supported by the Government through the National Rural Health Mission program . These include provision of equipment and staff to health facilities, training, and outreach camps . The integration of project activities with the Rural Health Mission program during the project period also helped to support institutional capacity development, and numerous project staff are expected to remain in the Ministry . Government commitment to the sectoral priorities is also likely to be sustained, as reflected by its continued financial and policy support to health programs . However, as noted in the ICR (page 24), "more profound policy changes are still needed to improve service quality over the long -term, e.g. better drug prescribing, changes in the structure of incentives faced by doctors to keep good doctors in remote areas, and possibly, more managerial autonomy at the facility level to manage in a quality - and results-oriented fashion." a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project design reflected lessons learned from previous projects, in particular through a 2002 Quality Enhancement Review (QER) of several state-level health systems development projects in the country . As cited in the ICR (page 7), the QER highlighted the importance of demand -side interventions, private sector engagement, institutional development, governance and policy reforms, and adjusting of projects to fit state-specific context and capacity . The design was also informed by several preparatory studies (such as a social assessment study, tribal development plan, draft gender strategy, facility survey, institutional assessment, review of clinical service norms, analysis of community -based health insurance schemes, public -private partnership feasibility study, study of pilot programs for reaching the poor, and a study of existing larger -scale programs for reaching the poor), although as noted in the ICR (page 24), the PAD did not describe clearly how all these studies informed project design . Risks related to the flow of funds (i.e. financial management, procurement) were appropriately rated as Substantial; while mitigation measures were identified, they were inadequate as the project still experienced significant procurement delays during the first few years of the project . The project preparation period was relatively lengthy (27 months), due to the long time needed to secure government support for certain aspects of the project design and to set up implementation arrangements . Shortcomings in the M&E framework included the lengthy list of indicators, a number of which were not realistic or measurable (given the available data), and the lack of baseline or target figures . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank team was flexible in adapting the project design to the Government's newly -introduced health programs, including responsibilities of the Bank vs . Ministry staff. The ICR (page 25) reports that the "staff of the PMU speak highly of the commitment of the task team, " including developing precise follow -up actions for each mission and conducting weekly calls . However, there were significant disbursement delays due to inadequate institutional capacity on the part of the Borrower (see Section 11), and the project was flagged as a problem project at the Mid-Term Review (July 2007) and remained at such status for the duration of the second half of the original project period; the disbursement grace period was also extended after project closing (to March 31, 2012). Implementation performance subsequently improved, and as a large amount of the credit was yet to be disbursed, the project was extended for two years . However, most of the new activities planned for the extension period were not initiated, and others were only partially completed . According to the ICR (page 10), this was due to procurement and human resources issues that resurfaced during the extension period . Safeguard and fiduciary compliance were effectively monitored, including proactive efforts to prevent fraud and corruption (which had been flagged as a significant issue in the country, though not necessarily with this project ). However, there were no proactive steps taken to address shortcomings in M&E until the 2006 project restructuring. Although the ICR (page 13) notes that the Bank team strongly encouraged the use of M&E data to improve activities, there was still limited use of M&E by the implementing agencies even though monitoring data was provided by facilities regularly . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government provided an overall supportive policy and legislative environment for the project . It supported ongoing reforms such as subsidies to BPL patients, contracting out specialized services, addressing medical staff shortages in rural areas, and creating an enabling environment for partnership with the private sector. However, significant time was taken in securing government support for some aspects of the project design (health system strengthening measures and targeted interventions for the poor and disadvantaged ), which led to a lengthy preparation period (27 months). Compliance with fiduciary guidelines and covenants was satisfactory. However, staffing needs were not effectively managed (as reported below ), including failure to provide adequate resources to monitor project implementation during the extension period . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The Department of Medical, Health and Family Welfare at the state level was the primary implementing agency and was overall effective in implementing project activities . The ICR (page 27) highlights success in taking over procurement responsibilities from contracted procurement agencies and in regular monitoring through hospital activity formats. However, leadership of the PMU changed five times during the project period, which had a disruptive effective on implementation particularly during the extended project period . In addition to the frequent turnover, project directors had multiple responsibilities outside the project, and vacancies were frequent (particularly during the extension period when core staff were lost and never replaced ). The ICR (page 8) reports that these factors led to increased procurement processing time and delays in critical decision -making. At the district level, project coordinator positions had little turnover and retained capacity over time, although a long time was taken to fill these positions in the beginning . Procurement was extremely slow during the first few years of the project due to inadequate capacity (see Section 11), with significant disbursement delays during the first part of the project period. District-level PMUs were considerably varied in performance, although capacity improved over time. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The original M&E framework had significant shortcomings, as there were numerous indicators (11 outcome indicators and 32 intermediate outcome indicators), the indicators were not measurable or realistic (i.e. difficult to measure with available data, not consistent with data sources or timeframe of sources ), and there was a lack of baseline data and targets. It is also not clear whether the M&E design included arrangements for capturing utilization rates or other outcomes among the target populations . Some pilot and operational research activities were planned; however, the design did not include an overall evaluation of project impact (i.e. project-specific surveys at baseline and end-project). The Bank and Government agreed to the revised results framework in April 2006. Targets for three of the indicators were subsequently revised downward in November 2009. b. M&E Implementation: As part of the 2006 project restructuring, the number of indicators was reduced to 7 outcome and 11 intermediate outcome indicators (some of which were modified to more measurable definitions ), and baseline and target figures were identified. As the HMIS did not contain the information needed to track progress towards the project's specific objectives, nor disaggregated utilization data by poor or vulnerable groups, a separate hospital activity format (HAF) was developed to monitor results . By October 2007, more than 90% of project facilities were submitting timely and complete reports. However, this resulted in a parallel monitoring system (which created additional reporting responsibilities for project facilities ), rather than a strengthening of the existing HMIS . An unsuccessful attempt was made in 2009 to integrate the two systems. Lastly, several of the planned assessments were conducted (i.e. patient satisfaction surveys, patient counselor intervention, quality of civil works, equipment audits ), although the ICR (page 12) notes that these were more useful for understanding implementation issues rather than assessing project impact . c. M&E Utilization: With a few exceptions due to the proactive approach of some Health Systems Improvement Teams and District Project Coordinators in which HAF information was used to inform decision -making, the ICR (page 13) reports that due to multiple reporting requirements, staff time limitations, and insufficient interest within the Government, the "substantial information that was generated by the HAFs was not used to its full potential ." M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was classified as a Category "B" project, triggering OP/BP/GP 4.01 on environmental assessment, due to medical waste issues, and OP4.10 on Indigenous Peoples. A health care waste management plan and a tribal development plan were developed and implemented according to the project covenants, and compliance on both was satisfactory. Slow progress on common waste treatment facility connectivity led to an unsatisfactory rating in May 2007; however, this was resolved within six months and the environmental assessment rating was eventually improved to satisfactory. b. Fiduciary Compliance: Financial management: There were no major financial management issues reported . Bank financial management specialists assessed performance on each supervision mission and identified clear follow -up issues. There is no information in the ICR on the results of external audits . Procurement: Procurement was extremely slow during the first few years of the project due to inadequate capacity of the Borrower. Procurement support agencies (PSAs) had been contracted at the beginning of the project in order to avoid procurement difficulties and strengthen PMU procurement capacity; however, despite the preparation of detailed procurement materials, the PSAs still had inadequate capacity . Contractors in desert and tribal areas were not available to fulfill bid criteria. During this initial period, procurement was rated unsatisfactory on two occasions . PMU and DPMUs took over procurement responsibility in 2006-2008, which improved the pace of procurement . However, delays returned during the extension period when procurement responsibility was solely placed in the PMU, which had difficulty in managing large international competitive bidding procurements and was slow to make decisions, aggravated by frequent staff turnover and lengthy vacancy periods . Minor issues noted in the ICR (page 14) included instances of small misprocurements (due to misunderstanding of allowable expenditures ); a number of (frequently unjustified) procurement complaints by competing bidders; slow responses to complaints; failure to complete timely procurement post reviews; and occasional failure to satisfactorily address review recommendations . By the end of the project period, all procurement -related issues were resolved and, according to the project team, all project audits were unqualified. c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons from the ICR: The long-term payoffs of integration and harmonization with government programs need to be balanced with the short-term needs of project implementation. In the case of this project, careful integration with the government's newly introduced health programs helped to promote complementarities and sustainability; however, it reduced the PMU's singular focus on Bank project implementation . Capacity of procurement staff needs to be rigorously assessed . In the case of this project, the extensive preparation of procurement documents prior to effectiveness was not effective in preventing major procurement delays, given the existing capacity levels . Lessons drawn by IEG: The monitoring and evaluation framework needs to be robust (i.e. realistic and measurable) in order to be useful. In the case of this project, the M&E outputs were of limited usefulness to the implementing agencies and led to data quality issues for the purposes of evaluating the project's impact . In particular, indicators were inadequate for capturing intended outcomes among the targeted populations . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: ICR Quality is Satisfactory. The ICR was well-written and concise; of particular note was the analysis which was outcome-driven and the use of multiple sources of data, including from outside the project . The ICR does not include the results of external audits, including whether project audits were unqualified or not . a.Quality of ICR Rating : Satisfactory