Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) Report Number: ICRR0021634 1. Project Data Project ID Project Name P127245 BR Rio de Janeiro Mun. Strengthening PSM Country Practice Area(Lead) Brazil Macroeconomics, Trade and Investment L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-82710 30-Apr-2018 12,441,313.34 Bank Approval Date Closing Date (Actual) 14-Jun-2013 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 16,200,000.00 0.00 Revised Commitment 12,441,313.34 0.00 Actual 12,441,313.34 0.00 Prepared by Reviewed by ICR Review Coordinator Group Paul Holden Judyth L. Twigg Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The objective of the project was to support institutional capacity strengthening of the Borrower to enhance public service delivery, including in health, education and environmental management (Financing Agreement, p. 5). ”Unpacking” the PDO suggests four distinct outcome areas: Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) Supporting institutional capacity strengthening of the Borrower to enhance public service delivery in: I. Health; II. Education; III. Environment management; and IV. Other areas, derived from the non-exclusivity of the listed areas (implied by the word “including"). This technical assistance project loan aimed to provide follow-on support for the World Bank Fiscal Consolidation for Efficiency and Growth Development Policy Loan (DPL, P111665, US$1,045 million) that had been approved in July 2010 and closed on June 30, 2013. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project had four components. Component 1: Improving Medium Term Fiscal and Public Investment Management (appraisal estimate US$1.5 million; actual cost at closing US$3.95 million). This component aimed to strengthen processes connected with fiscal and public investment management in the medium term through: 1. Integrating the results of impact evaluation and information on performance with proposed policies related to planning and budgets; and 2. Undertaking diagnostic evaluations and implementing recommendations so obtained. Some of these included: • Developing criteria to account for fiscal risks; • Developing costing methodologies for future policies and programs; and • Capacity building to formulate multi-year revenue estimates. At a Level 2 restructuring in December 2017, the name of Component 1 was changed to Improving Medium Term Fiscal Management, and the budget was increased to US$3.95 million. At this restructuring, the first subcomponent was dropped. Component 2: Innovating in Service Provision (appraisal estimate US$25.4 million; actual cost at closing US$22.07 million). This component had two sub-components: 1. Health: Improving health service delivery through: • Developing an effective referral system for hospital and diagnostic services and a hospital bed managing system; • Evaluating reforms in a specific health district related to the expansion of primary care, 24-hour emergency services, and the hospital system; Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) • Conducting operational research on prenatal mortality and the quality of prenatal care; • Evaluating the impact of interventions to improve chronic disease care; and • Supporting family health clinics. At the December 2017 restructuring, all activities under this subcomponent except development of the referral/bed management system and support for family health clinics were dropped. 2. Education: Improving education quality through: • Enhancing capacity building activities for learning assessments; • Establishing a new office for information management to track and utilize data assessments of students; • Supporting the creche monitoring system; • Monitoring early childhood development; • Managing early childhood education through an integrated quality assurance system; • Supporting the integration of early childhood services in education, health, and social systems; and • Designing and implementing teacher training programs. At the December 2017 restructuring, all three early childhood support activities were dropped. Component 3: Strengthening Social Governance and Monitoring and Evaluation (appraisal estimate US$3.3 million; actual cost at closing US$3.85 million). This component was aimed at promoting the integration of secured favelas and strengthening evidence-based policy making. It had two sub- components: 1. Strengthening the operating systems for a social Police Peacekeeping Units program through promoting community participation with Police Peacekeeping Units in secured favelas and improving the implementation of sectoral policies for service provision in these favelas; and 2. Design a Monitoring and Impact Evaluation System for the municipality, which included: (a) improving the expertise of the Peacekeeping units in the collection of data in order to improve service delivery; b) designing an integrated monitoring and evaluation system; and (c) strengthening the technical skills of the Police Peacekeeping Units for carrying out impact evaluations for evidence-based policy making and undertaking four evaluation assessments. At the restructuring, the number of assessments was reduced to three in order to focus more on implementation efforts. Component 4: Innovating in Environmental Management (appraisal estimate US$1.04 million; actual cost at closing US$0). This component was to implement Rio de Janeiro’s Climate Data Bank and Green House Gas Emission Monitoring System, to track progress on meeting emissions targets. However, at the December 2017 restructuring, this component was dropped. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The original project cost was US$32.4 million, of which US$16.2 million was an International Bank for Reconstruction and Development (IBRD) loan and US$16.2 million was to be financed by the Borrower. The revised amount of the project at the restructuring was US$28.6 million, of which US$12.4 million was the IBRD loan and US$16.2 million was to be supplied by the Borrower. The actual amounts disbursed were US$12.4 million from the IBRD loan and US$16.2 million from the Borrower. The project was approved on June 14, 2013 and became effective on October 23, 2013. It was originally scheduled to close on April 30, 2018 and actually closed on June 30, 2018. The project was restructured (Level 2) on December 21, 2017, at which point Component 4 and some other activities (as described above) were dropped. In the project restructuring, activities that would probably not be completed prior to closing were dropped, although the PDO remained unchanged because any changes would have required the approval of Brazil’s Senate. As a result, US$2.025 million of the project funds were cancelled. 3. Relevance of Objectives Rationale The World Bank Country Assistance Strategy (CAS) for FY2012-FY2015 had objectives to increase the efficiency of public and private investments, improve the provision of public services for low-income households, promote regional economic development and improve sustainable natural resource management and climate resilience. Except for component 1, the PDOs were broadly consistent with the second and fourth of these objectives, and also partly with the third because of the focus on Rio de Janeiro, Although component 1 was relevant to the objectives of the CAS and the CPF, it was not aimed at any single sector, nor was is directly related to service delivery. At closing, the PDOs were consistent with the Country Partnership Framework (CPF) for FY2018-FY2023, which had 3 focus areas: (i) fiscal consolidation and government effectiveness; (ii) private sector investment and productivity; and (iii) equitable and sustainable development. The ICR (p. 16) pointed out that the CPF maintains a focus on improved service delivery that was the focus of the previous CAS. The World Bank Systematic Country Diagnostic (2016) points out the importance of social inclusion and socially progressive allocations of government expenditure, which is consistent with the PDO. However, a weakness of the PDO was that its scope was so broad as to justify virtually any public sector intervention in the areas of health, education, or environmental management. Furthermore, the way that the PDO was presented, other public service delivery could be included, so that the PDOs were not effectively Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) operationalizable or monitorable. Nevertheless, the loan was granted a Federal Guarantee indicating that the Federal Government approved the objectives of the project. The rating of relevance is therefore "substantial". Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Supporting institutional capacity strengthening of the Borrower to enhance public service delivery in health Rationale The focus of the PDO was institutional capacity strengthening for the enhancement of public service delivery. Because the PDO statement was so broad, it is difficult to identify its exact focus. For example, the first component focused on strengthening processes connected with public investment management, which does not appear in the PDO. The second component focused on innovating service provision, but the PDO focused on “supporting” the strengthening of service provision, and it is unclear whether the “innovations” strengthened service provision, although the sub-components did focus on improving the delivery of health services and improving education quality. The third component aimed to strengthen social governance, defined as the integration of secured favelas, although exactly what this meant was unclear. The other goal of component 3 was to strengthen M&E although how this related to improved service delivery was not specified. The fourth component aimed to “innovate” environmental management, although it is unclear whether this encompassed support for institutional capacity strengthening. The ICR (p. 8) pointed out that the project was prepared under earlier guidelines that did not require an articulation of a theory of change. The ICR therefore “retrofitted” a theory of change to the activities, outputs, and outcomes of the project. The objective in the health area was to be achieved through technical assistance that supported and implemented a series of outputs: • Developing an effective referral system for managing access to specialists, hospitals, and diagnostic services and for managing hospital beds; • Reforms to expand primary care, 24-hour outpatient care, and emergency services and to evaluate the hospital system in Health District 5.2; • Pilot information and communications technology for the remote monitoring of patients with diabetes, hypertension, and tuberculosis; and • Progress in implementing the Family Health Clinic initiative. Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) Outputs Only the family health clinics were actually implemented. Outcomes The efficacy section of the ICR did not discuss the achievement of public service delivery in health. The Restructuring Document (p. 5) pointed out that delays in procurement processes and lack of ownership by the municipality resulted in only the Family Health Clinics initiative being implemented through the use of counterpart funding. However, the indicators (ICR p. 42) do not contain any measure of improvements in public service delivery in health. The achievement of this objective is therefore rated as Negligible. Rating Negligible OBJECTIVE 2 Objective Supporting institutional capacity strengthening of the Borrower to enhance public service delivery in education Rationale The objective in the education area was to be achieved through technical assistance supporting and implementing a series of outputs that related to improved education outcomes and constituted a plausible causal chain: • Enhancing capacity building activities for learning assessments; • Establishing a new office for information management to track and utilize data assessments of students; • Supporting the creche monitoring system; • Monitoring early childhood development; • Managing early childhood education through an integrated quality assurance system; • Supporting the integration of early childhood services in education, health, and social systems; and • Designing and implementing teacher training programs. Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) Outputs Although not part of the project results indicators, the ICR (pp. 17-18) described “key outputs and results”: • Provision of capacity building activities for learning assessments, which the ICR stated was fully achieved; • The implementation of consistent writing test grading criteria, which was fully achieved; • Support for the establishment of a new office of strategic management to track and analyze data from the Ministry of Education’s comprehensive student assessments, which was partially achieved, although the office employed only two specialists; • The design and implementation of an integrated quality assurance system for early childhood education, which was fully achieved; and • The development of a new teacher training school that could deliver distance learning, which was fully achieved. However, these indicators were included in neither the project design nor in the monitoring and evaluation framework, so it is difficult to consider them as being "key." Outcomes The outcome indicator for this objective was the improvement in education results of Rio municipal students as measured by the Rio de Janeiro Education Development Index. Initial grades improved from 5.4 at the 2012 baseline to 5.7 in 2018, not reaching the target of 6.0. Final grades improved from 4.4 at the 2012 baseline to 4.7 in 2018, not reaching the target of 5.0. With neither outcome target reached, achievement of this objective is rated Modest. Rating Modest OBJECTIVE 3 Objective Supporting institutional capacity strengthening of the Borrower to enhance public service delivery in environment management Rationale Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) This objective was meant to be achieved through developing a monitoring system for emissions, and providing technical assistance to set up the system and training in how to operate it. The ICR (p. 21) stated that no activities were implemented under this objective, although results were achieved under a different project. The rating for this objective is therefore Negligible. Rating Negligible OBJECTIVE 4 Objective Supporting institutional capacity strengthening of the Borrower to enhance public service delivery in other areas Rationale The actions underlying this objective were to indirectly support sector policies through “cross sectoral improvements in public-sector management as well as impact evaluation capacity" (ICR, p. 8). Under this objective, anticipated outcomes included: • Improved fiscal and public investment management processes. This included improved collection times for urban property taxes and an improved data warehouse system that integrated through digitization such data as property taxes, urban maps, property sales data, and garbage collection routes, through real time monitoring; • Better coordination of the Social Police Peacekeeping Units (UPPs) in the Pacified Favelas. Social UPP is the flagship program promoting the integration of the favelas with the rest of the City of Rio de Janeiro, with the goal of eliminating the influence of criminal organizations; • A strengthened program for managing the Social UPPs; • Improved implementation of sectoral policies in the Pacified Favelas; • Strengthening the Instituto Periera Passos, which collected household and survey data to improve service delivery targeting and performance and conducted impact evaluations for evidence-based policy making. The causal connections through the results chain for the achievement of this objective are hard to discern. The theory of change articulated in the ICR is not convincing. While the outcomes undoubtedly are connected to inputs for institutional capacity strengthening, the direct relationship with improved public service delivery is Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) obscure. Presumably, the various capacity development interventions in the components would have led to institutional capacity strengthening, but there are no specific actions that link the objective with the outcomes. Outputs • The number of users that are social organizations and firms/entrepreneurs registered in the Portal RCI – Community Integrated Network (a community portal information site). The target was 200, compared with a baseline of 0, and the actual achievement was 62. • The number of users that are individuals registered in the Portal RCI – Community Integrated Network. The target was 1500, compared with a baseline of 0, and the actual achievement was 920. • The number of active users of the Armazem de Dados portal (Rio’s Data Warehouse), The target was 5500, compared with a baseline of 0, and the actual achievement was 5122. • The number of impact evaluation concept notes produced and published by the Pereira Passos Institute. The target was 6, compared with a baseline of 0, and the actual achievement was 6. Outcomes The number of ongoing impact evaluations implemented, with a target of three compared with a baseline of zero. The actual number achieved was three. • The average time of “solution for the payment of tributary (property) taxes" was targeted to decline from a baseline of 129.2 days to 109.82 days. The actual figure achieved was 141.35 days. The ICR (p. 17) stated that the increase in time was not due to problems with the digitization, but rather that because of greater transparency, protests against increased property taxes rose. • The number of page views on the Armazem de Dados portal was to increase from a baseline of zero to a target of 9000 in 2012 by the end of the project. The actual number achieved was 27,744. The project's stated outcome indicators all relate to inputs, without any measure of their effect on the quality of public service. Page views by firms and citizens, the number of impact evaluations completed, or the resolution of property tax disputes do not in themselves reflect improved provision of services. The same reasoning applies to the improved implementation of sectoral policies in the favelas and to the actions of the UPPs. There is no evidence presented to support the contention that services improved. Rating Negligible OVERALL EFF TBL OLD Rationale With three of the objectives rated Negligible and one rated Modest, the overall efficacy rating is Negligible. Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) Overall Efficacy Rating Primary reason Negligible Low achievement 5. Efficiency The ICR contained an economic and financial analysis that attempted to evaluate the net benefits from the investment of the project funds. The ICR pointed out (p. 46) that “the lack of a clearly defined causality chain of project activities and expected outcomes made the efficiency analysis a very challenging exercise." As in the efficacy analysis, the ICR focused on the components, rather than on the PDO. It is therefore questionable as to whether the results of the estimates are relevant. However, a brief summary of the calculations is provided here. The ICR adopted a cost effectiveness analysis (CEA) approach (although the PAD estimated efficiency based on cost benefit analysis) because of the difficulties of placing a value on all of the outcomes. It identified the quantifiable benefits as: • Higher property tax collections with a discounted present value of US$36.9 million over a 15-year period, which the ICR estimated as providing a 99% internal rate of return (IRR) (ICR, p. 47); • Indirect benefits arising from the health sector activities, including creating an information system for epidemiology, a new regulatory system for prenatal care, and the support of family health, with coverage of family health clinics rising to 70% of the municipal population compared with 30% in 2013. However, the ICR (p. 48) stated, “of the five health activities originally planned, only one...financed with the Municipality of Rio funds was implemented – Bank financing did not have any direct impact” (ICR, p. 48), which would exclude it from being included in the CEA for the project;; • Innovation in the education sector sub-component, estimated to yield an IRR of 88%. The ICR stated that this activity: (i) raised the quality of education throughout the system “through the focus on the definition of results-based policies” (p. 49); (ii) extended the quality of education through the system by “the focus on the definition of results-based policies"; and (iii) improved the effectiveness of teachers in the classroom ”through training and support in the vocational training of the faculty of the municipality” (ICR, p. 49). Overall, the ICR pointed out (p. 23) that, while some sub-components registered highly satisfactory results, overall efficiency was either below expectations or not rated at all because of lack of data or because activities were dropped at the restructuring. The ICR went on, however, to say that all of the sub-components that were analyzed had a positive net present value, based on the analysis described above. This analysis is not credible because of the questionable indicators used as the basis for the IRR calculations. The indicators are poorly defined, and the discussion in Appendix 4 of the ICR made little analytical sense. It appeared to be based on expectations, rather than on hard evidence. Implementation efficiency was negligible because of serious project delays (see ICR, p. 15, paras 39-40), a high rate of personnel turnover, and low capacity. The project involved the expenditure of substantial amounts while achieving very little. Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome With a Modest rating for relevance of objectives, Negligible ratings for 3 of the 4 development objectives and an overall rating of Negligible for efficacy, and a Negligible rating for efficiency, the outcome rating is Highly Unsatisfactory. These ratings are consistent with severe shortcomings. a. Outcome Rating Highly Unsatisfactory 7. Risk to Development Outcome Factors that will likely contribute to the sustainability of the project (ICR, p. 33) were: • The contribution of the Borrower to the project, amounting to more than 50% of the final cost, although this is offset by the possibility that funding will not continue (see following paragraph); • The anchoring of the project in Rio de Janeiro’s development objectives that included improving fiscal and public management and improved service delivery; • Learning from the World Bank with respect to improved technical and administrative procedures; and • Improved coordination among the agencies. However, there are several substantial risks related to the national economy: (i) uncertainty regarding continued funding for the Municipality of Rio de Janeiro; (ii) fiscal constraints that will reduce the likelihood of Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) continued funding because of expenditure increases and revenue uncertainties; (iii) a significant chance that government commitment will wane and that political volatility will reduce the commitment to the program; and (iv) some chance that staff turnover will reduce capacity to administer the program. 8. Assessment of Bank Performance a. Quality-at-Entry Although the ICR (p. 26) stated that the project reflected the lessons learned from the prior DPL, the design of the program at entry did not recognize significant legal, capacity and institutional barriers to the effectiveness of the loan (ICR p. 32). In particular, there was a failure to identify the lack of readiness of the counterparts at entry, which adversely affected the implementation of policy actions as well as project implementation and led to serious delays. Furthermore, both the broad nature of development objective and the design of components that were not consonant with the PDO meant that much of the work of the Bank team did not directly map into the PDO. Furthermore, the ICR (p. 34) reported that the Municipality expressed low interest in the project during preparation and only agreed to it because of strong advocacy by the Bank team and management. . Quality-at-Entry Rating Highly Unsatisfactory b. Quality of supervision . The ICR reported (p. 32) that, although there were timely supervision missions and nine Implementation and Results Reports, the Bank team did not closely monitor and supervise implementation over the first three years of the project. In particular, the ICR reported that internal coordination among the Bank team during the initial phases of the project was weak. In addition, the fact that the TTL was located in Washington and could not communicate effectively with Brazilian counterparts because of a lack of fluency in Portuguese hampered the effectiveness of supervision. The ICR (p. 32) states “Having DC-based TTLs with no fluency in Portuguese limited the Bank team’s ability to have more frequent follow-ups with the Government to ensure that any issues causing implementation delays are addressed in a timely fashion.” Furthermore, there was also significant turnover of TTLs, which hampered the continuity of supervision and had a negative effect on project management even though there were 9 supervision missions during the Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) project. These factors contributed to the failure to flag low government commitment to the project, which should have been identified at the outset. Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Highly Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The Municipality of Rio de Janeiro was responsible for data collection and reporting and providing evidence for the M&E framework. The design of the M&E framework was inconsistent with the PDOs, which in turn were loosely defined. The design of monitoring and evaluation failed to include any outcome indicators for the health sub-component. While the education indicators were relatively precisely defined, others were not. The number of impact evaluations was an input into improvement in health and education, which would have required implementing their findings, but there was little evidence that this occurred. Solving tributary tax processes was not related to the PDOs, so the average time indicator was not relevant. Similarly, the number of page views on the Internet portals gives no indication of whether the PDO were being achieved. The environmental indicator regarding the monitoring of emissions was dropped. A major weakness of the design of the program and the M&E framework was that its scope was so broad as to justify virtually any public sector intervention in the areas of health, education, and environment management. Furthermore, the way that the PDO was presented, other public service delivery could be included, so that it was not effectively monitorable b. M&E Implementation The ICR (p. 30) reported that data regarding intermediate and outcome indicators was generally not available, particularly during the initial implementation phase. Delays in signing consulting contracts were at least partly the cause of evidence not being available. The Municipality of Rio de Janeiro was only able to track project indicators at the time the final supervision mission occurred. The ICR (p. 30) stated, “with the long delays in the project implementation, there were no products to be assessed in the World Bank's Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) supervision missions, and it was not possible for the teams to track results during most of the ISR missions.” c. M&E Utilization The ICR (p. 30) stated that although data on the intermediate and outcome indicators were not available, "overall M&E was carried out and monitoring reports were used appropriately for project management and to inform decision making by the agencies responsible for implementation of project activities." This statement is difficult to reconcile with the absence of M&E data. It suggests that the M&E framework was not relevant to the progress of the operation. M&E Quality Rating Negligible 10. Other Issues a. Safeguards The project had a category C rating for environment impact, which did not trigger an environmental analysis. The project did not trigger social safeguard policies. b. Fiduciary Compliance The ICR reported (p. 31) that a comprehensive review of the Municipality’s public financial management uncovered no irregularities, although the lack of capacity in contract management was flagged as being an issue because of the management team’s lack of familiarity with World Bank procurement processes. At closing, project financial management was rated Satisfactory, and procurement was rated Moderately Satisfactory due to lack of capacity in contract management. c. Unintended impacts (Positive or Negative) None reported. Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Severe shortcomings in efficacy Outcome Highly Unsatisfactory Unsatisfactory and efficiency. Severe shortcomings in quality Moderately Bank Performance Highly Unsatisfactory at entry, and major shortcomings Unsatisfactory in supervision. The M&E framework was not Quality of M&E Modest Negligible used, and the choice of indicators was weak. The ICR did not evaluate the Quality of ICR --- Modest project against the PDOs. 12. Lessons The following lessons are drawn from the ICR, with some adaption of language by IEG:  Lack of strong ownership by government entities and implementing agencies compromises the potential success of a project. Considering not proceeding with a project under these circumstances is a viable option for the Bank. In this case, the ICR (p. 34) reported that the Municipality expressed low interest in the project during preparation and only agreed to it because of strong advocacy by the Bank team and management.  Weak institutional capacity of counterparts compromises the potential success of a project and should be incorporated into design. In the case of this project, the ICR (p. 15) stated that “the Municipality showed low capacity to prepare and implement activities.”  Implementation support, particularly when counterparts are not familiar with World Bank procurement practices, is a critical factor in determining the likelihood of success. In the case of this project, the ICR (p. 24) stated that shortcomings in procurement management jeopardized project implementation. IEG adds the additional lesson: Ensuring that the PDOs and the components of a project are consistent is a critical factor in project design. In the case of this project, there were substantial gaps between the two. 13. Assessment Recommended? Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Rio de Janeiro Mun. Strengthening PSM (P127245) No 14. Comments on Quality of ICR The ICR rated the project on the achievement of its components, rather than on the achievement of the PDO. It did not attempt to parse the development objective to derive the outcome areas. As a result, much of the information regarding outcomes was not easily accessible, since the ICR did not rate outcomes against the PDO. The ICR also did not acknowledge the lack of consonance between the PDOs and the M&E outcome indicators. a. Quality of ICR Rating Modest Page 16 of 16