IEG Report Number: ICRR14900 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 06/08/2016 Country: Rwanda Is this Review for a Programmatic Series ? Yes No How many operations were planned for the 3 series? How many were approved ? 4 Series ID: S117495 First Project ID : P117495 Appraisal Actual Project Name: Seventh Poverty Project Costs (US$M): 104.4 104.4 Reduction Support Grant (PRSG) L/C Number: Loan/Credit (US$M): 104.4 104.4 Sector Board: Economic Policy Cofinancing (US$M): Cofinanciers: Board Approval Date : 02/24/2011 Closing Date: 06/30/2012 06/30/2012 Sector(s): General public administration sector (40%); General energy sector (21%); General finance sector (13%); General education sector (13%); General agriculture; fishing and forestry sector (13%) Theme(s): Public expenditure; financial management and procurement (45%); Infrastructure services for private sector development (20%); Other Private Sector Development (15%); Other rural development (10%); Education for the knowledge economy (10%) Second Project ID :P113241 Appraisal Actual Project Name: Sixth Poverty Project Costs (US$M): 115.8 115.8 Reduction Support Grant L/C Number: Loan/Credit (US$M): 115.8 115.8 Sector Board: Economic Policy Cofinancing (US$M): Board Approval Date : 03/30/2010 Cofinancers: Closing Date: 06/30/2011 06/30/2011 Sector(s): General public administration sector (31%), General water, sanitation and flood protection sector (31%), General education sector (15%), General transportation sector (15%), Crops (8%) Theme(s): Urban services and housing for the poor (32%), Administrative and civil service reform (23%), Infrastructure services for private sector development (15%), Public expenditure, financial management and procurement (15%), Education for the knowledge economy (15%) Third Project ID :P106083 Appraisal Actual Project Name: Fifth Poverty Project Costs (US$M): 80.0 80.0 Reduction Support Grant L/C Number: Loan/Credit (US$M): 80.0 80.0 Sector Board: Economic Policy Cofinancing (US$M): Board Approval Date : 03/17/2009 Cofinancers: Closing Date: 06/30/2010 06/30/2010 Sector(s): Vocational training (20%), Central government administration (20%), General water, sanitation and flood protection sector (20%), Irrigation and drainage (20%), Sub-national government administration (20%) Theme(s): Rural services and infrastructure (29%), Public expenditure, financial management and procurement (29%), Infrastructure services for private sector development (14%), Education for all (14%), Water resource management (14%) Fourth Project ID :P104990 Appraisal Actual Project Name: Fourth Poverty Project Costs (US$M): 70.0 70.0 Reduction Strategy Grant L/C Number: Loan/Credit (US$M): 70.0 70.0 Sector Board: Poverty Reduction Cofinancing (US$M): Board Approval Date : 03/27/2008 Cofinancers: Closing Date: 12/31/2008 12/31/2008 Sector(s): General public administration sector (47%), General water, sanitation and flood protection sector (17%), Irrigation and drainage (12%), General education sector (12%), Health (12%) Theme(s): Public expenditure, financial management and procurement (33%), Education for all (17%), Health system performance (17%), Water resource management (17%), Rural services and infrastructure (16%) Evaluator: Panel Reviewer: ICR Review Group: Coordinator: Lev Freinkman Clay Wescott Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: The Program Document (PD) for the Fourth Poverty Reduction Support Grant (PRSG-4) prepared in 2008 was not very specific about policy objectives of the PRSG series. The PD defined the policy development objective (PDO) in terms of broad support for implementation of the Government’s own development strategy. In particular, the PD stated (p.2): “The series designed to assist the Government of Rwanda (GoR) to implement key policy actions that will form part of its second Poverty Reduction Strategy Paper (PRSP), called the Economic Development and Poverty Reduction Strategy (EDPRS).” Then it clarified (p. 28): “The proposed second PRSG series will support programs in the EDPRS’s three flagships i.e. Growth for Jobs and Exports, Vision 2020 Umurenge, and Governance.” This PDO was further specified in the follow-up operation, PRSG-5 (2009). The respective PD stated (p. v) that the operation will support two (instead of three) EDPRS flagship programs – Growth for Jobs and Exports, and Governance, and “Support to these two flagship areas will be aimed at: (i) raising agricultural production in a sustainable manner; (ii) increasing access to infrastructure services; (iii) strengthening the framework for post-basic education, with the aim of building a skilled workforce and promoting science and technology; (iv) deepening the financial sector, including capacity building measures to facilitate increased access to finance by micro-enterprises; and (v) strengthening the management of public resources at the central and local levels.” In the two follow-up operations, PRSG-6 and -7, the above description of the PDO remained substantively unchanged (while some wording was modified and streamlined). In light of the broad objective of this series, the IEG review is assessing the program’s efficacy on the basis of its contribution towards five specific policy objectives: (i) enhancing the skills of the population; (ii) building economic infrastructure; (iii) raising agricultural productivity; (iv) deepening the financial sector; and (v) enhancing government capacity, accountability, and transparency. Such decomposition of the original PRSG series objective is consistent with the program’s revised Policy and Results Matrix (see PD for PRSG-7, pp. 53-56). b. If this is a single DPL operation (not part of a series), were the project objectives/ key associated outcome targets revised during implementation? No c. Policy Areas: The policy reforms supported under the PRSG series were grouped in five policy areas consistent with the program’s objectives (according to the Policy Matrix in PD for PRSG-4, pp. 64-68). The first of them, which originally covered both health and education reforms, was significantly modified in PRSG-5 to focus exclusively on education. Health policy actions moved under the Bank's Community Living Standard Grant Program, which was a parallel programmatic series of three single-tranche policy operations (2009-2012) that in addition to health also covered social protection. Four other policy areas remained unchanged in the course of the PRSG program implementation. (i) Investing in human capital to increase the supply of a skilled and health workforce Policy actions included the development of the new government’s Technical and Vocational Training (TVET) policy, the revision of the Education policy to include greater emphasis on post-basic education, and undertake school assessment to identify districts with the largest needs. In the health sector (under PRSG-4) the program supported the adoption of the new Health Insurance Law, accompanied by the introduction of adequate financing and regulatory framework for its implementation. (ii) Improving economic infrastructure  Energy: Measures included the adoption of the Law on establishing the power sector utility to improve its governance and performance; adoption of new basic Law on electricity and gas to expand private sector participation in the sector; strengthening capacity to implement the investment program in power; preparation of the new tariff structure to improve sustainability of the power sector; and removal of taxes on liquefied petroleum gas (LPG) to promote its use and reduce dependence on charcoal.  Water and sanitation: Measures included a decision to increase budget allocations for water to support implementation of the Integrated Water Management Program; establishment of a water and sanitation management information system; approval of the national strategy for water supply and sanitation services; adoption of key performance indicators to improve monitoring of performance of the national water utility; and design of a tariff framework for urban water services to promote efficiency and sustainability.  Road transport: Supported policies included adoption of local Road Maintenance and Decentralization strategy; and implementation of the decision to increase the fuel levy. (iii) Improving agriculture productivity Supported policy actions were strengthening watershed management on the basis of completion of feasibility studies for particular sites; and strengthening the role of the private sector in procurement and distribution of fertilizers and other sectoral inputs, particularly at the local level. (iv) Deepening and broadening the financial sector Covered policy actions included: A decision to facilitate expansion in rural credit under the second Rural Investment Facility; adoption a Law on microfinance; strengthening the legal and regulatory framework for national payment systems; and upgrading/consolidation of the government system of SMEs support. (v) Enhancing government capacity, accountability, and transparency  Public financial management: Development and rollout of the Integrated Financial Management Informational System (IFMIS) in the central government, including the payroll module; and adoption of the new accounting software at the district level.  Public procurement: Strengthening the regulatory framework for public procurement (adoption of procurement standards and model bidding documents); amending the Procurement Law to increase efficiency and transparency; publication of the report by the independent review (appeal) panel; and implementation of the decision that requires government agencies to submit their procurement plans in a timely manner.  Fiscal decentralization: Improvements in districts’ budget reporting and disclosure; undertaking an assessment of quality in service delivery at the local level; adoption of a 5-year capacity building strategy for local governments.  Civil service reforms: Completion of functional reviews in six key ministries; adoption of new staffing levels and ministerial structures for all central government ministries; adoption of amendments to the Pay and Remuneration policy to strengthen the link between pay and performance. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The Program was financed under a second programmatic series of four single-tranche development policy operations, PRSG-4, -5, -6 and -7. Total program cost was US$372.6 million (out of which US$30.4 million was provided as International Development Association (IDA) credit, while the rest was IDA grants). IDA financing consisted of three grants and one, PRSG-7, mixed operation (a combination of grant and credit financing). The four operations disbursed fully and each of them closed on time. The originally approved program financing was US$370.2 million equivalent, the difference with the actual program cost was due to exchange rate fluctuations of US dollar versus SDR basket. There was no government contribution. PRSG-4 was appraised in December 2007, approved for US$70 million equivalent by the Board on February 28, 2008, was declared effective and disbursed on March 27, 2008, and closed on schedule on December 31, 2008. The disbursed amount was US$72.4 million due to exchange rate fluctuation of US dollar versus SDR basket. PRSG-5 was appraised in January 2009, approved for US$80 million equivalent by the Board on March 17, 2009, was declared effective and disbursed in full on April 7, 2009, and closed on schedule on June 30, 2010. PRSG-6 was appraised in December 2009, approved for US$115.8 million equivalent by the Board on March 30, 2010, was declared effective and disbursed in full on June 2, 2010, and closed on schedule on June 30, 2011. PRSF-7 was appraised in January 2011, approved for US$104.4 million equivalent by the Board on February 24, 2011, was declared effective and disbursed in full on April 5, 2011, and closed on schedule on June 30, 2012. Initially, the second PRSG series was to include three operations, covering the years 2008-2010. However, an additional operation (PRSG-7) was added at the time of preparation of PRSG-6 (2009) to bridge the transition to a new fiscal calendar in Rwanda while ensuring full alignment between the PRSG support and the Country Performance Assessment Framework (CPAF). The Rwanda Country Assistance Strategy (CAS) FY09-12 originally suggested (p. 21) that the overall financing under these four PRSG operations would amount to US$307 million, which is 20% less than the actual disbursement. In the event, the Bank had leveraged the PRSG program to identify additional concessional funding for Rwanda, including US$10 million provided through the Global Food Crisis Response Program as additional financing for PRSG-5, US$29.8 million from the Crisis Response Window for PRSG-6, and US$24 million from the IDA-15 buffer fund for PRSG-7 (ICR, p.19). This incremental funding was mobilized in response to various external shocks Rwanda was facing in 2008-2010, as well as a reflection of its strong reform performance. The PRSC support was provided as part of the single multi-year budget support framework, established on the basis of the Joint Budget Support Memorandum of Understanding, signed in September 2008 by eight multi- and bilateral donors, including the Bank (ICR, p.15). PRSG financing amounted to about 20% of total official budget support to Rwanda during the period. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Rating: High The objectives of the second PRSG series were fully consistent with the key priorities and expected outcomes of the World Bank Group CAS, FY09-12, which was under preparation simultaneously with the PRSG-4 and was approved by the Board in September 2008. The PRSG program was seen as the overall umbrella for Bank support and policy dialogue with the government. It was aligned with the first strategic theme of the CAS – promotion of economic transformation and removal of bottlenecks to growth. The PRSG series was explicitly mentioned in the CAS document (p. 21). The principal objective of the second PRSG series was to support the implementation of the Rwanda’s second Poverty Reduction Strategy Paper (PRSP), the EDPRS, which was also a key objective of the CAS. The EDPRS was launched in September 2007 and elaborated a development plan for 2008-2012 (ICR, p. 5). The focus of the second PRSG series was on support for two EDPRS flagship programs – Growth for Jobs and Exports, and Governance. The PRSG program was also highly relevant to the country’s conditions as it intended to address the several core interrelated growth bottlenecks and help Rwanda to sustain high growth in the medium term. b. Relevance of Design: Rating: Modest The Program's causal chain was broadly plausible, as there was a logical link between the program’s objectives, the policy areas and most outcomes and prior actions. It linked policy support from the Bank to broad regulatory improvements in the core economic sectors and to various improvements in governance. A strong feature of the program design relates to complementarity between PRSC policy reforms and a number of parallel Bank-supported investment projects and advisory activities that provided technical assistance for preparation and implementation of the PRSG-supported policies. The use of a programmatic instrument through a series of Development Policy Operations was appropriate in supporting selected sector policies. Overall the Program under the five policy areas included 39 prior actions. The following table summarizes the breakdown of actual prior actions in the four PRSG operations across the above policy areas. PRSG- PRSG- PRSG- PRSG- Total 4 5 6 7 1. Human Capital 2 1 0 1 4 2. Economic Infrastructure 2 3 4 3 12 3. Agricultural productivity 1 1 1 1 4 4. Financial sector 1 1 1 2 5 5. Governance 4 2 4 4 14 TOTAL 10 8 10 11 39 The prior actions largely followed the triggers that for each PRSG were identified in the PD for the previous operation. However, in the two operations (PRSG-5 and -6) the disconnect between the original triggers and actual prior actions was significant. For instance, out of nine original triggers proposed in the PD for PRSG-5 for the next operation PRSG-6 only four were converted into prior actions without much change, while four were either replaced or significantly modified, and one was dropped completely. At the same time, two new prior actions were added to PRSG-6. Overall, out of 29 triggers originally identified for PRSG-5,-6, and -7 only 13 (less than 50%) were kept as actual prior actions, indicating considerable instability of the program policy framework. At the same, these modifications do not necessarily suggest a serious weakening of the reform program, supported under PRSG series. As stated in the program documents, most policy triggers, as originally formulated, were eventually met, while some with delay. For instance, in the case of PRSG-6, out of five its original triggers that were modified/dropped, four were substantively met, and only one (“signing at least two performance contracts in road maintenance”) remained incomplete at the time of ICR preparation (Table 7 in PD for PRSG-6). However, there were significant design weaknesses that affected the quality of results chain. The core of these shortcomings was derived from the difficulty to align the PRSG reform program with the Common Performance Assessment Framework (CPAF) agreed by all development partners, including the Bank, for the evaluation of reform performance. Beginning with PRSG-5 (2008), it was agreed that all prior actions, triggers, expected outcomes as well as related indicators, used by the donors in their budget support operations, would be taken only from the CPAF. The original policy matrix used in the PRSG-4 was abandoned to align the rest of the program with the CPAF. In retrospect, this Bank commitment to full alignment had proved to be problematic for a number of reasons, including that (i) some CPAF actions were weak; (ii) CPAF actions did not correspond to the most critical reforms needed; (iii) the CPAF did not reflect the totality of government reforms undertaken; (iv) certain CPAF outcome indicators were difficult to measure, etc. It was felt, especially in the course of PRSG-6 preparation, that the alignment with the CPAF might weaken the perception of government performance. More generally, it created tensions with the Bank’s own lending policy (OP 8.60), which emphasizes criticality of selected prior actions and triggers, while too many CPAF-backed prior actions were process oriented. For instance, out of 11 prior actions in PRSG-7 seven could be considered as process oriented as they provided for government studies, strategies and frameworks that did not have immediate regulatory traction. This tension also resulted in significant annual modifications of the policy matrix, as discussed above. The quality of the results framework also shows significant weaknesses, given that there were frequently only weak links between the program’s policy actions and selected outcome indicators. See section 10a for additional detail. Additionally, the program paid insufficient attention to the rationalization of expenditure policies at the sectoral level. Given the PRSG provided budget support and aimed at protecting the critical government expenditure programs, more emphasis on quality of spending, informed by the recommendations of various PERs, should be expected. Macroeconomic framework. The program design took account of Rwanda’s macroeconomic management, which has been satisfactory. Rwanda’s overall macroeconomic situation remained stable over the program period despite external shocks and was supportive of economic growth. Fiscal policy was consistent with macroeconomic stability, and the budget deficit was largely financed by donor assistance. The Central Bank’s monetary policy stance has been largely appropriate, while the level of international reserves was adequate. The medium term macro-economic framework was supported by the IMF’s three-year Policy Support Instrument (PSI), approved in June 2010, and before that by a three-year Poverty Reduction and Growth Facility (PRGF), which was approved in June 2006. Performance under both IMF programs remained strong for most of the period. 4. Achievement of Objectives (Efficacy): The IEG’s assessment of efficacy is based primarily on the data (when available) for the end-June 2012, the closing date of the program (PRSG-7). These outcome data were separately requested by IEG and provided by the team in August 2015. The ICR, prepared in early 2011, reports the outcome indicators only as of end-June 2010, which were the latest available data at that time. That data point, however, is not considered appropriate for assessment of the program that was closed two years later. The 2012 actual values for the outcome indicators were compared with the program targets for June 2012 as presented in the Policy Document for PRSG-6 (pp. 45-46). The PD for PRSG-7, for an unclear reason, did not contain any targets beyond 2010. While the Bank provided its budget support within closely coordinated donor partnership (Budget Support Harmonization Group), there was clear division of labor across various donors, with limited overlap in policy conditionality. The program indicators also had considerable overlap with several investment lending that were supported by the Bank and in some instances used the same indicators (as in the case of energy and education). Thus, the results below raise questions on attribution. Objective 1: Enhancing the skills of the population – Substantial The Government continued implementation of its policy to improve access and quality of basic education; at the same time it made a stronger focus on upgrading post-basic education and training, the linkages between the education system and labor demand. The key policy developments included: (i) the development of a fully costed strategy for post-basic education; (ii) adoption of the teacher development management policy (TDMP); (iii) the establishment of the Workforce Development Authority in 2009 with the mandate to implement the TVET policy, etc. Progress towards the specific program targets was as follows:  Teacher-to-pupil ratio in primary education decreased from 72.7 in 2006 to 62.1 in 2012 surpassing the target of 65.  Transition from basic to upper secondary education increased from 82% in 2006 to 90.2% in 2012 exceeding the target of 85%.  Percentage of qualified teachers has changed from the average of 87% in 2006 to 95.6% (primary education) and 68.6% (secondary education) in 2014, below the target of 100%.  Indicator “Employer satisfaction with performance of TVET graduates” had no baseline, and was not measured during the program implementation. Objective 2: Building economic infrastructure - Substantial The Government made broad policy progress in several infrastructure sectors, including water, power, and roads. Specific policy highlights comprise consolidation of responsibility for all aspects of water and sanitation management under the single Ministry; institutional strengthening of water and power utilities; adoption of a new tariff structure to ensure financial viability of the newly restructured water utility; updating the policy on private participation in power generation and off-grid distribution to encourage private investments; and adoption of the new Road Act. Progress towards the specific program targets was particularly strong in the area of power generation and distribution, but it was mixed in other sectors:  The share of classified district road network in good condition increased from 15% in 2006 to 37% in 2012 not meeting the target of 43%.  The share of population with access to clean drinking water increased from 64% in 2006 to 71% in 2012 not meeting the target of 83%.  The share of population with access to hygienic sanitation increased from 38% in 2006 to 65% in 2012 meeting the target of 50%.  The number of households and firms with access to electricity increased from 91,332 in 2006 to 308,326 in 2012 surpassing the target of 270,000.  Electricity generation increased from 45 MW in 2006 to 157 MW in 2012 surpassing the target of 130 MW. Objective 3: Raising agricultural productivity - Substantial Policy progress in this area included preparation of the irrigation master plan, increase in budget allocations for irrigation and water management, development of the fertilizer strategy and crop intensification plan, and steps towards expanding the role of private sector in fertilizer distribution. These policy helped to improve availability of key sectoral inputs (fertilizers, water, protected arable land), which have been major drivers of agricultural productivity in Rwanda. Progress towards the specific program targets was as follows:  The share of arable land protected against soil erosion increased from 40% in 2006 to 92% in 2012 surpassing the target of 70%.  Mineral fertilizer use increased from 14,000 in 2006 to 46,000 in 2012, close to the target of 47,600. Objective 4: Deepening and broadening the financial sector – Modest The core area of policy reforms supported under the program was modernization of the national payment system, seen as critical factor to facilitate monetization of the economy. The critical new pieces of legal and regulatory framework, governing the payment system, were adopted and implemented by Parliament and the Central Bank, respectively. Other important financial sector reform included adoption of the regulatory framework for credit bureaus. The specific target under this policy objective was met - credit to the private sector increased from 10% of GDP in 2006 to 14.5% in 2012 exceeding the target of 13.9%, indicating significant deepening of the financial sector. However, no evidence was provided in the ICR with respect to attaining other part of this objective, which was “broadening of the financial sector”, specifically by facilitating expansion in rural credit. Objective 5: Enhancing government capacity , accountability, and transparency - Modest The Government of Rwanda pursued a broad governance reform strategy with a special emphasis on strengthening public financial management (PFM), public procurement, and local government capacity. In the PFM area, the focus under the program was on strengthening the budget monitoring and reporting on the basis of development and rollout of the IFMIS. The specific target under the program was to make the IFMIS fully operational in all government agencies in 2010 and have it integrated with payroll payment system. In the event, this timetable proved to be too optimistic. The ICR reports that the IFMIS implementation was delayed by at least a year against the original timeline, in part due to weaknesses in the original system design. By early 2011 the IFMIS was only piloted in key central ministries, while the payroll system still contained large data gaps. However, since the time of ICR was finalized, the Government sustained its progress in the area of IFMIS rollout. As reported by the team, by 2014 the IFMIS was implemented in 267 government entities, including 60 central government agencies and 31 Districts and Kigali City administrations. This indicates that the respective IFMIS program target was substantively met, while with a considerable delay. Another PFM-related target was linked to improvements in budget accounting and reporting. It was formulated in terms of ratings for the Public Expenditure and Financial Accountability (PEFA) indicator PI-25 (Index on quality of timeliness of annual financial statements), which was expected to improve from C+ in 2007 to B+ in 2011. This target was not met. In the event, the respective indicator deteriorated to D+, as documented by the PEFA report (2010). This was in part driven by the delays in the IFMIS rollout as noted above. As of September 2015, the more recent PEFA report (2014) was not completed. It is worth noting that, despite missing both PFM targets, the PEFA (2010) documented broad-based improvements in the quality of PFM system in Rwanda relative to baseline assessment in the previous PEFA (2007). Specifically, it underlined significant progress (but from a low starting point) in the areas of public procurement, external audit, public access to information, control over the budget arrears, etc. Rwanda has made good overall progress in establishing a modern procurement system, based on comprehensive reforms to strengthen institutional, legal and regulatory framework. Specifically, the Government undertook various steps to establish an institutional framework for procurement oversight, promote transparency and strengthen appeal procedures in procurement, as well as implement a national capacity building strategy. However, the specific procurement indicator under the program remained unmet: the share of public procurement by value that tendered competitively increased from 73% in 2006 to only 75% in 2012 missing the target of 87%. Two other outcome indicators under this objective were as follows:  All 100% of districts achieved at least 80% of their service delivery and development targets in 2012, meeting the target of 75%.  The percentage of government budget agencies achieving at least 80% of their performance targets in the annual work plan was not measured (the target was 35%). The ICR also reports performance for three health outcome indicators, and provides a Satisfactory Outcome rating for progress under the health sector reform. However, under PRSG-5 support to the health policy area was moved to a parallel DPO, the Community Livening Standard Grant (P122157), and has already been accounted for and reviewed under that DPO. Thus, this ICR Review does not assess health outcomes (see above Section 2c). 5. Efficiency (not applicable to DPLs): 6. Outcome: The outcome rating of the program is Moderately Satisfactory reflecting high relevance of objectives and modest relevance of design, modest achievement of two objectives, and a substantial rating for the remaining three. The program helped the government to advance the broad policy reform program aimed at ensuring sustainability of high rates of economic growth over the medium term. While there were improvements in almost all specific areas monitored under the program, in a number of cases progress fell short of the agreed targets. Performance was relatively weaker in the fiscal management area and financial sector that are critical for growth sustainability. More generally, in IEG’s assessment, additionality of Bank support was limited: the Bank demonstrated limited ability to strengthen the Government own reform program, despite providing significant amount of budget support. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The potential risk of policy reversal is reduced given Rwanda’s strong reform track record. However, there remains a residual risk of slowdown in reform implementation due to limited implementation capacity. The country’s internal political situation is generally stable. The political risk to the reform program derives from the remaining regional instability. The major source of vulnerability, however, relates to Rwanda’s high dependence on donor funding. The recent cuts to budget support made by several donors confirmed this risk. Moreover, the country remains vulnerable to external shocks, such as e.g. increases in global food and fuel prices. According to the debt sustainability analysis, the country’s risk of debt distress is moderate, due in part to Rwanda’s narrow export base. On balance, risk to development outcome is Moderate. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: Preparation of the program reflected the recommendations of the Bank’s prior analytical work, including Country Economic Memorandum, Agriculture Policy Note, Public Expenditure and Financial Accountability Assessment, Science, Technology and Innovation Needs Assessment, Investment Climate Assessment, and PEFA Assessment. The PRSG series complemented activities undertaken by a number of Bank investment projects, including Human Resources Development Project, Public Sector Capacity Building Project, Decentralization and Community Development Project, etc. (PD for PSRG-4, pp. 24-26). The program was prepared in close coordination with other development partners within the CPAF donor coordination framework. The team made an extra effort to accelerate the preparation of individual operations under the series in order to accommodate the change in the government’s budget calendar. The results achieved under the series were helped by strong sector dialogue through the sector working groups . This helped to build on policy agreements reached at the sector level, strengthen capacity of sector ministries to present their priorities, and to facilitate timely Bank inputs into CPAF discussions. The risk assessment developed during the project preparation was largely adequate, but it did not envision the forthcoming risk related to the difficulty of full alignment of Bank’s own policies for budget support with the commitments under the CPAF, which was finalized after PRSG-4 approval. The attempt of full alignment resulted in weakening of both the program design and results framework, as explained in sections 3b and 10. Specifically, there was considerable instability in the program policy framework, reflected in significant annual adjustments of triggers and prior actions. While the policy program supported by the PRSG series was rather broad, there is a sense that some critical reforms were not adequately covered, such as e.g. the rationalization of budget expenditure in core sectors. The PD of the last operation in the series, PRSG-7, did not contain the end of the program targets, which posed difficulties in assessing the level of achievement of program outcomes. Quality-at-Entry Rating: Moderately Satisfactory b. Quality of supervision: Program supervision was closely coordinated with other development partners through the team’s active participation in the Joint Budget Support Reviews, which was the main agreed monitoring and supervision tool under the CPAF. Additional supervision and implementation support, including technical assistance, was provided under several parallel Bank investment projects and advisory activities. The stationing of team leader in the Bank’s country office helped to improve communication with the authorities. The Bank also leveraged the PRSG program to provide additional concessional funding for Rwanda and expand the program’s financing envelope. However, only two self-standing supervision reports (ISRs -- for PRSG-6 and -7) for the series of four operations were prepared, and they provided insufficient analysis of progress with policy reforms supported under the program. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government showed strong ownership and commitment to the implementation of its development program, EDPRS, in general and specifically to the policies supported under the PRSG series that mostly derived from the EDPRS. There were no instances of any substantial policy reversal. The Government also succeeded in maintaining macroeconomic stability despite significant external shocks at the time of global financial and food crises. The PRSG series improved donor coordination, harmonization, and greater support for reforms by budget support donors, including the African Development Bank, European Union, and bilateral donors. There were regular consultations with main stakeholders in the context of regular bi-annual EDPRS reviews, including increased dialogue among line ministries; while policymakers used the PRSG series, along with other budget support, to move the reform process forward. However, there were some important delays in the implementation of the agreed reforms, such as e.g. roll-out of the IFMIS, in part driven by the existing Government’s capacity constraints, and also because the original implementation timetable was over-optimistic. Government Performance Rating : Moderately Satisfactory b. Implementing Agency Performance: The main implementing agency for the PRSG program was the Ministry of Economy and Finance (MINECOFIN), which successfully coordinated preparation and implementation of the broad reform agenda across the government. In addition, the MINECOFIN’s Strategic Planning Unit had responsibility for the overall monitoring of PRSG (and more broadly for the EDPRS) implementation. The monitoring reports were produced regularly as part of bi-annual program reviews. However, for several outcome indicators, agreed as part of the results framework for PRSG, respective monitoring data were never collected, despite these indicators being part of the CPAF. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Monitoring arrangements for the PRSG program were based on the use of government systems. The Government, in close collaboration with development partners, including the Bank, developed the monitoring framework for its own development program, EDPRS. Starting from the PRSG-5, monitoring of the Bank-supported program relied exclusively on monitoring indicators that were a sub-set of the indicators used to monitor the EDPRS. The MINECOFIN’s Strategic Planning Unit had responsibility for the overall monitoring. The validation of actual progress was done once a year through a mechanism of Joint Budget Support Reviews, which incorporated views of development partners and civil society. Such monitoring arrangements have reduced the government’s transactions costs associated with monitoring programs of multiple donors. The original results framework, presented in the PD for PRSG-4, contained 23 specific program targets to be attained by the end-2010, the original closing date of the program. This set of indicators and targets was replaced in PRSG-5 driven by the Bank’s commitment to use only the monitoring indicators agreed under the CPAF. There were additional changes to the results framework in the two remaining operations of the series. As a result, out of 20 indicators reported in the ICR, only four were part of the original results framework agreed at the time of PRSG-4. Moreover, the program documents never presented an explicit and complete set of outcome targets for the new closing date of the program (June 30, 2012). This complicated validation of the program’s efficacy. The specific shortcomings in the results framework that weakened the links between the program’s policy actions and selected outcome included: • In several policy areas there was a gap between the program policy objectives and selected prior actions. For instance, in the area of fiscal decentralization the PRSG-supported policies aimed at capacity building of local governments, but not at fiscal decentralization per se. • In several cases there was a disconnect between the actual prior actions and the outcome indicators used to monitor reform progress. For instance, in the area of agriculture productivity the core supported policies aimed at expansion of the private sector’s participation in the distribution of fertilizers. But the indicator used to measure progress in this area was “growth in total amount of mineral fertilizers used”, without breakdown between private and public distribution channels. There was no indicator to measure progress towards broadening the financial sector under Objective 4. • Some outcome indicators in the Results Framework proved to be impractical – it was impossible for the Government to measure them during the entire program period (e.g. “proportion of employees who are satisfied with the performance of TVET graduates”). • There were no any forward looking targets in the final operation of the series, PRSG-7, which, while approved in early 2011, contained targets for mid-2010 (PD for PRSG-7, pp. 53-56). b. M&E Implementation: The M&E implementation was based on the data provided through the process of Joint Budget Support Reviews and coordinated by the MINECOFIN. Monitoring updates were done annually, but information was not collected for several indicators agreed as part of the results framework. Two ISRs prepared by the team as part of program supervision presented data on 12 (out of 20) M&E indicators. As part of the dialogue under the PRSG program, the Bank supported strengthening the national M&E framework for monitoring the EDRSP, including establishment of the M&E secretariat at the MINECOFIN. Alongside the PRSG, the Bank implemented two grants aimed at enhancing national statistical capacity (ICR, p.16). c. M&E Utilization: M&E framework introduced under the PRSG program was used as integral part of the broader EDRSP monitoring effort, which has been fully owned by the government and is well integrated into the national planning and budgeting processes M&E Quality Rating: Modest 11. Other Issues a. Safeguards: No safeguard policies were triggered by this DPL. b. Fiduciary Compliance: The ICR does not discuss fiduciary issues related to the program implementation. The ICR points to recent decline in fiduciary risks in Rwanda driven by ongoing reforms in financial management and procurement, especially at the local level. c. Unintended Impacts (positive or negative): d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Moderately Relevance of objectives is rated high, Satisfactory relevance of design is rated Modest, efficacy of achieving two program’s objective are rated Modest, while the three remaining are rated Substantial. Risk to Development Negligible to Low Moderate Rwanda remains highly dependent on Outcome: donor support and vulnerable to external shocks, which elevate the overall risk for program’s sustainability. Bank Performance: Satisfactory Moderately Performance downgraded due to Satisfactory moderate shortcomings in both the design and results framework.. Borrower Performance : Satisfactory Moderately Performance downgraded due to Satisfactory moderate delays in implementation of the agreed critical reforms and missing several core targets under the program. Quality of ICR: Unsatisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following lessons are taken from the ICR with some adaptation of language:  Given the capacity constraints in a small, low-income economy, such as Rwanda, successful implementation of the program of budget support that covers a broad range of sectors requires significant investments in capacity building. Such capacity support might have a larger impact if it is provided through the entire reform cycle (selection and formulation of specific reforms, prioritization, implementation, and M&E).  Success of the Bank’s support for a multi-sector reform program depends upon the quality of policy dialogue at the sectoral level and extent to which the substantive agreements with sectoral counterparts in the government are reflected in the design of reform program at the central level. The following lessons are additionally suggested by IEG:  The programmatic budget support, such as PRSG, could be an effective instrument to sustain broad structural reforms in the environment of strong government ownership but limited capacity, as it would help to design and implement a reform process through a sequence of small incremental steps. This is especially important in the policy areas with high capacity requirements, such e.g. procurement, decentralization, and civil service reforms.  While participating in donor coordination agreements (e.g. on shared conditionality for budget support), Bank teams need to maintain some degree of operational discretion in their decisions regarding the core programs’ design elements, such as prior actions and monitoring indicators. Otherwise, they are exposed to a risk of tension between the Bank’s commitments under donor agreements and Bank’s own operational policy on budget support. 14. Assessment Recommended? Yes No Why? Assessment is recommended for this PRSG series to be undertaken jointly with the follow-up PRSG operation. Rwanda has accumulated an interesting experience of donor coordination and alignment of donor budget support programs with the national development plan. The proposed assessment would provide lessons about the Bank's contribution to this process, specifically about the additionality of the Bank's program (vs those by other development partners) as well as about practical difficulties of simultaneous compliance with the Bank’s internal policies for budget support and commitments imposed on the Bank by participation in donor coordination agreements at the national level. 15. Comments on Quality of ICR: The ICR presents a detailed analysis of policy developments and implementation experiences under the PRSG series. The ICR is candid in its presentation of practical difficulties of aligning the design of the program with the Bank’s commitments made under the donor coordination framework, CPAF. It also suggests several useful insights regarding the lessons learned from the program. However, the ICR has two major shortcoming in the assessment of the program’s outcomes that led to a disconnect between the ICR’s ratings and those presented in this review. First, the ICR made its assessment of the program’s efficacy on the basis of the data for mid-2010, which are considered by IEG to represent just intermediary program results as this date was two years before the official closing date for the program. The team decided to have the main ICR mission in early 2011, immediately after the approval of PRSG-7. At that time, because of peculiarities of Rwanda’s budget calendar, the only available outcome data were for mid-2010. While, the ICR was ultimately finalized in late 2012, its core evidence regarding the program outcomes was never updated. Second, the ICR’s way of assessing program’s outcomes is inconsistent with the evaluation guidelines. It rated progress under the program for each individual outcome indicator separately, not per broader program development objective. This greatly fragmented the results of evaluation in the ICR. In addition, the ICR incorrectly used the 6-point scale for assessing the efficacy instead of the 4-point scale suggested by the guidelines. The ICR exceeds (28 pages) the 15 page limit for the main text. a.Quality of ICR Rating : Unsatisfactory