95123 PROGRAM-FOR-RESULTS: TWO-YEAR REVIEW Operations Policy and Country Services March 17, 2015 ACRONYMS AND ABBREVIATIONS ACGs Anticorruption Guidelines BP Bank Procedures CAS Country Assistance Strategy CODE Committee on Development Effectiveness CPF Country Partnership Framework CPS Country Partnership Strategy CSOs Civil Society Organizations DLIs Disbursement-Linked Indicators DPF Development Policy Financing ESIA Environmental and Social Impact Assessment ESSAs Environmental and Social Systems Assessments FCS Fragile and Conflict-Affected Situations FSA Fiduciary Systems Assessment GoM Government of Maharashtra GoU Government of Uruguay GPs Global Practices GRMs Grievance Redress Mechanisms IBRD International Bank for Reconstruction and Development IDA International Development Association INT Integrity Vice-Presidency IPF Investment Project Financing IRAFs Integrated Risk Assessment Frameworks ISRs Implementation Status and Results Reports M&E Monitoring and Evaluation MDG Millennium Development Goals MDTF Multi-Donor Trust Fund MOU Memorandum of Understanding OC Operations Committee OP Operational Policy P&PF Policy and Procedure Framework PADs Program Appraisal Documents PAPs Program Action Plans PDO Program Development Objective PforR Program-for-Results ROC Regional Operations Committee SORT Standardized Operational Risk-Rating Tool WBG World Bank Group PROGRAM-FOR-RESULTS: TWO-YEAR REVIEW Contents Executive Summary ..................................................................................................................... iii I. Introduction .........................................................................................................................1 II. Background .........................................................................................................................3 III. Overview of Progress ..........................................................................................................7 A. Preparation, Approval, and Implementation of Operations ...........................................7 B. Feedback on Overall Experience with Use of PforR Instrument .................................10 C. Country and Sector Dialogue and Choice of Financing Instrument ............................12 IV. Experience with Specific Features of PforR Operations ...............................................15 A. Program Definition ......................................................................................................15 B. Results Frameworks and Disbursement-Linked Indicators .........................................17 C. Technical Assessments ................................................................................................22 D. Fiduciary Assessments .................................................................................................24 E. Anticorruption Guidelines ...........................................................................................27 F. Environmental and Social Assessments.......................................................................28 G. Risk Assessment and Management ..............................................................................30 H. Institutional Strengthening, Capacity Building, and Program Action Plans ...............32 I. Partnerships ..................................................................................................................34 J. Transparency, Accountability, and Citizen Engagement .............................................35 K. Implementation Support ..............................................................................................36 V. Internal Organization and Management ........................................................................38 VI. Summary of Consultations ...............................................................................................40 VII. Summary of Findings and Proposed Way Forward .........................................................41 A. Responding to the Demand for PforR ..........................................................................41 B. Limited Exception to Procurement Exclusions............................................................43 C. Technical and Editorial Revisions to OP/BP 9.00 and ACGs .....................................44 VIII. Next Steps and Request for Approval .............................................................................46 Annexes Annex 1. Results-Based Initiatives .............................................................................................. 49 Annex 2. Surveys: Methodology and Results .............................................................................. 53 Annex 3. Report on Qualitative Data ............................................................................................ 87 Annex 4. PforR Two-Year Review Consultations ...................................................................... 101 Annex 5. Program-for-Results Pipeline ...................................................................................... 107 Annex 6A. Draft Policy .............................................................................................................. 109 Annex 6B. Technical, Editorial, and Substantive Changes ........................................................ 117 Annex 7A. Draft Directive .......................................................................................................... 119 Annex 7B. Technical, Editorial, and Substantive Changes ........................................................ 134 Annex 8A. Guidelines on Preventing and COMBATING FRAUD and Corruption in Program- for-Results Financing ........................................................................................... 137 Annex 8B. Summary of Changes to ACGs ................................................................................ 143 Boxes Box 1. Expanding Opportunities for Development Effectiveness with PforR ............................... 5 Box 2. Lessons from the Rollout of PforR in Morocco, Tanzania, and Vietnam ......................... 14 Box 3. Aligning PforR Programs with Government Programs .................................................... 16 Box 4. Ethiopia Health PforR DLIs .............................................................................................. 20 Box 5. PforR Expenditure Frameworks ........................................................................................ 23 Box 6. Fiduciary System Assessments: Supporting Institutional Strengthening .............................. 26 Box 7. What does an Environmental and Social Systems Assessment Entail? ............................ 28 Box 8. Program-Level Opportunities to Improve Environmental and Social Risk Management 30 Box 9. PforR Potential for Supporting Fragile States ................................................................... 33 Box 10. Complaint and Grievance Mechanisms........................................................................... 35 Box 11. Citizen Engagement in PforR Operations ....................................................................... 36 Box 12. Use of IT to Enhance Citizen Engagement ..................................................................... 36 Box 13. Lessons learned from Vietnam Rural Water Supply and Sanitation PforR .................... 38 Figures Figure 1. PforR Portfolio & Pipeline by Region............................................................................. 7 Figure 2. PforR Portfolio & Pipeline by Global Practice ............................................................... 8 Figure 3. Actual Bank Disbursements .......................................................................................... 10 Figure 4. Actual vs. Expected Cumulative Bank Disbursements ................................................. 10 Figure 5. Likelihood of Using PforR Instrument Again in Next 2-5 Years Among PforR Users 11 Figure 6. Greatest Strengths of PforR Instrument as Identified by PforR Users .......................... 11 Figure 7. Greatest Challenges of PforR Instrument as Identified by PforR Users ....................... 11 Figure 8. Likely Demand for PforR as Reflected in Survey Responses from Clients ................. 12 Figure 9. Value of PforR by Sector, According to PforR Users ................................................... 12 Figure 10. Verification Protocols for DLIs by Type..................................................................... 21 Figure 11. PforR as a Percentage of IBRD/IDA Commitments (Two-Year Basis) ..................... 41 Figure 12. PforR as a Percentage of IBRD/IDA Commitments (Fiscal Year Basis) ................... 42 Tables Table 1. Complementary Financing Instruments ............................................................................ 4 Table 2. IBRD/IDA Lending, Total and for PforR Operations, FY12 to FY14 ............................. 7 Table 3. Summary of Approved PforR Operations ........................................................................ 9 Table 4. Awareness/Familiarity with PforR among Non-Users .................................................. 12 Table 5. PforR Programs and Government Programs .................................................................. 15 Table 6. Ethiopia Health MDGs PforR: Results Chain ................................................................ 17 Table 7. Characteristics of DLIs for Approved PforR Operations ............................................... 18 Table 8. Examples of DLIs for Selected Global Practices............................................................ 19 Table 9. Formulation of DLIs and Associated Disbursement Mechanisms ................................. 19 Table 10. Perceptions of Results Frameworks, DLIs, and Verification Protocols ....................... 22 Table 11. Risk Ratings at Board Approval for Approved PforR Operations ............................... 31 ii PROGRAM-FOR-RESULTS: TWO-YEAR REVIEW EXECUTIVE SUMMARY 1. In early 2012, the World Bank introduced a new financing instrument, Program-for-Results (PforR) Financing, to complement the Development Policy Financing (DPF) and Investment Project Financing (IPF) it already provided. PforR has the following features: (a) Finances and supports borrowers’ programs of expenditures and activities. (b) Disburses upon achievement of program results. (c) Focuses on strengthening the institutional capacity and systems needed for programs to achieve their desired results. (d) Provides assurance that Bank financing is used appropriately and that the environmental and social impacts of the program are adequately addressed. 2. Review of Experience. This two-year review of PforR has two major objectives: (a) to assess the early experience with the design and implementation of PforR operations and the challenges faced by borrowers, development partners, and Bank staff; and (b) to identify emerging early lessons and recommend any proposed changes to the PforR framework that would strengthen the instrument. The review has involved a broad literature and desk review, internal and external surveys of those who have used the PforR instrument and those who have not, structured interviews, and stakeholder consultations. The review has been under the oversight of both an internal advisory group and an independent external panel of experts. A. Overview of Progress 3. Development and Implementation of Operations. As of December 31, 2014 the Board has approved 22 PforR operations, providing $3.5 billion of Bank financing to support a total of $8.5 billion in government programs. The pipeline is strong, with another 17 operations at an earlier stage of preparation (i.e., having already completed the concept stage), totaling $4.5 billion in Bank financing. All Regions now have at least two approved PforR operations, and most have additional operations under preparation. PforR operations also cover a wide range of the Global Practices (GPs). 18 of the 22 approved operations are now effective and, as of December 31, 2014, Bank disbursements were on track relative to expectations, with $640 million disbursed, of which $442 million was against prior and achieved results. Overall implementation of PforR operations is also on track, with performance ratings of satisfactory or moderately satisfactory for all but one operation and risk ratings stable through early implementation. 4. Overall Feedback. For both government officials and Bank staff who have experience in using the PforR instrument, feedback has been generally positive. All the government officials and 78 percent of Bank staff who responded to the survey of PforR users reported that they were somewhat satisfied or very satisfied with the instrument. In addition, the majority of both groups of respondents indicated that it is somewhat likely or very likely that they would use the instrument again in the next two to five years. Both groups said that the instrument’s greatest strengths are its focus on results, use of government systems, and focus on program institutional strengthening. They identified as its challenges its novelty and issues related to its newness, the heaviness of the assessments, and challenges in the articulation and application of some aspects of the Anticorruption Guidelines (ACGs). The structured interviews of government officials, development partners and senior Bank managers have reconfirmed both the positive survey feedback and the main challenges identified. In addition the survey of Bank country directors/country managers and their counterparts for countries that have yet to use the PforR instrument shows potential interest in using the instrument. B. Experience with Specific PforR Features 5. The review has looked carefully at each of the specific features of the PforR instrument with a view to determining whether the instrument’s policy requirements have been met, assessing experience to date, and identifying needed changes and improvements. 6. Experience to Date. Both government officials and Bank staff that have used the PforR instrument have indicated satisfaction with most of the specific features of the instrument. The review confirms that the policy requirements of the instrument are being met. All the approved PforR operations support government programs of expenditures for which high development impact is expected. All PforR operations set out a results framework for the PforR Program that includes clearly defined objectives and indicators and provide details on the agreed monitoring and evaluation arrangements. Disbursement-linked indicators (DLIs), verification protocols, and disbursement arrangements are discussed extensively in the documentation, in the Bank’s reviews of PforR operations during preparation, and during negotiations. The technical, fiduciary and environmental and social assessments that have been carried out have been comprehensive in scope and rigorous and have ensured that high value procurements and activities likely to have significant adverse environmental or social impacts have been excluded from PforR operations. The Bank’s ACGs have been applied to all of the approved PforR operations. All of the approved PforR operations highlight specific and priority capacity building activities, and include system- strengthening activities. Partnerships with governments, other development partners, and other stakeholders have been strengthened. Citizens now have mechanisms to know what results government programs are expected to achieve, how those results are verified, and how government programs are performing in relation to these results. Finally the fact that PforR requires a different kind of implementation support from what is expected under IPF operations (because of the focus on reviewing progress on achievement of results and DLIs, providing support on capacity building, and monitoring the performance and adequacy of program systems) is recognized. 7. Issues. While the policy requirements of the PforR instrument are being met, in some cases this has come at a cost. The application of the ACGs has frequently required extensive discussions between the Bank and borrowers during the preparation and negotiations of PforR operations. While borrowers have been supportive of cooperating with the Bank on measures to prevent and combat fraud and corruption in programs, the current articulation of the ACGs has led in some cases to a reduction in the scope of the PforR Program or a decision not to use the instrument. Concerns have been raised about the potential impact of procurement exclusions on the overall integrity of some PforR Programs as well as about the perceived inflexibility of the instrument’s audit requirements. These factors and the broader concerns about the high risks of involving the Bank in specific activities have also resulted in only six of the 22 approved operations defining the iv PforR Program as the same as the government program—with potential consequences for the leverage and impact of the instrument. Moreover, the process of formulating DLIs has been reported as challenging and requires additional training and guidance. In addition and through the surveys, structured interviews and other feedback, government officials and Bank managers and staff have asked for more training, guidance and sharing of best practices with respect to each of the specific features of the instrument. C. Internal Organization and Management 8. Following approval of the PforR instrument and taking account of the five percent cap on PforR commitments agreed between Management and the Board, there was no official launch of the instrument. Instead, enhanced corporate oversight arrangements were put in place to help ensure that PforR operations were processed in accordance with policy requirements and emphasis was placed on intensive, focused hand-holding for a set of operations put forward by the Regions, with task teams receiving advice from a corporate support team as well as a range of other guidance, support, and training. This learning approach has helped all involved to learn and adapt, has provided good support to teams, and has helped in the development of the tools necessary to institutionalize the instrument. It has also fostered a small but engaged community of practice that has contributed to developing the instrument and to sharing information across sectors and Regions. The Bank has also used special materials, information sessions, and other means to inform clients, development partners, and other external stakeholders about PforR. Still it is clear from the survey results and other feedback that wide disparities in the extent of knowledge of the instrument remain both across countries as well as within the Bank. D. Consultations 9. Consultations on the preliminary findings of this review were carried out with PforR teams in government, development partners, civil society organizations, the private sector and other interested parties. A dedicated website was created and stakeholders were invited to submit their comments in a variety of formats on the background documents and presentations. In addition, face to face and videoconference sessions were organized with 15 countries that have used the PforR instrument and with several bilateral development partners and multilateral organizations. The feedback from this consultation process has been largely positive, revealing significant support for the PforR instrument. The feedback also confirmed that the main findings of the review were consistent with stakeholders’ experience. The clear objectives of PforR, the focus on results and improved collaboration between the Bank and governments were consistently highlighted as positive features. Challenges identified included: how best to define and agree on DLIs; the articulation and application of the ACGs; the impact of “exclusions”; and the heaviness of the assessments. Countries also asked for more capacity building, communications and training. Feedback from the consultations has been summarized and is reflected in the review’s findings and recommendations. E. Findings and Proposals 10. The review finds that the PforR instrument has been successfully rolled out across a broad range of countries and sectors, policy requirements have been met, and implementation for all but one of the approved operations is broadly on track. Those who have used the instrument have v positive opinions, and findings suggest there is interest in the instrument among those who have not yet used it. No major changes in the design of the instrument appear to be necessary at this time, but some adjustments and areas for improvement have been identified. 11. PforR Commitments. While the number of approved PforR operations has not yet reached the five percent cap, the growth in demand indicates that the cap will be a constraint to the future processing of PforR operations. While IPF is expected to remain the Bank’s dominant instrument for many years to come, easing the constraint on PforR commitments is critical to respond to clients’ demand and to provide the needed support and investments for the instrument to reach its full and desired potential. In these circumstances, Management’s preference has been to proceed without a cap. However, from the consultations that Management has undertaken it is clear that there are different views on this matter among Executive Directors—some Executive Directors support proceeding without a cap, while others prefer to proceed more cautiously. In these circumstances, Management will continue to invest in enhanced quality assurance and risk management for PforR operations and will carefully monitor the growth of PforR commitments in total and in relation to the growth in total IBRD/IDA commitments. Management also proposes that if and when it is anticipated that the three year average of PforR commitments could exceed 15 percent of the three year average of total IBRD/IDA commitments, Management will inform the Executive Directors and discuss how best to address the future demand for the instrument. 12. Policy Modification: Limited Exceptions to the Procurement Exclusion. The PforR policy (OP 9.00) provides that activities that involve procurement of goods, works and services under high-value contracts may not be included in the program financed by the instrument. However, in some instances this exclusion requirement may affect the development impact and integrity of the Program. Thus, in light of experience, Management proposes that the current procurement exclusion policy requirement be amended to provide that in exceptional cases such activities may be included in the Program financed by PforR if they are deemed to be of critical importance to the integrity of the Program and the monetary value of the contracts is deemed to be modest in relation to the overall size of the Program. Such exceptions would be subject to the approval of the Managing Director and Chief Operating Officer. 13. Technical and Editorial Revisions of OP/BP 9.00 and the ACGs. While the above constitutes the one proposed substantive policy change there are other additional necessary editorial and technical changes, which do not constitute a change in policy. First in accordance with the new Policy and Procedure Framework (P&PF), the paper proposes a new PforR Policy and Directive which will replace Operational Policy (OP) and Bank Procedures (BP) 9.00. Second, while both staff and borrowers appreciate the goals and principles of the ACGs, the current articulation of the ACGs has made them a point of unnecessary contention for many borrowers. Accordingly, some modifications to the ACGs are being proposed as part of this review and should help ensure the further successful rollout of the PforR instrument, while maintaining the overall approach and principles set out in the ACGs. 14. Enhancing the Rollout of the PforR Instrument. The review has found a number of areas in which staff, borrowers, and other stakeholders expressed a wish for more information or guidance. It recommends a range of measures to provide support to Regions and GPs for the further rollout of the instrument; revise and update guidance, templates, instructions, and systems; communicate with and better inform government counterparts and other stakeholders about the vi instrument; strengthen and expand training programs; and continue with the learning approach to the instrument. 15. Request for Endorsement/Approval. Management requests that the Board of Executive Directors (a) endorse the new proposal regarding the PforR IBRD/IDA commitments as described in paragraph 91 of the paper, and (b) approve the limited exception to the policy requirement to exclude from PforR operations activities that involve the procurement of goods, works and services under high-value contracts, as described in paragraph 93 of the paper. vii PROGRAM-FOR-RESULTS: TWO-YEAR REVIEW I. INTRODUCTION 1. In early 2012, the World Bank introduced a new lending instrument, Program-for-Results (PforR), with several innovative features. It would: (a) finance and support borrowers’ programs of expenditures and activities; (b) disburse against achievement of program results rather than against inputs; (c) focus on strengthening institutional capacity to implement the program; and (d) provide assurance that the Bank’s financing is used appropriately and that the environmental and social impacts of the program are adequately addressed.1 2. Review of Experience. The Board paper recommending approval of the new instrument stated: “After about two years of implementation, Management will carry out an initial review of experience with the new instrument.” Management recognized that it would take time for programs to be prepared, approved, and launched, and that implementation experience in the first two years would be limited. Nonetheless, it expected that some lessons could be learned from an initial review of the preparation of operations and early implementation experience. For instance, were clients and teams able to apply the policy, support government programs, and improve the focus on results? An annex to the Board paper listed some of the questions the review might address and outlined some of the methodologies that might be used. 3. Objectives of Review. The two major objectives of this review are to: (a) assess the early experience with the design and implementation of PforR operations and the challenges faced by borrowers, development partners, and Bank staff; and (b) identify early lessons and recommend any proposed changes to the PforR framework that would strengthen the instrument. These objectives reflect the discussions with the Committee on Development Effectiveness (CODE) on the concept note2 as well as with the Board at the time the instrument was approved. 4. Methodology and Process. The review was conducted using the following methodological approach and process:  A literature and desk review was carried out by five working groups organized thematically around the following areas: overall management and organization, results and disbursement-linked indicators (DLIs), fiduciary aspects, environmental and social aspects, and program scope and related capacity building efforts and partnerships.  Two surveys were conducted, both internally and externally. One survey was sent to country directors/country managers and their counterparts in countries that have no PforR operations under implementation or preparation, to assess the nature and extent of the knowledge and dialogue around the PforR instrument. The other survey was sent to Bank managers, task teams and their country counterparts, asking about their perspectives on PforR operations that have been approved or that are under preparation. 1 A New Instrument to Advance Development Effectiveness: Program-for-Results Financing (R2011-0282, IDA/R2011-0350), December 29, 2011 (hereafter PforR Board paper). 2 Program–for-Results Two Year Review: Concept Note (CODE2013-0044), November 11, 2013.  Following the quantitative surveys, to provide more qualitative insights about the experience and lessons learned, structured interviews were conducted with government officials and with senior Bank managers.  Consultations on the findings of the review were undertaken with key stakeholders.  Task teams and managers who have worked on the instrument participated as members of the working groups and of a sounding board that vetted findings of the review to ensure that they are consistent with the experience of front-line staff working on the operations.  Two oversight groups were established: an internal advisory group of directors representing various Regions and Practices from across the Bank, who provided guidance and advice on the review at various milestones; and an Independent External Panel of Experts3 comprised of technical experts with relevant skills and experiences. In addition to monitoring the process and providing guidance to the Bank team working on the review, the Independent External Panel of Experts has provided a statement to the Board to accompany the staff report. 5. Review by CODE. In November 2013, CODE considered the concept note for the review. CODE broadly supported the scope and methodology of the proposed review and endorsed the concept note. In October 2014, CODE met to discuss the findings of the review4 and the comments of the External Panel.5 Members expressed broad support for the key features of the PforR instrument in light of the WBG strategy and twin goals, and were encouraged that the Review affirms PforR’s potential to focus on delivery of development results, capacity- and system- building for clients, greater country ownership and enhanced partnership with development partners. Members noted the positive assessment of the early implementation experience, as evidenced by the feedback from government officials, Bank staff and the External Panel. Members acknowledged the Review’s findings that the PforR instrument has been successfully rolled out across a range of countries and sectors, that early PforR operations have met Bank policy requirements, and that implementation is broadly on track. Members concurred that PforR operations should support augmented transparency, accountability and citizen engagement. Members and speakers discussed the various proposals put forward in the Review, including lifting the cap on PforR commitments, certain exceptions to the procurement exclusions, and editorial revisions to the Anticorruption Guidelines (ACGs) and noted the External Panel’s concurrence with Management’s findings and proposals. CODE encouraged Management to explore options and suggestions for moving forward and requested Management to come back to the Board with proposals for consideration. This paper responds to that request and takes into account the broader discussion in the CODE meeting. 3 Panel members: K.Y. Amoako, President, African Centre for Economic Transformation; Alison Evans, Policy Advisor, Evaluator in International Development; Alan Gelb, Senior Research Fellow, Center for Global Development; Paul O’Brien, Vice President for Policy and Campaigns, Oxfam; Jan Piercy, Executive Vice President, ShoreBank International (former US Executive Director at the World Bank); and Miguel Székely, Public Policy Advisor, Mexico. 4 Program-for-Results: Two Year Review (CODE2014-0038). 5 Comments of the External Panel (CODE2014-0039). 2 6. Structure of Paper. Following this introduction, Section II of the paper provides background on the rationale for the new instrument and its main features. Section III provides an overview of progress on the preparation, approval, and implementation of PforR operations and on the broader dialogue on the instrument across client countries, and feedback on the use of the instrument. Section IV summarizes experience with specific features of PforR operations and reviews how the PforR instrument has been rolled out. Section V reviews how the Bank is organized and managed internally to make effective use of the instrument. Section VI summarizes the consultations on this review. Section VII presents the summary of findings and proposed way forward. Section VIII focuses on next steps and the request for approval. Annexes provide additional and supporting information. II. BACKGROUND6 7. The PforR instrument is designed to enable the Bank to better respond to changing development needs, meet demand from client countries, and enhance development effectiveness by supporting a government program of expenditures, building institutional capacity, and tying financing to achievement of results. Client countries are increasingly implementing their own programs for development and poverty reduction that are rooted in their own legal, policy, regulatory and institutional environments and are asking development partners for financing and expertise to improve the programs’ effectiveness and efficiency in achieving results. The Bank was not able to adequately meet this demand with existing instruments, hence the introduction of PforR. 8. Key Features of the PforR Instrument. The PforR instrument has the following features: (a) Finances and supports borrowers’ programs. These programs, comprising expenditures and activities, can be ongoing or new, sectoral or sub-sectoral, and national or subnational programs, as well as community development programs. (b) Disburses upon achievement of program results. Disbursements are determined by the achievement of monitorable and verifiable indicators, rather than by inputs. (c) Focuses on strengthening the institutional capacity and systems needed for programs to achieve their desired results. A priority area for both preparation and implementation support is to strengthen the capacity and systems of the institutions that implement the program, thereby enhancing development impact and sustainability. (d) Provides assurance that Bank financing is used appropriately and that the environmental and social impacts of the program are adequately addressed. The Bank assesses the program’s fiduciary and environmental and social management systems and agrees with the borrower on any additional measures needed to provide the necessary assurances. 9. Complementary Instruments. With PforR, the Bank now has three complementary financing instruments to offer client countries: policy and institutional support through Development Policy Financing (DPF), project support through Investment Project Financing (IPF), and program support through PforR. While all Bank instruments focus on development results, borrowers are able to choose from a wider range of instruments to suit their objectives, the 6 Drawn mainly from the Executive Summary of the December 2011 PforR Board paper. 3 development challenge, results, and risks they face (see Table 1). The Bank extends financing that a client requests under the different instruments, taking into account its Country Assistance Strategy/Country Partnership Strategy/Country Partnership Framework (CAS/CPS/CPF) process; its assessment of the country’s policies, programs, and institutional capacity; and the specifics of each proposed operation. Table 1. Complementary Financing Instruments Category Project support: IPF Program support: PforR Policy support: DPF Purpose Supports specific investment Supports government Supports policy and operations programs or subprograms institutional actions Disbursement Disburses against specific Disburses upon achievement Disburses against policy and mechanism expenditures that support the of results according to institutional actions operation performance indicators Implementation Bank IPF rules and procedures Program systems Country policy processes mechanisms Funds for specific expenditures Funds for specific expenditure Non-earmarked funds for programs general budget support 10. Instruments Respond to Different Development Challenges. IPF is the Bank’s main instrument to finance investment projects. It supports a heterogeneous range of operations, but their common characteristic is typically the financing of specific investment activities that involve a set of expenditure transactions, most of which are used for the purchase of works, goods and services. IPF is a practical and effective way of supporting the achievement of results when risk management is on the inputs side (e.g., construction of a large infrastructure project or purchase of expensive and technically complex equipment) and when technical design and implementation challenges are critical bottlenecks to achieving results. DPF will remain the primary Bank instrument to support policy and institutional actions. It focuses on discrete actions that are under the direct control of governments, and it links disbursements to evidence that those actions have been taken. DPF is a practical and effective way of supporting policy actions that help create the enabling conditions for improving results – for example, when new regulations are required for better functioning of markets or new policy frameworks are necessary to improve government efficiency. PforR is the instrument of choice when the objective is to support the performance of a government program using the government’s own systems and institutions, when the results require expenditures, and when the risks to achieving the program’s objectives relate to the capacity of the systems to achieve better results. 11. Building on Experience. The design of the PforR instrument benefited from the experience the Bank has gained in designing and implementing IPF and DPF operations—specifically, experience with technical and design issues, results definition, fiduciary systems, environmental and social impacts, risk management, and improving policy environments. It also built on long experience—both inside and outside the Bank—with sectorwide, program-based, and results- focused operations. As the instrument was rolled out, the Bank team has continued to engage with development partners to share experiences and lessons. At the same time, a number of other partners (including the Asian Development Bank) have been looking to PforR as an example of how to expand their menu of instruments to include a results-based financing option. This knowledge sharing among the community of practitioners is proving to be very useful and needs 4 to continue. (Annex 1 includes details on a range of initiatives by other development partners and the Bank that are relevant to PforR). 12. PforR in the Context of the World Bank Group Strategy. When PforR was approved, it was envisaged as an instrument for the future. Its focus on results and its support of government programs and capacity represented a new approach to financing development. Since then, the World Bank Group (WBG) Governors have endorsed two goals to guide the institution and a WBG Strategy that maps out broad directions for the institution. The PforR instrument is fully in line with the WBG Strategy and can be an important element of its implementation. By emphasizing delivery of government programs, the PforR epitomizes problem-solving engagement and the “moving from a project mentality to a development solutions culture” that is a cornerstone of the Strategy. The PforR also has an explicit objective to foster partnerships in support of borrower programs—also central to the Strategy—recognizing that alignment of goals, deliverables, and results of development partners enhances development effectiveness. With its ability to measure results, the PforR represents an approach to accountability that also features in the WBG Strategy (see Box 1). Box 1. Expanding Opportunities for Development Effectiveness with PforR Development Solutions Culture. By using PforR to support local government operations, Bank operations have been able to both focus on improving local government performance and ensure that investment plans are properly implemented. For example, the Government of Tunisia is moving from a very centralized government system to a more decentralized structure. The Bank is supporting this transformation with the Urban Development and Local Governance PforR, which aims to improve municipalities’ performance and local governance, empowering them to make decisions, raising their capacity to deliver services, and making them more accountable to their citizens. The use of the PforR instrument has been indispensable for supporting this transformational initiative as it allowed for a shift in the planning and implementation process of the local grants to focus on results and institutional capacity building and identified changes needed at the institutional/system level that made this transformational operation possible. Fostering Partnerships. In Ethiopia, several development partners have supported the creation of the Millennium Development Goals (MDG) Performance Fund to support the country in achieving the MDG health sector goals. One of the priorities of the MDG Performance Fund is to achieve full harmonization and alignment of all development partners, including the Ministry of Health, through an effective planning system that makes decisions about how all resources are to be used and that monitors overall implementation through one regular reporting system. Using an IPF instrument, the Bank was unable to participate in the MDG Performance Fund, as any Bank-financed procurement would have to follow separate Bank procedures rather than the agreed common procedures. Using PforR for a Bank operation financing the health sector has allowed the Bank to join Ethiopia’s MDG Performance Fund (although, the Bank cannot finance large procurement contracts, such as large vaccine purchases, and these are excluded from the operation). Accountability for Results. In all PforR operations, the focus on results, the fact that disbursements are determined by achievement of monitorable indicators (rather than by inputs), and the establishment of specific and transparent verification protocols are in line with current thinking on ways to enhance development effectiveness and key to effective implementation of the WBG Strategy. In this regard the structured interviews have pointed to the importance of the instrument as a potential “game changer” as it shifts the focus of the dialogue among the client, development partners, and the Bank and ensures a strong sense of accountability regarding achievement of results. 13. Processing of PforR Operations. For each PforR operation, the Bank works with the client to carry out a process of identification, preparation/assessment, appraisal, and implementation support. A central focus of preparation is identifying DLIs and ensuring that each has a credible 5 mechanism in place for monitoring and verifying achievement. Appraisal of each operation is informed by rigorous assessments of three main aspects of the overall program and its expenditures. The technical assessment focuses on the strategic relevance and technical soundness of the program and its expenditure framework, the results framework, and monitoring and evaluation (M&E) arrangements. The fiduciary assessment, covering the procurement and financial management arrangements, seeks to make sure the program funds are used appropriately. The environmental and social systems assessment focuses on making sure that potential environmental and social impacts and risks are adequately addressed. These assessments identify measures to enhance performance, build capacity, and mitigate risks, and the resulting action plan is reflected in the legal agreement between the Bank and the government. Together, the assessments are used to inform an integrated risk assessment.7 14. Implementation Support. During implementation, the Bank team monitors the overall progress of the program, the associated expenditures, and the achievement of results (including the DLIs). It monitors progress in implementing the action plan, changes in the program’s risks, and compliance with the provisions of the legal agreements. Technical support from the Bank focuses on improving systems performance and resolving implementation issues. PforR operations are subject to the same corporate oversight functions as other Bank operations, and the Bank retains the right to carry out investigations that it judges to be necessary and to sanction entities that are found to have engaged in fraud or corruption. The Bank’s debarment list applies to PforR operations. 15. Capacity Building. The PforR instrument focuses on the behavioral and institutional changes that are required to achieve results and manage the associated risks. Hence many PforR operations are expected to require some level of capacity building activities, which are informed by the technical, fiduciary, and environmental and social systems assessments. Capacity building support is provided through different modalities, from direct technical assistance and training to specific actions that may be identified as part of the system assessments or DLIs to strengthen performance. 16. Management/Board Oversight of the Rollout of PforR. When the Board approved the PforR instrument, it was agreed that the Bank would roll out the instrument cautiously, seeking a balance between responding to potentially strong client demand and learning from implementation experience. This approach built on experience with introducing other initiatives, such as Additional Financing. Staff were to be provided with tailored and targeted training, technical guides, and other tools. In addition, Management would enhance corporate oversight arrangements during the initial phase to help ensure appropriate and consistent application of the new instrument, and would put in place a strong feedback mechanism to capture lessons of experience and use them in guidance to staff and in the review of new operations. Finally, Management agreed to limit commitments for PforR operations to five percent of total IBRD/IDA commitments for the first two years from Board approval—a cap that Management would propose to lift if justified by a review of implementation experience. 7 With the introduction of a harmonized risk rating approach across all instruments, this approach will be updated for PforRs (see section IV, G – Risk Assessment and Management). 6 III. OVERVIEW OF PROGRESS A. Preparation, Approval, and Implementation of Operations 17. As of December 31, 20148 the Board has approved 22 PforR operations, providing $3.5 billion of Bank financing to support a total of $8.5 billion in government programs (see Table 2 and 3). The total volume of approved operations at the end of the PforR’s initial two years after approval was $1.8 billion or 2.6 percent of total IBRD/IDA commitments over that period. Over the various fiscal years, there has been a noted increase in the share of PforR commitments from $0.4 billion in FY12 (1.1%) to $0.8 billion in FY13 (2.5%) to $1.7 billion in FY14 (4.2%). In addition, 17 operations are at an early stage of preparation (i.e., have completed the concept stage), totaling $4.5 billion in Bank financing. The schedule for preparation of these and other operations indicates that the cap will be a constraint to the future processing of PforR operations in the pipeline (see Table 2). Table 2. IBRD/IDA Lending, Total and for PforR Operations, FY12 to FY14 (US billion) Commitments FY12 FY13 FY14 FY15 estimates PforR- IBRD lending 0.3 0.1 0.5 1.1* PforR- IDA lending 0.1 0.7 1.2 1.7* Total PforR IBRD/IDA lending 0.4 0.8 1.7 2.8 IBRD lending 20.5 15.2 18.7 20.4 IDA lending 14.8 16.3 22.2 16.6 Total IBRD/IDA lending 35.3 31.5 40.9 37.0 *As of December 31, 2014 a total of $622 million ($300 million IBRD and $322 million IDA) has been approved. 18. Country Distribution. During consultations on the design of the instrument, many observers expected there to be the strongest pick up of the instrument in IBRD countries. However, to date, 67 percent of approved operations have been in IDA countries, although the pipeline is starting to show more pick up in IBRD countries (see Table 2). Survey results and focused interviews point to two issues that have contributed to this pattern: (a) IDA Figure 1. PforR Portfolio & Pipeline by Region countries value highly the instrument’s institutional and capacity building dimension; and (b) some of the PforR elements—particularly the articulation of the ACGs—limited interest in the instrument’s potential to support the larger programs that are more common in some middle-income countries and larger IDA countries. (This is discussed in more detail in section IV.) 8 Numbers presented in the text and tables in this paper are as of December 31, 2014 unless otherwise noted. 7 19. Regional and Sectoral Distribution. All Regions now have at least two approved PforR operations, and most have additional operations under preparation (see Figure 1 and Table 3). PforR operations are also used across a broad distribution of Global Practices (GPs) in which the Bank traditionally provides financing (see Figure 2 and Table 3). Figure 2. PforR Portfolio & Pipeline by Global Practice 20. PforR/IPF Hybrids. PforR policy provides flexibility through the use of hybrids with other instruments. To date three approved operations9 have made use of this option and in all three cases, the IPF component is used only to support technical assistance activities. During interviews teams have indicated they like having the option of a hybrid; however, consultations also revealed the complexities involved with such approach, as it requires the use of two distinct sets of policies and procedures during preparation and implementation. This is consistent with experiences of other instruments with hybrid operations (i.e., IPF and DPF). Still, some countries have appreciated the availability of this option which makes it easier for them than having to prepare two different operations when there is a need for such an approach. 21. Partnerships. Most operations have emphasized the importance of partnerships with other development partners. Six of the approved operations are being financed with other donors.10 In Ethiopia, World Bank support for the MDG Performance Fund for the health sector is being complemented by support from nine other multilateral and bilateral partners. The share of Bank financing varies widely from 13 to 100 percent of the Program in approved operations. Moreover, Bank financing and development partner financing, as a share of these overall government programs, averages 15 and 24 percent respectively (see Table 3). The Bank would not have been able to participate in some of these partnerships—for example, the Ethiopia health and the Morocco human development operations—without PforR. 9 Brazil Strengthening Service Delivery for Growth, Poverty Reduction and Environmental Sustainability in the State of Ceará Program, Mexico Oaxaca Water Supply and Sanitation Sector Modernization Program, and Moldova Health Transformation Program. 10 Uruguay Road Rehabilitation and Maintenance Program, Ethiopia Health MDG Program, Kenya National Integrated Safety Net Program, Tanzania Big Results for Education Program, Rwanda Transformation of Agriculture Sector Program, and Rwanda Public Sector Governance Program. 8 Table 3. Summary of Approved PforR Operations (as of December 31, 2014) (US$ million) PforR Other Total Associated Board IDA/ Bank donor program gov. Country Program name approval Global Practice IBRD financing financing cost program Bangladesh Revenue Mobilization Program 5/9/2014 Governance IDA 60 0 73 73 for Results: VAT Improvement Program Brazil Strengthening Service Delivery 11/21/2013 Trade & IBRD 315 0 416 477 for Growth, Poverty Reduction Competitiveness and Environmental Sustainability in the State of Ceará Croatia Health System Quality and 5/8/2014 Health, Nutrition IBRD 103 0 248 409 Efficiency Improvement Program & Population Ethiopia Health Millennium Development 2/28/2013 Health, Nutrition IDA 100 556 676 4610 Goals Program & Population Ethiopia Local Government Development 5/2/2014 Social, Urban, IDA 380 0 557 557 II Program Rural & Resilience India Third Maharashtra Rural Water 3/12/2014 Water IDA 165 0 235 2100 Supply and Sanitation Program Kenya National Integrated Safety Net7/23/2013 Social Protection IDA 250 261 952 952 Program & Labor Mexico Oaxaca WSS Sector 6/6/2014 Social, Urban, IBRD 45 0 93.5 800 Modernization Program Rural & Resilience Moldova Health Transformation Program 5/22/2014 Health, Nutrition IDA 28.7 0 114 1,964 & Population Morocco National Initiative for Human 6/28/2012 Social, Urban, IBRD 300 0 1,115 2100 Development (INDH) Phase II Rural & Resilience Mozambique Public Financial Management for 6/24/2014 Governance IDA 50 0 130.6 287.6 Results Program Nepal Results-Based Bridges 6/28/2012 Transport & ICT IDA 60 0 148 1250 Improvement Program Pakistan Punjab Governance Reforms for 11/14/2013 Governance IDA 50 0 77 407 Service Delivery Rwanda Transformation of Agriculture 10/31/2014 Agriculture IDA 100 800 1200 1200 Sector Program Rwanda Public Sector Governance 10/31/2014 Governance IDA 100 30 172 250 Program Tanzania Urban Local Government 10/23/2012 Social, Urban, IDA 255 0 255 620 Strengthening Rural & Resilience Tanzania Big Results in Education 07/10/2014 Education IDA 122 130 416 416 Tunisia Urban Development and Local 07/24/2014 Social, Urban, IBRD 300 0 363 751 Governance Program Rural & Resilience Uganda Support to Municipal 3/28/2013 Social, Urban, IDA 150 0 160 341 Infrastructure Development Rural & Resilience Uruguay Road Rehabilitation and 11/13/2012 Transport & ICT IBRD 66 160.5 510 1127 Maintenance Vietnam Results-Based Rural Water 11/1/2012 Water IDA 200 0 260 1313 Supply and Sanitation Vietnam Northern Mountains Urban 6/5/2014 Social, Urban, IDA 250 0 300 300 Program Rural & Resilience 9 22. Implementation. Figure 3. Actual Bank Disbursements Implementation support for PforR is ($M) As of December 31, 2014 very important, but experience is still limited. 18 of the 22 approved operations are now effective. As of December 31, 2014, Bank disbursements were on track relative to expectations, with $640 million disbursed, of which $442 million was against prior ($114 million) and achieved ($328 million) results (Figures 3 and 4). Of the 18 effective operations, Figure 4. Actual vs. Expected Cumulative Bank Disbursements ($M) As of December 31, 2014 nine have been effective for over a year and have on average disbursed 30 percent. Except for the Uruguay roads operation11, overall implementation of PforR operations is broadly on track, with performance ratings satisfactory or moderately satisfactory and risk ratings stable through early implementation. Performance ratings are assessed on a regular basis and if progress is not as expected, teams will downgrade such ratings. 23. Efficiency of PforR Operations. The average Bank financing of a PforR operation (including both approved operations and those under preparation) is about $203 million, which is larger than the average IPF financing. Preparation time for a PforR operation ranges from 6.9 months to 28.7 months, with an average time of 13.6 months (the average for Track II IPF operations in FY14 was 15 months). For the 22 approved PforR operations, the average expense directly related to preparation is at $555,724, and the median is $508,291.12 By comparison, preparation of IPF operations (Track II) had an average cost of $548,000 and a median of $466,000 for FY13 and FY14. During the interviews, teams noted that introducing the instrument to clients and learning for both the client and Bank staff were factors that increased preparation costs and time. B. Feedback on Overall Experience with Use of PforR Instrument 24. For both government officials and Bank staff who have experience in using the PforR instrument, initial feedback has been generally positive. All the government officials and 78 percent of Bank staff who responded to the survey of PforR users (see Annex 2) reported that they were somewhat satisfied or very satisfied with the instrument. In addition, the majority of Bank staff and clients indicated that it is somewhat likely or very likely that they would use the 11 See paragraph 74. 12 Some operations were converted from other instruments; for those for which teams did not change the code, the costs were broken down and the numbers reflect those related to PforR aspects only. 10 instrument again in the next two to five years (Figure 5). This positive feedback has been validated through feedback from the structured interviews of government officials and development partners as well as senior Bank managers (see Annex 3). Figure 5. Likelihood of Using PforR Instrument Again in Next 2-5 Years Among PforR Users External Fig 5 External Internal Fig 5 Internal 12% 0% 8% 9% 6% Very unlikely 7% Very unlikely 15% Somewhat unlikely Somewhat unlikely Somewhat likely Somewhat likely Very likely 27% 51% Very likely Don’t know Don’t know 65% 25. Strengths and Challenges. The government officials and Bank staff surveyed also have very similar perspectives on the strengths and challenges of the instrument. Both groups see the focus on results as the instrument’s greatest strength; they also regard the use of government systems and the focus on program institutional strengthening as major strengths (see Figure 6). The instrument’s novelty and issues related to its newness are seen as the biggest challenge (accounting for 47 percent of the responses of government officials and 26 percent of the responses of Bank staff). Other significant challenges (noted by staff in particular) relate to the exclusions, heaviness of the assessments and the difficulties in applying the ACGs (Figure 7). Figure 6. Greatest Strengths of PforR Instrument as Identified by PforR Users External Internal Fig 6 Fig External 6 External Fig 6 Internal 0% 5%5% 0%4% 2% 2% 9% 1% 4% Efficiency Efficiency in processing in processing the preparation the preparation of the operation of the operation Efficiency i 9% 9% Use of government program systems 25% Use of gov 23% Use of government program systems 23% 16% 16% Focus on results 20% Focus on r Focus on results Focus on program institutional strengthening Focus on p Focus on program institutional strengthening Enhanced World Bank staff support to government in Enhanced identifying and addressing key issues under the Program identifying Enhanced World Bank staff support to government in Usefulness of World Bank assessments in designing the Usefulness identifying and addressing key issues under the Program Program Program 43% Usefulness of World Bank assessments in designing the Other Program 43% 41% Fig 7 External The instrument’s novelty and issues related to 7.7Greatest FigureFig External its newnessof PforR Instrument as Identified by PforR Users Challenges Fig 7 External The instrument’s novelty and issues related to External Fig 7 ExternalFig 7 External The The its The instrument’s instrument’s instrument’s newness novelty novelty noveltyand and andissues issues issues related relatedto related to its newness to Internal Fig 7 Internal 11% Fig 7 External its newness Delays its The in processing newness instrument’s the preparation novelty of the to and issues related 11% 11% 9% The instrume 11% 47% operation its newness 8% 47% Delays processing in processing Delays Delays in thepreparation in processing the preparation of the of the operation the preparation of the 8% 11%8% Delays in processing the preparation of the operation 26% 47% 47% operation Delays in pro 8% 11% 8% Exclusions operation of high-value contracts and activities 11% 47% Delays in processing the preparation of the 11% 8% due to their potential adverse impact on the 11%8% 11% 47% operation Exclusions Exclusions Exclusions Exclusions of of of of high-value high-value high-value high-value contracts contracts contractscontracts and and and and activities due to 20% activities activities activities 11% environment and/or affected people Exclusions of due their their to their Exclusions due theirof potential potential potential high-value potential adverse adverse adverse adverse impact impact impact contracts impact on and on on on the thethe environment activities the 11% their potentia Heaviness environment environment due and/or to their Exclusions environment in of conducting affected and/or and/or potential high-value and/or World people affected affected adverse affected Bank people people impact contracts andon people the activities 11% affected peop due Heaviness assessments Heaviness environment in to their in conducting conducting and/or World affected World Bank assessments Bank people Heaviness in Heaviness Heaviness inpotential in conducting conducting adverse World World impact Bank on the Bank 10% assessments and/or affected people environment 10% assessments Heaviness assessments in conducting World Bank Difficulties inconducting applying the World Bank’s PforR 13% 10% 10% 10% assessments Heaviness in Difficulties World in applying Bank the World Bank’s PforR Anti- Difficulties in 13% Difficulties in applying Anti-Corruption the World Guidelines Bank’s PforR Corruption G 10% 13% assessments Difficulties Corruption Difficulties in applying in applying the Bank’s PforR the World Bank’s GuidelinesWorld PforR 23% Anticorruption Guidelines 10% 13% Anti-Corruption Guidelines Difficulties in applying Anti-Corruption Guidelinesthe World Bank’s PforR 9% Other 13% 13% Other 13% Other Other Anti-Corruption Difficulties Guidelines in applying the World Bank’s PforR 13% Anti-Corruption Guidelines Other Other Other Other 11 Table 4. Awareness/Familiarity with Figure 8. Likely Demand for PforR as Reflected in PforR among Non-Users Survey Responses from Clients External Internal Very likely Somewhat likely Don't know Somewhat unlikely Very unlikely Awareness of PforR 61% 100% 80% Familiarity with the instrument 65% Very familiar 17%* 42% 60% Somewhat familiar 49%* 58% Not very familiar 32%* 0% 40% 34% Not at all familiar 2%* 0% 22% 24% Discussions about the instrument 20% 15% 12% 8% Very informative 29%* 16% 5% 7% 0% discussions 0% Some discussions 41%* 52% Non-Users/know PforR Users of PforR No discussions 29%* 32% Note: 8% of non-users prefer not to respond. *Of the 61% external respondents who are aware of PforR. C. Country and Sector Dialogue and Choice of Financing Instrument 26. The survey of World Bank country directors/country managers and their counterparts in countries that have yet to use the PforR instrument provides evidence that non-users have interest in using the instrument in the future.13 Over half of the non-user counterparts who are aware of the instrument say that they are somewhat likely or very likely to use the instrument in the next two to five years (see Figure 8). Two-thirds of the country directors/managers surveyed also think it is somewhat likely or very likely that the instrument will be used in their countries in the next two to five years. However, country directors/managers have somewhat different ideas than their counterparts on the sectors in which PforR would be most valuable: country directors/managers see the instrument as being mainly of value in the social sectors, while counterparts emphasize its value for a much broader range of sectors (see Figure 9). It is also clear from the survey of non-users that much more information sharing and dialogue about the PforR instrument is necessary; while all country directors/managers who responded to the survey indicate that they are aware of the PforR instrument, only 61 percent of their counterparts are (see Table 4). Moreover, of those counterparts who are aware of the instrument, relatively few have discussed it with the Bank enough to feel well informed. External Figure 9. Value of PforR by Sector, According to PforR Users Internal External Education External Internal Education Health, nutrition, and population 8% 5% Education 17% 0% 6% Health, nu 13% 48% Governance Health, nutrition, and population 6% 13% 8% 5% 61% Governan 13% 48% Agriculture Governance 13% Energy and extractives Agriculture 28% Agricultur 15% 15% and Markets Energyand Finance extractives Energy an Transport andMarkets Finance and ICT Finance a 20% 33% 20% 33% Transport and ICT Transport Poverty Poverty 28% Social Protection and Labor Poverty Social Protection and Labor 20% 20% Water Social Pro Water 0% 28% 28% Macroeconomics and Macroeconomics management fiscalmanagement and fiscal 6% Water 20% 20% Urban, rual Urban, and rural and socialdevelopment social development 0% 67% Macroeco 23% 23% 25% 25% Environment Environment and natural and natural resources resources 6% 22% Urban, ru Trade and competitiveness Trade and competitiveness 6% Environm Trade and 13 See Annex 2 for detailed survey results. 12 27. Country Strategies and PforR. The broad level of interest in using the PforR instrument is reflected partly, but not entirely, in the CAS/CPS/CPF process. Of the 110 CPSs/CASs discussed by the Board since the approval of the PforR instrument, 32 percent explicitly mention PforR and another third refer to programmatic lending more generally or mention sectorwide approaches or other forms of programmatic IPF. This suggests that interest in programmatic lending has stayed about the same over recent years (65 percent of the 51 CPSs/CASs discussed by the Board between 2008 and 2010 proposed some form of programmatic financing, program-based approach, or sectorwide approach in addition to regular IPF or DPF14). However, referring to PforR in the CPS/CAS is not necessarily a good predictor of the actual delivery of PforR operations; a number of countries that mentioned PforR in their CPSs/CASs have yet to begin preparing specific operations. In other cases, PforR operations have been prepared and approved without being explicitly mentioned in the CPS/CAS—a reflection of the fact that the dialogue on PforR is taking place at many different levels and stages, and most notably at the concept note stage for an operation where the choice of financing instrument is a key focus of discussion. The importance of effectively using the discussion of the concept note for specific operations to firm up the choice of instrument has been emphasized in the interviews with senior Bank managers. 28. Regional, Country, and Sectoral Considerations. The introduction of PforR to specific countries appears to be influenced by a broad range of factors. In Africa, for example, it has been influenced by the perspective that PforR is a good instrument for supporting the regional agenda of building stronger institutions and delivering better services. In other countries, alignment with country strategies and the focus on results is clearly driving use of the PforR instrument. In Morocco, for example, the instrument is now regarded as a good fit for efforts to increase the emphasis on results and institutional capacity building. In still other countries, the PforR instrument’s alignment with sectoral or operational strategies is critical—for example, in the Ethiopia health operation the instrument has provided the flexibility to enable the Bank to support multipartner initiatives. 29. Internal Constraints. Bank managers and staff have provided feedback that final decisions on the use of PforR are also affected by a range of internal constraints and trade-offs: (a) comfort in dealing with IPF versus the uncertainty of dealing with a new instrument which some perceive as being cumbersome; (b) differing views and perceptions about the assessments, especially those related to fiduciary and environmental and social issues; (c) concerns about the extensive ‘investigative’ right afforded by the ACGs, which are seen as allowing the Bank to investigate entire government programs; and (d) uncertainties about achieving results with the PforR instrument, when the likelihood of impact using results-based IPF operations (particularly in the human development sector) is known. A broader set of institutional constraints have also emerged from the review: (a) the cultural change that is required for handling PforR; (b) the differing skills requirements of PforR compared with IPFs and DPFs; (c) the related issue of awareness and learning; and (d) the Bank’s soft rollout of PforR (in part because of the financing cap) and its limited investment in introducing the instrument. (These internal institutional issues are discussed in more detail in section V.) 30. Rollout of PforR. The existence of these concerns suggests that while a country’s choice of financing instrument should be based on the specifics of a particular operation and its development challenges and risks, in reality it is also a function of the knowledge and perceptions about available instruments and of the readiness for trying out something that is new and innovative. This is not 14 See PforR Board paper. 13 surprising, particularly in the early stages of introducing a new instrument. But it reinforces the need for a rigorous and well-informed discussion of the instrument at both country and sector levels. Experience from three countries where PforR has now been rolled out in a substantial way gives a sense of the nature and intensity of the effort that may be required (see Box 2). The dialogue on the desirability of using the new instrument is more likely to be productive if it can draw on a substantial body of analytic work and discussion around programs that governments clearly own and are committed to improving. The dialogue needs to involve country and GP management from both the country and the Bank, as well as experts at the task team level, and it is likely to involve a lot of interactions across these different groups and an intensive process of consensus building. Finally, it is critical to have both country and Bank experts who understand the instrument; and even then, a great deal of information sharing and learning by doing will likely be necessary. The structured interviews with government officials have confirmed the intensive nature of the required effort and the importance of sharing experiences/good practices and clearly explaining the value added of PforR compared with other instruments. Box 2. Lessons from the Rollout of PforR in Morocco, Tanzania, and Vietnam Tanzania and Vietnam already have two PforR operations under implementation (Urban Local Government Strengthening and Big Results in Education in Tanzania and Results-Based Rural Water Supply and Sanitation and Northern Mountains Urban Program in Vietnam) while in Morocco the first PforR operation is under implementation (National Initiative for Human Development Phase II) and a second operation is expected to receive Board approval in the near future. A review of the experience with the rollout of the PforR instrument in these three countries suggests that attention needs to be focused on the following three broad dimensions. First, it is important to build the discussion of the use of the instrument on a solid foundation in terms of potential PforR Programs. In all three countries there are strong national programs in place that formed the basis for the discussion of possible PforR Program support. Concerned government agencies are strongly committed to enhancing the results orientation of the Programs. The sectors and programs under consideration for use of the PforR instrument have also been the subject of substantial dialogue between the government and the Bank for a considerable time. Second, broad engagement and collaboration is needed around the potential value added of the PforR instrument and at different levels. The necessary time to build consensus needs to be taken. Country directors, managers, and task teams need to be engaged with key counterparts including Ministries of Finance and Planning and sectoral ministries–including through workshops and follow-up discussions. Frequent back and forth discussions are also needed within the Bank between country and sector management and within the government between line ministries and concerned sector ministries (and, where appropriate, local governments) to build understanding and consensus on the way forward. It is also critical to reach out to other partners and stakeholders and share information at all levels. Third, at the operational level, integrated teams need to be put in place and appropriately supported. Within the Bank, a team that brings together complementary skills in policy dialogue, technical design, and implementation support is needed, including experts with knowledge on specific features of the PforR instrument. A strong field presence to these teams is helpful. It is also critical that a joint government-World Bank team is set up that recognizes that it is working together to build something new; is open, flexible and willing to learn together; and does not lose sight of the development outcomes/results it is trying to achieve. Where appropriate, development partners should also be included as members of these teams. 31. Additional Guidance. This review of the dialogue about the use of the PforR instrument suggests that more information sharing and a deeper understanding of PforR is needed by both Bank managers and staff and their country counterparts. The importance of greater information sharing and understanding also comes through strongly in the structured interviews of government 14 officials, development partners and senior Bank managers. Accordingly, the Bank will increase its outreach efforts so that country and GP managers and staff, and their counterparts, are supported in becoming fully familiar with PforR as one of the Bank’s three available financing options and understanding how it compares with the other options. IV. EXPERIENCE WITH SPECIFIC FEATURES OF PFORR OPERATIONS A. Program Definition 32. The approved PforR operations support government programs that are part of a broader government development strategy. In all cases, the government programs described in the Bank’s operations documents are programs of expenditures (as opposed to programs of reforms or discrete sets of activities) and are generally well defined with respect to: (a) overall long-term objectives and key results to be achieved; (b) the estimated timeframe and costs/financing to achieve such objectives; (c) an evaluation of progress to date and challenges to be addressed; and (d) overall implementation arrangements. In some cases, weaknesses or gaps in government programs are also being identified, and the Bank is discussing with clients how to address such weaknesses/gaps. While the majority of the approved PforR operations support and strengthen existing government programs, a few support either new government programs (e.g., Tanzania Urban Local Government Strengthening) or government programs conceived in a new way (e.g., Vietnam Results-based Rural Water Supply and Sanitation). 33. Boundaries of PforR Program. Of the 22 approved operations, only six have defined the boundaries of the PforR Program as being the same as the overall government program; all the other operations have defined the PforR Program more narrowly (see Table 3). The PforR Board paper recognized that it would be necessary to carefully consider the appropriate definition of the PforR Program in relation to the overall government program, partly because the “exclusions” can affect the scope of PforR Programs. For instance, in Uruguay, Ethiopia, and the Vietnam Northern Mountain operations, activities or components have Table 5. PforR Programs and Government Programs been excluded from the PforR Programs Percentage breakdown because of concerns about identified or 15 Factors accounting for differences of responses potential high-value procurements. The between PforR Programs and Government results of the survey of PforR users show government programs Bank staff officials’ that risks associated with perceptions about responses responses specific activities, as well as the articulation Exclusion of high-value procurement and current application of the ACGs, also contracts 27 18 limit PforR Programs (see Table 5). A Exclusion of activities with number of the senior Bank managers potentially significant 27 20 interviewed as part of this review environmental and social impacts emphasized the challenges and concerns Perceptions of high risk in involving with respect to the ACGs. In Brazil and Bank in specific activities 17 31 India, for example, the ACGs were an Perceived scope of Bank’s ACGs 7 19 important factor in reducing the size of the Other factors 22 12 15 Anecdotal evidence suggests that the impact of the exclusion goes beyond reducing the scope of specific Programs as in some cases it has been a key factor leading to countries and Bank teams deciding not to use the instrument. 15 Programs. Other factors mentioned in the survey and in other feedback that constrain the boundaries of PforR Programs are: (a) the desire to pilot/test a new instrument before fully rolling it out; (b) the degree of “readiness” of all aspects of the government program; (c) challenges related to the scope of the technical and other assessments needed for the full government program; (d) the large size of the government program in relation to the amount of Bank financing available; and (e) concerns about perceived implications for implementation support if the scope of the PforR Program is very large. 34. Strengthening Links between PforR Programs and Overall Government Programs. The Program supported by PforR should be as closely aligned with the government program as possible (see Box 3). Such alignment allows PforR operations to support a holistic approach to achieving results. However, as noted above this is not always the case. While some identified concerns that have limited Programs are understandable, these issues need to be looked at carefully. If these concerns result in a PforR Program that is very narrowly defined, or that is defined in terms of subsets of activities, they could affect the leverage and potential impact of the instrument. During Program preparation, the Bank and the government need to discuss the relationship of the PforR Program to the overall government program, so that appropriate decisions can be taken about the size and scope of the PforR Program. The Bank needs to provide managers and staff with further guidance on this critical dimension of the instrument. Box 3. Aligning PforR Programs with Government Programs A review of approved PforR operations suggests that teams have adopted a range of approaches to aligning PforR Programs with government programs. Kenya National Safety Net Program (NSNP). This operation has been able to define the boundaries of the PforR Program as the same as a national government program. It covers all five of the main cash transfer programs and is supporting the NSNP in helping reinforce its goals of strengthening the targeting methods for vulnerable poor households, improving the payments systems, and improving monitoring and evaluation frameworks. Bangladesh Revenue Mobilization Program For Results. This operation has also defined the boundaries of the PforR Program as the same as a government program. But the government program is a relatively narrowly defined Value Added Tax (VAT) Improvement Program that aims to raise VAT revenues through implementation of a new VAT Law, improved business processes and automation and greater transparency. The government is prioritizing strengthening the administration of the VAT given its potential to contribute to revenue mobilization. Croatia Health Systems Quality and Efficiency Improvement Program and Moldova Health Transformation Program. Both of these operations are supporting national health reform strategies aimed at providing quality health services at an affordable cost. The Croatia operation covers five out of the eight priorities defined in the government’s 2012-20 strategy and the Moldova operation covers five of the seventeen sub-programs of the 2008-17 National Health Service Development Strategy. The choice of priorities and sub-programs took into account such factors as: (i) the strategic importance of the issues; (ii) the level of the World Bank’s past engagement and added value; and (iii) the need for a focused Program. Pakistan Punjab Governance Reforms for Service Delivery. This operation is focused on one province and covers a subset of the provincial government’s Governance Reform Roadmap that focuses on addressing critical constraints to service delivery. The boundaries of the PforR Program are defined in terms of the activities to be supported, the public management systems and institutions to be strengthened and a functional focus on implementation of activities in seven departments and key identified services. Uruguay Road Rehabilitation and Maintenance Program and Nepal Bridges Improvement and Maintenance Program. Both operations support a subset of activities within national infrastructure development programs. These activities are explicitly defined to exclude any activities assessed to be likely to have a significant adverse impact on the environment and/or affected people as well as works, goods and consultancy contracts above the limits stated in the PforR policy. 16 B. Results Frameworks and Disbursement-Linked Indicators 35. PforR places strong emphasis on results, and as it links disbursements to the achievement of program results, rather than to inputs, the preparation of PforR operations needs to involve a wide-ranging process of discussing results and results chains and preparing results frameworks. 16 All PforR operations set out a results framework for the PforR Program that includes defined objectives and indicators that the government can monitor, measure, and report on routinely. These results frameworks currently use a standard template drawn from the IPF instrument. While this template provides specific information on program development objectives and intermediate results (with associated indicators for each), it is not always easy to apply to large and complex programs. Therefore, the Program Appraisal Documents (PADs) of several of the approved operations have added another section for presenting the results chain (Table 6 shows one example) in addition to the results framework. This is leading to some duplication of effort, and teams will need both additional guidance and updated templates. Table 6. Ethiopia Health MDGs PforR: Results Chain Priority area Activities Intermediate results Outcomes Accelerate  Supplying equipment & commodities  Health centers offer basic Increased progress for providing emergency obstetric care emergency obstetric care  Skilled care at child birth towards  Supplying contraceptives  Woredas have functional maternal  Antenatal care  Providing ambulances to all Woredas ambulance services. health MDG  Contraceptive prevalence  In-service training of midwives &  Midwives receive in- training of health officers in service training emergency surgical and obstetric skills  Health officers are trained  Capacity building of health extension in emergency surgical and workers in clean and safe delivery obstetric care Sustain the  Strengthening of cold chain systems  Health centers have  Increased immunization gains made in  Supplying of vaccines functional cold chain coverage child health equipment MDG  Holding immunization campaigns  Outreach campaigns are  Supplying bed nets held  Long lasting insecticidal nets are distributed Strengthen  Constructing health centers  Health centers are built  Improved HMIS health  Supplying essential medical products  Health facilities report  Rolled out balanced score systems and equipment HMIS information in time card & institutional  Validating HMIS semi-annually  Annual Facility Readiness performance incentives  Undertaking surveys and studies Assessment is undertaken  Improved facility readiness Note. The indicators highlighted in bold/italics are examples of DLIs in this operation. 36. Monitoring and Evaluation. All the PADs for approved PforR operations provide details on the agreed M&E arrangements for the Programs. About two-thirds of the approved operations make explicit mention of the institutional strengthening and capacity building support for M&E 16 In some cases PforR has helped build capacity in this area, and the discussions with counterpart governments have resulted in the introduction of or improvements to the results framework for the overall government program—as in the case of the National Initiative for Human Development PforR in Morocco. 17 that will be provided under the Program, providing details, including on the importance of strengthening sectorwide M&E systems. The importance of data and systems for generating reliable data and reporting is also emphasized in PforR operations and we have noted increased demand for assistance and support in this area. Moreover, as a new instrument, there is an important role for impact evaluation; however, such evaluations have to be applied selectively. To date, we find that over forty percent of the operations emphasize Program support for impact evaluations.17 Moving forward, the findings of these evaluations will provide important lessons for future review of the instrument. In addition, about three quarters of the approved operations emphasize greater disclosure of the results of M&E processes and more engagement of civil society in the carrying out of the required M&E activities. In all of these respects, it appears that approved operations are living up to the letter and intent of the PforR policy. 37. DLIs. DLIs are discussed extensively in the documentation for PforR operations and are explicitly mentioned in three tables (the results framework, the DLI matrix and the DLI verification protocol table). To date, most DLIs are linked to intermediate results (Table 7). This is because of the need for regular disbursements throughout the implementation of a PforR operation. Most DLIs are scalable (disbursement is proportional to progress in achieving the DLI) which helps provide a more stable basis for disbursements than threshold-type conditions. On average a higher level of financing has been attached to Program Development Objectives (PDO)-level DLIs than to intermediate results DLIs. Over half of the DLIs are also of a capacity building and/or institutional strengthening nature. Table 8 provides some specific examples of DLIs for selected GPs. Government officials and Bank staff agree that the formulation of DLIs (and associated disbursement mechanisms) is one of the most challenging aspects of the preparation of PforR operations, with more than three quarters of both groups saying it was very difficult or somewhat difficult (Table 9). Feedback from the structured interviews also points to the formulation of DLIs being a difficult process. These difficulties stem in part from the challenge of ensuring that the agreed DLIs strike a reasonable balance between being ambitious but not “too” ambitious. A stock take of corporate reviews and interviews confirm that the “ambitiousness” of the DLIs for PforR operations is a key focus of PforR preparation, is carefully reviewed during meetings of the Regional Operations Committee (ROC) and is a major focus of the negotiations of each operation. Table 7. Characteristics of DLIs for Approved PforR Operations Of which Average Of which capacity Financing financing amount Type of DLI Total scalable building amount ($M) per DLI ($M) DLIs linked to PDOs 34 30 12 942 28 DLIs linked to intermediate results 122 80 65 2219 18 Other DLIs 26 19 21 297 11 Total 182 129 98 3458 19 17 Brazil Strengthening Service Delivery for Growth, Poverty Reduction and Environmental Sustainability in the State of Ceará; Ethiopia Health MDG Program; India Maharashtra Rural Water Supply and Sanitation Program; Kenya National Integrated Safety Net Program; Morocco National Initiative for Human Development II Program; Nepal Results-Based Bridges Improvement Program; and Tanzania Urban Local Government Strengthening Program, Rwanda Transformation of Agriculture Sector Program, Rwanda Public Sector Governance Program. 18 Table 8. Examples of DLIs for Selected Global Practices Country and Program Global Practice Example of DLI Recipient has met the annual target for number of schools Tanzania Big Results in Education that have conducted Student and Teacher Enrichment Education Program Program activities India Maharashtra Rural Number of house connections to a sustainable water supply Water Supply and Water system receiving regular water service Sanitation Program Croatia Health System Health, Nutrition Percent of rationalized hospitals subject to technical audit in Quality and Efficiency & Population the preceding calendar year Improvement Program Brazil Strengthening Service Trade and Index of Environmental Enforcement quality, i.e., increase in Delivery in the State of Competitiveness effectiveness of environmental monitoring and enforcement Ceará Social, Urban, Vietnam Northern Enhanced Annual City Plans approved and disclosed to the Rural and Mountains Urban Program public Resilience Nepal Bridges Transport and Strengthened performance management in bridge sector Improvement and ICT (percent of works completed on schedule) Maintenance Program Rwanda Transformation of Annual increases of land protected against soil erosion, Agriculture Agriculture Sector Program based on agreed technical standards 38. Verification Protocols. The Table 9. Formulation of DLIs and Associated Disbursement verification protocols for all DLIs— Mechanisms highlighting who will carry out the How difficult is it to formulate Government verification, how it will be carried DLIs and associated officials’ Bank staff out, and how often—are described in disbursement mechanisms? responses (%) responses (%) the verification protocol tables and Very difficult 0 18 summarized in the main text of the Somewhat difficult 83 58 PAD. All the approved PforR Somewhat easy 17 16 operations have credible verification Very easy 0 0 protocols for DLIs. For more than Don’t know 0 8 half the DLIs of approved PforR Total 100 100 operations, verification will be done by an independent third party, and for the remainder it will be done by a government agency that is separate from the implementing agency (see Figure 10). The data sources to be used in verifying the achievement of DLIs, which are specified in the protocols, include reports from government agencies, data from information systems, annual performance assessments, surveys, audits, and physical inspections/spot checks. In general, verification protocols have been designed to use and strengthen existing government systems for oversight and monitoring, and they are well linked in with broader measures to strengthen M&E (Box 4 provides an example). 19 Box 4. Ethiopia Health PforR DLIs Ethiopia is committed to achieving the MDGs and is among the first countries to sign the International Health Partnerships Compact to harmonize support from all partners. The health PforR operation, approved in 2013, responds to Ethiopia’s commitments and has enabled the Bank to join other partners in supporting the country’s efforts to reach the MDGs. Fundamental to the operation are the DLIs, many of which measure improvements in coverage for key maternal and child health services. An important first step during the preparation of the operation was to use evidence for setting the targets. The team studied Ethiopia’s own achievements and trends in average annu al improvements among developing countries, and extensive consultations were held to agree on targets that were both realistic and ambitious. The second step was to plan and implement population-based surveys that would provide data needed to monitor the indicators and their progress and provide data that the government and all development partners can use. The use of the common approach and data sources also helps ensure synergy among development partners and effective use of country institutions. To this end, the program relies on two important sources of information. First, the ongoing multi-cluster survey supported by the Gates Foundation was used to establish a more recent baseline for immunization coverage among children. Second, small scale (mini) Demographic and Health Surveys (DHS) funded by the multi-donor-supported Promotion of Basic Services Project will provide population-level data on antenatal care, skilled care at childbirth, and contraceptive prevalence rates. Moreover, the team took into account the timing and availability of data in setting the DLIs and the verification protocols to ensure that the appropriate data will be available at the time needed to verify the DLIs. Implementation has proceeded as planned and the findings of the mini-DHS show that targets set for 2014 were achieved. The experience points to the importance of building on country efforts, including the use of population- based surveys for setting DLIs and verification protocols. Partnerships are critical in supporting such M&E mechanisms in countries. The role of the technical assistance component to help build M&E capacity in the country is also important. Finally, having common sources of information for the program helps to ensure consistency in setting and measuring progress towards targets and achievement of results. 39. Disbursement Arrangements. The DLIs formulae (key for disbursements) are described in the Bank disbursements schedule for each PforR operation. As relevant, the schedule also identifies the specific DLIs against which the Bank and the borrower have agreed prior results and/or advance financing. In practice, all but two of the approved operations are using the advance financing option. On the other hand, there has been very limited use of prior results, despite the expectation that it would be an attractive feature for clients, as heard during the consultations process on the design of the instrument. This is due, at least in part, to the current cap on the combined use of prior results and advances which limits the total to 30 percent. This link between prior results and advances implies that countries needing the full advance to allow the Program to start or facilitate the achievement of DLIs cannot also take full advantage of financing for prior results that have already been achieved. In addition, although PforR operations do not link disbursements to individual expenditure transactions, task teams are required to ensure that the amount of Bank financing disbursed does not exceed the total amount of expenditures under the Program. However, determining how to effectively meet this requirement has proved to be challenging and in some cases confusing, particularly where there are multiple sources of funding (including private, national, subnational, etc.), and has added a level of complexity that was not the intent of the policy. 20 Figure 10. Verification Protocols for DLIs by Type Independent third party breakdown Government agencies breakdown Type of Data Source 30% 26% 24% 25% 20% 15% 11% 10% 10% 10% 8% 5% 5% 3% 2% 2% 0% Report Administrative Several types Annual Physical Population Audits Annual Strategy Published data of source Performance Inspection based surveys capacity document decree Assessment building plan 40. Adjustments. The overall process of developing results frameworks, strengthening M&E systems, identifying DLIs and associated verification protocols, and specifying disbursement arrangements appears to be working as intended. Indeed, while both government officials and Bank staff are finding the process of formulating DLIs and associated disbursement mechanisms to be challenging, they are uniform in their support for the overall approach towards results, DLIs, and verification protocols under the PforR instrument (Table 10). Bank and country teams have asked for help in preparing DLIs; responding to the request will require additional guidance, training, and capacity building—an important and welcome challenge, as this area is fundamental to the instrument’s overall effectiveness. The approach to verification, which is the government’s 21 responsibility, has proved to be workable, but some questions have been raised about the level and sources of funding for such activities. Task teams have also asked for more guidance on verification of achievement of DLIs that involve official (e.g., legislative) action from government. Other adjustments are needed at the Bank Procedure level—in particular, removing the artificial and unnecessary link between prior results and advances. This link is: (a) counter to the flexibility borrowers need to make Bank financing a key driver of the continuation of their programs; and (b) inconsistent with other Bank instruments, in particular IPF, in which project/program financial needs drive advance financing, and retroactive financing is determined independently of advances. In addition, advances have to be repaid if not fully utilized against DLIs before the Program closes. Further clarification of the requirement with respect to Bank financing as a proportion of total program expenditures would also be helpful. Table 10. Perceptions of Results Frameworks, DLIs, and Verification Protocols Has preparation of Is the approach of using Can the mechanisms results framework helped DLIs as the main for verifying strengthen results focus mechanism of achievement of DLIs of Program? disbursement useful? function effectively? To a very low/ Gov. officials 4% 0% 4% somewhat low degree Bank staff 16% 6% 16% To a somewhat high Gov. officials 44% 22% 70% degree Bank staff 40% 27% 44% To a very high degree Gov. officials 52% 74% 17% Bank staff 35% 54% 21% Don’t know Gov. officials 0% 4% 9% Bank staff 9% 13% 19% C. Technical Assessments 41. The technical assessments require that, with borrowers’ support, task teams evaluate the adequacy of Program arrangements and their performance in four areas: strategic relevance and technical soundness; the expenditure framework; the results framework and M&E capacity; and economic justification. Task team leaders usually take direct responsibility for the conduct of these assessments. A review of the assessments carried out to date finds that they have been comprehensive in scope and particularly strong with respect to addressing each Program’s strategic relevance and technical soundness. Technical assessments have also provided the basis for the solid dialogue around results and M&E discussed above. It is noteworthy that in PforR, the Bank carries out all assessments in collaboration with clients and, when applicable, other development partners. This is an important aspect of capacity building and needs to be recognized as part of the design and support of the instrument. 42. The technical assessments are valued by both clients and staff. The survey results show that 87 percent of government officials and 76 percent of Bank staff have a somewhat high or very high understanding of the technical assessment process; 91 percent of government officials and 77 percent of staff are somewhat or very satisfied with the quality of the assessments; and 86 percent of government officials and 62 percent of Bank staff feel the assessments have been useful. Discussions with staff point to the importance of more systematic learning that can make the process of preparation of the assessments easier and more focused, and hence more useful. In addition, some 22 asked for more direct examples in their area of expertise. With more experience with the instrument, more knowledge products and learning will be made available and hence should help address this concern. At the same time, regular interactions with teams directly and through the community of practice will provide additional feedback as the instrument moves forward. 43. Expenditure Framework. The assessment of the Program’s expenditure framework is a critical element to assess the efficiency and effectiveness of the Program. It is expected to include reviewing the budget structure and classification; assessing the Program’s financial sustainability and funding predictability; assessing the extent to which the budgeted Program expenditure and its execution adhere to the government’s priorities; and focusing on the efficiency of Progra m expenditures (as a complement to the Program’s economic analysis). The scope of the assessment depends on the specific Program needs and results. Moreover, data on various components of the analysis may vary by country and sector (or sub-sector). The analysis of the expenditure framework was carried out in all cases; however, the depth and coverage varied (Box 5 provides two examples). This can be explained in part by data availability in the country, another important area where institutional capacity building is needed. Informal feedback suggests that clients particularly value the work to ensure that Program expenditures adhere to government priorities and to assess the overall efficiency of Program expenditures. Box 5. PforR Expenditure Frameworks The analysis of the expenditure framework for PforR operations involves assessing the Program’s efficiency and effectiveness as well as its financial sustainability and the alignment of expenditures with government’s priorities and results. Below are two examples of how task teams have carried out such analysis as part of the preparation of PforR operations. Ethiopia Second Urban Local Government Development Program (ULGDP II). The technical assessment focused on municipal planning and budgeting and on the reporting structure of Program expenditures. The analysis of the expenditure framework provided a clear budget structure, showing the classifications of Government programs and sources of funds; discussed the expenditure performance under the first ULGDP; and addressed funds flow and execution issues as well as the Government’s capability to finance the execution of the Program. Improvements in the technical design of the second ULGDP were based on this analysis. In addition, the Program design also ensured that the Program funds would be protected from any budget cuts, and promoted the future rollout of the support to urban local governments as part of the intergovernmental fiscal framework. Vietnam Results-Based National Urban Development Program in the Northern Mountains Region. The technical assessment set up a framework under which the team identified appropriate measures to mitigate the risk of delays in implementation of the Program due to inadequate availability of capital funds to complete Program activities. It also evaluated the financial impact of the Program on the recurrent budgets of city administrations. In addition, DLIs were organized to better accommodate Program cash flow needs, and sustainability was addressed by carefully assessing operation and maintenance funding arrangements. The technical assessment also adopted three approaches to assessing the efficiency of expenditures: comparison with published cost norms, the size of variations in unit costs between similar subprojects, and comparison with similar outputs supported by other Bank projects in the country. Finally, the assessment also discussed the extent to which the use of competitive procurement can be relied on to ensure cost-effectiveness. 44. Economic Justification. The economic evaluation of a PforR operation focuses on the following key issues: whether public provision and/or financing is appropriate for the Program; the economic impact of the Program, as it is currently implemented or planned; the economic impact of the improved Program with PforR support; and the Bank’s added value. For all the approved operations, teams have carried out evaluations of the economic impact of the program. The 23 methodology for these evaluations and the ability to quantify varies by sector. For instance it is easier to provide quantitative assessments in sectors like transport than in some of the social sectors. Assessing the economic benefits of the capacity building and institutional strengthening measures is another methodological challenge that is faced in all Bank operations. To this end, it is important to build a greater knowledge base and derive lessons from experience. As for the issue of the appropriateness of public provision, as most of the PforR operations support governments’ existing public programs, consideration of the issue has been more implicit than explicit in some operations. Teams have often addressed this issue as part of the overall strategic/technical relevance and soundness of the Program. 45. Adjustments. Staff undertaking technical assessments would benefit from more training and guidance. The guidance should focus on all aspects of the technical assessment: strategic relevance and technical soundness; expenditure analysis; results; and economic justification. More importantly, the technical (and other) assessments need to be recast as capacity building efforts and not just a requirement. This is a core feature and an advantage of the PforR instrument. More work is needed to better explain the instrument and related guidance in this manner. In addition, as PforR moves forward, more outreach to GPs and Regions to explain this vision and how it should work in practice is important. D. Fiduciary Assessments 46. All of the PforR operations that have been approved or authorized for negotiations have benefited from a rigorous fiduciary systems assessment (FSA) process. Both government officials and Bank staff appear to be generally satisfied with this process: 91 percent of government officials and 82 percent of Bank staff surveyed say they have a good understanding of the purpose and contribution of the FSA; 90 percent of government officials and 73 percent of Bank staff are somewhat or very satisfied with the quality of the FSA; and 86 percent of government officials and 69 percent of Bank staff feel the assessment has been useful to a somewhat high or very high degree. One should note that this approach is new and requires a lot more judgment than what some staff are used to with IPF operations, and this may explain some of these numbers (see paragraph 49 for more details). All of the FSAs for approved operations have concluded that the fiduciary systems used by the programs supported by PforR financing provide reasonable assurance that Bank financing will be used appropriately. The FSAs have also been the primary means of identifying potential high-value procurement contracts that need to be excluded from PforR operations. 47. Concerns have been raised in several areas. The first concern relates to the procurement exclusions. These exclusions have resulted in a number of cases of either avoidance or reducing the overall scope of the PforR Program. During the consultations, clients also pointed to the need to allow in exceptional cases for the inclusion of contracts whose value is above the exclusion thresholds but that are critical to the integrity of the program and are a relatively modest proportion of the overall program. In such cases, excluding critical contracts may affect the integrity of the overall program and put the client at a higher risk of not achieving results. In practice most teams simply shied away from this dilemma. In one case (Ethiopia Health MDG Program) while there are no such exclusions in the program at present, there was a risk that some important contracts for the health program would not be eligible because they could rise above the exclusion thresholds, even though the specific contracts posed a low risk from a procurement point of view as they were being handled through a UN agency. In this case, the government addressed the 24 potential problem by creating a sub-account in the Health MDG Fund for the Bank’s funding which in turn meant that the PforR could not fully pool funds with those of other development partners. This is not in line with the spirit of the instrument and its objectives. At least two operations under preparation are currently facing this challenge which will occur more often in the future and especially in the health sector. The real risk is that the excluded activities are critical to achieve results. For example, excluding contracts for vaccines may curtail the government’s ability to achieve results when these are related to child health or immunization, whereas including them would be accompanied by a careful analysis of the procurement approach and methodology as well as associated risks and appropriate risk management and mitigation measures. On the financial management side, there have been less critical issues and all PforR operations complied with the audit requirements. However, clients have suggested that in some cases the verification processes could fulfill a similar function; hence there should be flexibility in terms of applying the audit requirements, similar to what is currently the case for IPF. 48. System Strengthening and Risk Mitigation Measures. The PforR FSAs have helped identify needed system and capacity improvements to either support the operation’s development objectives or mitigate risk in areas of weakness (Box 6 provides two examples). Some of the capacity improvements and fiduciary risk mitigation measures identified in the FSAs have been addressed as DLIs. In fact, in the PforR operations that have been approved, 19 percent of DLIs relate to fiduciary actions or processes that support the achievement of the operation’s development objectives, and another seven percent of DLIs relate to fiduciary risk mitigation measures. In other operations, these measures have been addressed in the Program Action Plans (PAPs): 24 percent of the actions in PAPs for approved PforR operations relate to fiduciary measures that support the achievement of the development objectives, and 18 percent address fiduciary risk mitigation measures. Fiduciary system/capacity improvements and risk mitigation measures may also be addressed as legal covenants or through parallel technical assistance financed by the Bank or by other development partners. In addition, fiduciary weaknesses identified in the assessments are being incorporated as key components of the integrated risk assessment of the operation. 49. Lessons Learned. The approach to FSAs in PforR is new; it involves a change in thinking from the financial management and procurement assessments that staff are accustomed to undertaking as part of IPF projects. Any new approach involves an element of interpretation, experimentation, and learning, and this has certainly been true of PforR FSAs. Those carried out in the first two years of PforR have varied in rigor and clarity and in length —from concise and focused assessments of as few as 14 pages to much more wide-ranging assessments of up to 94 pages, leading to understandable concerns about the “heaviness” of many of the FSAs. Over time, as FSAs build on experience and staff gain greater knowledge about them, their quality appears to be improving. However, staff need more training in the substance of PforR FSAs, including how to better integrate financial management, procurement, and related governance issues and how to better integrate the findings of the FSA into the overall design and capacity building focus of the PforR operation. Additionally, staff should be encouraged to rely more on existing information and assessments rather than conduct lengthy assessments afresh in countries and sectors with which the Bank is familiar. It is also important that the focus on “exclusions” does not lead to a more generic shift from a systems-strengthening approach and that activities deemed to be of critical importance to the integrity of the Program are supported. Finally, there should be the same flexibility in applying audit requirements as there is for the IPF instrument. 25 Box 6. Fiduciary System Assessments: Supporting Institutional Strengthening The fiduciary system assessments have been designed to look at the common systematic issues that affect the performance of government programs in a manner that enhances the efficiency and effectiveness of such programs and their ability to get best value for money and achieve results. The approach focuses on key principles that guide these assessments and a framework that should help identify key constraints. The expectation is that such assessments help focus on common root causes of performance issues in the public sector by looking at procurement, financial management, and governance issues together and provide a good platform for institutional strengthening and capacity building. Two examples from India and Uruguay show how the assessments can be important tools for clients as part of the PforR engagement. India Third Maharashtra Rural Water Supply and Sanitation Program. Maharashtra State has a population of 113 million, of whom 62 million live in rural areas where access to good rural water and sanitation systems (RWSS) is important for improving peoples’ lives. The Government of Maharashtra (GoM) has invested significant resources in RWSS over the years, and the World Bank has supported the sector with two investment projects. The GoM’s strategy for RWSS increasingly focuses on achieving results. The GoM was enthusiastic about using the PforR instrument for new World Bank support as it matched the Government’s strategy, used the government’s own systems, and focused on institutional strengthening as well as service delivery. The GoM found the FSA useful in providing an independent perspective on the state of the government systems. The FSA identified a number of areas for strengthening fiduciary systems, which the GoM has decided to implement over the short and medium term. These included creation of a Manual for procurement in the RWSS sector, conducting a procurement review of at least 15 percent of contracts issued each year in the RWSS Sector, establishing double-entry accounting systems at all levels in GoM, and establishing a system to report information on expenditures at village level. Uruguay Road Rehabilitation and Maintenance Program. The density of Uruguay’s road network is among the highest in Latin America, but the condition of the road network has deteriorated over the years. The Government of Uruguay (GoU) is working to reduce the backlog of road rehabilitation and maintenance. The GoU considered that the PforR instrument provided a holistic approach to World Bank support for road sector issues, balancing the road investment program with a multifaceted institutional strengthening plan. Drawing on the World Bank’s global knowledge and expertise in the roads sector, the FSA identified opportunities for institutional strengthening in the areas of financial management and controls and procurement processes, including streamlining procedures for bid evaluation and contract award, diversifying suppliers and contractors, and improving procurement planning for high-volume items. More specifically, the FSA was not seen as a Bank requirement but rather as a useful tool for the Ministry of Transport and Public Works to better understand its procurement performance. Uruguay promotes transparency and makes public, on-line, a database including the main features of all the state agencies’ procurement processes. This database made it possible for the team to carry out a quantitative analysis of all the procurement processes for road works that had taken place over the past five years, considering indicators such as number of bidders per process and average size of the contract. One of the most interesting findings was related to the procurement process average turnaround time. It was perceived, within the Ministry of Transport and Public Works that procurement is long, but there was no concrete evidence. The fiduciary assessment highlighted that, on average, road works procurement required 285 days between bid notice publication and contract award: the assessment put a number on a "perception". This indicator then became one of the fiduciary system indicators and the Program Action Plan defined a follow-up action building on this finding. 50. Adjustments. The continuing objective of the FSA will be to evaluate whether the program fiduciary systems provide reasonable assurance that Program expenditures will be used appropriately to achieve their intended purpose. The experience and lessons from FSA, to date, will provide valuable inputs to update guidance and enhance training aimed at making the FSA process focused and consistent. It should also emphasize the importance for FSA staff to work in the integrated manner required by the PforR instrument. One change to the policy is proposed: to allow the PforR Program to include high-value contracts in exceptional cases when such contracts are deemed to be of critical importance to the overall integrity of the Program. In addition, minor edits to the Bank Procedure (BP) are proposed to provide the needed flexibility in audit requirements (similar to IPF). 26 E. Anticorruption Guidelines 51. The Bank’s Guidelines on Preventing and Combating Fraud and Corruption in Program-for- Results Financing (PforR Anticorruption Guidelines or ACGs) apply to the whole of the Program supported by the instrument. The application of the ACGs has been a point of contention in several PforR operations. Survey results show that 43 percent of government officials and 59 percent of Bank staff indicated that applying the requirements of the ACGs has been somewhat or very difficult. While all borrowers in PforR operations have been supportive of cooperating with the Bank on measures to prevent and combat fraud and corruption in programs, many have been disappointed that the ACGs do not focus more strongly on supporting governments in improving their own systems, in accordance with the overall thrust of the PforR instrument. 52. Some borrowers have been concerned that the ACGs give the Bank extensive investigative rights over government programs and have decided either not to use the PforR instrument or to significantly limit the scope of the Program. The ACGs were one of the factors that limited the scope of the PforR operations in a number of cases, like in Tanzania and Uganda. There have also been concerns that specific provisions in the ACGs may be inconsistent with national legislation, in particular with respect to the conduct of investigations, or may encroach on the jurisdiction of national law enforcement authorities. The inconsistency with national laws has been reported in a number of countries including Brazil, Morocco, and Uruguay. In Brazil, for example, to avoid this inconsistency at the local government level, certain activities were taken out of the scope of the PforR operation. The use in the ACGs of the term “investigation” in relation to allegations of fraud and corruption in connection with the PforR Program has been interpreted as encroaching on the exclusive rights and duties, under national laws, of police or anticorruption commissions to investigate fraud and corruption in government programs (e.g., Pakistan and Ethiopia). These apparent inconsistencies are particularly problematic for borrowers because the ACGs do not explicitly recognize the role of a country’s legal and judicial framework in dealing with integrity issues in PforR operations. Of course, it is not the Bank’s intention to replace national criminal law systems, but these concerns have resulted in extensive discussions between the Bank and borrowers during the preparation and negotiations of PforR operations to clarify misunderstandings. 53. Adjustment. Going forward, therefore, some technical adjustments to the ACGs should be made. Key issues that would benefit from clarification include: framing the ACGs in the context of the Bank’s and borrower’s common interest in and commitment to preventing and combating fraud and corruption in the Program, not just as an expression of Bank policies and interests; more explicit recognition that responsibility for the implementation of the Program under the legal agreement, including the primary responsibility for preventing and combating fraud and corruption, rests with the borrower; clarifying that the ACGs are not intended to restrict or otherwise affect the country’s sovereign right to investigate, prosecute, or take any other action in furtherance of its own laws and regulations, or to obligate countries to undertake investigations and follow up actions in contradiction with the country’s applicable law; and clarifying that any “investigations” undertaken by the Bank are “administrative inquiries” for the purpose of determining compliance with its own policies, directives, and procedures, and to fulfill its fiduciary duty, and are not a replacement for national law enforcement. All of this can be achieved while maintaining the principles and application of the ACGs and without changing the current substantive legal rights of the Bank or legal undertakings of borrowers. 27 F. Environmental and Social Assessments 54. With respect to the environmental and social dimensions of PforR operations, the instrument is working as intended, and Bank program teams are following PforR requirements specified in the policy for PforR. All operations have conducted the required Environmental and Social Systems Assessments (ESSAs) (see Box 7 for more details). ESSAs are used to (a) assess the Program systems against the principles of PforR Policy; and (b) review and screen for any exclusions required by the policy regarding environmental and social aspects. The review confirms that supported program activities generally present low to moderate environmental or social risks. As required, Program designs and screening processes have excluded activities judged to be likely to have significant adverse impacts that are sensitive, diverse, or unprecedented on the environment and/or affected people investments. In two cases where risks were rated as substantial (Mexico Water and Vietnam Northern Mountains) due to contextual reasons, adequate risk management and mitigation measures were put in place. In another case, the Nepal Results-Based Bridges Improvement and Maintenance Program, the three percent of activities that ran through or along national parks was excluded. Program teams have consistently undertaken stakeholder consultations on the ESSA, and have met ESSA disclosure requirements. The timing of the consultation is important as is the format of the materials presented to convey the system assessments and proposed actions well to the stakeholders. ESSAs have also included review of grievance redress arrangements, and the Programs have established grievance redress mechanisms Box 7. What does an Environmental and Social Systems Assessment Entail? The Environmental and Social Systems Assessment (ESSA) is the primary analytical instrument used in PforR operations to evaluate social and environmental risks and benefits associated with a given PforR operation. The ESSA is prepared by World Bank staff in collaboration with program counterparts and is a key element of the Bank’s overall evaluation of programs proposed for Bank financing. Unlike a conventional Environmental and Social Impact Assessment (ESIA) the ESSA does not evaluate the direct impacts and risks of specific transactions. Instead, the ESSA focuses on the counterpart systems which apply to a given Program and whether they are sufficient to manage risks associated with the proposed Program. The ESSA provides an assessment of the applicable systems against a set of core principles defined in the policy. The ESSA provides the following information: (i) an assessment of the environmental and social risks and benefits associated with types of activities required to achieve program objectives against a set of five risk criteria; (ii) a description of the counterpart systems (policies, procedures, legal framework) which apply to the program; (iii) an assessment of the adequacy of these systems; and (iv) an assessment of the implementation experience of the program to examine the effectiveness of implementation of the systems in place. The ESSA is also a screening tool for activities that are judged to be likely to have significant adverse impacts that are sensitive, diverse, or unprecedented on the environment and/or affected people as those are excluded from the operations. Based on this assessment the Bank team may, when appropriate, recommend specific counterpart actions to: (i) minimize or mitigate risks and enhance program benefits; (ii) improve the counterpart systems to more closely align with core principles defined in the policy; and/or (iii) strengthen implementation capacity to achieve improvements in performance where necessary. Relevant actions are then included in the program action plan, legal covenants or DLIs, as appropriate for the specific operation. ESSAs provide the basis for consultations and are disclosed before appraisal. To ensure that consultations are done in a consistent manner, staff have been given guidance on the consultation process. During implementation the Bank provides implementation support to ensure the counterparts are putting agreed actions into practice and may from time to time adjust the actions to respond to changing conditions or experience on the ground. 28 (GRMs)—for example, traditional community institutions, program-level GRMs, or national ombudsmen. All projects generally commit to strengthening grievance redress. 55. System Strengthening and Risk Mitigation Measures. All of the ESSAs include an assessment of the legal and regulatory frameworks relevant to the Program, and a systems capacity assessment. About three-quarters of the government officials and Bank staff who responded to the survey indicated a high or very high level of understanding of the purpose and contribution of these assessments; 90 percent of government officials and 69 percent of Bank staff were also somewhat satisfied or very satisfied with the quality of the assessments that have been undertaken; and 90 percent of government officials and 62 percent of Bank staff felt to a somewhat or very high degree that the assessments have been useful. However, only 48 percent of Bank staff (compared with 90 percent of government officials) felt that the consultations had informed the assessment’s findings and conclusions. Among the issues that affected staff views are the need for more focus and prioritization in the ESSAs as they seem to be excessive at times (see paragraph 56 for more details). In some cases, the assessments’ findings have been used to discuss and agree on a set of measures to address technical, policy, organizational, and program accountability issues at a systems level (see Box 8 for examples). Programs may also include capacity building and training activities as well as development of Environmental and Social Management Manuals. DLIs or PAPs may address institutional strengthening and risk mitigation measures; in fact, in the PforR operations that have been approved, ten percent of the DLIs and 15 percent of the PAP actions relate to ESSA measures. 56. Lessons Learned. The review has identified two areas, in particular, where improvements are warranted. First, while all ESSAs fulfill all necessary requirements, they are uneven in their depth of assessment, focus, and clarity of presentation. Also, they could do more to identify the range of potential issues considered, prioritize potential issues according to their relevance, devise appropriate actions to address priority issues, and emphasize measures to strengthen borrowers’ overall environmental and social risk management systems. Some ESSAs have been too “heavy” and have not adequately drawn on existing knowledge. Second, more integration at the team level can help ensure that the ESSA is part of the overall focus on capacity building (similar to the FSA) and an integral part of the broader dialogue. 57. Adjustments. As noted in the FSA section earlier, more training and guidance is needed for both staff and clients. Such efforts should stress the need for specialists to improve the focus and relevance of the ESSA, and better establish priorities among potential issues. In this regard, more regular use of the environment and social risk assessment matrix would promote increased focus and attention to the broader Program context. Training should stress the importance of implementation and how best to follow-up on recommended actions and how, to the extent possible, to link those recommendations to results. This in turn would help strengthen the linkages between the environmental and social issues raised in the ESSA and the design and performance of the PforR Program. As with other aspects of PforR, more integration of team members is to be encouraged. There is also demand for more synthesis and dissemination of lessons from existing PforR operations and identification of good practice examples regarding various aspects of the ESSA. 29 Box 8. Program-Level Opportunities to Improve Environmental and Social Risk Management By focusing on environmental and social risk management at the system level, the PforR approach sometimes has advantages over the narrower transactional focus on site-specific impacts that characterizes IPF projects. Some examples: Addressing a complex technical problem. The ESSA for the Vietnam Rural Water Supply and Sanitation Program addresses arsenic contamination of groundwater in a more integrated and comprehensive manner than likely would have been done in a project focusing on mitigation of site-specific impacts in a project-level physical footprint. The program assessment process provided a more complete understanding of the nature and challenges of the technical problem and allowed the Government and the Bank team to focus on systematic solutions by carrying out technical design work, structuring improved monitoring programs, and presenting the issues to a range of stakeholders. Promoting policy change. The PforR process may create opportunities to devise policy-level improvements that are applicable throughout the borrower’s scope of operations. In the Mexico Oaxaca Water and Sanitation Sector Modernization Program, one of the DLIs pertains to the institutional modernization of the sector, which includes revision of the State Water and Sanitation Sector Law and its accompanying regulations. The reforms will aim to incorporate social management issues in line with laws on the rights of indigenous peoples in the state, transparency and access to information, and citizen engagement and participation. This legislation will have impacts at both the municipal and state levels of government. Strengthening organizational structure. Engagement at the sectoral program level may create or reinforce incentives for organizational reform. The ESSA for the Nepal Bridges Improvement and Maintenance Program concluded that a key constraint to effective risk management was the absence of an authorized and resourced entity to conduct screening, assessment, planning, and monitoring. As a consequence, the program will transform an existing advisory body into a formal environment and social management unit within the Department of Roads, with clear lines of authority and dedicated resources for staffing and operations. Strengthening program accountability. Engagement at the program level creates opportunities to improve coordination and effectiveness among multiple actors or modalities. In the Kenya National Integrated Safety Net Program, the ESSA found high variability in public awareness of, and system responsiveness in, the GRMs associated with various cash transfer schemes. The ESSA recommended measures to provide clearer lines of responsibility, institute common formats and processes, improve communications and outreach, and improve overall responsiveness to grievances. G. Risk Assessment and Management 58. Risk assessment is not a separate exercise; it is an integral part of the PforR instrument and informs every stage of the design of an operation. The survey feedback suggests that both government officials and Bank task teams understand this, and that most believe the process has been useful and has identified the major risks that could prevent achievement of Program results. Each of the technical, fiduciary, and environmental and social assessments identifies potential risks and risk management actions to inform decision-making on the scope and content of PforR Programs. The various risk assessments are summarized in risk ratings that are incorporated into the integrated risk assessment frameworks (IRAFs), disclosed as part of the PforR documentation, and monitored during implementation. The risk assessment is influencing the design of PforR Programs through the selection of DLIs and certain exclusions. Indeed, if there is a concern at this early stage of the rollout of the instrument, it is that risk aversion may be having too great an effect in terms of limiting the scope and content of particular Programs. 59. Identification and Rating of Risks. In general terms, technical, fiduciary, environmental and social, and DLI risks seem to be reasonably well identified. But there may be a tendency to overrate them: none of the 22 approved operations has an overall risk rating of low (see Table 11). 30 These overall risk ratings are heavily influenced by fiduciary risk ratings (59 percent are substantial and 18 percent are high; only 23 percent are moderate and none are low). By contrast, 55 percent of the technical risks are rated as moderate and only 9 percent as high; environmental and social risks are largely moderate (70 percent, with 20 percent rated low and 10 percent substantial); DLI risks are also largely moderate (68 percent). It is interesting to note that in spite of efforts to reduce the scope of PforR Programs during the preparation process, only two of the approved operations had an overall risk rating that was reduced between the concept stage and Board approval; for five others the rating increased, and for the remaining operations it stayed the same.18 The overall distribution of risk ratings for approved PforR operations is very similar to that for IPF operations: 2 percent low, 34 percent moderate, 48 percent substantial, and 16 percent high for IPF operations approved in FY13, compared with 0 percent low, 32 percent moderate, 50 percent substantial, and 18 percent high for PforR operations approved to date. Table 11. Risk Ratings at Board Approval for Approved PforR Operations Low (0) Moderate (7) Substantial (11) High (4) Brazil Ceara, Croatia Health, Bangladesh VAT, Ethiopia Kenya Safety Net, Mexico India Water, Moldova Health, Ethiopia Local Gov, Oaxaca Water, Pakistan Health, Morocco INDH II, Mozambique PFM, Nepal Governance, Tunisia Urban Rwanda Agriculture, Bridges, Rwanda Development Uruguay Roads Governance, Tanzania Local Gov, Tanzania Education, Uganda Urban, Vietnam Water, Vietnam Urban 60. Risk Mitigation Measures. In general, based on the desk review, the risk mitigation measures embedded in the design of PforR operations and usually included in PAPs seem relevant and achievable. Moreover, they are also a reflection of a process in which teams (on both the government and Bank sides) continue identifying possible risks and related mitigation measures throughout the preparation of operations. However, because risks are discussed at various stages of the process and in varying contexts—while designing specific aspects of the operation, conducting the assessments, drafting legal agreements, and so on—the management/mitigation measures tend to be scattered across various elements of an operation’s preparation, and it is difficult to consolidate them. In addition, feedback from teams points to the difficulty at times in distinguishing between capacity building efforts and risk management since the two are often closely related. Consequently, for some operations, the totality of the agreed risk mitigation and management measures goes beyond what is summarized in the IRAF. 61. Adjustment. Going forward, the IRAF will be replaced by the Standardized Operational Risk-Rating Tool (SORT), which will apply to all financing instruments. While SORT is a simpler template than IRAF, it provides the scope to include most of the risk ratings reflected in the IRAF. Still, managers and staff will need guidance on how SORT will substitute for IRAF and be integrated into the overall PforR approach to risk assessment and management. It may also be useful to provide additional guidance on how to more effectively use the risk assessment and management dimensions of the instrument in decision-making about the scope and ambitiousness of PforR Programs. 18 The ratings for Ethiopia Health and Morocco INDH II fell; those for Ethiopia Local Government, India Water, Pakistan Punjab Governance, Tunisia Urban Development, and Vietnam Water increased. 31 H. Institutional Strengthening, Capacity Building, and Program Action Plans 62. PforR focuses on the behavioral and institutional changes that are required to achieve results and manage associated risks. Hence it is expected that many PforR operations will require some level of capacity building activities, which will be informed by the technical, fiduciary, and environmental and social systems assessments. The assessments that are carried out as part of preparation identify, as appropriate, key program improvements and actions that can be undertaken over the time period of the operation. Since the assessments focus on the whole program, their recommended measures will help reduce the fragmentation of capacity building, focus attention on core issues affecting the performance of the program, and thus enhance the effectiveness and sustainability of these measures. The action plan should also consider improvements to enhance the long-term efficiency of a borrower’s program while taking into account the feasibility of implementation during the operation’s timeframe. 63. Drawing on the technical, fiduciary, and environmental and social systems assessments, all of the approved operations highlight specific and priority capacity building activities. All approved operations also include system strengthening activities. These activities and measures may be linked to specific DLIs, included in legal covenants, or made part of the PAP. While realizing the instrument’s full potential in helping government agencies with capacity building and broader system and institutional strengthening will likely take some time, it is already clear that both the government and Bank teams are taking this dimension of the PforR instrument very seriously: the survey results show that 91 percent of government officials and 71 percent of Bank staff believe that, to a somewhat high or very high degree, priority capacity building needs are being identified under the Program. The importance of this dimension of the PforR instrument is also strongly emphasized in the structured interviews with government officials. When we examine DLIs for various country groupings, we note that in IDA countries a significant portion of DLIs are associated with capacity building (56 percent as opposed to 49 percent in IBRD countries). With this strong emphasis on capacity building, PforR also holds potential for support to fragile and conflict-affected situations (FCS). Box 9 describes how PforR could be applied in FCS. 64. Financing of Capacity Building. The survey results show that securing financing for Program capacity building needs is often a challenge; 31 percent of government officials and 45 percent of Bank staff believe to a somewhat high or very high degree that it is a serious challenge. In principle, priority capacity building measures should be fully integrated into the Program expenditure framework and financed in the same way as other Program expenditures. However, in practice there may be cases where alternatives may be needed. For example, in some countries, government regulations may constrain the inclusion of a full program of capacity building measures or the procurement/selection of high-quality technical assistance and training programs. In such cases, PforR offers a number of modalities for providing the necessary assistance: technical assistance can be provided through an IPF (either as a stand-alone IPF or as a PforR/IPF hybrid), or through parallel efforts financed by development partners. Of the 22 approved PforR operations, 16 made special arrangements to finance capacity building—putting in place a PforR/IPF hybrid, using another World Bank IPF operation, using trust funds supported by other development partners or by the Bank, or relying on projects separately funded by other development partners. While there is no one modality or answer to address this issue, it is important to explore it in the context of each operation and to have options for countries where this may be a real constraint. 32 Box 9. PforR Potential for Supporting Fragile States Fragile states, characterized by weak institutions, poor access to basic services and vulnerability to conflict have increasingly become an area of focus for the World Bank. For example, the World Development Report 2011: Conflict, Security and Development called for a paradigm shift in the development community’s approach to fragile and conflict-affected situations (FCS), based on the premise that violence and other challenges cannot be resolved by short-term or partial solutions in the absence of institutions that provide people with security, justice, and jobs. It is precisely in the area of institution strengthening where PforR can make a significant contribution to supporting fragile states. With its focus on strengthening the institutional capacity and systems needed for government programs to achieve their desired results, PforR can facilitate a shift in incentivizing service delivery to citizens and foster greater accountability between, policy makers, service providers and citizens. All PforR operations highlight specific capacity building and system strengthening activities. The design of the PforR instrument also allows for it to be adapted to specific situations, hence, one would expect to see more emphasis on capacity building efforts in FCS. The program scope can also be scaled up or down depending on the specific sector and the associated risk. There are no PforR operations yet in FCS, but there are some under preparation. Given the importance of fragile states for achieving the WBG strategy, efforts to ensure that those working on the fragile states agenda receive the needed training and support will be made. As the portfolio grows, lessons will also be drawn to learn how this instrument works in this environment and how we can best support countries facing fragility. 65. Program Action Plans. Each of the approved PforR operations includes a PAP. The PAP is a key feature of the preparation of PforR operations, is reviewed and finalized as part of the appraisal and negotiations for each operation, and is closely monitored during implementation. The PAPs for approved operations are structured in a range of ways—for example, according to the major themes of the PforR operation, according to the structure of the PforR assessments (technical, fiduciary, environment and social), and according to an overall listing of specific actions. PAPs enable task teams to focus on additional measures not already included in DLIs or legal covenants. Almost half of the actions reflected in the PAPs for approved operations are neither DLIs nor legal covenants. Many of the actions are of a short-term nature (44 percent of the actions are due by the end of year one), and 38 percent of all PAP actions are in the fiduciary area. Realizing the full potential of PAPs is directly linked to enhancing the focus of PforR Programs on forward-looking institutional strengthening and capacity building measures over the full four or five years of the implementation of a PforR Program. 66. Adjustment. The Bank has never found it easy to address the critical issue of institutional strengthening and capacity building, so the progress already made on this critical dimension of the PforR instrument is welcome. Going forward, it would be useful to identify some good practices in operations that are ongoing or under preparation. Additional guidance and options for mobilization of additional resources for clients would be welcome. And efforts will be made to enhance resources for the Multi-Donor Trust Fund (MDTF) established to support capacity building for PforR)19. 19 The MDTF received initial funding from Germany for a total of $2 million to provide support for new PforR operations and related capacity building. The MDTF has already supported preparation of 12 PforR operations and will also support activities related to the institutionalization of the instrument. While relatively small in size, it has been an important stimulus for the instrument and has demonstrated the value of such a fund moving forward. 33 I. Partnerships 67. The PforR instrument was expected to enhance the Bank’s ability to work in partnership with governments and other development partners around strengthening the design and enhancing the implementation of government programs to achieve better results. In this regard, the overall feedback from government officials directly involved in the preparation of PforR operations has been positive, with all respondents saying that to a somewhat high degree (41 percent) or very high degree (59 percent) the teams working on the PforR operations (government, World Bank, development partners, other stakeholders) have been working together effectively in preparing the Program. A similarly positive response was received to the question of how effectively the teams have been working together in the implementation of the PforR Program: 32 percent responded to a somewhat high degree and 54 percent said to a very high degree (the remaining 14 percent responded that the question was not relevant, probably because their operation had not yet reached the implementation stage). 68. Governments. In general, the preparation of PforR operations has enhanced partnerships between government agencies and the Bank, as Bank staff focus on helping counterparts better translate overall strategies into specific programs with clearly identified expenditures, a well- understood results chain, and specific actions for strengthening systems and enhancing implementation capacity. On the basis of such discussions, the specific Program that the Bank will support is then determined. In informal feedback and structured interviews, government officials have recognized and welcomed the increased sense of partnership in the preparation and implementation of PforR operations. In their survey responses, government officials have also indicated a high level of satisfaction with how the Bank has organized itself to support the preparation and implementation of PforR operations: 43 percent say that to a somewhat high degree and 57 percent that to a very high degree they believe the World Bank task team has the right skills mix to prepare/implement the PforR operation, and 38 percent are somewhat satisfied and 62 percent very satisfied with the information/support the World Bank provides to help government officials prepare and implement PforR operations. Interviews and consultations confirmed an increased sense of partnership in a number of PforR operations. 69. Development Partners. By working with governments to better specify their own programs and enhance their overall effectiveness and efficiency, the dialogue and discussions inherent in the preparation of PforR operations should help governments take leadership of their development programs and discussion of those programs with the broader development community. This is already happening in different ways and to different extents in the context of many of the approved PforR operations. As has been noted, in six approved PforR operations other partners are formally involved in co-financing the overall PforR Program. In a further six operations, other partners have been involved in preparation and are providing technical support for implementation either directly or through trust funds that the Bank is administering. In three operations, the PAD notes the possibility that other development partners will provide additional financial support during implementation. These are all encouraging developments in helping realize the potential of PforR as an instrument that can help increase the efficiency and enhance the impact not just of Bank support but of support from the broader development community. But it is also clear (and confirmed through the structured interviews of government officials and development partners) that progress in this area is constrained by a range of factors—for example, relatively few development partners are able to use government program systems instead of their own and there 34 are different approaches to results-based financing. The Bank has been using the opportunity of the two-year review to engage with development partners on the progress that has been made and discuss how best to enhance the partnerships that are being established or strengthened in the context of PforR. The review has also enabled the Bank to step up its engagement and dialogue with other development partners around the overall issue of program and results-based financing, the experience of other partners with using instruments similar to PforR, and the potential for further use and strengthening of government systems. J. Transparency, Accountability, and Citizen Engagement 70. The PforR instrument was expected to support increased transparency in engagements with other stakeholders, and this indeed is what is happening. First, PforR operations have been disclosing all assessments and program documents, a level of disclosure and transparency that is higher than what has been done in the past. Second, all results used as DLIs are subject to a verification protocol that involves independent government agencies or third-party verification of the achievement of DLIs. Third, as noted above, all PforR Programs have established GRMs appropriate to the activities supported under the Program (see Box 10 for examples). Hence, citizens have clear mechanisms to know what results government programs are expected to achieve, how those results will be verified, the performance of government programs in relation to these results, and a mechanism to voice their complaints, where needed. Box 10. Complaint and Grievance Mechanisms Kenya National Safety Net Program (NSNP). Two of the five programs included in the NSNP already have complaint and grievance mechanisms, which are being strengthened. For the other three programs, a harmonized complaint and grievance mechanism has been designed; a hotline has also been established by the Department of Gender and Social Development to provide a way for people to raise concerns with the programs that is independent from front- line implementers; and the M&E framework for the NSNP includes indicators to monitor the functioning of the complaint and grievance mechanisms, including an assessment of the time it takes to give feedback to people who lodge complaints. Mexico Oaxaca WSS Sector Modernization Program. Guidelines for the establishment of GRMs at the water operator level are being included in the Environmental and Social Management Manual, which also covers other issues relating to citizen engagement, transparency, and accountability. The Ministry of Finance has hired a reputable nongovernmental organization to help prepare the Manual, thereby strengthening the engagement between civil society and the Government. Local Government Programs in Ethiopia, Tanzania, and Uganda Urban. Each participating urban/local government has or is putting in place a grievance handling system that includes reporting of complaints, receiving and addressing complaints, and reporting on corrupt practices and complaints. If a participating government does not put such a mechanism in place, it will not receive any funds from the operation. Health Programs in Croatia and Moldova. In Croatia, the Ministry of Health has introduced a free telephone service for patients’ complaints. In Moldova, an integrated complaint mechanism is being created and the capacity of the hotline is being strengthened. 71. Citizen Engagement. Many PforR operations are helping more broadly to increase the engagement of the public at large in planning operations (including selecting subprojects), monitoring implementation, and holding governments more accountable (see Box 11 for examples). In addition, PforR operations have also been using technology to help the verifications and the transparency of the system (see Box 12). Collectively, these various examples from 35 approved PforRs suggest that while the operations are still in the early stages of implementation, the instrument has significant potential to enhance transparency, accountability, and engagement of a broad range of stakeholders. Box 11. Citizen Engagement in PforR Operations Uganda Support to Municipal Infrastructure Development. The Municipal Development Forum is a multi- stakeholder forum in each of the municipalities that was piloted under a Cities Alliance trust fund and has now been rolled out under the PforR operation. Each forum provides a platform for the municipal government to engage with citizens on selection of subprojects, resource allocation, and accountability. The forum has allowed the voices of slum-dwellers to be heard, and they are the most active in the forum. Since the forum is apolitical, it brings in stakeholders from all walks of life who are interested in the development of the municipality —religious leaders, academicians, business communities, slum-dwellers, and so on. Ethiopia Local Government Development II Program. Citizens’ participation in the planning process under the Local Government Program is a performance measure that helps determine the financial allocation from the Program. Other performance measures include dissemination of annual city budgets, approved projects, expenditures, audited accounts, and results of the procurement decisions in offices and other public places or in newspapers, and webpages. Mexico Oaxaca WSS Sector Modernization Program. Under the Oaxaca Water and Sanitation Modernization Program, guidelines on engagement with neighborhood committees have been prepared to strengthen citizen engagement. Box 12. Use of IT to Enhance Citizen Engagement The Punjab Public Sector Management Reform Program (PPMRP) in Pakistan builds on several ongoing provincial government efforts to use cell phones to improve citizen engagement and performance monitoring. The Citizen Feedback Monitoring Program, run by the Punjab Information Technology Board and the Chief Secretary’s Office, monitors around 20 government services by reaching out to citizens who have received the targeted government service anywhere in Punjab. When a citizen accesses a government facility for a service, his/her cell phone number and the details of the transaction are logged and transmitted to an outsourced vendor who seeks feedback through a mixture of automated calls, SMS, and calls. As of August 2014, over five million citizens have been contacted, and around a quarter million citizens have replied with positive and negative feedback. The collected SMS responses are manually categorized, and call agents make random calls for further inquiry. The results are used to benchmark services across the province, and “district scorecards” are sent to district managers from the Chief Secretary’s Office. Nearly 4,000 corrective actions have been reported in the dashboard, based on the feedback received. The model is now being replicated in the provinces of Sindh and Khyber Pakhtunkhwa. The PPMRP is taking the Citizen Feedback Monitoring Program deeper into other areas of service delivery. Initially, the third-party vendor has collected and organized feedback from farmers receiving selected livestock and agricultural services whose details have been logged into the system by service providers who have been provided smart phones to collect and transmit transaction details. The ongoing stream of responses from beneficiaries enables managers to identify patterns of recurring problems and to tag them to the service provider or facility that remains problematic. Intensive citizen engagement also helps to improve citizen-state trust. K. Implementation Support 72. While to date the implementation of PforR operations has generally been proceeding well, experience is still limited. Implementation support is expected to include: (a) reviewing implementation progress and achievement of Program results and DLIs; (b) providing support on implementation issues and on institutional capacity building; (c) using Program monitoring 36 reports, audit reports, and field visits to monitor systems’ performance to ensure their continuing adequacy; and (d) monitoring changes in risks to the Program and compliance with legal agreements and, as needed, the PAP. This is a fundamentally different kind of implementation support from what is expected under IPF operations, and it requires a change in mind set —and even different skills—on the part of Bank managers and staff. 73. Implementation Support Plans. Implementation support plans have been prepared for all of the approved PforR operations, giving details on the main focus of implementation support and task team skills mix requirements. All of the plans emphasize the support and monitoring roles that teams need to play, and some go into considerable detail on the types of support to be provided. About half of the plans describe the role of other partners in implementation support. Some also note the connections between Bank support for the PforR operation and Bank support being provided in the context of related operations. Many plans stress the importance of having a significant presence in the country office so that PforR implementation support can be provided more regularly. The need for significant in-country presence is also emphasized in the structured interviews with government officials. 74. Implementation Status and Results Reports (ISRs). ISRs have now been prepared for 20 of the approved PforR operations, and fourteen of the operations have multiple ISRs. Summaries of these ISRs have also been publicly disclosed. The ISRs confirm that operations are performing at a satisfactory level, except for the Uruguay Roads PforR, where implementation progress was downgraded to moderately unsatisfactory in the latest ISR due to shortcomings in the implementation of the monitoring and evaluation framework and delays in PAP implementation. The Bank and government are discussing remedial actions to be undertaken. All other PforR operations are currently rated satisfactory or moderately satisfactory on both progress toward PDOs and implementation progress. ISRs are prepared by task teams and reviewed by sector and country management. Performance ratings are reviewed and updated regularly, and if progress is not as expected, teams are expected to downgrade the ratings. The performance of the Nepal operation, for instance, was downgraded for a period of time as the actions/results agreed at the time of appraisal were not achieved; following agreement between the Bank and government on remedial actions, the operation is now back on track. The ISRs prepared to date are focused particularly on progress toward achievement of the DLIs and the PAPs and on ways to address capacity constraints. Management comments recognize the generally good progress that is being made and the importance of proactive and high-quality technical support from the Bank teams. 75. Learning from Experience. Lessons are already being learned from implementation experience and reflected in the design of new operations. In Vietnam, for example, the task team has highlighted a range of issues for which lessons of experience with Vietnam’s first PforR operation have informed the design of the second operation (see Box 13). Other task teams have pointed to other important lessons that are being learned, including moving quickly to determine the agency responsible for verification of DLIs, and striking the right balance between detail and timeliness with respect to the scope of implementation reports. In the structured interviews, government officials have also emphasized the importance of building capacity from the very beginning, reducing and simplifying reporting requirements, and ensuring sufficient flexibility is built into the operations to allow for adjustments to indicators and other dimensions of the operations based on the realities of implementation. As implementation experience expands, it is clear that regular synthesis of the lessons learned will be helpful to teams and clients. 37 76. Adjustments. After only two years, implementation experience with PforR operations is limited. But even at this early stage, it appears that some additional guidance to managers and task teams as well as clients with respect to the expectations of implementation support under PforR would be helpful. Clients and teams have also asked for additional guidance in areas that are likely to emerge in the near future—additional financing, restructuring, midterm reviews, and so forth. Guidance in these areas will be developed as part of the updates of the guidance to staff following the two-year review and to be shared with government counterparts. Moving forward, as lessons from implementation emerge, they will also be mainstreamed into training and learning events to support clients and task teams and share lessons learned. Box 13. Lessons learned from Vietnam Rural Water Supply and Sanitation PforR The Vietnam team has undertaken two PforRs. The second operation, Vietnam Results-Based Northern Mountain Urban Development Program in the Northern Mountains Region (RBNUDP-NM), incorporated a number of lessons learned from the first:  Integration of technical assistance. The first operation supported technical assistance in parallel through a separate bilateral trust fund, which was not the most effective way to mainstream capacity building in a PforR as it followed different procedures and approaches. The RBNUDP-NM integrated capacity building activities into the operation through a DLI.  Flexibility through the Program Operational Manual. The team emphasized the development of a user- friendly and informative manual that can be adapted to operational realities during implementation.  Strengthening PAP compliance. While PAP actions are part of the financing agreement and are thus legally mandated, RBNUDP-NM further strengthened the PAP by integrating key actions into related DLIs. Disbursements are made only if the relevant DLIs have been achieved by conforming to the related PAP actions.  Engaging more at the policy level during preparation. Learning from the previous operation, the RBNUDP- NM task team engaged the higher-level stakeholders from the onset and obtained their buy-in and ownership, which facilitated the Program dialogue and ensured smooth preparation; it is also expected to lead to better implementation.  Institutionalization of verification. The verification agency chosen for the first PforR operation was the State Auditor of Vietnam, whose primary capacity lay in financial audits. Effective capacity building and transfer of technical skills to the State Audit of Vietnam have strengthened its capacity and broadened its skill-set to undertake technical audit as well. Having a strong and well-skilled State audit body will significantly benefit future operations across sectors in the country.  Deploying an experienced, field-based task team. Building on the experience of the first PforR operation, the RBNUDP-NM also has a country-office-based task team that is both experienced with PforR operations and readily accessible by the client. It has helped in both preparation and in day-to-day implementation support to the government at both national and local levels. V. INTERNAL ORGANIZATION AND MANAGEMENT 77. Following Board approval and taking account of the five percent cap on PforR commitments in the first two years, no official launch process was put in place. Instead, emphasis was placed on intensive, focused hand-holding for a set of operations put forward by the regions as possible candidates for the new instrument. This learning approach has helped all involved to learn and adapt, has provided good support to teams, and has helped in the development of the tools necessary to institutionalize the instrument. The approach has also fostered a small but engaged community of practice that has contributed to developing the instrument and to sharing 38 information across sectors and Regions. While this approach was judged as most feasible in light of the five percent cap on financing for the instrument, the downside, as noted earlier, has been that many clients are still unaware of PforR, and within the Bank there are wide disparities in the extent of knowledge of the instrument among both managers and staff. 78. Corporate Oversight. As Management agreed with the Board, it enhanced the corporate oversight arrangements during this initial phase. All proposed PforR operations are reviewed by a dedicated corporate-level team that includes technical, procurement, financial management, environmental, and social specialists. All operations have been required to have a corporate review (ROC or Operations Committee) meeting. In addition, the first two PforR operations in each Region have had a full Board discussion. Management has also closely monitored the build-up of PforR operations to ensure that commitments for them remain below five percent of total IBRD/IDA commitments. All of these arrangements have helped ensure that PforR operations are being processed in accordance with policy requirements. 79. Corporate Support. In addition to receiving advice from the corporate support team, staff have been provided with a range of other guidance, support, and training. An internal PforR website has been populated with guidance and learning materials as they became available. OPCS has met with many groups (e.g., regional management teams, sector boards, and hub trainings) to present the key features of the instrument and respond to questions. Templates have been prepared to help teams carry out their tasks; and systems, instructions, and guidance notes have been prepared to provide staff with a framework for the preparation of PforR operations. Finally, an MDTF has been established to support capacity building efforts, and demand for these resources has far exceeded the size of the trust fund. Feedback from the interviews with senior Bank managers suggests that the extent and nature of corporate support is well recognized and appreciated. 80. Other Organizational Support. In keeping with the cautious rollout of PforR, relatively few changes were introduced at the regional or network level to address the requirements of the new instrument. The senior managers interviewed for this report, representing regions and global practices, have all stressed the importance of the new instrument and their commitment to supporting its effective utilization. Some Regional Vice Presidents are actively promoting the instrument while others would prefer to see it spread more organically. They also stressed the importance of rolling out the instrument more both internally and with clients, creating communities of practice and other mechanisms for sharing of experiences, and ensuring there are skilled and experienced teams for PforR. 81. Learning. An overview e-learning module was made available in July 2012, and a full suite of six e-learning modules was launched in November 2013. Peer-to-peer learning events have been organized both for sharing overall experience and for capturing early findings on specific topics (such as ESSAs and legal issues); in addition, a community of practice has been established to help share lessons across operations. PforR clinics were offered to every PforR team that wished to have an intensive introduction to the instrument before starting PforR preparation. A PforR Academy is also under development and should be ready after the two-year review is completed. 82. External Communications and Outreach. Following approval of the new instrument, communications with clients and other external stakeholders were stepped up. Communication materials (e.g., leaflets) were produced and made available through country offices and the Bank’s 39 external website. Managers and staff were encouraged to discuss the merits of the new instrument with their counterparts. In connection with both the Annual and the Spring Meetings, special PforR events were organized for government delegations and other stakeholders. OPCS reached out to other development partners to inform them of progress in implementing the new instrument and obtain their feedback, including learning more about partners’ experience with similar types of instruments. Despite these efforts, in countries that are not yet involved in PforR operations, 40 percent of key counterparts in the dialogue on the Bank Group’s overall CPS report that they are not aware of the instrument; thus it is clear that more outreach and knowledge-sharing is still required. 83. Adjustments. Two areas of change and improvement are envisaged. First, more outreach, knowledge dissemination, and training activities are needed both internally and externally to raise awareness and understanding of the PforR instrument. In this regard, the community of practice will be strengthened to facilitate effective learning across teams and operations. Consideration will also be given to instituting a more rigorous learning program for task teams preparing PforR operations and staff in the advisory units commenting on PforR operations. Second, in the context of the Bank-wide reorganization, careful consideration needs to be given to what knowledge and capacity are required in the Regions and in the GPs to facilitate the further mainstreaming of the instrument and the preparation and implementation of successful PforR operations across all Regions and all GPs. However, enhanced outreach, knowledge dissemination, and training and strengthened capacity in the Regions and GPs will make sense only if the five percent cap on PforR commitments is eased and the choice of the PforR instrument can be made at the country level. VI. SUMMARY OF CONSULTATIONS 84. External consultations were carried out with PforR teams in government, bilateral development partners, multilateral development banks, civil society organizations (CSOs), the private sector and other interested stakeholders. Regular dialogue sessions on the review were also held for government delegations, staff from development partners, and international CSOs at the Annual and Spring meetings in Washington. The team also sought feedback from multilaterals including the Asian Development Bank, Inter-American Development Bank, African Development Bank and interested stakeholders in Europe. Internally, the team consulted major groups inside the Bank involved in the PforR process. Multiple engagements and inputs from directors, managers, country program coordinators and task teams helped shape the review exercise. In addition to face- to-face consultations, a dedicated consultation website was created providing details about the consultations. To facilitate the discussions, the team translated presentation materials into French, Portuguese, and Spanish, and videoconferences to connect multiple sites were organized. 85. The feedback from the consultations helped shape the recommendations of the review and helped validate the main findings from the surveys and interviews. Clear objectives of PforR, focus on results, and improved collaboration between the Bank and governments (working as one team, close communication to solve problems) were consistently highlighted as positive features. Countries asked for more in terms of capacity building, communications and training, and noted the importance of having field based staff for PforR operations. 86. The consultation process was also an opportunity for participants to provide additional feedback and some highlighted challenges. For instance, how best to define and agree on DLIs, 40 the articulation and application of the ACGs, the need for more capacity building, limitations of the exclusions, and the heaviness of the assessments were some of the issues raised. The summary of the consultations is presented in Annex 4. VII. SUMMARY OF FINDINGS AND PROPOSED WAY FORWARD 87. Although the PforR instrument was approved only three years ago, this review has been able to draw on a significant body of experience and on feedback from a broad range of stakeholders. At this relatively early stage in the rollout of the instrument, the overall story is a positive one.  While remaining below the five percent financing cap, the instrument has been successfully rolled out across a broad range of countries and sectors, policy requirements have been met, and implementation for all but one of the approved operations is broadly on track.  Feedback from those who have had direct experience in using the instrument (both government officials and Bank staff) is positive; the great majority is satisfied with their experience and expects to use the instrument again in the next two to five years.  Survey findings suggest interest among those who have yet to use the instrument; the majority of users also indicate that it is likely that they will use the instrument again in the next two to five years.  While no major changes in the design of the instrument appear to be necessary at this time, some improvements have been identified.  There is a clear need for more information sharing, exchange of good practices, and additional guidance and training for both Bank staff and government officials. A. Responding to the Demand for PforR 88. The five percent cap is a constraint. Figure 11. PforR as a Percentage of IBRD/IDA Commitments for approved PforR operations Commitments (Two-Year Basis) remain below the five percent cap of IDA and IBRD commitments (on a two year basis) 12% that was agreed with the Board when the 10% instrument was approved. However, a review 10.6% of planned operations for FY15 (see Annex 8% 5) shows that the cap will be reached during 6% the current two-year period (see Figure 11), 6.5% towards the end of this fiscal year. Moreover, 4% in view of the interest in using the 3.5% instrument, from both current and potential 2% 1.7% future users of the instrument, demand for 0% PforR operations is expected to continue FY12 - FY13 FY13 - FY14 FY14 - FY15 FY15 - FY16 41 increasing in the coming years. It is also clear that the presence of the cap has constrained the introduction of the instrument and its broader rollout and related investments in training and capacity building. 89. Strong demand expressed by clients. While the future share of the instrument as part of the World Bank portfolio will be influenced by a number of factors, the current pipeline and the survey results give a broad indication of the potential demand. As noted above, based on the existing pipeline, the cap will be reached in Q4 of FY15. And the pipeline for FY16 suggests that PforR operations will be 12-15 percent of the total IBRD/IDA portfolio (see Figure 12) in that year. Additionally, using the survey results to predict how many new PforR operations there will be in future years suggests that the demand could potentially double by FY19. Taking into consideration clients’ responses to the survey when asked about the likelihood that they will use PforR in 2-5 years, it was possible to derive that about 60 countries are likely to want to use PforR by FY19. Projections of demand then considered the average size of a PforR operation, regional variations, and estimated number of operations per country, to come up with an indicative measure of 24-27 percent by FY19. While this is not a target or an estimation of what the Bank will do, it provides an idea about the space that may be needed if the Bank is to effectively respond to client demand. 90. Going forward. In response to the strategic importance of the instrument, the positive response to date and the demand from clients, Management proposes a two pronged approach in the upcoming phase: (a) a revision of the constraint to allow the Bank to respond to demand over the next 2-3 years while at the same time ensuring that the instrument continues to be rolled out ‘gradually,’ and (b) continued investment in enhanced quality assurance and risk management while institutionalizing the instrument further. These are elaborated below. 91. PforR IBRD/IDA Commitments. Clearly the five percent cap is a major constraint to the development of an instrument that is critical to the implementation of the WBG strategy, that has already been successfully rolled out across a broad range of countries and sectors, and that is increasingly being demanded by both IBRD and IDA countries. In these circumstances, Management’s preference has been to proceed without a cap. However, from the consultations that Management has undertaken it is clear that there are different views on this matter among Executive Directors—some Executive Directors support proceeding without a cap, while others prefer to proceed more cautiously. In these circumstances, Management will continue to invest in enhanced quality assurance and risk management for PforR operations and Figure 12. PforR as a Percentage of IBRD/IDA Commitments (Fiscal Year Basis) will carefully monitor the growth of PforR commitments in total and in FY15 – PforR 15% operations approved relation to the growth in total IBRD/IDA or scheduled for 12% 13.6% commitments. Management also proposes Board presentation that if and when it is anticipated that the by June 30, 2015: 9% three year average of PforR commitments $2.8 billion 6% 7.6% 5 could exceed 15 percent of the three year average of total IBRD/IDA commitments, FY16 – PforR 3% Management will inform the Executive operations in pipeline: 0% Directors and discuss how best to address $5.1 billion. FY15 FY16 the future demand for the instrument. 42 92. Continued emphasis on quality assurance and risk management. While the overall story of the early experience with the PforR instrument is a positive one, the experience is still limited. Accordingly Management intends to carefully manage the further roll out of the instrument. Management also intends to continue the learning approach to PforR. Furthermore, Management proposes to provide the Board and stakeholders with updates on the instrument and its implementation through regular PforR retrospectives. More specifically, the following actions will be taken:  All proposed PforR operations will continue to be reviewed by a dedicated corporate- level team that will include technical, procurement, financial management, environmental and social specialists;  PforR operations will have corporate oversight and corporate inputs at various decision points;  Decisions being made at the country level concerning the choice of financing instrument will be tracked, with a view to consolidating experience and lessons learned and providing further guidance to managers and staff;  Management will carry out regular retrospectives on a periodic basis (about every 3 years) to take stock of the instrument and consolidate the learning during that period. Such reviews will incorporate the results of implementation completion and results reports and also build on work done by IEG and other groups (internal and external). In addition, an effort will be made to utilize impact evaluations on selected PforRs to facilitate operation specific learning. B. Limited Exception to Procurement Exclusions 93. One risk of the policy requirement to exclude from PforR operations activities that involve the procurement of goods, works and services under high-value contracts, highlighted at the time of the approval of the instrument, is that in some instances the exclusion may affect the development impact and integrity of the Program This has proven to be the case in certain instances. Thus, in light of experience, Management proposes that the current procurement exclusion policy requirement be revised to provide that, in exceptional cases, high-value contracts may be included in a PforR operation if the activities are deemed to be essential to the integrity of the Program and the monetary value of the contracts is deemed to be modest in relation to the overall size of the Program. Such exceptions would be subject to the following requirements:  At the earliest possible time during the preparation phase of an operation, the task team would prepare a request for such an exception that includes the justification, risk analysis and, as needed, risk management and mitigation measures.  The exception will be based on the “criticality” of the activity in relation to the supported Program. The approval of such exceptions would be at the level of the Managing Director and Chief Operating Officer. 43  Management would ensure that if such activities/contracts are included in a PforR operation, the activities/contracts, the justification for the exception, and agreed risk mitigation measures would be clearly described in the PforR documents. Any such activities would be subject to enhanced implementation support and teams will be asked to review their status as part of implementation support missions.  Management proposes to define the term “modest” (in the PforR Directive) as an individual contract for which the estimated monetary value is equal to or less than 25 percent of the overall size of the Program. Instructions for staff would be updated to set out the mandatory steps to be followed; and specific guidance will be developed to illustrate when this might be an option.  Management would review the experience with implementation of this exception in the next retrospective/review of the instrument. C. Technical and Editorial Revisions to OP/BP 9.00 and ACGs 94. Technical and Editorial Revisions of OP/BP 9.00 and ACGs. While the above procurement-related proposal constitutes the one substantive policy change that requires Board approval, there are several additional necessary technical and editorial changes that do not constitute a change in policy. First, in accordance with the P&PF,20 a substantive change to OP 9.00 (which would be the case if the procurement proposal is approved) requires that it be retrofitted in a P&PF compliant form, and be issued as a Policy (rather than OP). This requires a number of editorial and formatting changes. The draft PforR Policy that would replace OP 9.00 is attached in Annex 6A. Annex 6B provides a more detailed description of the proposed changes. 95. In accordance with the P&PF, if OP 9.00 is being retrofitted, BP 9.00 must also be retrofitted in a P&PF compliant form, and be issued as a Directive (rather than BP). This requires a number of editorial and formatting changes, in addition to the one substantive change in paragraph 93 above. The draft PforR Directive that would replace BP 9.00 is attached in Annex 7A. In addition, and based on the outcome of the review, a number of other technical and editorial changes will be made. Such changes aim to further clarify specific issues, address the concerns about linking prior results and advances, ensure consistency across instruments (e.g., audits), and/or minimize duplication (e.g., if they are covered in the Policy or ACGs). These are explained in detail in Annex 7B. 96. Finally, in light of the importance of the feedback received, Management has determined that some clarifications to the ACGs would help ensure the further successful rollout of the PforR instrument (see Annex 8A). These changes would in no way change the Bank’s substantive rights under the current ACGs. The proposed changes frame the ACGs in the context of the Bank’s and borrower’s common interest in and commitment to preventing and combating fraud and corruption in the Program (not just as an expression of Bank policies and interests) and clarify that the ACGs are not intended to usurp the rights of sovereign borrowers. Annex 8B provides a more detailed description of the proposed changes. 20 Bank Procedure: Policy and Procedure Framework, Catalogue Number LEG4.01PROC.01 (January 8, 2014). 44 D. Implementation Plan 97. PforR has been rolled out cautiously and successfully. But as paragraph 12 explained, PforR is an important element in the implementation of the WBG’s Strategy. Accordingly it is important that the instrument now be fully rolled out and integrated into the Strategy implementation process at the corporate, Regional, and GP levels. There is also a clear need for more knowledge sharing and enhanced understanding of the PforR instrument at two levels. First, at the overall level of the country dialogue, counterparts in many countries would like to be better informed about the instrument so they can effectively consider its potential value in the context of the development of their partnerships with the WBG. Country directors and country managers in countries that have not yet used the PforR instrument have also indicated their desire for more knowledge and for learning from teams with direct PforR experience. Second, at the level of the task teams working on PforR operations, there is a desire for enhanced support and more cross- team learning. 98. Below is an outline of the plan for enhanced roll out of the instrument. This builds on the efforts that have been put in place to date as well as the lessons learned from the initial implementation phase. It is important to note that the level of investment in these areas depends on the decision regarding the cap.  Templates, instructions, and systems. Templates and instructions will be updated to take into account lessons learned from the review and other changes in the institution. These changes also include updating the risk framework, the SORT (see paragraph 61), and IT Systems to better institutionalize the monitoring and reporting on the instrument.  Guidance Materials. All of the Guidance notes will be updated and revised to take into account the lessons of experience and feedback from clients and staff. Attention will also be given to take into account the recommendations in the various sections of this review. The revisions are intended to provide more direct operational guidance and “how to” direction to staff. Where appropriate, best practice notes and examples will also be provided to teams to facilitate the learning process.  Training. For both clients and staff, there is a strong demand for learning events and materials about PforR. Core training and orientation programs will be designed for countries that can be made available through a variety of vehicles. For staff, the formal operational training that is currently available for other instruments will be expanded to include PforR. In addition, a PforR Academy will soon be launched. For both clients and staff, learning events upon demand will also be available.  Community of practice. The established community of practice will be strengthened and its outreach efforts expanded, thereby strengthening learning across teams and GPs and helping to generate knowledge and good practice notes that can help clients and task teams with future PforRs. The community of practice is considered an important vehicle for enhancing the efficiency of learning about PforR, as well as an extension of the corporate support for the instrument. This will help to ensure that staff working on PforR operations have adequate access to those with experience and knowledge of the instrument. 45  Corporate Support. Support at the corporate level will continue (see paragraph 79). Working with the community of practice as well as the Regions and GPs, this will facilitate the effective further rollout of the instrument.  Communications and outreach. A package of communication materials that clearly describe the instrument and how it functions will be developed for clients. A package of communication materials will also be prepared for staff, presenting the instrument and discussing how and when to use PforR and identifying available resources. As part of the outreach and training efforts, the materials and activities will incorporate clear guidance to staff and information for clients to help them make informed choices of instrument. In addition, outreach to external stakeholders including civil society and development partners, will continue.  Transparency and access to information. PforR has adopted a very open approach where all documents, including the Bank’s assessments, have been made available to the public. Moving forward, these efforts will continue and other mechanisms to make information more accessible to various stakeholders in countries, and hence further enhance transparency, will be explored.  Change Management Process. PforR requires a change in mind set and as such investments in the change management process are critical for its success. Users of PforR, as well as country directors and GP managers have an important role to play. Outreach to these groups as well as targeted events and engagement will be very important. Senior Management (at Regional, GP and Corporate levels) is also critical as staff will look to them for guidance and direction. The systematic outreach to these key decision makers and influential figures will also ensure that we have good understanding of and have more systematic and consistent approaches for the instrument in a broader set of countries, Regions, and GPs.  Supporting capacity building. Efforts will be made to expand and replenish the MDTF for PforR to provide additional support regarding capacity building efforts, especially for the client.  Learning approach. Management will continue with the learning approach by establishing a strong corporate support team to help with PforR operations and overall learning and by introducing regular PforR “retrospectives” through which it can assess and draw lessons from experience with the instrument. VIII. NEXT STEPS AND REQUEST FOR APPROVAL 99. Next Steps. If Executive Directors approve Management’s proposal as described in paragraph 100 below, Management will undertake the following actions to enhance the rollout of the PforR instrument: a. Management will issue the new Policy and Directive that will replace OP/BP 9.00 as described in paragraphs 94 and 95 of the paper. 46 b. Management will issue the revised ACGs as described in paragraph 96 of the paper. c. Management will update the instructions, guidance, templates, and improve the communication and training as described in paragraph 98 of the paper. 100. Recommendation. Management recommends that the Executive Directors: a. Endorse the new proposal regarding the PforR IBRD/IDA commitments as described in paragraph 91 of the paper. b. Approve the limited exception to the policy requirement to exclude from PforR operations activities that involve the procurement of goods, works and services under high-value contracts, as described in paragraph 93 of the paper. 47 ANNEX 1. RESULTS-BASED INITIATIVES 1. Development of the PforR instrument took into account the lessons learned from the Bank’s own experience with various lending operations that incorporate programmatic features as well as other agencies’ experience with program-based aid. In the course of implementation the PforR instrument continues to draw on internal21 and external experiences. In addition, this process has given the Bank the opportunity to be more engaged in results-based approaches across institutions. The knowledge- sharing among the community of practitioners is a very important way to build on and expand our knowledge in this area. To this end the Bank has participated via video-conferencing in a seminar hosted by the European Commission, met with the Management of the African Development Bank, consulted with the Asian Development Bank and taken the opportunity to exchange experiences with results-based financing with bilaterals during visits to client countries. 2. This annex highlights some of the lessons learned and some of the experiences of Bank engagement, and builds on the work done at the time the policy paper was prepared.22 A. UK Department for International Development (DFID) 3. DFID has been one of the world leaders in pioneering innovative payment-by-results (PBR) programs. PBR has been a core part of DFID’s work, and the experience with PBR informed the development of the PforR instrument. In June 2014, DFID scaled up its PBR strategy—“Sharpening Incentives to Perform: DFID’s Strategy for Payment by Results”—taking into account the lessons of experience:  PBR works best when performance is within the control of the management or staff and the incentives provide the motivation to produce desired results.  PBR should be tailored to each situation and requires up-front investment and careful risk analysis.  Given the challenges to implementation of PBR, programs should include an internal M&E capability.  Internal M&E capacity is important, and the “hands off” approach should not be seen as a barrier to independent evaluation.  Financial planning for PBR programs can be more challenging because of the inherent uncertainty about the results the implementing organization will be able to deliver. B. Inter-American Development Bank (IDB) 4. IDB adopted its policy on performance-driven loans (PDLs) in 2003 for a 6 year pilot. The main features of PDLs are (a) disbursement based on meeting results targets, except for advance 21 A review of 781 Investment Project Financing that were approved over the last two years revealed that approximately 21.77 percent of these operations supported some form of results based lending (including conditional cash transfers, DLIs, results-based financing, SWAps, and output-based aid). 22 World Bank, 2011, A New Instrument to Advance Development Effectiveness: Program-for-Results Financing. See http://documents.worldbank.org/curated/en/2011/12/15590386/new-instrument-advance-development-effective ness-program-for-results-financing 49 payment, which can be up to 20 percent of IDB financing; (b) results defined by outcomes; (c) results audited by independent performance reviewers; and (d) use of country systems for procurement. The demand for PDLs was low—17 operations from 2003 to 2009—for two main reasons. Effectively, PDLs have placed double requirements on the governments and IDB staff by adding the result requirements on top of investment lending (that is, both results-based disbursement and tracking of funds). This increased transaction costs and contributed to significant delays in disbursements. More importantly, the PDLs’ combination of results- and transaction-based approaches was not optimal for project design and implementation. The last PDL was approved in November 2009 and in effect the pilot program has been allowed to lapse, pending further review and evaluation. 5. In 2010 the IDB’s Board of Governors approved the 9th General Capital Increase of the IDB (IDB-9). The IDB-9 Agreement laid out a series of reforms intended to strengthen the IDBs’ strategic focus, development effectiveness, and efficiency to help it remain competitive and relevant in the years ahead. IDB’s Office of Evaluation and Oversight (OVE) was charged with conducting a midterm evaluation, to be presented to the Board of Governors in March 2013. In discussing PDLs, that report concluded: “This raises some real issues for IDB in that the portfolio now lacks an instrument that disburses against results. The solution here might be to move to the approach of the World Bank, which has introduced loans that disburse against program results.” IDB’s implementation experience served as an important input in the design of the PforR. C. European Commission 6. In 2009, at the start of European Development Fund (EDF) 10, the EC launched the MDG Contract, a longer-term, more predictable form of general budget support, in a number of countries as part of its international commitments to provide more predictable assistance to developing countries. MDG-Contracts have a strong focus on MDG-related results. About two-thirds of indicators relate to the health and education sectors (three-quarters when considering the financial weight of different indicators), with water, agriculture, business climate and infrastructure, public financial management, and social vulnerability accounting for the rest. 7. MDG-Contracts were agreed in eight countries (Burkina Faso, Ghana, Mali, Mozambique, Rwanda, Tanzania, Uganda, and Zambia). Collectively these accounted for €1.8 billion, or about 50 percent of all general budget support commitments in EDF 10 national programs, and some 14 percent of all EDF 10 national programs. D. Asian Development Bank (ADB) 8. ADB’s policy on piloting Results-Based Lending for Projects became effective on June 6, 2013, and is being piloted for six years. 9. The objectives of Results-Based Lending are to increase accountability and incentives to deliver and sustain results, improve the effectiveness and efficiency of government-owned sector programs, promote institutional development, and enhance development effectiveness. What distinguishes Results-Based Lending from other ADB modalities is that disbursements of loan proceeds are directly linked to the achievement of program results. 10. The first Results-Based Lending program ($200 million loan) was approved by the Board on June 27, 2013, to help the Government of Sri Lanka revamp its secondary school system to cut youth 50 unemployment and meet the country’s urgent need for a skilled workforce. The second program ($100 million), also in Sri Lanka’s education sector, was approved on March 29, 2014. There are nine other operations in the pipeline currently under preparation in different sectors and countries. E. Millennium Challenge Corporation (MCC) 11. The MCC was created in 2004 as an independent aid agency of the United States to help in the fight against global poverty in innovative ways. The MCC emphasizes achieving and measuring results, concentrating on identifying activities that have the greatest promise of poverty reduction through growth, measuring progress during implementation, and learning from experience. 12. As of late 2013, 25 countries have received funding through 27 compacts23 totaling $9.2 billion, and 22 countries have received funding through 24 threshold programs. About 53 percent of compact funding has gone to countries in sub-Saharan Africa. The MCC has disbursed over US$2 billion. 13. Recent MCC publications highlight the main lessons learned from the agency’s experience: (a) Focus on results  Transparency is a big part of accountability for results.  “Results” means different things at different times in the project cycle. Detailed project design in the early years produces better results in the long-term.  Data quality is the cornerstone of robust focus on results. (b) Country ownership  is a partnership based on mutual accountability that benefits from structure and clear expectations.  is a balancing act between MCC principles and operational approaches.  goes beyond national governments, both in setting investment priorities in compact development and in implementing compact programs.  includes capacity building, but not everything has to be about capacity building. F. Output-based Aid 14. Output-based aid (OBA), broadly defined, seeks to tie disbursement to the achievement of specific outputs. The Bank-administered Global Program on Output-based Aid (GPOBA) started in 2000 to both support OBA activities through trust funds and promote the wider use of OBA in the development community. The GPOBA portfolio includes 147 closed and active projects, and the OBA subsidy portfolio is $4 billion. Most of the projects are in AFR (39% of active or closed projects) and LCR (72% of the total volume of subsidies). 23 A Compact is a multi-year agreement between the MCC and an eligible country to fund specific programs targeted at reducing poverty and stimulating economic growth. 51 15. Lessons learned:  There has been increased transparency through the explicit targeting of subsidies. If outputs are explicitly defined, targeting can be made more precise. The process of output verification can serve as an additional check on the targeting of subsidies.  Increased accountability because performance risk is shifted to service providers: OBA pays providers only after delivery of verifiable access and service.  OBA leverages some private funding, but because of its generally pro-poor nature, private financing leveraged is limited by the extent to which user fees can incorporate investment costs while remaining affordable. 16. With the introduction of PforR, an effort has been made to coordinate and guide staff as to when to use each instrument. Moving forward a guidance note will be developed to clarify further this issue for staff. G. Results-based Financing for Health 17. The Bank supports results-based financing for health (RBF) to help achieve the health-related Millennium Development Goals (MDGs). As part of this effort the Health Results Trust Fund (HRITF) was created in 2007—a multi-donor trust fund supported by the governments of Norway and the United Kingdom with commitments totaling US$551 million equivalent from 2009-2022. The HRITF plays a major role in scaling up RBF in many countries. It supports Country Pilot Grants (CPGs) linked to eligible new and existing International Development Association (IDA) projects, learning and dissemination activities in IDA countries, and evaluation activities contributing to RBF work. One of the PforR operations (Ethiopia Health) was co-financed with the HRITF; the overall goals and objectives were aligned, but there were some operational issues (including what is eligible as results) that will need further work. 18. As of May 22, 2014, 30 RBF projects have been approved by the Board, for which IDA has provided US$1.3 billion in financial support and HRITF US$346 million in cofinancing. 19. Lessons learned:  Implementing an RBF scheme cannot be successful without a strong design that responds to local constraints and bottlenecks for service delivery, and strong implementation support.  Robust institutional set-up is critical, including the need for separation of functions to ensure no conflicts of interest.  Continuous monitoring throughout project implementation is vital to identifying issues and any need for real-time course correction.  An understanding of the financial incentive payment mechanism is important. 52 ANNEX 2. SURVEYS: METHODOLOGY AND RESULTS 1. To capture the early lessons of the PforR instrument, OPCS conducted surveys to collect opinions from clients and staff. Separate surveys were designed for two distinct groups of recipients: a longer one for users of the PforR instrument and a shorter one for non-users. Each survey focused on the experience internal and external clients had so far had with the new instrument and on their expectations for the future. The team made an effort to ensure that most of the questions were the same for the different groups being surveyed to facilitate direct comparisons, although some questions were unique to particular stakeholder groups. The annex starts with a description of the survey and then presents the results of the short survey (for non- users) followed by an analysis of the long survey (for the users of the instrument). I. SURVEY DESCRIPTION A. Questionnaire Design Methodology 2. The core team established to work on the questionnaire design included members of the original PforR design team, who had significant knowledge about PforR. The surveys were designed through a rigorous process: brainstorming by the core team, review under the guidance of the Bank’s Public Opinion Research Group (PORG), consultation with the Independent Evaluation Group (IEG), and pilot-testing with country program coordinators. After the feedback from the consultations was incorporated into the survey questionnaires, the links to the surveys were circulated to selected OPCS members to detect any technical problems. The final surveys were made available in four languages: English, French, Spanish, and Russian. B. The Two Surveys 3. Short Survey. The short survey targets country directors/country managers and their country counterparts to assess their awareness of the instrument, and to evaluate the extent and nature of the dialogue about PforR in all the countries that have lending programs with the Bank but had not yet used the instrument (92 countries as of the launch date, listed below). The survey had seven multiple-choice questions and one section for comments, and took about 5 minutes to complete. Minor adjustments were made to adapt to internal/external audiences, and the links to the surveys were sent to 66 Bank managers and 216 clients. Countries that have not used PforR Afghanistan, Albania, Arab Republic of Egypt, Armenia, Azerbaijan, Belize, Benin, Bhutan, Bosnia and Herzegovina, Botswana, Bulgaria, Burkina Faso, Burundi, Cambodia, Cape Verde, Central African Republic, China, Colombia, Comoros, Côte d'Ivoire, Costa Rica, Democratic Republic of Congo, Djibouti, Dominican Republic, El Salvador, Georgia, Ghana, Grenada, Guatemala, Guinea, Guinea-Bissau, Guyana, Haiti, Honduras, Iraq, Jamaica, Jordan, Kazakhstan, Kiribati, Kosovo, Kyrgyz Republic, Lao People's Democratic Republic, Lebanon, Lesotho, Liberia, FYR Macedonia, Madagascar, Malawi, Maldives, Mali, Mauritania, Mauritius, Mongolia, Montenegro, Myanmar, Namibia, Nicaragua, Niger, Panama, Papua New Guinea, Philippines, Poland, Republic of Congo, Republic of Yemen, Romania, Russian Federation, Samoa, Senegal, Serbia, Seychelles, Sierra Leone, Solomon Islands, South Africa, South Sudan, Sri Lanka, St. Lucia, St. Vincent and the Grenadines, Sudan, Swaziland, Tajikistan, Thailand, The Gambia, Timor-Leste, Togo, Tonga, Tuvalu, Uzbekistan, Vanuatu, West Bank and Gaza, Zambia, Zimbabwe 53 4. Long Survey. The long survey focuses on the perspectives of Bank task teams and their country counterparts concerning PforR operations that are already approved or under preparation, for which they are responsible. This survey endeavors to enrich the understanding both of what is going well with respect to PforR operations and of what the concerns and challenges are from the perspective of those who have been deeply involved in their design and implementation. The survey had 45 multiple-choice questions and 4 open-ended questions, and took 10-15 minutes to complete. Minor adjustments were made to adapt to internal/external audiences, and the links to the surveys were sent separately to the internal group (380 potential participants: task team members, country managers, country directors, country program coordinators, and sector managers from 30 operations) and the external group (71 potential respondents from 24 different operations). 5. The list below shows the PforR operations that were surveyed. The gray shading identifies the six operations that were in very early stages and did not have any external potential respondent. Country – Operation Bangladesh - Enabling Open Government - P129770 Bolivia - National Road Sector Management Program - P144597 Brazil - Ceará (MST) PforR - P127463 Croatia - Health System Quality and Efficiency Improvement - P144871 Ethiopia - Health Millennium Goals Program-for-Results - P123531 Ethiopia - Local Government Development Project II - P133592 India - Enhancing Teacher Education Bihar - P132665 India - Third Maharashtra Rural Water Supply and Sanitation Project (Jalswarajya-II) - P126325 Indonesia - Road Asset Preservation Program - P113936 Kenya - Devolution Support Project - P149129 Kenya - National Integrated Safety Net Program - P131305 Mexico - Oaxaca WSS Sector Modernization - P145578 Moldova - Health Transformation Operation - P144892 Morocco - Health Sector Support - P148017 Morocco - Integrated Risk Management - P144539 Morocco - National Initiative for Human Development (INDH) Phase II - P116201 Morocco - Urban Transport - P121497 Mozambique - Public Financial Management for Results Program - P124615 Nepal - Results-Based Bridges Improvement and Maintenance Project - P125495 Nigeria - Program to Support Saving One Million Lives - P146583 Pakistan - Punjab Governance and Service Delivery - P132234 Rwanda - Agriculture Growth Corridors - P148927 Rwanda - Public Sector Governance Program - P149095 Tanzania - Big Results Now in Education - P147486 (9) Tanzania - Urban Local Government Strengthening Program - P118152 Tunisia - Local Government - P130637 Uganda - Support to Municipal Infrastructure Development - P117876 Uruguay - Road Rehabilitation and Maintenance Program - P125803 Vietnam - Northern Mountains Urban Program - P143596 Vietnam - Results-Based Rural Water and Sanitation under the National Target Program Project - P127435 54 C. Survey Launch, Closure, and Analysis 6. The short survey opened to internal and external audiences on January 16, 2014, and was sent in batches as the contacts for the external potential respondents were received. It concluded on April 4, 2014. The long survey was launched on March 6, 2014, and closed on May 9, 2014. It was also sent in batches to the external clients as the names became available. For both external surveys, the links were sent from an OPCS vice presidency service account to ensure the proper monitoring of response rates. 7. OPCS and PORG worked closely on monitoring the surveys throughout the process. PORG reported regularly on the response rates, enabling OPCS to decide to whom and when to send reminders. When both of the surveys were closed, PORG consolidated the results (ensuring that the data were anonymous) and sent the raw data to OPCS for analysis. II. SURVEY RESULTS AND ANALYSIS A. Short Survey The internal survey was sent to country Internal External managers and country directors covering Number of people to whom the survey 66 216 client countries that had had no was sent experience with the new instrument, and Respondents 19 67 the external survey was sent to Response rate 29% 31% counterparts in those countries. Both Number of countries covered 98 78 surveys were very similar (only question Number of countries with respondents 19 51 3 was different) and very short, with only Rate of countries with respondents 19% 65% seven multiple-choice questions. 1. External Short Survey This table compares absolute responses Are you aware of the World Bank Group’s PforR lending and responses weighted by country. This instrument? same analysis was made for all the other Number of % out of Number of % out of 67 responses questions, and no significant differences Response 51 responses responses weighted by were found. Therefore, only the analysis of country countries absolute responses is presented for the Yes 41 61% 34.27 67% remaining questions. This first question in No 26 39% 16.73 33% the survey served as a filter: only those who responded that they were aware of the instrument proceeded to answer the rest of the questions. The respondents for the rest of this survey totaled 41 from 36 countries. Number of How would you rate your familiarity with the PforR lending instrument? % responses Not at all familiar 1 2% Not very familiar 13 32% Somewhat familiar 20 49% Very familiar 7 17% 55 Which of the following best describes the discussions you have had with the Number of % World Bank Group abour the PforR lending instrument? responses I have had very informative discussions about the PforR instrument 12 29% I have had some discussions about PforR but do not feel very informed 17 41% I have had no discussions with the Bank about PforR 12 29% Prefer not to respond 0 0% Type of discussion.  All 7 respondents who felt very familiar with the instrument had had very informative discussions.  7 respondents had no discussions with the Bank about PforR but were somewhat familiar with it. Which other groups were involved in discussions that you have had with the World Bank Number of Group about PforR? (choose all that apply) responses Other government representatives 25 Other development partners 7 Other non-government stakeholders 6 No other groups were present in those discussions 5 I do not recall 2 Prefer not to respond 6 Based on your knowledge of the PforR lending instrument, in which sectors would it be Number of most valuable for it to be used in your country? (choose up to three) responses Education 19 Health, nutrition, and population 13 Governance 11 Agriculture 10 Energy and extractives 9 Finance and markets 8 Transport and ICT 8 Poverty 8 Social protection and labor 6 Water 5 Macroeconomics and fiscal management 5 Urban, rural, and social development 3 Environment and natural resources 2 Trade and competitiveness 0 Most valuable sectors.  In general, respondents indicated that PforR would be most valuable in the education and the health, nutrition, and population sectors. 56  However, respondents who were very familiar with the instrument reported it as applying to a broader range of sectors. How likely is it that your country will use the PforR instrument in the next two to Number of % five years? responses Very unlikely 3 7% Somewhat unlikely 2 5% Somewhat likely 9 22% Very likely 14 34% Don’t know 10 24% Prefer not to respond 3 7% Likelihood of using PforR.  The majority of respondents felt that their country is somewhat or very likely to use PforR in the next 2-5 years.  Respondents who claimed to be very familiar with the instrument believed that it was very likely that their country will use PforR in the next few years.  Of the 15 respondents who either thought their country is unlikely to use the instrument or were not certain, most were somewhat to not at all familiar with the instrument. Which ONE of the following best describes why you think that it is unlikely that your Number of country will use the PforR instrument in the next two to five years? (Only those who think responses the country is unlikely to use the instrument in the next 2-5 years answered this question.) No new operations are being planned 1 World Bank’s other lending instruments are more suitable 2 Some features of the PforR instrument would need to be modified to make it usable for the 0 country Don’t know enough about the instrument to answer 4 2. Alternative Analysis of External Short Survey a. By country income Response IBRD IDA Total Aware of PforR 21 20 41 Awareness of PforR. Of the 67 external respondents, Not aware of PforR 12 14 26 33 were from 21 middle-income countries and 34 from Number of respondents 33 34 67 30 low-income countries. The first question about the awareness of PforR yielded 21 (36%) respondents from middle-income countries and 20 (41%) from low- Response IBRD IDA Total income countries. Not at all familiar 1 0 1 Not very familiar 7 6 13 Familiarity with PforR. The responses from the IBRD Somewhat familiar 11 9 20 and IDA respondents had similar distributions; the Very familiar 2 5 7 majority of respondents were somewhat or very familiar Prefer not to respond 0 0 0 with the instrument. Number of respondents 21 20 41 57 Type of discussion. When asked about the discussions with the World Bank Group about the PforR lending instrument, more low-income country respondents (40%) than middle-income country respondents (19%) reported very informative discussions. However, combining the answers “very informative discussions” and “some discussions,” both groups had a similar number of responses. Response IBRD IDA Total I have had very informative discussions about the PforR instrument 4 8 12 I have had some discussions about PforR but do not feel very informed 10 7 17 I have had no discussions with the Bank about PforR 7 5 12 Prefer not to respond 0 0 0 Number of respondents 21 20 41 Participants in the discussions. Only low-income country respondents indicated that the discussions included other development partners. According to respondents from middle-income countries, no groups were present in the discussions other than the government representatives. Response IBRD IDA Total Other government representatives 12 13 25 Other development partners 0 7 7 Other non-government stakeholders 2 4 6 No other groups were present in those discussions 5 0 5 I do not recall 1 1 2 Prefer not to respond 1 5 6 Number of respondents (Respondents could choose all that applied) 21 18 39 Most valuable sectors for using PforR. IDA country respondents rated health, nutrition, and population as the most valuable sector in which to use PforR, while IBRD respondents chose education (see figure below). Education Governance Transport Finance Energy Health Water Social Poverty Agriculture Macro- Environment Urban and and Protection economics Rural and Markets Extractives and Labor Social Dev 58 Likelihood of using PforR in the next 2-5 Response IBRD IDA Total years. Half (50%) of the respondents from low- Very unlikely 3 0 3 income countries thought their country is very Somewhat unlikely 1 1 2 likely to use the instrument in the future, Somewhat likely 4 5 9 whereas more than a third (38%) of respondents Very likely 4 10 14 from middle-income countries did not know Don’t know 8 2 10 how to answer when asked about the likelihood Prefer not to respond 1 2 3 of using the instrument. All the respondents Number of respondents 21 20 41 who said their country would be very unlikely to use the PforR instrument in the next two to five years were from middle-income countries. b. By Region Awareness of PforR. The table below shows the breakdown of respondents by Region. Only those who were aware of PforR went on to answer the rest of the survey. The largest share of respondents (37%) who were aware of the instrument were in ECA. SAR was not very well represented in the short external survey, with only 2 participants. Region AFR EAP ECA LCR MNA SAR Number of countries 16 6 13 7 7 2 Number of respondents 18 6 25 7 9 2 Familiarity with the instrument. The majority of respondents in all Regions were at least somewhat familiar with PforR, except in MNA, where respondents seemed to be less familiar with PforR, on average. Discussions about PforR. In all Regions, the majority of respondents had some discussion or very informative discussion about PforR. Participants in the discussions. For all the Regions except SAR, a majority of respondents indicated that “other government representatives” were involved in discussions with the Bank. Response AFR EAP ECA LCR MNA SAR Other government representatives 5 3 8 5 4 0 Other development partners 1 2 2 1 0 1 Other non-government stakeholders 0 2 3 0 0 1 No other groups were present in those discussions 2 0 2 0 1 0 I do not recall 0 0 1 0 1 0 Prefer not to respond 0 1 2 1 1 1 59 Most valuable sectors. The most valuable sectors for using PforR, looking at the responses by Region, were:  AFR: Education and poverty  EAP: Agriculture and poverty  ECA: Education  LCR: Transport  MNA: Education and transport  SAR: Governance Likelihood of using PforR in the next 2-5 years. In all Regions except ECA, the majority of respondents thought that it is very or somewhat likely their country will use PforR in the future. Response AFR EAP ECA LCR MNA SAR Very unlikely 14% 0% 7% 0% 14% 0% Somewhat unlikely 0% 0% 7% 0% 14% 0% Somewhat likely 14% 25% 13% 0% 43% 100% Very likely 43% 75% 20% 67% 15% 0% Don’t know 29% 0% 40% 17% 14% 0% Prefer not to respond 0% 0% 13% 16% 0% 0% 3. Internal Short Survey Are you aware of the World Bank Group’s PforR lending instrument? All 19 country managers and country directors who responded to the survey said that they were aware of PforR; 47 did not respond. How would you rate your familiarity with the PforR lending instrument? 11 (58%) respondents were somewhat familiar, and 8 (42%) very familiar, with the instrument. Which of the following best describes the discussions that you have had with Number of % your government counterparts about the PforR lending instrument? responses I have had very detailed discussions about the instrument with government 3 16% counterparts I have had some brief discussions about PforR with government counterparts 10 52% I have had no discussions with government counterparts about PforR 6 32% Prefer not to respond 0 0% 60 Type of discussion.  The majority of respondents had some brief discussions with government counterparts about PforR, but only respondents who were very familiar had very informative discussions.  Three respondents were very familiar with the instrument but did not have any discussions. Which other groups were involved in discussions that you have had with your government Number of counterparts about PforR? (choose all that apply) responses Other government representatives 9 Other development partners 7 Other non-government stakeholders 4 No other groups were present in those discussions 4 I do not recall 1 Prefer not to respond 3 Based on your knowledge of the PforR lending instrument, in which sectors would it be Number of most valuable for it to be used in your country? (choose up to three)Response responses Health, nutrition, and population 12 Education 11 Water 5 Social protection and labor 5 Governance 4 Urban, rural, and social development 3 Agriculture 1 Energy and extractives 1 Transport and ICT 1 Trade and competitiveness 1 Macroeconomics and fiscal management 1 Finance and markets 0 Environment and natural resources 0 Poverty 0 Most valuable sectors. Like external respondents, internal respondents also selected the health, nutrition, and population and education sectors as those in which the use of PforR would be most valuable. 61 How likely is it that your country will use the PforR Instrument in the next two to Number of % five years? responses Very unlikely 3 16% Somewhat unlikely 3 16% Somewhat likely 9 47% Very likely 4 21% Don’t know 0 0% Which ONE of the following best describes why you think that it is unlikely that your country Number of will use the PforR instrument in the next two to five years? (Only those who think the responses country is unlikely to use the instrument in the next 2-5 years answered this question) No new operations are being planned 2 World Bank’s other lending instruments are more suitable 2 Some features of the PforR instrument would need to be modified to make it usable for the country 2 Don’t know enough about the instrument to answer 0 Modifying features. Respondents identified the following features that made it unlikely the countries would use PforR:  It is very costly to appraise and supervise a PforR because of the need to look at the whole system in the country.  The anticorruption guidelines’ insistence on the Bank’s right to audit all contracts even if it funds a tiny amount will need to be changed. B. Long Survey The internal survey was sent to task Internal External team members, sector managers, Number of people to whom the survey 380 71 country managers, and country directors was sent with experience on PforR, and the Respondents 141 26 Response rate 37% 37% external survey was sent to counterparts Number of countries covered 21 19 in countries with PforR operations Number of countries with respondents 19 12 under preparation or implementation. Rate of countries with respondents 90% 63% Both surveys were very similar, with Number of operations covered 30 24 around 45 multiple-choice questions Number of operations with and 5 open-ended questions.24 The 26 14 respondents number of respondents and the response Rate of operations with respondents 87% 58% rates are shown below. 24 Written comments in response to open-ended questions were collated and analyzed by an independent expert in qualitative data. 62 1. External Long Survey a. Overall experience with PforR instrument Number of How knowledgeable do you consider yourself about the PforR instrument? % responses Not knowledgeable enough to understand the PforR operation 0 0% Somewhat knowledgeable but would need more to understand the PforR operation 7 27% I have enough knowledge to understand the PforR operation 19 73% Number of respondents 26 100% What were the key reasons for deciding to use the PforR instrument? (Choose no Number of As % of total more than TWO) responses responses Support to improvements under government programs 10 22% Use of government systems 10 22% Focus on results 19 42% The fact that this is a new instrument 1 2% Prospect for rapid implementation 3 7% Other factors (please specify):______________________________ 2 5% Number of respondents (responses) 26 (45) 100% Reasons for using the instrument. Focus on results was rated as the main reason for using PforR, and support to improvements under government programs and use of government systems were second in importance. Based on your experience with the PforR instrument, how likely is it that you will Number of % use the PforR instrument again in the next two to five years? responses Very unlikely 0 0% Somewhat unlikely 2 8% Somewhat likely 4 15% Very likely 17 65% Don’t know 3 12% Number of respondents 26 100% Based on your experience, which of the following represent what you consider Number of % the greatest strengths of the PforR instrument? (Choose no more than TWO) responses Efficiency in processing the preparation of the operation 2 8% Use of government program systems 10 42% Focus on results 19 79% Focus on program institutional strengthening 7 29% Enhanced World Bank staff support to government in identifying and addressing 4 17% key issues under the Program Usefulness of World Bank assessments in designing the Program 2 8% Other (please specify):______________________________ 0 0% Number of respondents 24 Strengths of PforR. Respondents identified focus on results, use of government program systems, and focus on program institutional strengthening as the top three strengths of PforR. 63 Based on your experience, which of the following represent what you consider Number of % the greatest challenges of the PforR instrument? (Choose no more than TWO) responses The instrument’s novelty and issues related to its newness 18 75% Delays in processing the preparation of the operation 5 21% Exclusions of high-value contracts and activities due to their potential adverse 4 17% impact on the environment and/or affected people Heaviness in conducting World Bank assessments 4 17% Difficulties in applying the World Bank’s PforR Anti-Corruption Guidelines 3 13% Other (please specify):______________________________ 4 17% Number of respondents 24 Challenges. Respondents pointed to issues related to the newness of the instrument as the biggest challenge. Delays in processing the preparation of the operation, the second challenge identified, may also be due to the novelty of PforR. b. The PforR Program Which of the following are the most useful features in helping to define Number of % programs supported by PforR? (Choose no more than TWO) responses Flexibility to define government programs based on desired results that can be 16 70% sectoral, sub-sectoral, or cross-sectoral Ability to work on various levels of government (sub-national) 3 13% Ability to focus on institutional strengthening results 13 57% Ability to include capacity building measures and focus on incentives 8 35% Other (please specify):______________________________ 0 0% Number of respondents 23 Most useful features. The external survey identified flexibility to define government programs based on desired results that can be sectoral, sub-sectoral, or cross-sectoral and ability to focus on institutional strengthening results as the top two most useful features in helping to define programs supported by PforR. If the PforR program does not fully cover the entirety of the government program, Number of % which of the following do you attribute this to? (Choose no more than TWO) responses The exclusion of high-value procurement contracts 8 36% The exclusion of potentially significant and irreversible adverse impacts on the 8 36% environment and/or affected people Clients’ and World Bank staff’s perceptions of the high risks of involving the Bank 5 23% in specific activities The perceived scope of the World Bank’s anti-corruption guidelines 2 9% Other (please specify):______________________________ 7 32% Number of respondents 22 Why Program does not fully cover the government program. The exclusion of high-value procurement contracts and exclusion of potentially significant and irreversible adverse impacts on the environment and/or affected people were the two main reasons that the PforR Program does not cover the entire government program. Other reasons offered were the newness of the instrument and the low amount financed by the Bank. 64 The PforR program relies on the use of country program systems. To what extent Number of % has this been beneficial? responses To a very low degree 0 0% To a somewhat low degree 0 0% To a somewhat high degree 10 43% To a very high degree 10 43% Don’t know 3 13% Total 23 100% Use of program systems. The majority of respondents thought that the use of program systems has been beneficial to a somewhat high or very high degree. To what extent has the PforR instrument helped to identify priority capacity- Number of % building needs under the Program? responses To a very low degree 0 0% To a somewhat low degree 2 9% To a somewhat high degree 15 65% To a very high degree 6 26% Don’t know 0 0% Total 23 100% To what extent has securing financing for Program capacity-building needs Number of % posed a serious challenge? responses To a very low degree 2 9% To a somewhat low degree 10 43% To a somewhat high degree 5 22% To a very high degree 2 9% Don’t know 4 17% Total 23 100% Capacity building. A large majority of respondents thought that the PforR instrument has helped to identify priority capacity-building needs under the Program to a somewhat high or very high degree. Nearly one-third indicated that securing financing for Program capacity-building needs was a serious challenge, while over half thought it posed challenges to a somewhat low or very low degree. c. Results Frameworks and disbursement-linked indicators (DLIs) To what extent has preparation of the Results Framework helped strengthen the Number of % focus on results of the Program? responses To a very low degree 0 0% To a somewhat low degree 1 4% To a somewhat high degree 10 44% To a very high degree 12 52% Don’t know 0 0% Total 23 100% Preparation of Results Framework. Nearly all respondents said that the preparation of the Results Framework strengthens the focus on results to a somewhat high or very high degree. 65 DLIs are the main mechanisms for disbursements in a PforR operation, and they Number of focus on incentives or key results. Based on your experience, to what extent do % responses you believe this approach has been useful? To a very low degree 0 0% To a somewhat low degree 0 0% To a somewhat high degree 5 22% To a very high degree 17 74% I do not have enough information to respond 1 4% Total 23 100% To what extent do the DLIs provide incentives to the government to achieve Number of % desired improvements under the Program? responses To a very low degree 0 0% To a somewhat low degree 3 13% To a somewhat high degree 5 22% To a very high degree 14 61% I do not have enough information to respond 1 4% Total 23 100% Would you say that formulating the DLIs and associated disbursement Number of % mechanisms was …? responses Very difficult 0 0% Somewhat difficult 19 83% Somewhat easy 4 17% Very easy 0 0% Don’t know 0 0% Total 23 100% DLI approach. Nearly all respondents said that the DLI approach had been useful to a somewhat high or very high degree, and a majority think it has provided incentives to the government to achieve desired improvements. However, 83% found it somewhat difficult to formulate these indicators, and only 17% found it somewhat easy. To what extent do you believe the mechanisms for verifying the achievement of Number of % the DLIs can function effectively? responses To a very low degree 0 0% To a somewhat low degree 1 4% To a somewhat high degree 16 70% To a very high degree 4 17% Don’t know 2 9% Total 23 100% Verification mechanisms. Most respondents believed that the mechanisms for verifying the achievement of DLIs can function effectively. 66 d. Program assessments Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program technical assessments. responses Very low level of understanding 0 0% Somewhat low level of understanding 2 9% Somewhat high level of understanding 9 39% Very high level of understanding 11 48% No understanding at all (if answer is e, skip to question 20) 0 0% Not applicable (if answer is f, skip to question 20) 1 4% Total 23 100% Number of How satisfied are you with the quality of the technical assessment? % responses Very unsatisfied 0 0% Somewhat unsatisfied 0 0% Somewhat satisfied 10 48% Very satisfied 9 43% Don’t know 2 9% Total 21 100% Number of To what extent has the technical assessment been useful? % responses To a very low degree 0 0% To a somewhat low degree 1 5% To a somewhat high degree 9 43% To a very high degree 9 43% Don’t know 2 9% Total 21 100% Technical assessment. Respondents indicated a good level of understanding of the purpose and contribution of the technical assessment and of satisfaction with its quality and usefulness. Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program fiduciary systems assessments. responses Very low level of understanding 0 0% Somewhat low level of understanding 1 5% Somewhat high level of understanding 11 50% Very high level of understanding 9 41% No understanding at all (if answer is e, skip to question 23) 0 0% Not applicable (if answer is f, skip to question 23) 1 5% Total 22 100% Number of How satisfied are you with the quality of the fiduciary systems assessment? % responses Very unsatisfied 0 0% Somewhat unsatisfied 1 5% Somewhat satisfied 8 38% Very satisfied 11 52% Don’t know 1 5% Total 21 100% 67 Number of To what extent has the fiduciary systems assessment been useful? % responses To a very low degree 0 0% To a somewhat low degree 2 10% To a somewhat high degree 10 48% To a very high degree 8 38% Don’t know 1 5% Total 21 100% Fiduciary assessment. Respondents indicated a good level of understanding of the purpose and contribution of the fiduciary assessment and of satisfaction with its quality and usefulness. Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program environmental and social systems assessments. responses Very low level of understanding 0 0% Somewhat low level of understanding 3 13% Somewhat high level of understanding 6 27% Very high level of understanding 11 50% No understanding at all (if answer is e, skip to question 27) 1 5% Not applicable (if answer is f, skip to question 27) 1 5% Total 22 100% How satisfied are you with the quality of the environmental and social systems Number of % assessment? responses Very unsatisfied 0 0% Somewhat unsatisfied 1 5% Somewhat satisfied 9 45% Very satisfied 9 45% Don’t know 1 5% Total 20 100% To what extent has the environmental and social systems assessment been Number of % useful? responses To a very low degree 1 5% To a somewhat low degree 0 0% To a somewhat high degree 12 60% To a very high degree 6 30% Don’t know 1 5% Total 20 100% To what extent have the consultations on the draft environmental and social Number of % systems assessment informed the assessment’s findings and conclusions? responses To a very low degree 1 5% To a somewhat low degree 0 0% To a somewhat high degree 13 65% To a very high degree 5 25% Don’t know 1 5% Total 20 100% 68 Environmental and Social Systems Assessment. Of the 22 respondents, most had some understanding of the purpose and contribution of the Environmental and Social Systems Assessment (ESSA) in PforR, but only 77% have a somewhat or very high level of understanding. Most are satisfied with its quality and find it to have been useful to a somewhat or very high degree. In addition, 90% considered the consultations on the draft ESSA informative for the assessment’s findings and conclusions. e. Program risk assessment Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program risk assessments. responses Very low level of understanding 0 0% Somewhat low level of understanding 2 9% Somewhat high level of understanding 13 59% Very high level of understanding 7 32% No understanding at all (if answer is e, skip to question 31) 0 0% Not applicable (if answer is f, skip to question 31) 0 0% Total 22 100% To what extent has the risk assessment identified the major risks that could Number of % prevent the achievement of Program results? responses To a very low degree 0 0% To a somewhat low degree 1 5% To a somewhat high degree 15 68% To a very high degree 5 23% Don’t know 1 5% Total 22 100% Number of To what extent has the risk assessment been useful? % responses To a very low degree 0 0% To a somewhat low degree 1 5% To a somewhat high degree 15 68% To a very high degree 5 23% Don’t know 1 5% Total 22 100% Program risk assessment. Most respondents expressed some level of understanding of the purpose and contribution of the PforR Program risk assessment, and 91% said the assessment had to a somewhat or very high degree identified the major risks that could prevent the achievement of Program results, and that it had been useful. 69 What has been your experience with adhering to the requirements of the World Number of % Bank’s PforR Anticorruption Guidelines? responses Very difficult 0 0% Somewhat difficult 9 43% Somewhat easy 6 29% Very easy 3 14% Don’t know 3 14% Total 21 100% Anticorruption Guidelines. Of those who answered the question, half had a somewhat difficult experience, and half thought it was easy. f. Collaboration To what extent have the teams working on the PforR operations (government, Number of World Bank, relevant development partners, and other stakeholders) worked % responses together effectively in the PREPARATION of the Program? To a very low degree 0 0% To a somewhat low degree 0 0% To a somewhat high degree 9 41% To a very high degree 13 59% Not relevant 0 0% Total 22 100% To what extent are the teams working on the PforR operation (government, Number of World Bank, relevant development partners, and other stakeholders) working % responses together effectively in the IMPLEMENTATION of the Program? To a very low degree 0 0% To a somewhat low degree 0 0% To a somewhat high degree 7 32% To a very high degree 12 54% Not relevant 3 14% Total 22 100% To what extent is the World Bank team working on the PforR operation Number of % responsive to the GOVERNMENT? responses To a very low degree 0 0% To a somewhat low degree 1 5% To a somewhat high degree 7 32% To a very high degree 14 63% Don’t know 0 0% Total 22 100% Collaboration. All 22 respondents said that the teams collaborating on PforR worked together somewhat or very effectively during preparation, and 19 that they did so during implementation. In addition, almost all respondents thought the World Bank team had been very responsive to the government. 70 g. Initial program implementation Has the implementation of the PforR operation with which you are/have been Number of % involved, started? responses Yes 16 73% No (participants responding “no” to skip this section and go to Section VIII) 6 27% Total 22 100% Number of How satisfied are you with the Program implementation? % responses Very unsatisfied 0 0% Somewhat unsatisfied 1 7% Somewhat satisfied 9 60% Very satisfied 3 20% Don’t know 2 13% Total 15 100% To what extent do you believe the progress is on target toward achieving Number of % Program results? responses To a very low degree 0 0% To a somewhat low degree 1 7% To a somewhat high degree 10 67% To a very high degree 2 13% Don’t know 2 13% Total 15 100% Satisfaction with implementation and progress toward results. Of the 15 respondents to these questions, 12 (80%) said that they were somewhat or very satisfied with implementation and that progress toward achieving Program results is on target to a somewhat high or very high degree. To what extent have World Bank task teams been able to provide adequate technical Number of advice to the government (i.e., implementing agreed actions, strengthening Program % responses capacity, and monitoring the evolution of Program risks)? To a very low degree 0 0% To a somewhat low degree 1 7% To a somewhat high degree 9 60% To a very high degree 5 33% Don’t know 0 0% Not applicable 0 0% Total 15 100% Task team support during implementation. Nearly all respondents to this question (14 out of 15) indicated that the World Bank task teams were able to provide appropriate technical advice to the government to a somewhat or very high degree. 71 h. Organization and management To what extent do you believe that the World Bank task team has the right skills Number of % mix to prepare/implement the PforR operation? responses To a very low degree 0 0% To a somewhat low degree 0 0% To a somewhat high degree 9 43% To a very high degree 12 57% Don’t know 0 0% Total 21 100% How satisfied are you with the information/support that the World Bank Number of % provides to help you prepare and implement the PforR operation? responses Very unsatisfied 0 0% Somewhat unsatisfied 0 0% Somewhat satisfied 8 38% Very satisfied 13 62% Don’t know 0 0% Total 21 100% Task Team Skills Support. All respondents to this section thought the World Bank task teams had the right skills mix to prepare and implement the PforR, and were satisfied with the support provided by the Bank to help prepare and implement the operation. i. Final thoughts Considering your experience, how satisfied are you, overall, with the PforR Number of % Program? responses Very unsatisfied 0 0% Somewhat unsatisfied 0 0% Somewhat satisfied 13 62% Very satisfied 8 38% Don’t know 0 0% Total 21 100% Overall satisfaction. Everyone who answered this question (21 respondents) were somewhat or very satisfied with the PforR Program. When you first engaged with the PforR instrument, what were your expectations Number of % of the Program? responses Very pessimistic 0 0% Somewhat pessimistic 0 0% Somewhat optimistic 13 62% Very optimistic 8 38% Don’t know 0 0% Total 21 100% 72 Number of To what extent have your expectations been met for the PforR Program? % responses To a very low degree 0 0% To a somewhat low degree 2 10% To a somewhat high degree 12 60% To a very high degree 6 30% Don’t know 0 0% Total 20 100% Expectations of PforR. All 21 respondents were optimistic about the Program when they first engaged with it, and 90% found their expectations met to a somewhat or very high degree. 2. Internal Long Survey The internal survey had 141 responses, of which 114 were fully completed and 27 partially completed. Overall, the respondents had very positive opinions about the instrument, especially because of its focus on results, and were satisfied with such features as the results framework and DLIs approach, the quality of the assessments, and the support provided to them. They also noted some negative points, such as difficulty in formulating the DLIs and adhering to the requirements of the World Bank’s PforR anticorruption guidelines. The table below shows the profile of the respondents to the internal survey. Function Number Country manager or director 10 Sector specialist 53 Financial management specialist 10 Procurement specialist 11 Environmental/social safeguards specialist 9 Legal specialist 6 Other 12 Preferred not to answer 10 The next table shows the distribution of respondents by operation. The survey was sent to task team members and managers of 30 operations. No responses were received from staff of 4 operations; also, 4 people preferred not to specify in which operation they were involved. Responses were received from 7-12 staff for 11 operations, and from 1-4 staff for another 15 operations. 73 The sections following the table present the results for the multiple-choice questions in the survey. Number of respondents by Country - Operation operation Ethiopia - Health Millennium Goals Program-for-Results - P12353 12 Vietnam - Results-Based Rural Water and Sanitation under the National Target Program Project - P127435 12 Croatia - Health System Quality and Efficiency Improvement - P144871 11 Morocco - National Initiative for Human Development (INDH) Phase II - P116201 10 Moldova - Health Transformation Operation - P144892 9 Tanzania - Urban Local Government Strengthening Program - P118152 8 Brazil - Ceará (MST) PforR - P127463 8 India - Third Maharashtra Rural Water Supply and Sanitation Project - P126325 8 Uganda - Support to Municipal Infrastructure Development - P117876 7 Mexico - Oaxaca WSS Sector Modernization - P145578 7 Pakistan - Punjab Governance and Service Delivery - P132234 7 Ethiopia - Local Government Development Project II - P133592 4 Kenya - National Integrated Safety Net Program - P131305 4 Uruguay - Road Rehabilitation and Maintenance Program - P125803 4 Bangladesh - Enabling Open Government - P129770 4 Nepal - Results-Based Bridges Improvement and Maintenance Project - P125495 4 Mozambique - Public Financial Management for Results Program - P124615 3 Vietnam - Northern Mountains Urban Program - P143596 3 Bolivia - National Road Sector Management Program - P144597 3 Indonesia - Road Asset Preservation Program - P113936 2 Morocco - Urban Transport - P121497 2 Rwanda - Agriculture Growth Corridors - P148927 1 Rwanda - Public Sector Governance Program - P149095 1 Tanzania - Big Results Now in Education - P147486 1 Morocco - Health Sector Support - P148017 1 Morocco - Integrated Risk Management - P144539 1 Kenya - Devolution Support Project - P149129 0 Nigeria - Program to Support Saving One Million Lives - P146583 0 Tunisia - Local Government - P130637 0 India - Enhancing Teacher Education Bihar - P132665 0 Preferred not to answer 4 TOTAL 141 a. Overall experience with PforR instrument Number of How knowledgeable do you consider yourself about the PforR instrument? % responses Not knowledgeable enough to understand the PforR operation 1 1% Somewhat knowledgeable but would need more to understand the PforR 22 16% operation I have enough knowledge to understand the PforR operation 117 84% Number of respondents 140 100% Knowledge about the instrument. The great majority of respondents said they have enough knowledge to understand the PforR operation. 74 What were the key reasons for deciding to use the PforR instrument? (Choose no Number of % more than TWO) responses Support to improvements under government programs 63 46% Use of government systems 60 44% Focus on results 103 76% The fact that this is a new instrument 13 10% Prospect for rapid implementation 8 6% Other factors (please specify):______________________________ 12 9% Number of respondents 136 Reasons for using the instrument. Focus on results was identified as one of the key reasons for deciding to use PforR. Support to improvements under government programs and use of government systems were second in importance. Based on your experience with the PforR instrument, how likely is it that you will Number of % use the PforR instrument again in the next two to five years? responses Very unlikely 9 6% Somewhat unlikely 11 8% Somewhat likely 37 27% Very likely 69 50% Don’t know 12 9% Number of respondents 138 100% Likelihood of using PforR. More than three-quarters of respondents said it is somewhat or very likely they will use the instrument again in the next few years. Based on your experience, which of the following represent what you consider Number of As % of total the greatest strengths of the PforR instrument? (Choose no more than TWO) responses responses Efficiency in processing the preparation of the operation 4 2% Use of government program systems 64 25% Focus on results 105 41% Focus on program institutional strengthening 52 20% Enhanced World Bank staff support to government in identifying and addressing 23 9% key issues under the Program Usefulness of World Bank assessments in designing the Program 3 1% Other (please specify):______________________________ 4 2% Number of respondents (responses) 133 (255) 100% Strengths of PforR. Respondents identified focus on results, use of government program systems, and focus on program institutional strengthening as the top three strengths of PforR. Based on your experience, which of the following represent what you consider Number of % the greatest challenges of the PforR instrument? (Choose no more than TWO) responses The instrument’s novelty and issues related to its newness 64 49% Delays in processing the preparation of the operation 32 24% Exclusions of high-value contracts and activities due to their potential adverse 21 16% impact on the environment and/or affected people Heaviness in conducting World Bank assessments 58 44% Difficulties in applying the World Bank’s PforR Anti-Corruption Guidelines 50 38% Other (please specify):______________________________ 23 18% Number of respondents 131 75 Challenges. Respondents identified issues related to the newness of the instrument, the heaviness in conducting assessments, and difficulties in applying the World Bank’s PforR anticorruption guidelines as the top three challenges. b. The PforR Program Which of the following are the most useful features in helping to define Number of % programs supported by PforR? (Choose no more than TWO) responses Flexibility to define government programs based on desired results that can be 92 71% sectoral, sub-sectoral, or cross-sectoral Ability to work on various levels of government (sub-national) 35 27% Ability to focus on institutional strengthening results 58 45% Ability to include capacity building measures and focus on incentives 46 36% Other (please specify):______________________________ 2 2% Number of respondents 129 Most useful features. Flexibility to define government programs based on desired results that can be sectoral, sub-sectoral, or cross-sectoral were identified as the most useful feature. Ability to focus on institutional strengthening results and ability to include capacity-building measures were also important features of PforR. As % of If the PforR program does not fully cover the entirety of the government program, Number of total which of the following do you attribute this to? (Choose no more than TWO) responses responses The exclusion of high value procurement contracts 33 18% The exclusion of potentially significant and irreversible adverse impacts on the 36 20% environment and/or affected people Clients’ and World Bank staff’s perceptions of the high risks of involving the Bank in 56 31% specific activities The perceived scope of the World Bank’s anti-corruption guidelines 34 19% Other (please specify):______________________________ 24 12% Number of respondents (responses) 117 (183) 100) Why Program does not fully cover the government program. Nearly half of respondents said the Program does not cover the entire government program because of the client’s and World Bank staff’s perceptions of the high risks of involving the Bank in specific activities. The PforR program relies on the use of country program systems. To what extent Number of % has this been beneficial? responses To a very low degree 5 4% To a somewhat low degree 16 12% To a somewhat high degree 49 37% To a very high degree 49 37% Don’t know 12 9% Number of respondents 131 100% Use of program systems. The majority of respondents (74%) thought that the use of program systems has been beneficial to a somewhat or very high degree. 76 To what extent has the PforR instrument helped to identify priority capacity- Number of % building needs under the Program? responses To a very low degree 2 1% To a somewhat low degree 27 21% To a somewhat high degree 43 33% To a very high degree 50 38% Don’t know 9 7% Number of respondents 131 100% To what extent has securing financing for Program capacity building needs posed Number of % a serious challenge? responses To a very low degree 19 15% To a somewhat low degree 36 29% To a somewhat high degree 40 32% To a very high degree 16 13% Don’t know 15 12% Number of respondents 126 100% Capacity building. Most respondents (71%) thought that the PforR instrument has helped to identify priority capacity-building needs under the Program to a somewhat or very high degree, but 45% found it a challenge to secure financing for those capacity-building needs. c. Results Frameworks and disbursement-linked indicators (DLIs) To what extent has preparation of the Results Framework helped strengthen the Number of % focus on results of the Program? responses To a very low degree 1 1% To a somewhat low degree 19 15% To a somewhat high degree 51 40% To a very high degree 44 35% Don’t know 11 9% Number of respondents 126 100% Preparation of Results Framework. Three-quarters of respondents thought that the preparation of the Results Framework strengthens the focus on results to a somewhat or very high degree. DLIs are the main mechanisms for disbursements in a PforR operation, and they Number of focus on incentives or key results. Based on your experience, to what extent do % responses you believe this approach has been useful? To a very low degree 1 1% To a somewhat low degree 6 5% To a somewhat high degree 34 27% To a very high degree 67 54% I do not have enough information to respond 16 13% Number of respondents 124 100% 77 To what extent do the DLIs provide incentives to the government to achieve Number of % desired improvements under the Program? responses To a very low degree 3 2% To a somewhat low degree 10 8% To a somewhat high degree 39 32% To a very high degree 51 42% I do not have enough information to respond 19 16% Number of respondents 122 100% Would you say that formulating the DLIs and associated disbursement Number of % mechanisms was …? responses Very difficult 23 18% Somewhat difficult 72 58% Somewhat easy 20 16% Very easy 0 0% Don’t know 10 8% Number of respondents 125 100% DLI approach. Most respondents said that the DLI approach has been useful to a somewhat high or very high degree (81%) and that it has provided incentives to the government to achieve desired improvements (74%). However, 76% found it somewhat or very difficult to formulate DLIs. To what extent do you believe the mechanisms for verifying the achievement of Number of % the DLIs can function effectively? responses To a very low degree 2 2% To a somewhat low degree 17 14% To a somewhat high degree 56 44% To a very high degree 26 21% Don’t know 23 19% Number of respondents 124 100% Verification mechanisms. Two-thirds of the respondents said that the mechanisms for verifying the achievement of DLIs can function effectively. d. Program assessments Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program technical assessments. responses Very low level of understanding 0 0% Somewhat low level of understanding 13 10% Somewhat high level of understanding 47 38% Very high level of understanding 62 49% No understanding at all (if answer is e, skip to question 20) 1 1% Not applicable (if answer is f, skip to question 20) 3 2% Number of respondents 126 100% 78 Number of How satisfied are you with the quality of the technical assessment? % responses Very unsatisfied 0 0% Somewhat unsatisfied 10 9% Somewhat satisfied 50 43% Very satisfied 39 34% Don’t know 16 14% Number of respondents 115 100% Number of To what extent has the technical assessment been useful? % responses To a very low degree 2 2% To a somewhat low degree 26 22% To a somewhat high degree 37 31% To a very high degree 36 31% Don’t know 17 14% Number of respondents 118 100% Technical assessment. Of the respondents with some level of understanding about the technical assessment, 77% were satisfied with its quality, but only 62% though it had been useful to a somewhat or very high degree. Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program fiduciary systems assessments. responses Very low level of understanding 1 1% Somewhat low level of understanding 14 11% Somewhat high level of understanding 41 34% Very high level of understanding 58 48% No understanding at all (if answer is e, skip to question 23) 0 0% Not applicable (if answer is f, skip to question 23) 8 6% Number of respondents 122 100% Number of How satisfied are you with the quality of the fiduciary systems assessment? % responses Very unsatisfied 3 3% Somewhat unsatisfied 16 14% Somewhat satisfied 50 43% Very satisfied 34 30% Don’t know 12 10% Number of respondents 115 100% Number of To what extent has the fiduciary systems assessment been useful? % responses To a very low degree 4 3% To a somewhat low degree 22 19% To a somewhat high degree 51 44% To a very high degree 29 25% Don’t know 10 9% Number of respondents 116 100% 79 Fiduciary assessment. Of the respondents with some level of understanding about the fiduciary assessment, 73% said that they were satisfied with its quality, and 69% said it had been useful to a somewhat high or very high degree. Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program environmental and social systems assessments. responses Very low level of understanding 1 1% Somewhat low level of understanding 14 12% Somewhat high level of understanding 50 41% Very high level of understanding 43 36% No understanding at all (if answer is e, skip to question 27) 4 3% Not applicable (if answer is f, skip to question 27) 9 7% Number of respondents 121 100% How satisfied are you with the quality of the environmental and social systems Number of % assessment? responses Very unsatisfied 2 2% Somewhat unsatisfied 14 13% Somewhat satisfied 48 45% Very satisfied 26 24% Don’t know 17 16% Number of respondents 107 100% To what extent has the environmental and social systems assessment been Number of % useful? responses To a very low degree 5 5% To a somewhat low degree 20 18% To a somewhat high degree 49 45% To a very high degree 19 17% Don’t know 16 15% Number of respondents 109 100% To what extent have the consultations on the draft environmental and social Number of % systems assessment informed the assessment’s findings and conclusions? responses To a very low degree 5 5% To a somewhat low degree 24 22% To a somewhat high degree 39 36% To a very high degree 13 12% Don’t know 27 25% Number of respondents 108 100% Environmental and Social Systems Assessment. Of the respondents who were aware of the purpose and contribution of the ESSA in PforR, 69% were satisfied with its quality and 62% thought it had been useful to a somewhat or very high degree. Nearly half (48%) said that the consultations on the draft ESSA informed the assessment’s findings and conclusions, but 25% didn’t know how to answer. 80 e. Program risk assessment Please rate the level of your understanding of the purpose and contribution of Number of % the PforR Program risk assessments. responses Very low level of understanding 1 1% Somewhat low level of understanding 11 9% Somewhat high level of understanding 54 45% Very high level of understanding 54 45% No understanding at all (if answer is e, skip to question 31) 0 0% Not applicable (if answer is f, skip to question 31 1 1% Number of respondents 121 100% To what extent has the risk assessment identified the major risks that could Number of % prevent the achievement of Program results? responses To a very low degree 2 2% To a somewhat low degree 21 18% To a somewhat high degree 63 52% To a very high degree 24 20% Don’t know 10 8% Number of respondents 120 100% Number of To what extent has the risk assessment been useful? % responses To a very low degree 4 3% To a somewhat low degree 28 24% To a somewhat high degree 54 46% To a very high degree 22 18% Don’t know 11 9% Number of respondents 119 100% Program risk assessment. Nearly all participants have some level of understanding of the purpose and contribution of the PforR Program risk assessment, and 90% had a somewhat or very high level of understanding. Most thought it had, to a somewhat or very high degree, identified the major risks that could prevent the achievement of Program results (73%) and that it had been useful (63%). What has been your experience with adhering to the requirements of the World Number of % Bank’s PforR Anticorruption Guidelines? responses Very difficult 25 21% Somewhat difficult 45 37% Somewhat easy 29 24% Very easy 3 3% Don’t know 18 15% Number of respondents 120 100% Anticorruption Guidelines. Of those who answered the question about their experience with adhering to the requirements of the World Bank’s PforR Anticorruption Guidelines, 59% said it was somewhat or very difficult and 27% said it was somewhat or very easy. 81 f. Collaboration To what extent have the teams working on the PforR operations (government, Number of World Bank, relevant development partners, and other stakeholders) worked % responses together effectively in the PREPARATION of the Program? To a very low degree 5 4% To a somewhat low degree 19 16% To a somewhat high degree 45 36% To a very high degree 47 39% Not relevant 6 5% Number of respondents 122 100% Preparation. Most respondents (76%) thought that the various teams had worked together somewhat or very effectively during the preparation of the Program. To what extent are the teams working on the PforR operation (government, Number of World Bank, relevant development partners, and other stakeholders) working % responses together effectively in the IMPLEMENTATION of the Program? To a very low degree 3 3% To a somewhat low degree 9 8% To a somewhat high degree 38 32% To a very high degree 23 19% Not relevant 45 38% Number of respondents 118 100% Implementation. Only half of respondents said that all the stakeholders in their PforR operation were working effectively together, but 38% said the question was not relevant, because they had not started implementation yet. g. Initial program implementation Has the implementation of the PforR operation with which you are/have been Number of % involved, started? responses Yes 71 57% No (participants responding “no” to skip this section and go to Section VIII) 53 43% Number of respondents 124 100% Number of How satisfied are you with the Program implementation? % responses Very unsatisfied 1 1% Somewhat unsatisfied 8 12% Somewhat satisfied 27 39% Very satisfied 16 23% Don’t know 17 25% Number of respondents 69 100% Satisfaction with implementation. Only respondents involved in operations that were under implementation answered this section. Of the total of 71 respondents, 62% were somewhat or very satisfied with the Program’s implementation. 82 To what extent do you believe the progress is on target toward achieving Number of % Program results? responses To a very low degree 1 1% To a somewhat low degree 9 13% To a somewhat high degree 21 31% To a very high degree 17 25% Don’t know 20 30% Number of respondents 68 100% Progress toward results. Of those who answered this question, 56% said that progress toward achieving Program results is on target to a somewhat or very high degree, while 29% didn’t know how to answer. To what extent have World Bank task teams been able to provide adequate technical advice to the government (i.e. implementing agreed actions, Number of % strengthening Program capacity, and monitoring the evolution of Program responses risks)? To a very low degree 0 0% To a somewhat low degree 8 12% To a somewhat high degree 23 34% To a very high degree 25 36% Don’t know 12 17% Not applicable 1 1% Number of respondents 69 100% Task team support during implementation. Most respondents (69%) said that teams had been able to provide adequate technical advice to the government to a somewhat or very high degree. h. Organization and management To what extent do you believe that the World Bank task team has the right skills Number of % mix to prepare/implement the PforR operation? responses To a very low degree 3 2% To a somewhat low degree 19 16% To a somewhat high degree 52 43% To a very high degree 41 34% Don’t know 6 5% Number of respondents 121 100% Task team skills. Three-quarters (77%) of respondents said that task teams had the right skills mix to prepare and implement the PforR operation to a somewhat or very high degree. How satisfied are you with the information/support that the World Bank Number of % provides to help you prepare and implement the PforR operation? responses Very unsatisfied 2 2% Somewhat unsatisfied 20 16% Somewhat satisfied 50 41% Very satisfied 36 30% Don’t know 13 11% Number of respondents 121 100% 83 How satisfied are you with the information/support that OPCS provides to help Number of % you prepare and implement the PforR operation? responses Very unsatisfied 7 6% Somewhat unsatisfied 13 11% Somewhat satisfied 44 36% Very satisfied 46 38% Don’t know 11 9% Number of respondents 121 100% How satisfied are you with the information/support that OTHER CORPORATE Number of UNITS (including Legal and Controller’s) provide to help you prepare and % responses implement the PforR operation? Very unsatisfied 3 3% Somewhat unsatisfied 15 13% Somewhat satisfied 42 34% Very satisfied 40 33% Don’t know 20 17% Number of respondents 120 100% Support. Seventy-one percent of respondents were satisfied with the support provided by the Bank to help prepare and implement the operation, 74% with the support from OPCS, and 68% with the support provided by other corporate units (including Legal and Controllers). How effective has the World Bank’s management been in promoting the PforR Number of % instrument? responses Very ineffective 3 3% Somewhat ineffective 20 17% Somewhat effective 59 49% Very effective 31 25% Don’t know 7 6% Number of respondents 120 100% Management promotion of PforR. A majority of respondents (75%) said that the World Bank’s management has been promoting the PforR instrument effectively. i. Final thoughts Considering your experience, how satisfied are you, overall, with the PforR Number of % Program? responses Very unsatisfied 2 2% Somewhat unsatisfied 20 17% Somewhat satisfied 54 45% Very satisfied 40 33% Don’t know 3 3% Number of respondents 119 100% Overall satisfaction. Most respondents (79%) said that, overall, they are somewhat or very satisfied with the PforR Program. 84 When you first engaged with the PforR instrument, what were your expectations Number of % of the Program? responses Very pessimistic 1 1% Somewhat pessimistic 19 16% Somewhat optimistic 58 49% Very optimistic 39 32% Don’t know 2 2% Number of respondents 119 100% Number of To what extent have your expectations been met for the PforR Program? % responses To a very low degree 5 4% To a somewhat low degree 29 24% To a somewhat high degree 46 39% To a very high degree 29 24% Don’t know 10 8% Number of respondents 119 100% Expectations with PforR. When respondents first engaged with the PforR instrument, 82% were optimistic about the Program; 63% said their expectations were met to a somewhat or very high degree. 3. Alternative Analysis of Internal Long Survey a. By IDA and IBRD For the internal long survey there were 49 respondents from 10 IBRD operations and 88 from 16 IDA operations. A comparison of the disaggregated results of all the multiple-choice questions shows that the distribution of most of the answers was very similar among these two groups. One question for which there was a significant difference in the answers was the one about the Response IBRD IDA likelihood of using the instrument in the next two Very unlikely/Somewhat unlikely 21% 9% to five years. On the whole, IDA operations Very likely /Somewhat likely 64% 86% respondents considered it more likely that they Don’t know 15% 5% would use PforR in the next few years. Also, IDA operations respondents seem to have Response IBRD IDA found it more difficult than IBRD operations Very difficult/ Somewhat difficult 67% 82% respondents to formulate the DLIs. Very easy/Somewhat easy 20% 13% Don’t know 13% 5% Similarly, a higher percentage of IDA operations Response IBRD IDA respondents thought Management had been Very ineffective/ Somewhat 26% 16% effective in promoting the instrument. ineffective Very effective/Somewhat effective 67% 81% Don’t know 7% 4% 85 b. By Region The results were also broken down by Region to identify Regional differences in experience with PforR. For this analysis, the average of the answers provided by Region was calculated for each question. The table below shows the distribution of the survey respondents and operations by Region. Africa Region had the highest number of respondents, since it had the highest number of operations in the PforR portfolio and pipeline when the survey was launched. Region AFR EAP ECA LCR MNA SAR Number of respondents 41 17 20 22 14 23 Number of operations 9 3 2 4 4 4 In general, the data confirmed that in all six Regions, staff are optimistic about PforR. SAR gave more positive answers than staff in the other Regions. Staff in ECA and LCR gave lower average answers. 86 ANNEX 3. REPORT ON QUALITATIVE DATA Section 1: Introduction 1. The purpose of conducting in-depth, structured, face-to-face interviews as part of the PforR Two-Year Review process was to receive feedback about the instrument through direct interaction with clients and Bank staff. The interviews also sought suggestions to help improve implementation of this new lending instrument. 2. The open-ended nature of qualitative interviews allows respondents to provide richer and more nuanced accounts than the responses provided on structured surveys. This form of data collection also allows interviewers to probe previously unexplored responses and provides a way for respondents to explain their thoughts and perspectives. 3. This report is structured as follows: Section 2 provides details on the number of interviews and the countries covered, and Section 3 explains how interview materials were developed, interview teams set up, and data collected and analyzed. Section 4 gives an overview of the focus areas covered in the interview guides. Finally, Section 5 presents an overview of findings. Section 2: Coverage 4. External Respondents. External interviews have been Table 1. External Interviews: carried out in 10 countries. All respondents were senior officials Countries Covered and Number of in the key ministries involved in the PforR operation, usually Interviews in Each Country the Ministry of Finance and the relevant sectoral ministries. Where possible, interviews were also conducted with Number of Country interviews development partners. Table 1 shows the countries and the number of interviews conducted in each country. Bolivia 5 Croatia 4 5. Internal Respondents. A total of 11 interviews were Ethiopia 5 conducted with senior World Bank managers (see Table 2). India 3 Kenya 5 Section 3: Methodology Morocco 4 Pakistan 4 6. The qualitative data collection and analysis were Tanzania 4 carried out as follows. Tunisia 2 a. External Interviews Vietnam 4 7. Preparation of Interview Guide and protocol. An Table 2. Internal Interviews: Interview Guide and protocol were prepared with input on key Positions held by Internal World questions from various members of the OPCS team working Bank Respondents on the PforR Two-Year Review. The guide was shared with Number of Position several World Bank colleagues, including IEG’s Country, interviews Corporate and Global Evaluations Department (IEGCC) and Vice-President 6 Public Sector Evaluation Department (IEGPS), as well as the Senior Director 3 Independent External Panel of Experts. The questions were Director 2 further refined to reflect these colleagues’ comments. TOTAL 11 87 8. Selection and recruitment of respondents. The OPCS team in Washington, DC, worked with country staff to select key respondents from across the different relevant ministries, based on their familiarity with and knowledge of their country PforR operations and on their availability and convenience. An information sheet describing the purpose of the interviews and the Interview Guide were provided to respondents ahead of time. 9. Interview team. The interview team comprised at a minimum a primary interviewer and a primary note-taker. Depending on country context, the interviewer either had relevant language proficiency or hired the services of a translator. Members of the Independent External Expert Panel overseeing the Two-Year Review were also informed about the plans and invited to participate. Panel members participated in the interviews in two countries; participation was planned in a third country but did not materialize because of delays in visa processes. 10. Interview data. The primary note-taker was responsible for writing up the notes; the primary interviewer provided input on accuracy and missing information. 11. Interview data analysis. For the data on external respondents, the analyst first created separate folders for each of the thematic areas. Within each thematic area, subfolders were created for each country. The relevant sections from interviews from each country were placed in the relevant folder and subfolder. The interview data were then analyzed for common themes and patterns that emerged across all interviews on a given theme in each country. Once each country’s interview data were analyzed by theme, an overview section captured the patterns that emerged across countries. b. Internal Interviews 12. All internal respondents were senior Bank managers, selected because of their knowledge of, and experience with, the PforR across regions and sectors. Interviews were conducted face-to- face in Washington, DC, or via Skype/telephone. All interviews were conducted by a single interviewer. The interview notes were analyzed by the analyst for common themes and patterns that emerged across all interviews. Section 4: Scope 13. The interviews with external government respondents covered the following five focus areas: a) Awareness and knowledge of the PforR instrument. This section asked respondents about their current levels of awareness and knowledge, how information was provided to them, and suggestions for improvement. b) Choice of instrument. Questions in this section focused on the decision-making process in choosing the PforR instrument; any early concerns about using PforR; the reasons respondents found PforR to be a favorable choice; and suggestions to improve the process of selection. c) PforR preparation. Respondents were asked about aspects of the preparatory process that had gone well and aspects that had been challenging, and about suggestions for improvement. 88 d) PforR implementation. Respondents with implementation experience were asked to reflect on aspects of the implementation that have gone well and aspects that have been challenging, and to offer suggestions for improvement. e) Collaborations and partnerships. This section asked respondents whether there had been a change in the interaction between the World Bank and government counterparts, other stakeholders, and development partners. The interviews with senior Bank managers asked respondents to reflect on the following: 1. Importance of PforR 2. Experience with PforR 3. Role of respondent in promoting PforR 4. Institutional support for PforR 5. Ongoing challenges and suggestions for improvement Section 5: Findings a. Findings of Interviews with Senior Government Officials 1. Levels of Awareness 14. Overall, government respondents’ level of awareness and understanding of the PforR instrument was high. They attributed their good knowledge of the new instrument to the following factors: a strong working relationship with the World Bank team, the PforR being a good fit for their country’s overall development strategy of focusing on results, prior experience with similar instruments, and the series of meetings and workshops organized to disseminate information on the PforR (see illustrative quotes below). Interviews also revealed that in some cases, there may be an initial period when awareness is still developing; but over time, levels of understanding increase as familiarity with the instrument grows. On strong working relationships with the WB team: “The government teams have considerable experience with the Bank. The relationships were strong to begin with and the government has confidence that the Bank staff were suggesting an instrument that was relevant and would work to the benefit of the programs involved… Overall level of knowledge of PforR is good. The Ministry has been working with the Bank since 2008 on an earlier project which followed the Bank’s investment project lending procedures. When the new project was being prepared there was a lot of discussion with the Bank task team on the new PforR instrument including sharing of how it had been applied in the other similar government operations.” On a focus on results aligning with the government’s approach: “The strategy is to move from a very centralized state to a decentralized model. We heard about PforR two years ago and thought that this was just what we wanted for our operation. It fits with our results focus and helps with understanding the PforR mechanisms.” 89 On previous experience with similar types of instruments: “We were pleased when the World Bank eventually adopted a results-based approach. During the preparation of the PforR we came to understand the specific elements of the instrument very well, including issues such as the exclusions of high-value procurement contracts because of our earlier experience with a similar instrument.” On continuous workshops and meetings to explain the PforR: “At the beginning, [the sector ministry] did not understand the operation mechanism of the PforR. 3-4 workshops were organized with participation of the sector ministry, participating provinces and other related stakeholders to introduce PforR. With big support of WB TT and through a lot of meetings and references to the first PforR, we step by step understood the operation mechanism of PforR. We often told WB TT that if we do not fully understand PforR, we will not approve this program….By the end of the year, leader and key persons of participating provinces basically understood the operation mechanism of PforR.” “Many officials now understand PforR very well. PforR is now very famous.” Evolving understanding of the new instrument 15. Some respondents shed light on how an understanding evolved over time, as initial confusion gave way to more clarity and a better appreciation of the advantages of the instrument. “Initially [when PforR was proposed], we were not sure especially as it was a new instrument but decided to give it a try. We have a good understanding of the PforR instrument now but initially the information on PforR was not as well organized despite several Bank missions. There are definitely advantages to disburse against achieved results or percentage of results for scalable DLIs. In the past, there was some confusion on what procedures, Bank vs Country, to use but not anymore…. This [PforR] is better than IPF.” Suggestions 16. Several suggestions were offered to enhance awareness levels: when introducing the PforR into a new country, it should be used on a small project as a test case; more materials should be provided on the pros and cons of different instruments; more case studies/good practices on ongoing and completed operations should be developed; presentations on PforR should be tailored to the country context; explanation should be provided regarding the value-added of the PforR compared to other instruments, both World Bank instruments and those of other development partners; and need for improved information-sharing within the country to ministries and stakeholders that are not currently involved in a PforR operation. 2. Choice of Instrument 17. Respondents from all countries agreed that the PforR was the appropriate choice for their respective operations, even when the initial push was more from the Bank’s side and the country required convincing before both sides reached agreement on using the instrument. 90 18. Respondents highlighted various features of PforR that they found favorable. Contrasting the PforR operation with preceding operations that used other funding mechanisms, interviewees noted the following advantages: A focus on results “Our current plan focuses on results and so using a PforR instrument integrates the Bank financing into the government’s systems. The Bank team discussed the new PforR instrument with us even before the PforR instrument had been approved and it seemed like an ideal fit for our results based approach.” Use of government systems, institutional strengthening, and sustainability “The previous Bank project has been very successful in supporting the development of local government. However, there was a drawback that it did not use government systems, which caused some issues and inefficiencies. Bank systems and processes were imposed and when the project closed there was no continuity with the continuing government operations…The PforR approach shifts responsibility and accountability from the Bank to the government and strengthens institutional performance and sustainability.” Decentralization One respondent noted that the country’s PforR operation had given a more active role to participating provinces, allowing local governments to prioritize their infrastructure investments for five years in a planned manner. An official from another country highlighted the advantage of motivating agencies to work in their own specialized division rather than burdening one implementing agency. For example, in this country’s PforR operation, one agency is implementing two DLIs and another is in charge of one DLI. This has also resulted in developing more strategic and operation-level interaction between the government’s and the Bank’s representatives once the program is signed off, instead of focusing on getting approvals at each step from the Bank. PforR gives countries more independence. “We think it is a good tool for the “new” Bank, especially for countries that are advancing. IPF preparation and implementation is very focused on details with the Bank alongside us all the time. But once you have more capacity, you should have more autonomy and also be able to take on larger loans. PforR brings institutional development support that the DPL does not. This is very important to us… It gives the country the flexibility to implement using its own regulations.” 19. Respondents raised concerns about the predictability of funding streams, resistance to the Bank’s requested changes, the requirement for a well-defined government program, and the difficulty of applying PforR to cross-sectoral activities when there is no coordinating body. 91 Suggestions 20. Suggestions were offered to improve the process of choosing the PforR: presentations and examples of good practice; more clarity on how PforR is different from other lending instruments; and more thought to how issues are framed and worded. 3. PforR Preparation 21. Overall, respondents across all countries were positive about the preparation process. With regard to specific aspects of the preparatory process, some respondents found the assessments useful, while others did not. Assessments “The Fiduciary Systems Assessment and the Environmental and Social Systems Assessment were very useful in identifying areas for strengthening technical areas and sharing best practice from elsewhere…and compiling the rules and procedures on procurement, including for selection of consultant services, into a comprehensive procurement manual.” “We found that the Fiduciary Systems Assessment was unnecessarily burdensome and duplicative. As part of another project, the participating Development Partners commissioned a comprehensive fiduciary risk assessment…we expected that the PforR would be based on the already completed FSA and would update where necessary. We were very disappointed that the World Bank insisted on carrying out a full Fiduciary Systems Assessment from a zero base without using the findings of the FSA report. The PforR Fiduciary Systems Assessment took a long time to prepare.” DLIs 22. A number of respondents found the process of identifying and deciding on appropriate DLIs to be challenging. At the same time, there was also broad agreement that the process of selecting DLIs was useful and necessary, even if the discussions took time and the negotiations became tense at times. The involvement of implementing agencies (including local governments where appropriate) in crafting relevant DLIs was critical, and sometimes led to revising targets when indicators were deemed to be unrealistic. Several respondents spoke highly of how Bank staff led them through the difficult process of selecting indicators that were reasonable and achievable. “During preparation, we discussed thoroughly with WB TT and participating provinces. If there are not such thorough discussions, there would be difficulties during program implementation. The preparation of DLIs was the most difficult process for us. However, we also learnt a lot from the preparation of DLIs. In order to better understand between WB and the ministry, many meetings including working lunches were organized. Some meetings were very tense and straight. However, we were really satisfied with those meetings.” 92 Advantages of in-country staff 23. Respondents in two countries mentioned the challenges of working in contexts where key WB staff are not based in the country, while a respondent from a third country pointed to the advantage of the reverse situation: having strengthened country offices. “The worst thing you can do is have a DC-based TTL and try to move a project forward. In our [development partner’s] agency, all the heads of projects are in the country.” 24. A similar view was shared by another respondent: Interviewer: Any worries on reaching targets for DLIs? Respondent: Not really, we are trying to work together. Before the Bank’s strengthened presence in the country, we used to be scared when a Bank mission was announced. Now it is like working with ourselves. Interviewer: What percentage of the Bank’s missions are from country offices? Respondent: 90% or more. They are self-sufficient now. Length of preparation time 25. There was a variation in views across countries about the length of time it took to prepare the PforR operation; some had experienced delays and others were pleased with the fast pace. Anticorruption Guidelines 26. There were varying views across countries about the Bank’s ACGs. In two countries, respondents’ overall past experience with the Bank had been positive and constructive; as a result, this was not an area that anyone in the government focused on. However, in another context, respondents noted that they had expected that under PforR the country’s regulations on fraud and corruption would be followed. Even though the World Bank team and government counterparts worked hard to resolve the issue, concerns remain about problems that will arise during implementation, and how they will be addressed. Newness of instrument 27. Some respondents focused on the overall newness of the instrument as a challenge and on whether government officials, including those at the subnational level, had received enough outreach about, and developed awareness of, the instrument. Other respondents made more global comments – the challenges of changing mind-sets and internal procedures, and the need for continued training and awareness-raising. 28. Certain concerns were raised in one country: need to involve all relevant agencies from the start in the preparatory process, reconsider procurement exclusions, lack of clarity on the use of prior results, and duplicating the verification process of results. 93 Suggestions 29. Suggestions offered by respondents: provide a guide of practical information on how implementation agencies can best participate in the PforR operation (including local governments, where appropriate); institute small follow-up projects to ensure sustainability; increase capacity building during implementation, including at the local levels, where appropriate; allocate more time to identify and decide on DLIs; use simpler language in the PADs; provide more communication from the Bank on lessons learned from other PforR operations; in a changing environment, build in flexibility to change or modify some of the anticipated results; involve all departments and agencies that may be engaged in PforR operations at an early stage; ensure that the Ministry of Finance is closely involved during the preparation of PforR operations so that by the time of negotiations, there is an understanding of budget allocations, procedures, and resources to achieve DLIs; and finally, take into account country context and political climate when talking about institutional reform and offer a range of alternatives rather than a prescribed “best practice.” 4. PforR Implementation 30. Because several PforR operations were in early stages of implementation, respondents did not have many comments. Bank-Government partnership for pragmatic solutions to implementation issues 31. Those who provided feedback spoke about seeking pragmatic solutions to implementation issues, noting that challenges can be met if there is a good working relationship with the World Bank team. PforR implementation aids decentralization 32. Another aspect of implementation experience mentioned in the interviews was decentralization; respondents from a number of countries remarked that their PforR operations had aided in giving more autonomy to participating provinces and local governments, thereby improving the chances of sustainable outcomes. “The overall systems are improving because of the PforR project. Local governments now like the complaints-handling mechanism; also the central government is thinking about more decentralization of the Environmental Impact assessments as they cannot review all the assessments coming to them at the central level.” 33. One respondent said that the implementation experience had clarified the type of program that the PforR is suited for: when aligned with an existing program and for programs that have reasonable yet long-term targets. 34. There was some worry about inflexible requirements and reverting to an “IPF way of thinking.” 94 Suggestions 35. Suggestions made: having fewer indicators; building capacity in operations from the very beginning; more clarity on specifics—for instance, the types of indicators that can be used, whether it is possible to change indicators, and whether additional financing is an option; more support is required from the Bank for the implementation stage as this was the first PforR operation in the country and would be used as an example for other sectors; the creation of steering committees to coordinate the work of the different agencies; the ability to revise policies and definitions to reflect the local situation; organizing a dissemination meeting to share lessons learned; reducing and simplifying the reporting requirements; training staff in reporting skills; and finally, looking forward and planning for financial resources in the coming years, including, where appropriate, for participating provinces, which need to mobilize a lot of resources for program implementation. 5. Collaborations and Partnerships Partnerships between the World Bank and government counterparts 36. Government officials from some countries said that excellent prior working relationships with the World Bank had extended into collaborative partnerships on the PforR operations. However, the nature of the dialogue had changed, and respondents cited less frustration because of not having to use separate Bank processes, and engaging in discussions on results and DLIs rather than on expenditures and administrative processes. Respondents with implementation experience noted that without good relationships, the implementation would have proceeded poorly; collaborative engagement with the Bank team helped the government teams meet implementation challenges. Finally, a respondent noted that the PforR had not only changed the relationship between the Bank and the government but also within government ministries, as the implementation requires close coordination among several ministries. Relationships with development partners 37. There was widespread interest in greater collaborative partnerships and increased harmonization between the Bank and other development partners, and an agreement that the PforR is particularly suited for such fruitful partnerships. “Certainly [PforR] can help with harmonization. It helps by setting precedent for future operations because in a way they can certify operations. The joint evaluation of agencies can help save time, too.” 38. However, the issue of donor engagement is more complex. In one country, respondents pointed out that it is not always easy to harmonize donors because the government prefers working with a partner whose rules on safeguards and so on are simpler than those of development agencies such as the World Bank. In another country, the government was reluctant to bring in too many agencies because of problems with such initiatives in the past. Respondents in a third country expressed interest in collaborating with other partners, as long as the World Bank was in charge. 39. Representatives from various development partners interviewed across different country contexts expressed interest in working more closely with the Bank. For example, a respondent from a partner agency said that with the PforR, there is a better base for cooperation with the World 95 Bank because of the similarities of instruments and approaches. PforR is more suitable for collaboration than IPF. Another respondent provided the following feedback regarding the preparation of a PforR operation: “Our [development partner’s] team was very involved in the design and preparation of the operation. The team participated in the pre-appraisal of the operation which included a three day workshop that went over the operation as well as the PforR instrument. We received good information and cooperation from the Bank team. There was also coordination and discussions of the development of the DLIs which were done jointly. We are now conducting joint review missions and we see progress against two DLIs and there is learning by doing.” b. Findings of Interviews with Senior WB Officials 6. Importance of PforR 40. Senior Bank managers considered the PforR to be an important instrument, remarking that it “held a lot of promise”, was “a crucial step forward for the Bank” and that they were “excited about its potential.” One stated that it was “absolutely critical.” Another remarked that the PforR is “the perfect middle ground between the policy and project approaches to Bank operations”. One respondent noted that in spite of the high expectations, very few operations in the respondent’s sector had actually used the instrument. 41. Speaking more concretely of the advantages of the PforR, respondents said that the new instrument tackled the challenges posed by the IPF and helped strengthen local institutions. Some noted that for IBRD countries in particular, the PforR has improved the value proposition of the Bank overall. “PforR is an ideal instrument for tackling challenges facing the IPF instrument. IPF takes time away from focusing on institutional strengthening because you have to focus on the Bank’s procedural matters.” “For IBRD countries – we’re raising prices and our transaction costs are likely to increase, while quality will probably stay the same – so we are not improving the value proposition of the Bank. We need new instruments like the PforR.” 42. Two respondents said that the PforR was an important instrument but one of several types of mechanisms through which a results objective could be met. “The Bank is now much better focused on a higher-level objective of results. The PforR is one expression of this. Donors and governments are also much more focused on results than ever before, with a recognition that a focus on inputs is insufficient. The PforR is extremely important, but only one way of getting at a results objective. In fact, I would argue that a results focus is getting traction throughout our operations, including technical assistance operations as well as analytical and advisory activities.” 7. Experience with PforR 96 43. Respondents spoke about their generally good experience with PforR to date and of the promise of the instrument and the need to fulfill that promise as operations move into implementation. “[The type of program we have launched]…cannot do these kinds of operations without PforR.” “There’s lots of promise with PforR. And lots of evolution in the government’s thinking with respect to their own programs.” 44. Respondents spoke about PforR operations in four countries where initial hurdles were overcome, resulting in overall satisfaction, among government counterparts and within the Bank, with the current state of affairs. “[There were] no PforRs initially. Nobody wanted to be the guinea pig. There was concern about the burden of assessments. And fiduciary staff felt under the gun… and now we have got a lot of kudos from the PforRs that we have now done…” “Everything has teething problems, things that are not anticipated, such as dealing with Bank culture. The going was very rough. ACGs were a huge problem at the time and there was a big battle between OPCS and INT. But now [government officials] love the PforR, although we haven’t seen any test yet with regards to the role of INT.” “…PforR was a natural successor to results-based IPF. It was lot of work but an obvious thing to do. We struggled with DLIs and with ACGs. But once we got over the hurdles, everyone really liked it.” “The transactions costs for the first PforR were huge, but the second PforR was a breeze, including on ACGs. The second was prepared quickly and with a good policy dialogue. We can learn some interesting stuff now, but the really interesting stuff will be learned in 2-3 years when we see results, change in behaviors and whether we can expand programs nationwide. But there’s lots of promise with the PforR. And lots of evolution in thinking in the country with respect to their own programs.” 45. Respondents provided insight into two country contexts where they would like to do more with respect to the future use of PforR. “We’re doing bits and pieces with respect to PforR because there is also concern about INT. We would like to leverage government programs much more.” “Want to do a PforR. But there are challenges in getting counterparts to think out of IPF mode.” 46. Respondents referred to three countries that decided not to use the PforR, for the following reasons: “…government decided to stick with IPF because they felt they were not ready to use their own country systems.” 97 “[It was] suggested that the team think about PforR. But the task team wanted IPF with DLIs because they thought it would take too long to develop the PforR.” “INT was a big problem. It was the kiss of death.” 47. A few respondents noted that their direct experience with specific PforR operations was limited. 8. Role of Respondent in Promoting PforR 48. Respondents had different views of their own role vis-à-vis the instrument: some actively promote it while others would prefer to see it spread more organically. One respondent spoke of the importance of protecting the new instrument from too much bureaucracy. Yet another stated that after the Global Practices are properly organized, more attention should be paid to the different results-based options. Respondents also recognized that the time might not be right for PforR in all countries. “I’m trying to push the PforR instrument. When teams come with ideas for new lending, I push for the PforR.” “Personally I play a very active role in promoting PforR - both internally and within countries.” “I intend to let PforR spread organically – by sector and by country. Can’t go on a crusade for PforR because we know there are potential problems out there. Need to be patient. Expect to use PforR more in future. Need greater simplification of processes. But experience so far has been good. “ “My role is to fight the bureaucracy because bureaucracy could kill the instrument. At present, the PforR operations have to be reviewed by so many layers of bureaucracy. And the World Bank is very good at killing instruments.” “I haven’t done much marketing of the PforR. In some cases there are lots of uncertainties about country programs.” “With respect to my role on PforR…once the GP is better organized we need to think more clearly about results based approaches and what are the issues with different approaches and what are the real options. Managers have not kept up to speed on all of this.” 49. Respondents also shared strategies they use to highlight the new instrument: during reviews of Country Partnership Frameworks and Interim Strategy Notes, during senior management meetings of various kinds, and especially at the concept notes stage for operations, which provides an excellent opportunity for discussion of choice of instrument. One respondent intends to use the Two-Year Review report to highlight the profile of the instrument within the Global Practices. 9. Institutional Support for PforR 98 50. Respondents were appreciative of OPCS’s work and its “global perspective,” calling its support “outstanding,” “huge,” and “very helpful.” Two specifically singled out t he support provided by OPCS to sort out issues around ACGs. Some respondents noted that the advent of Global Practices should help the PforR more broadly because of an overall move toward a results agenda and “transformational projects.” Some respondents also pointed out that the Global Practices would help in the sharing of experiences across sectors and countries. 51. On issues related to the Bank’s budget, one respondent pointed out that the pressures on the budget as well pressures to implement projects with transformational impact suggests that programmatic lending options should become more prominent. Therefore, when comparing the budget costs of PforR to those of IPF, one needs to relate costs to impact and think about value for money. Looked at in this way, the PforR may cost more than IPF, but IPF often cannot scale up in the way that the PforR can. 10. Ongoing Challenges and Suggestions for Improvement Challenges 1. Assessments. Some respondents said that the guidance was too complex and assessments could be lighter. One respondent said that when the instrument was first introduced fiduciary staff were very conservative and preferred to do what they were familiar with. 2. DLIs. Challenges include how to pay for verifications, measurement difficulties, and the importance of ensuring that DLIs are focused on institutional performance as well as other indicators. 3. Cap. A number of respondents said that it was essential that the cap be removed; one said that “after a 5% cap on PforR was introduced by the Board, it became anti- climactic.” 4. ACGs. Most respondents brought up the ACGs. One stated that “without fixing the ACGs, it is difficult to recommend PforR, particularly for MICs.” 5. Clarity of options. Clarity about PforR relative to other instruments—DPF, output- based aid, results-based IPF, and so on is needed. 6. Technical Assistance. Operations that require a lot of technical assistance are difficult to do under PforR. 7. Uptake. There is a perception that PforR has been “slow to take off” – particularly in some sectors. Suggestions for improvements 1. More experience-sharing among front-line staff and with countries and sectors that have not yet used the instrument. 99 2. Create a community of TTLs familiar with PforR to advise future generations of TTLs as well as a cadre of people who do fiduciary and safeguards assessments who really understand the PforR. 3. Develop a decision tree to help determine choice of instrument during discussions of concept notes for specific operations. 4. Focus on institutional issues to fulfill promise of PforR. 5. Need strong country offices: fly-in, fly-out approach won’t work. 6. Roll out PforR instrument more and increase engagement with government officials. 7. Need for added push to ensure implementation goes well and operations can disburse. 8. Need for experienced TTLs to manage PforR operations. 9. Discussions on the appropriate instrument to use should begin well before the discussion of the concept note. 100 ANNEX 4. PFORR TWO-YEAR REVIEW CONSULTATIONS 1. To better understand the impact, perceptions and utility of the Program-for-Results (PforR) instrument, the PforR review team sought feedback from a broad range of stakeholders, including policy makers, officials in borrower governments, bilateral and multilateral development partners, Bank staff, civil society representatives, and other interested development groups and practitioners. 2. Consultations were held with members of government delegations, development partners, and representatives of international civil society organizations (CSO) at the Bank’s Annual (October 2013) and spring (April 2014) meetings. The team also reached out to the Asian Development Bank, Inter-American Development Bank, African Development Bank, as well as to interested stakeholders in Europe (Belgium, Germany, and the UK), for their views. 3. This annex focuses on the additional consultations held with countries and with CSOs and others at the Brookings Institution to discuss the process and findings of the review. Consultation process 4. During the summer of 2014, a series of consultations was organized with 15 countries that had experience with the PforR instrument. Participants included government officials, CSOs, the private sector and development partners in Bangladesh, Brazil, Croatia, Ethiopia, India, Kenya, Mexico, Moldova, Morocco, Nepal, Tanzania, Tunisia, Uganda, Uruguay, and Vietnam. In addition, a separate session was held for CSOs, think tanks and the private sector at the Brookings Institution (Washington, DC). 5. The overall structure of the consultations included a presentation of the main findings of the review, and an open discussion on experiences, alternate views and suggestions for enhancement of the instrument. To facilitate the discussions, the team translated presentation materials into French, Portuguese, and Spanish, and arranged videoconferences to connect multiple sites. A dedicated website was created to serve as a hub for information about upcoming consultations and allow visitors to provide comments and suggestions. 6. The following table presents the feedback received and the Bank’s response. 101 Table 1: Summary of Feedback from Public Consultations Feedback Bank Response Objectives of Program-for-Results and overall assessment Overall, most participants thought the findings reflected their own The review provides many details and experience with PforR and felt that the review process had provided examples of how the PforR instrument has them with a good opportunity to engage and provide feedback. Many been helpful in focusing on results and participants affirmed that what they liked best about the instrument was strengthening collaboration across different the focus on results and the ability to foster collaboration across various levels of government and with other levels of government to achieve results. Participants were stakeholders. knowledgeable about the instrument and its objectives, and made the Guidance can be enhanced on areas pointed following points: out by the consultations. The Bank agrees  More information sharing and outreach are needed for that this experience now needs to be shared countries at both national and sub-national levels. more widely as part of the further rollout of  Outreach should also involve Ministries of Finance (MoF) the instrument. and/or Treasury as PforR works differently than other Bank instruments.  Reporting requirements, especially when there is more than one sector involved, are important and should be clarified upfront.  Some features of PforR need to be better communicated, especially in the area of changing the culture of institutions, transparency, and accountability. The instrument has potential to be used more widely and across more sectors. Working with the Government and World Bank on PforR Participants who had experience with PforR operations noted that the The review includes examples of how the instrument changes the conversation and the interactions between the Bank is working more closely with World Bank and Governments. Participants shared examples of how the governments and other development focus had shifted and how this shift added value. They also pointed to partners during both the preparation and the some areas that need further consideration: implementation of PforR operations. The  Close communication between the Bank and the focus of discussion has shifted much more government is required to solve design and implementation to results and to helping strengthen the challenges. program systems and capacity necessary to achieve good results. The preparation time  Greater support is needed from the Bank during for approved PforR operations has varied implementation. Suggestions included considering shorter greatly but the Bank expects this to intervals between missions, and more use of stabilize as both government officials and videoconferencing for specific issues. Bank staff become more familiar with the  Building future partnerships requires changing the instrument. Implementation support will be interaction with other development partners, regular critical and requires a different mind-set communication, and a lot of work upfront. and even different skills as well as a  It is important that PforR find a way to leverage funds from different way of doing business. The Bank government and other development partners. recognizes this challenge and will be addressing it in the context of the further  Bank and government staffs working together more as a rollout of the instrument. team is critical, especially in terms of quality assurance and assessments. The Bank will continue to emphasize the importance of partnerships with other  Preparation of a PforR operation should take a specific and development partners, and step up its reasonable amount of time. outreach about the overall issue of program 102 Feedback Bank Response and results-financing with a view to realizing the potential of PforR as an instrument that can increase efficiency and enhance, i.e., leverage, Bank and broader development community support. Disbursement Linked Indicators (DLIs) Most participants thought that the review reflected their experience in The review recognizes the challenges this area – i.e., DLIs are very important and are taken seriously, but they associated with the formulation of DLIs are not so easy to identify and to set targets. Some specific reflections and associated disbursement mechanisms, and suggestions included: noting that 83 percent of government  Some DLIs require cooperation not only from MoF but from officials and 76 percent of Bank staff found line ministries, e.g., Ministry of Education, Ministry of Health. this aspect of PforR to be somewhat This can encourage better inter-governmental cooperation and difficult or very difficult. But it also notes coordination and help strengthen country systems. that both government officials and Bank staff strongly agree that the preparation of  In some programs, there may be a need to have targeted DLIs results frameworks has helped strengthen that focus on specific vulnerable or poor communities. the results focus of the Program, that using  It is important to understand the results chain and what DLIs as the main mechanism of interventions are needed to achieve results. disbursement is useful, and that the agreed  DLIs need to be set carefully; many could be scalable. mechanisms for verifying the achievement of DLIs can function effectively.  In some cases, the monitoring & evaluation (M&E) capacity in both the government and the private sector is minimal. It is In the further rollout of the instrument, the important to identify and build this capacity in government and Bank will provide additional guidance and private sector. training on this important feature of PforR, drawing on the experiences gained from  If the DLIs are designed very well and understood by all both approved operations and those under stakeholders then there will be a higher chance of success. preparation.  When other development partners are involved, understanding the design, formulation and flexibility of DLIs is important for harmonizing the approach.  Training and dissemination of good practices in the formulation of DLIs would be helpful.  It is important to consider the country’s forecasted cash flows and financing needs and to understand what will happen if DLIs are partially achieved or not achieved. Application of the Anticorruption Guidelines (ACGs) Participants noted that the ACGs: The review is recommending modifications  Were in conflict with the national law. Several went on to to the ACGs to: a) frame the ACGs in the explain how this issue was addressed in their operation; some context of the Bank’s and the borrower’s indicated that discussions on the ACGs delayed preparation of common interest in and commitment to the PforR operation. preventing and combating fraud and corruption under the Program; b) provide  Did not acknowledge that the country had its own processes for more explicit recognition that responsibility dealing with fraud and corruption allegations. for the implementation of the Program, including primary responsibility for preventing and combating fraud and corruption, rests with the borrower; c) clarify that the ACGs are not intended to restrict or otherwise affect the member country’s sovereign right to investigate, 103 Feedback Bank Response prosecute, or take any other action in furtherance of its own laws and regulations, or to obligate countries to undertake investigations and follow up actions in contradiction with the country’s applicable law; and d) clarify that any investigations the Bank undertakes are administrative inquiries for the purpose of determining compliance with its own policies, directives, and procedures, and to fulfill its fiduciary duty, not as a replacement for national law enforcement. Exclusions In a number of country consultations, participants raised questions about The review is recommending that the the exclusions and their rationale. They made the following procurement exclusions policy be amended observations: to provide that, in very limited exceptional  The policy requirement to exclude high-value contracts led to cases, large contracts that are deemed to be the exclusion of an important component of a program. It of critical importance to the integrity of the would be desirable to include these activities/contracts and have Program may be included in a PforR a more holistic approach, rather than an artificial program. operation. Any such large contracts/activities would be subject to  Potential for use in larger programs and other sectors is limited enhanced monitoring and supervision. due to the exclusions on the procurement of large contracts.  More flexibility is needed on procurement exclusions to support the use of the instrument where it can achieve most impact. Using Program Systems and assessments Participants made several points about program systems and The review documents the strong support assessments. of government officials and many other  PforR is helping the government focus more on improving stakeholders for the use of program implementation of its own programs. systems as a key feature of the PforR instrument.  PforR assessments should take into consideration existing reviews and assessments, and build on these. Recognizing the challenges associated with assessments and meeting audit  Using government policies and systems is good; this means the requirements, the review recommends assessments are critical, and that one needs to be realistic about providing more guidance and training. In the results. addition and building on the experience of  Fiduciary assessments were complicated, especially when other instruments, the review recommends subnational governments and agencies, all with their own flexibility in meeting audit requirements. forms, timelines, and formulations, were involved.  Audit and reconciliation of accounts (program expenditures) at the completion of the operation will be a challenge for some, given different budget structures and cycles and the many forms and sources of financing. Partnerships Participants commented on the impact of PforR regarding partnerships The review summarizes the positive and made the following remarks: experiences of the instrument in  PforR fosters closer collaboration between Bank and strengthening partnerships with government staff to drive the program forward. governments, development partners and stakeholders. It also recognizes the challenges of further enhancing 104 Feedback Bank Response  Continued engagement is also important as not all partners collaboration with other development have the same flexibility to use government systems, and many partners including the fact that relatively have different approaches to results-based financing. few development partners are able to use  Enhanced partnerships between development partners are government systems instead of their own, needed to jointly implement strategies for capacity and that there are different approaches to development. results-based financing. The Bank will continue to address these issues as part of  Synergy between the PforR instrument and Government its on-going dialogue with other procedures and processes is vital. development partners.  Cooperation with other PforR financing partners has become easier because of the instrument. Capacity Building Participants noted the importance of the capacity building aspect of The review stresses that strengthening the PforR for countries. Some key points raised include: institutional capacity needed for Programs  This aspect of PforR can create a strong incentive for local to achieve their desired results is one of the governments to focus on the preparation of investments; this is key features of the PforR instrument. All of a good starting point. However, it could be challenging for the approved PforR operations featured in larger investments that span several years. the review highlight specific and priority capacity building activities as well as  Technical assistance (TA) is important for success with PforR, system strengthening measures that may be but views diverged as to how and when it should be provided. linked to specific DLIs, included in legal Some participants said that it was best to mainstream capacity covenants, or made part of the Program building into the operation as a DLI to incentivize the desired Action Plan. institutional strengthening. Others expressed the view that TA is better programmed as non-lending TA to support the With respect to technical assistance, the government, as some countries have an issue with hiring review highlights the broad range of ways international consultants with their own systems. in which this issue has been addressed across the PforR operations approved to  It is important to focus on capacity building of the institutions date. It also notes the challenges with to bridge the gaps that exist in these institutions and also to respect to the financing of capacity facilitate publicizing the results and sharing the findings with building and alternative options in this the beneficiaries of the program. regard.  As the program focuses on specific indicators, this should be Going forward, maintaining a flexible supported by capacity building and, in selected cases, impact approach to how these issues are addressed evaluations. and learning from varied experiences will  Program Action Plans (PAP) are well aligned with the be key. government’s desire to be more transparent in management. The PAPs can help facilitate the implementation of some of these policies. Monitoring and verifications Participants noted that the findings on monitoring and verification The review confirms the importance of resonated with them and their experience. Other observations included: monitoring and verification under the  There is a need for an increased understanding of and more PforR operations that have been approved training about verification protocols. to date. It also pinpoints the need for more guidance, training and communications in  It is not clear who should cover the cost of the verification. this respect. As for financing of the  There were concerns about the impact of delayed/late verification process, that is the performance assessments, especially when they are also used responsibility of the Program. To date, for verification purposes. some monitoring and verification activities  Some participants also pointed out that it is not enough to have have been financed through trust funds or robust verification protocols; it is also important to other partners and government entities. communicate the results effectively and transparently. Verification delays can affect 105 Feedback Bank Response disbursements and it is important to set realistic expectations when identifying DLIs and specifying verification protocols. Communications and Training Participants noted the importance and demand for more communication, The review recognizes the urgent need and training and outreach. Some suggestions included: critical role of increasing communications,  Preparation of documentation/materials and training courses for training and outreach for government client countries that could be delivered at both national and officials and other stakeholders. subnational levels. Communication, dissemination, and outreach activities including tailoring  Preparation of outreach materials about the instrument, content to various audiences will be a explaining what PforR is and how it relates to other instruments priority for the Bank in the further rollout (including those of other development partners). of the PforR instrument.  Need to invest more in reaching out and training various groups As the PforR program portfolio matures, in the country and at all levels in the new instrument was the Bank will share lessons learned from stressed. teams and clients.  More learning across countries and operations, expand the peer learning and share more lessons learned. Other Issues  In some of the consultations, participants pointed out the The review recognizes the importance of importance of the political economy and understanding how developing PforR operations on the basis of decisions are made and their implications. strong partnerships with a range of  In other consultations, the liquidity issue of poor IDA countries government agencies and a solid was pointed out; participants proposed the Bank increase from understanding of government systems and 25% to 50% the amount of advance funding that can be how they can be strengthened. provided. The review does not recommend increasing the level of the advances above the current  In one consultation questions were raised about how robust the level. However, it does recommend complaints/grievance mechanisms are. delinking prior results financing from the  Clarification was sought on what policies applied in the case of provision of advances. This would provide hybrids. governments with more flexibility in addressing the liquidity needs of particular programs. The review also looked into the grievance redress mechanisms (GRMs) and while they are specific to each operation, this is an area where interviews showed value added from the instrument. Finally, regarding hybrids, if there is an Investment Project Financing (IPF) component, such a component would apply IPF policies. 106 ANNEX 5. PROGRAM-FOR-RESULTS PIPELINE (as of December 31, 2014) *Operations that have been initiated in the system but have not held a concept stage Program Financing IDA/ Country Program Name Global Practice FY ID (M$) IBRD Africa (AFR) Burkina Faso Public Sector Modernization P132216 Governance 25 IDA FY15 Cote Public Investments and Service P151507 Governance 50 IDA FY16 d’Ivoire* Delivery Ethiopia Service Delivery for the Poor P151432 Social Protection & Labor 600 IDA FY16 Kenya* Devolution Support P149129 Social, Urban, Rural & Res. 100 IDA FY16 Health, Nutrition & Kenya* Health Sector Support Program P152394 150 IDA FY16 Population Kenya Statistics Program for Results P149718 Poverty 50 IDA FY16 Lesotho* Education & Skills for Growth P151583 Education 50 IDA FY16 Health, Nutrition & Nigeria Saving One Million Lives P146583 500 IDA FY15 Population Education and Skills for Productive P152810 Tanzania* Education 120 IDA FY16 Jobs Program P152736 Health, Nutrition & Tanzania Big Results Now for Health 100 IDA FY15 Population East Asia and the Pacific (EAP) Vietnam Scaling Up Sanitation program P152693 Water 200 IDA FY16 Europe and Central Asia (ECA) Bosnia& BiH Business Environment P149755 Trade & Competitiveness 60 IBRD FY16 Herzegovina* EU Coastal Waters Compliance Croatia* P143251 Water 205 IBRD FY16 Program Latin America and the Caribbean (LAC) National Health Insurance System Health, Nutrition & Costa Rica P148435 400 IBRD FY16 Program Population Middle East and North Africa (MNA) Egypt Finance for Rural Egypt P151116 Finance & Markets 500 IBRD FY16 Egypt Inclusive Housing Finance P150993 Finance & Markets 500 IBRD FY15 Health, Nutrition & Egypt Health Care Support Program P149622 200 IBRD FY16 Population Egypt* Rural Sanitation Program P154112 Water 500 IBRD FY15 Health, Nutrition & Morocco Health Sector Support P148017 100 IBRD FY15 Population Morocco Integrated Risk Management P144539 Social, Urban, Rural and Res. 200 IBRD FY16 Morocco Urban Transport P149653 Transport & ICT 200 IBRD FY15 Leveraging Finance for Inclusive Morocco P153099 Finance & Markets 100 IBRD FY16 Growth Program Morocco* Municipal Support P149995 Social, Urban, Rural and Res. 100 IBRD FY16 Sylvo-Pastoral Ecosystems Co- Environment & Natural Tunisia* P151030 120 IBRD FY16 management Resources Tunisia Financial Sector Modernization P151011 Finance & Markets 250 IBRD FY16 South Asia (SAR) Results-based Rural Water Supply & Afghanistan P146943 Water 100 ARTF FY16 Sanitation India Swacch Bharat Sanitation P153251 Water 500 IDA FY15 Enhancing Teacher Effectiveness in India P132665 Education 250 IDA FY15 Bihar Program 107 ANNEX 6A. DRAFT POLICY Bank Policy Program-for-Results Financing Bank Access to Information Policy Designation Public Catalogue Number [to be completed by the LEG P&P F Administrator] Issued and effective ____, 2015 Content Establishes the Policy for the Program-for-Results Financing Applicable to IBRD and IDA Issuer Vice President and Head of Network, OPCS Sponsor Director, Operations Policy and Quality Department, OPCS Section I – Purpose and Application 1. This Policy sets forth the policy framework for the development and management for Program- for-Results Financing. 2. This Policy applies to the Bank. Section II – Definitions and acronyms As used in this Policy, the following capitalized terms and acronyms have the meanings set out below: 1. Bank: IBRD and IDA (whether acting in its own capacity or as administrator of trust funds funded by other donors). 2. Board: the Executive Directors of IBRD or IDA, or both, as applicable. 3. Borrower: includes the borrower of record and any other entities involved in Program implementation. 4. IBRD: International Bank for Reconstruction and Development. 5. IDA: International Development Association. 6. Management: the President or a Manager, or some or all of these persons, as applicable. 7. OPCS: Operations Policy and Country Services. 8. Policy: a Policy, as defined in “Bank Policy and Procedure Framework”, January 8, 2014, Catalogue No. EXC4.01POIL.01. 9. Program: is a program supported by Program-for-Results Financing. 10. Program-for-Results Bank Financing: refers to the provision of loans, credits, or grants financed by the Bank (including IBRD and IDA) from its resources or from trust funds financed by other donors and administered by the Bank, or a combination of these (hereinafter Program-for-Results Financing). 11. Program system or systems: refers to the systems of the institutions and relevant rules responsible for Program implementation, including the relevant rules, procedures, controls, risk identification and mitigation arrangements, and, when applicable, past implementation experience and record. Bank Policy, “Program-for-Results Financing” | Catalogue Number OPCS___ Page _ of 7 110 Section III – Scope 1. Program-for-Results Bank Ffinancing aims to promote sustainable development and improve the efficiency and effectiveness of expenditures by: (a) financing the expenditures of specific borrowerBorrower development programs; (b) disbursing on the basis of the achievement of key results (including prior results) under such programs; (c) using and, as appropriate, strengthening the program systems to provide assurance that program funds are used appropriately and that environmental and social impacts are adequately addressed by such programs; and (d) strengthening, where appropriate, the institutional capacity necessary for such programs to achieve their intended results. 2. Programs supported by Program-for-Results financing (hereinafter “Programs”) have expenditures, activities, and defined results, and promote sustainable development. The Programs may be: (a) new or already under implementation; (b) national, subnational, multisectoral, sectoral, or sub-sectoral in scope; (c) part of broader, longer-term, or geographically larger programs; and/or (d) carried out by governmental and/or nongovernmental parties. 3. Subject to the other applicable requirements of this OPPolicy, Program-for-Results Ffinancing may be extended to any type of expenditures, provided the Program is productive and Program oversight arrangements, including the fiduciary arrangements, provide reasonable assurance that financing proceeds will be used for the purposes for which the Ffinancing is granted, with due consideration of economy and efficiency. The amount of Program-for- Results Ffinancing should beis equal to or less than the total Program expenditures. If by the end of the Program, the amount of Program-for-Results financing disbursed exceeds the total amount of expenditures under the Program, taking into account contributions from other financing sources, the borrower is required to refund the difference to the Bank. 4. Subject to the other applicable provisions of this OPPolicy, Ffinancing proceeds are disbursed upon the achievement of verified results specified as disbursement-linked indicators (“DLIs”). Such disbursements are not dependent upon or attributable to individual transactions or expenditures. Under appropriate circumstances, such as to provide the borrowerBorrower with resources to allow the Program to start or to facilitate the achievement of one or more DLIsdisbursement-linked indicators, the Bank may agree to disburse a portion of the Ffinancing proceeds as an advance for DLIs disbursement-linked indicators that have not yet been achieved. Bank Policy, “Program-for-Results Financing” | Catalogue Number OPCS___ Page _ of 7 111 Considerations for Program-for-Results Financing 5. The Bank’s assessment of a proposed Program is based on various country and Program- specific strategic, technical, and risk considerations. These include the Program’s strategic relevance, technical soundness, expenditure analysis, economic rationale, results framework, fiduciary and environmental and social systems and risks., including governance arrangements.1 The assessments evaluate the relevant risks and the scope for improvements and managing such risks, including proposed institution strengthening activities to be undertaken before, if deemed appropriate, and during Program implementation. 6. The technical assessment considers, among other things, the Program’s rationale and its development objectives (taking into account consistency with the Bank’s overall assistance strategy for the member country in question); the borrowerBorrower’s commitment; relevant institutional and implementation arrangements, including governance; the Program’s activities and expenditures framework; the degree to which the Program aims to achieve specific, measurable, and verifiable results; the monitoring and evaluation arrangements; and the general policy, legal, regulatory, and institutional frameworks relevant to the Program. 7. The fiduciary systems assessment considers whether Program systems provide reasonable assurance that the Ffinancing proceeds will be used for intended purposes, with due attention to the principles of economy, efficiency, effectiveness, transparency, and accountability. The Program procurement systems are assessed as to the degree to which the planning, bidding, evaluation, contract award, and contract administration arrangements and practices provide reasonable assurance that the Program will achieve intended results through its procurement processes and procedures. The Program financial management systems are assessed as to the degree to which the relevant planning, budgeting, accounting, internal controls, funds flow, financial reporting, and auditing arrangements provide reasonable assurance on the appropriate use of Program funds and safeguarding of its assets. The fiduciary assessment also considers how Program systems handle the risks of fraud and corruption, including by providing complaint mechanisms, and how such risks are managed and/or mitigated. 8. The environmental and social systems assessment considers, as may be applicable or relevant in a particular country, sector, or Program circumstances, to what degree the Program systems: (a) promote environmental and social sustainability in the Program design; avoid, minimize, or mitigate adverse impacts, and promote informed decision-making relating to the Program’s environmental and social impacts; (b) avoid, minimize, or mitigate adverse impacts on natural habitats and physical cultural resources resulting from the Program; (c) protect public and worker safety against the potential risks associated with: (i) construction and/or operations of facilities or other operational practices under the Program; (ii) exposure to toxic chemicals, hazardous wastes, and other dangerous 1 In this OPPolicy, references to the Program governance arrangements and risks refer to: (a) good governance principles (transparency, integrity, accountability and effectiveness); (b) handling the risks of fraud, corruption, coercion, collusion, and similar activities (hereinafter “fraud and corruption”), and how such risks are managed and/or mitigated; and (c) compliance with the requirements set out in the Guidelines on Preventing and Combating Fraud and Corruption in Program-for-Results Financing (February 1, 2012), as such guidelines may be amended from time to time. Bank Policy, “Program-for-Results Financing” | Catalogue Number OPCS___ Page _ of 7 112 materials under the Program; and (iii) reconstruction or rehabilitation of infrastructure located in areas prone to natural hazards; (d) manage land acquisition and loss of access to natural resources in a way that avoids or minimizes displacement, and assist the affected people in improving, or at the minimum restoring, their livelihoods and living standards; (e) give due consideration to the cultural appropriateness of, and equitable access to, Program benefits, giving special attention to the rights and interests of the Indigenous Peoples and to the needs or concerns of vulnerable groups; and (f) avoid exacerbating social conflict, especially in fragile states, post-conflict areas, or areas subject to territorial disputes. 9. Activities that are judged to be likely to have significant adverse impacts that are sensitive, diverse, or unprecedented on the environment and/or affected people are not eligible for Program-for-Results Financing, and are excluded from the Program,. and aActivities that involve procurement of works, goods, and services under contracts whose estimated value exceeds specified monetary amounts (high-value contracts)2 are normally not eligible for Program-for-Results Ffinancing, and are also excluded from the Program. However, such contracts may be included in the Program if they are deemed to be important to the integrity of the Program and their monetary value in relation to the overall Program is modest. 10. The Program integrated risk assessment considers key risks to achieving the Program’s results and development objectives. The integrated risk assessment is informed by the results of the technical, fiduciary, and environmental and social systems assessments and provides a key input into the Bank’s decision to provide Program-for-Results Ffinancing. This decision takes into account country/sector/multisector-specific circumstances, potential benefits of the Program, the needs and capacity of the borrowerBorrower, and the degree to which the Program-for-Results Ffinancing and implementation support will contribute to the overall Program objectives and results. Borrower and Bank Roles and Responsibilities in Program Preparation and Implementation 11. The borrowerBorrower is responsible for preparing and implementing the Program. The Program’s scope and objectives and the borrowerBorrower’s contractual obligations to the Bank are set out in the legal agreements with the Bank. These obligations include the requirement to carry out the Program with due diligence, and to maintain appropriate monitoring and evaluation arrangements (including credible DLI disbursement-linked indicators verification protocols), fiduciary and environmental and social Program systems, and governance arrangements. The borrowerBorrower is expected to deal in a timely and effective manner with actual or alleged problems or violations (individual or systemic) in these areas. 12. The Bank assesses and appraises the Program in accordance with this OP, BP 9.00Policy, the Directive, and other applicable policy, technical, operational, and procedural requirements. In relation to the environmental and social systems assessment, the Bank consults with 2 See Section II, paragraph 7 and Section III, paragraph 5 of the BP 9.00Directive Program-for-Results. Bank Policy, “Program-for-Results Financing” | Catalogue Number OPCS___ Page _ of 7 113 Program stakeholders and discloses the results and recommendations of its assessment. As appropriate, the Bank agrees with the borrowerBorrower on specific actions to be taken and arrangements to be maintained during the period of the Program, including measures to address identified weaknesses and risks and to strengthen institutional capacity. 13. The Bank provides implementation support to the borrowerBorrower by reviewing implementation progress, achievement of the Program results and DLIsdisbursement-linked indicators, and associated Program risks. The Bank monitors the borrowerBorrower’s compliance with its contractual obligations, including actions to strengthen institutional capacity. Recourse, Remedies, and Sanctions 14. If the borrowerBorrower does not comply with its contractual obligations, the Bank consults with the borrowerBorrower, and requires the borrowerBorrower to take timely and appropriate corrective measures. The Bank’s legal remedies are specified in the relevant legal agreements and include the right to suspend disbursement and to cancel Program-for-Results Ffinancing. The Bank exercises such remedies when warranted and as it deems appropriate, taking into account, among other things, country-, sector-, and Program-specific circumstances, the extent of and possible harm caused by circumstances giving rise to the remedy, and the borrowerBorrower’s commitment and actions to address the identified problems. However, the Bank takes a graduated approach to suspension for non-payment, and when an IBRD loan or IDA credit payment from the borrowerBorrower to the Bank is overdue by 60 days, the Bank suspends all financings to or guaranteed by the country concerned. 15. The borrowerBorrower is responsible, among other things, for taking appropriate measures to prevent, detect, and respond to fraud and corruption or allegations of fraud and corruption in the Program. The Bank has the right to investigate allegations of fraud and corruption in the Program and to sanction parties that engage in sanctionable practices. Managing Program-for-Results Financing 16. Approval. The Executive DirectorsBoard decides whether to approve IBRD and IDA Program- for-Results Financing proposals. Except for IDA grants and trust-fund-financed grants explicitly requiring approval by the Executive DirectorsBoard, all other grants are approved by Management. 17. Signing. Signing of legal agreements for Program-for-Results Financing takes place after all required authorizations have been issued; and provided there are no payments on IBRD loans or IDA credits to the borrowerBorrower, or to or guaranteed by the member country, that are overdue by 30 days or more, unless, in exceptional circumstances, Management approves the signing and reports such information to the Executive DirectorsBoard. 18. Restructuring. During the implementation of the Program, and as part of Bank implementation support, the Program may, with the agreement of the Bank and the borrowerBorrower, be restructured to strengthen its development impact, modify its development objectives or Bank Policy, “Program-for-Results Financing” | Catalogue Number OPCS___ Page _ of 7 114 DLIsdisbursement-linked indicators, improve Program performance, address risks and problems that have arisen during implementation, make appropriate use of undisbursed financing, cancel unwithdrawn amounts prior to the Closing Date, extend the Closing Date, or otherwise respond to changed circumstances. A restructuring involving a modification of the original Program objectives is referred to as a Level One restructuring and is submitted for approval by the Executive DirectorsBoard. A restructuring involving any other modification of the Program is referred to as a Level Two restructuring. The authority to approve Level Two restructuring is delegated by the Executive DirectorsBoard to Management. Management periodically informs the Executive DirectorsBoard of Level Two restructurings. 19. Closing Date. The closing date (“Closing Date”) is the date after which the Bank may stop accepting withdrawal applications under the Program-for-Results Financing and cancel any undisbursed balance in the Ffinancing account. The Closing Date is not extended: (a) for Ffinancings subject to suspension of disbursements; or (b) for any Ffinancing to a borrowerBorrower with any outstanding audit reports or with audit reports which are not satisfactory to the Bank, unless the borrowerBorrower and the Bank have agreed on actions to address the deficiencies. Exceptionally, retroactive extensions of a Closing Date may be approved by Management. Additional Program-for-Results Financing 20. The Bank may agree to provide additional Program-for-Results Ffinancing to an ongoing Program to meet: (a) unanticipated significant changes to expenditures parameters required to achieve the original Program results or DLIsdisbursement-linked indicators; or (b) new or modified results, to be reflected in new or modified DLIsdisbursement-linked indicators, that aim to scale up the impact or development effectiveness of the original Program. The Bank provides additional Program-for-Results Ffinancing if it is satisfied with the overall implementation of the original (or restructured) Program. The Bank considers the proposed additional Program-for-Results Ffinancing on the basis of, as necessary, updated or additional technical, fiduciary, environmental and social impacts, and integrated risk assessments. Additional Program-for-Results Ffinancing is separate and distinct from the original Program- for-Results financing and is submitted for approval by the Executive DirectorsBoard or Management (as in paragraph 16 of this OPPolicy). Disclosure of Information During Program preparation and implementation support, the Bank discloses Program-related information in accordance with the World Bank Policy on Access to Information. Section IV – Waiver A provision of this Policy may be waived in accordance with Bank Policy, “Operational Policies Waivers”, April 7, 2014, Catalogue No. OPCS5.06-POL.01. Section V – Effective Date Bank Policy, “Program-for-Results Financing” | Catalogue Number OPCS___ Page _ of 7 115 This Policy is effective _________, 2015. Section VI – Issuer The Issuer of this Policy is the Vice President and Head of Network, OPCS. Section VII – Sponsor The Sponsor of this Policy is the Director, Operations Policy and Quality Department. Section VIII – Related Documents 1. Bank Directive, “Program-for-Results”, ____, 2015, Catalogue No. OPCS__. 2. Program-for-Results: Two-Year Review, _ 2015 3. A New Instrument to Advance Development Effectiveness, Program-for-Results Financing, December 29, 2011. 4. Guidelines on Preventing and Combating Fraud and Corruption in Program-for-Results Financing (__ 2015). 5. The World Bank Policy on Access to Information, July 1, 2013. 6. Program-for-Results operations are governed by this Policy, the related Directive, and the following OPs/Policies and BPs/Directive (including any relevant Operational Memoranda and Instructions/Procedures [internal use only]), as applicable: OP 1.00, Poverty Reduction; World Bank Group Directive Country Engagement; OP/BP 2.30, Development Cooperation and Conflict; OP/BP 3.10, Financial Terms and Conditions of IBRD Loans, IBRD Hedging Products, and IDA Credits; OP 4.07, Water Resource Management; OP/BP 4.20, Gender and Development; OP 4.76, Tobacco; OP 7.00, Lending Operations: Choice of Borrower and Contractual Agreements; OP7.20, Security Arrangements; OP /BP 7.30, Dealing with De Facto Governments; OP/BP 7.40, Disputes Over Defaults on External Debt, Expropriation, and Breach of Contract; OP/BP 7.50, Projects on International Waterways; OP/BP 7.60, Projects in Disputed Areas; OP/BP 8.45, Grants; OP/BP 10.20, Global Environmental Facility Operations; OP/BP; OP/BP 14.10, External Debt Reporting and Financial Statements; OP/BP 14.20, Cofinancing; OP/BP 14.40, Trust Funds; BP 17.30, Communications with Executive Directors; and BP 17.55, Inspection Panel. Questions regarding this Policy should be addressed to Operations Help Desk. Bank Policy, “Program-for-Results Financing” | Catalogue Number OPCS___ Page _ of 7 116 ANNEX 6B. TECHNICAL, EDITORIAL, AND SUBSTANTIVE CHANGES Draft Bank Policy, Program-for-Results Financing (currently OP 9.00) Technical, Editorial Reference to Relevant and Substantive Explanatory Notes Paragraph Concerned Changes (by topic) Entire document Compliance with In light of the Policy and Procedure Framework Policy, Directive and Policy and Procedure Procedure (P&P) (effective January 8, 2014), OP 9.00 is proposed to Framework be retrofitted to conform with the P&P rules. Specifically, the Policy is now formulated and organized in accordance with standard P&P templates and format. Accordingly, all definitions and acronyms are proposed to be listed in Section II of the Policy instead of within the body of the Policy text or within footnotes, as they are in OP 9.00; defined terms and acronyms are capitalized; undefined terms put in lowercase; references in the text to “OP” would be changed to “Policy” and the term “BP” would be changed to “Directive”; references to other OP/BPs in the Pedigree Box of OP 9.00 are proposed to be listed in Section VIII, Related Documents; and reference to Access to Information Policy is proposed to be in Section VIII, Related Documents. Section III, paragraph 3 Clarification of The last sentence in paragraph 3 has caused serious concerns due to (currently paragraph 3 Language the confusion in its phrasing. The reference to other sources of funds in OP 9.00) both mixes the level of expenditure with sources and has posed many challenges for clients in the interpretation. Clients have questioned whether this encompasses different levels of government (central or subnational) or even households –which would make it difficult to calculate. As the intent is to ensure that overall level of expenditure of the program is greater or equal to the WB financing, the sentence has been simplified to reflect the basic policy concept. Section III, paragraph 9 Substantive Change While the exclusions remain in place, the proposed change is (currently paragraph 9 intended to allow in very limited cases activities that are deemed to in OP 9.00) be of critical importance to the integrity of the Program and at the same time their monetary value is modest. Such included activities will be subject to enhanced monitoring and supervision. Section III, paragraph 5 Miscellaneous The phrase and accompanying footnote are defined more than once and footnote 1 (redundant text, and also duplicate what is included in the ACGs and Section III, (currently paragraph 5 consistency across paragraph 7 which defines fiduciary risks. The proposed revisions will and footnote 3 in OP policies; accuracy of ensure consistency of definitions and thus it is proposed to delete the 9.00) definitions) phrase “…including governance arrangements.” and the accompanying footnote. Section II, paragraph Reclassification The last part of the definition in paragraph 11 includes level of details 12 (currently in that is part Instructions and part guidance. Therefore it is proposed to footnote 2 in OP 9.00) retain the Policy aspects of the definition for “Program systems or system” and remove other details and include in the appropriate documentation. 117 Technical, Editorial Reference to Relevant and Substantive Explanatory Notes Paragraph Concerned Changes (by topic) Section II, paragraph Miscellaneous The proposal is to delete the term “Bank” and the phrase “hereinafter 10 and Section III, (Editorial Changes) Program-for-Results Financing.” paragraphs 1 and 2 (currently in paragraphs 1 and 2 and footnote 1 in OP 9.00) Section III, paragraph 3 The proposal is to change the phrase “should be equal” in the second (currently in paragraph sentence to “is equal.” 3 in OP 9.00) Section III, paragraphs It is proposed to change the term “Executive Directors” to “Board.” 16, 17, 18, 20 (currently paragraphs 16, 17, 18, 20 in OP 9.00) Section III, paragraphs The proposal is to spell out the acronym DLI (disbursement-linked 3, 4, 12, 20 and indicators). footnote 2 (currently paragraphs 3, 4, 12, 20 and footnote 4in OP 9.00) 118 ANNEX 7A. DRAFT DIRECTIVE Bank Directive Program-for-Results Financing Bank Access to Information Policy Designation Public Catalogue Number [to be completed by LEG P&PF Administrator] Issued and effective __________, 2015 Content Establishes the directions for the Program-for-Results Financing Applicable to IBRD and IDA Issuer Vice President and Head of Network, OPCS Sponsor Director, Operations Policy and Quality Department, OPCS 119 Section I – Purpose and Application 1. This Directive provides direction for Program-for-Results Financing. 2. This Directive applies to the Bank. Section II – Definitions As used in this Directive: (a) the capitalized terms have the meaning set out: (i) in Section II of the Bank Policy: “Program-for-Results Financing,” April 7, 2014, Catalogue No. OPCS___; or (ii) below; and (b) the acronyms stand for the terms set out below. 1. ACGs: Anticorruption Guidelines. 2. CAS: Country Assistance Strategy. 3. CPF: Country Partnership Framework. 4. CPS: Country Partnership Strategy. 5. Directive: a Directive, as defined in the “Bank Policy and Procedure Framework”, January 8, 2014, Catalogue No. LEG4.01DIR.01. 6. DLIs: Disbursement linked indicators are specific, measurable, and verifiable indicators related to and/or derived from the Program development objectives and the results framework. DLIs may be expressed as outcomes, outputs, intermediate outcomes or outputs, process indicators, or financing indicators. DLIs may also be defined as actions or process results deemed critical for strengthening performance under the Program (this could include actions for improving fiduciary, social and environmental issues and/or monitoring and evaluation), or as indicators of key institutional changes. 7. High-value Contracts: means contracts with estimated values exceeding the monetary amounts, as may be amended from time to time, that require mandatory review by the Bank’s OPRC.1 8. ICR: Implementation Completion and Results Report. 9. MOP: Memorandum of the President. 10. OPRC: Operations Procurement Review Committee. 11. PAD: Program Appraisal Document. 12. PCN: Program Concept Note. 13. PID: Program Information Document. 1 See OPRC Thresholds. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 120 Section III – Scope 1. The Bank assesses a program proposed by the Borrower2 for Program-for-Results Bank Ffinancing3 (hereinafter the “Program”) and, upon Program-for-Results Ffinancing approval, provides implementation support to the Borrower in accordance with the requirements set forth in OP 9.00the Bank Policy and this BPDirective. A. Preparation of the Program Operation 2. The preparation phase includes identification, assessment, and appraisal of the Program, and various interim processing and decision steps such as concept review, decision meeting, negotiations and approval. Identification to Concept Review 3. At the identification stage, the Bank consults with the Borrower on the Borrower’s development program and goals, and seeks to identify the Program’s overall parameters, objectives, financing requirements, possible level of Program-for-Results Ffinancing, and other general information. After the Bank and Borrower have reached preliminary understanding on the Program concept and parameters, Bank budget is made available for further Program preparation, and a task team is formed, led by a team leader and comprising relevant specialists. 4. The task team preliminarily, and in consultation with the Borrower: (a) defines the Program and assesses its development objectives, strategic relevance, rationale, and relation to the relevant CPF, Country Assistance Strategy (CAS), Country Partnership Strategy (CPS), or similar instrument; (b) identifies the results to be achieved under the Program; (c) identifies the overall Program expenditures, nature of activities and implementation arrangements; (d) estimates the possible scope of Program-for-Results Ffinancing; 2 In this BP, unless the context requires otherwise, references to the “Borrower” include the Borrower of record and any other entities involved in Program implementation. 3 Program-for-Results Bank financing refers to the provision of loans, credits, or grants financed by the Bank (including IBRD and IDA) from its resources or from trust funds financed by other donors and administered by the Bank, or a combination of these (hereinafter “Program-for-Results financing”). Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 121 (e) considers whether the Program systems4 in the fiduciary, environmental and social, and governance5 areas are broadly adequate; (f) when the proposed Program is a part of an ongoing program, considers that program’s performance to date and, if applicable, lessons learned from similar programs in the country; and (g) considers the risks to achieving the Program’s objectives and results. 5. In accordance with Section III, paragraph 9 of the Policy OP 9.00, the task team preliminarily assesses whether the proposed Program may involve activities that: (a) are judged to be likely to have a significant adverse impacts that are sensitive, diverse, or unprecedented on the environment and/or affected people; or (b) involve procurement of goods, works, and services under highHigh-value contractsContracts. For the purposes of Section III, paragraph 9 of the Policy, unless determined otherwise by Management, an individual contract is deemed to be modest in relation to the overall Program if its estimated monetary value is equal to or less than 25 percent of the estimated total Program expenditures. For the purposes of this BP, “high-value contracts” means contracts with estimated values exceeding the monetary amounts, as may be amended from time to time, that require mandatory review by the Bank’s Operations Procurement Review Committee (OPRC).6 Such activities are not eligible for Program-for-Results financing, and are excluded from the Program. 6. The Bank coordinates its preparatory work with development partners and other parties that may cofinance the Program or be otherwise involved in Program-related activities. 7. After the Program concept is developed, but before further preparation takes place, the task team prepares a Program Concept Note (PCN) that describes the proposed Program, including the information under paragraphs 4 and 5 above. The task team also prepares an initial Program Information Document (PID) that summarizes the main elements of the Program and the proposed Bank financing. The PCN and the PID are submitted for the Concept Review. The Concept Review decides, among other things: (a) whether the Bank should proceed with the preparation of the Program-for-Results Ffinancing; (b) the overall parameters of the assessments to be carried out; (c) the level of subsequent Bank review(s);7 and (d) the timetable and resources for Program preparation. 4 In this BP, the term “systems” or “Program systems” refers to the systems of the institutions responsible for Program implementation, including the relevant rules, procedures, controls, risk identification and mitigation arrangements, and, when applicable, past implementation experience and record. 5 In this BP, references to the Program governance arrangements and risks refer to: (a) good governance principles (transparency, integrity, accountability and effectiveness); (b) handling the risks of fraud, corruption, coercion, collusion, and similar activities (hereinafter “fraud and corruption”), and how such risks are managed and/or mitigated; and (c) compliance with the requirements set out in the Guidelines on Preventing and Combating Fraud and Corruption in Program-for-Results Financing (February 1, 2012), as such guidelines may be amended from time to time. 6 See OPRC Thresholds. 7 Decision reviews may take place at either: (a) the corporate review level, through a Regional Operations Committee, or the Bank’s Operations Committee; or (b) at a subcorporate level (review by country or regional management). For more details, see the Guidelines and Procedures for Corporate Review of Operations and Country Strategies. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 122 8. Upon receiving authorization to continue with the preparation of the operation, the task team submits the PID for public disclosure. Assessment and Appraisal 9. The Bank continues to carry out the Program technical, fiduciary systems, environmental and social systems, and integrated risk assessments (described in Part B of this BPDirective), taking into account the country-, sector-, and Program-specific circumstances, the scope for improvements before and during implementation, the risks to achieving Program objectives and results, and associated risk mitigation measures. The integrated risk assessment is an ongoing process that requires periodic consultations between the task team and Management, and between the Bank and the Borrower. 10. After the assessments have been carried out and most design and assessment issues have been resolved, the task team submits for Management’s Decision Review Meeting the draft Program documentation Program Appraisal Document (PAD), an updated draft PID, and the draft legal agreements.8 The Decision Review Meeting decides on, among other things, the following matters: (a) the ability to achieve Program results and the adequacy of the disbursement-linked indicators (DLIs), and their verification protocols; (b) the adequacy of the recommendations from the assessments, as well as the Program action plan (if required) for enhancing the Program systems and mitigating risks; (c) the overall Program risk assessment; (d) the exclusion from the Program of activities referred to in paragraph 5 of this BPDirective; (e) Program-for-Results financing conditions, including conditions of appraisal, negotiations, and Board presentation, and legal conditions; (f) the proposed implementation support arrangements; and (g) whether the task team should proceed with Program appraisal or whether the Program can be considered already appraised. 11. The task team appraises the Program-for-Results financing to confirm any relevant Program- and financing-related information and resolve any outstanding legal, design, and implementation issues, and then finalizes the draft PAD and draft legal documents. The updated PID is disclosed before the Program appraisal is completed. 12. Generally, the following information is finalized following the appraisal: (a) the Program’s definition, development objective, rationale, and scope, taking into account the provisions of Section III, paragraph 9 of OP 9.00the Policy and paragraph 5 of this BPDirective, planned expenditures, financing requirements, and implementation and funds flow arrangements; 8 When significant Program design or other legally relevant issues remain partially unresolved, the draft legal documents may be excluded from the documents submitted for the decision meeting. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 123 (b) the technical, fiduciary, environmental and social systems, and integrated risk assessments carried out and the conclusions of those assessments, and, as necessary, the relevant risk management and other actions undertaken or to be undertaken during the preparation of the Program operation, and/or during Program implementation, to enhance Program systems and performance; (c) the main legal terms and conditions, DLI verification protocols, and/or the Program action plan, as applicable; (d) the results framework and the monitoring and evaluation arrangements, and, as appropriate, baseline references to be used in monitoring implementation and Program systems performance and assessing the development effectiveness of the Program at completion; (e) the DLIs and their credible verification protocols,9 and other disbursement-related provisions; (f) cofinancing or other collaboration arrangements with other development partners and stakeholders; (g) any proposed exceptions to or waivers from Bank policies or procedures; and (h) implementation support arrangements. 13. DLIs and Disbursement. The DLIs are specific, measurable, and verifiable indicators related to and/or derived from the Program development objectives and the results framework.10 14.13. Subject to the other requirements of OP 9.00the Policy and this BPDirective, the Bank may agree to disburse up to 25 percent of Program-for-Results Ffinancing proceeds (unless a higher percentage is approved by Management) on account of the DLIs met by the Borrower between the date of the Program Concept Review and the date of the legal agreement for Program-for-Results Ffinancing. 15.14. To provide a Borrower with resources to allow the Program to start or to facilitate the achievement of one or more DLIs, the Bank may agree to make an advance payment (following the effectiveness of the Program-for-Results Ffinancing legal agreement) of up to 25 percent of Program-for-Results Ffinancing (unless a higher percentage is approved by Management) for one or more DLIs that have not yet been met (“advance”). When the DLI(s) for which an advance has been disbursed are achieved, the amount of the advance is deducted (recovered) from the amount due to be disbursed under such DLI(s). The advance amount recovered by the Bank is then available for additional advances (“revolving advance”). The Bank requires that the Borrower refund any advances (or portion of advances) if the DLIs have not been met (or have been only partially met) by the Program Closing Date. 9 The credibility of verification protocols is finally assessed and confirmed at appraisal, and described in the Program documents, as appropriate. Depending on the nature of the specific DLIs and the overall context of the Program and country circumstances, the verification of DLIs can be carried out in various forms and by various parties, which could include but not be limited to government agencies, semi-autonomous entities, statistical or audit entities, and third-party entities such as nongovernmental organizations and representatives of civil society. For disbursement purposes, the Bank retains the right to make the final decision whether DLIs have been achieved. 10 DLIs may be expressed as outcomes, outputs, intermediate outcomes or outputs, process indicators, or financing indicators. DLIs may also be defined as actions or process results deemed critical for strengthening performance under the Program (this could include actions for improving fiduciary, social and environmental issues and/or monitoring and evaluation), or as indicators of key institutional changes. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 124 16.15. The combined amount of financing referred to in paragraph 14 and paragraph 15 of this BP 9.00 may not exceed 30 percent of the Program-for-Results financing (unless a higher percentage is approved by Management). Negotiations and Approval 17.16. Following the appraisal, the task team submits for Management approval a request, supported by the relevant documents, to negotiate the Program-for-Results Ffinancing. After negotiations have been authorized, the Bank and Borrower and any other Program- related parties conduct the negotiations and seek to finalize agreement on the relevant issues and documents. If new substantive issues or significant changes in the design of the Program or Program-for-Results Ffinancing are raised during the negotiations, the task team consults with Management. If the negotiations of the financing requiring approval by the Executive DirectorsBoard are successfully completed, and there are no Board conditions, the task team finalizes the relevant draft Program-for-Results documents for submission to the Board. If any information in the PAD raises issues of confidentiality or sensitivity, or may adversely affect relations between the Bank and the Borrower, and this information is deemed to be relevant to the Executive Directors in their decision-making process, the task team, in consultation with Management, removes this information from the PAD and incorporates it in the draft MOP. 18.17. After all requirements for Board presentation of Program-for-Results Financing requiring Executive DirectorsBoard approval have been met, the Executive DirectorsBoard decide on whether to approve the proposed Program-for-Results financing.11 The final PAD is disclosed in accordance with the Bank’s Policy on Access to Information. B. Program Assessments Technical Assessment 19.18. The technical assessment is carried out in accordance with the provisions set out in Section III, paragraph 6 of OP 9.00this Policy and paragraph 9 of this BPDirective, and addresses the following matters: 20.19. Strategic relevance, technical soundness, and institutional arrangements. The task team assesses whether: (a) there is a clear rationale for the Program, with the Program addressing an important development goal, and whether there is potential to significantly improve Program performance; (b) the Program is consistent with the Bank’s overall assistance strategy for the member country in question (as expressed in the CAS, CPF, CPS, or similar instrument); 11 For more details, see the OPCS Processing Steps Guidelines and SECPO guidance detailing eSubmission of Board documents. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 125 (c) the technical approach proposed by the Borrower under the Program, and the Program’s structure and implementation arrangements (including governance arrangements), are adequate; (d) the Borrower’s institutional capacity is adequate; and (e) the general legal, regulatory, and institutional environment within which the Program will operate is adequate for Program implementation. 21.20. Expenditure Framework. The task team assesses the level, efficiency, transparency, and effectiveness of the expenditures included in the Program. This includes consideration of whether the planned expenditures are adequate to achieve the Program results, whether the medium-term budget is sustainable, and whether there are major discrepancies between budget allocations, releases, and actual expenditures. 22.21. Results Framework and DLIs. The task team assesses the Program’s: (a) results framework, which may include outcomes, outputs, intermediate outcomes or outputs, including the degree to which the Program aims to achieve these results; (b) monitoring and evaluation framework; and (c) DLIs and the related verification protocols. 23.22. Economic Evaluation. The task team undertakes an economic evaluation providing the rationale for the public provision of the Program, where appropriate, as well as its expected impact, describing, among other things, the expected added value of Bank support. Evaluation methods range from narrative comparisons of scenarios to quantitative methods. Fiduciary Systems and Environmental and Social Systems Assessments 24.23. The task team carries out the fiduciary and the environmental and social systems assessments in accordance with Section III, paragraphs 7 and 8 of OP 9.00 the Policy. The assessments consider whether the Program institutions have the capacity to carry out adequate planning, decision making, execution, reporting, monitoring and evaluation, auditing and information disclosure under the Program. The assessments include: (a) a review of the existing systems; (b) identification of areas in which the implementing entities should improve procedures and performance (which may be expressed through the Program’s action plan, as necessary); and (c) inputs to the integrated risk assessment. 25.24. Fiduciary Systems Assessment. The assessment is carried out in accordance with Section III, paragraph 7 of OP 9.00 the Policy. The task team evaluates the Program fiduciary systems to determine whether they provide reasonable assurance that the Program expenditures will be used appropriately to achieve their intended purpose. The fiduciary systems assessments consider the degree to which: (a) from a procurement perspective there is/are reasonable: (i) arrangements for planning and budgeting; (ii) procurement rules and such rules are easily accessible to the public; (iii) capacity for contract management and administration; (iv) complaint mechanisms, including clarity on how they are utilized; (v) systems for Program oversight and control; and (b) from a financial management perspective: (i) the budgeted expenditures are realistic, prepared with due regard to relevant policies, and executed in an orderly and predictable manner; (ii) reasonable records are maintained and financial reports Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 126 produced and disseminated for decision-making, management, and reporting; (iii) adequate funds are available to finance the Program; (iv) there are reasonable controls over Program funds; and (v) independent audit arrangements are in place. 26.25. As part of the assessments, and taking into account the requirements of the Guidelines on Preventing and Combating Fraud and Corruption in Program-for-Results Financing (February 1, 2012) (ACGs), the Bank also assesses the degree to which Program systems handle the risk of fraud and corruption, including complaint mechanisms, and how such risks are managed and mitigated. The Bank and the Borrower will, where required,may agree on a Program-specific protocol describing the operational arrangements to handle allegations, information, and investigations related to fraud and corruption. Such a protocol may be formulated in various forms, for example as a covenant in the legal agreement between the Bank and Borrower, letter or memorandum of understanding, or as part of the minutes of negotiations.12 27.26. The fiduciary systems assessment provides a reference that is used to monitor for fiduciary systems performance during Program implementation and identifies actions, as needed, to enhance the fiduciary systems during Program preparation and implementation (the latter are included in the Program’s action plan). The fiduciary risks and proposed mitigation measures are inputs to the integrated risk assessment. 28.27. Environmental and Social Systems Assessment. The task team assesses the Program systems for managing environmental and social effects, taking into account, among other things, the capacity to plan, implement, monitor, and report on the environmental and social mitigation measures, the scope for improvements, and the risks and related mitigation measures. 29.28. The assessment, carried out in accordance with Section III, paragraph 8 of OP 9.00 the Policy, considers, as may be applicable or relevant under particular country, sector, or Program circumstances, to what degree the Program systems: (a) operate within an adequate legal and regulatory framework to guide environmental and social impact assessments at the Program level; (b) incorporate recognized elements of environmental and social assessment good practice, including: (i) early screening of potential impacts; (ii) consideration of strategic, technical, and site alternatives (including the “no action” alternative); (iii) explicit assessment of potential induced, cumulative, and transboundary impacts; (iv) identification of measures to mitigate adverse environmental or social impacts that cannot be otherwise avoided or minimized; (v) clear articulation of institutional responsibilities and resources to support implementation of plans; and (vi) responsiveness and accountability through stakeholder consultation, timely dissemination of Program information, and responsive grievance redress measures; (c) include appropriate measures for the early identification and screening of potentially important biodiversity and cultural resource areas; 12 Such a protocol may be formulated in various forms, for example as a covenant in the legal agreement between the Bank and Borrower, letter or memorandum of understanding, or as part of the minutes of negotiations. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 127 (d) support and promote the protection, conservation, maintenance, and rehabilitation of natural habitats; avoid the significant conversion or degradation of critical natural habitats; and if avoiding the significant conversion of natural habitats is not technically feasible, include measures to mitigate or offset the adverse impacts of Program activities; (e) take into account potential adverse effects on physical cultural property and provide adequate measures to avoid, minimize, or mitigate such effects; (f) promote adequate community, individual, and worker safety through the safe design, construction, operation, and maintenance of physical infrastructure; or, in carrying out activities that may be dependent on such infrastructure, incorporate safety measures, inspections, or remedial works as appropriate; (g) promote the use of recognized good practice in the production, management, storage, transport, and disposal of hazardous materials generated under the Program; promote the use of integrated pest management practices to manage or reduce pests or disease vectors; and provide training for workers involved in the production, procurement, storage, transport, use, and disposal of hazardous chemicals in accordance with the relevant international guidelines and conventions; (h) include adequate measures to avoid, minimize, or mitigate community, individual, and worker risks when Program activities are located in areas prone to natural hazards such as floods, hurricanes, earthquakes, or other severe weather or climate events; (i) avoid or minimize land acquisition and related adverse impacts; identify and address economic and social impacts caused by land acquisition or loss of access to natural resources, including those affecting people lacking full legal rights to resources they use or occupy; provide compensation sufficient to purchase replacement assets of equivalent value and to meet any necessary transitional expenses, paid before taking land or restricting access; provide supplemental livelihood improvement or restoration measures if taking of land causes loss of income-generating opportunity (e.g., loss of crop production or employment); and restore or replace public infrastructure and community services that may be adversely affected by the Program; (j) undertake free, prior, and informed consultations if the Indigenous Peoples are potentially affected (positively or negatively), to determine whether there is broad community support for the Program activities; (k) ensure that the Indigenous Peoples can participate in devising opportunities to benefit from exploitation of customary resources or indigenous knowledge, the latter (indigenous knowledge) to include the consent of the Indigenous Peoples; (l) give attention to groups vulnerable to hardship or disadvantage, including as relevant the poor, the disabled, women and children, the elderly, or marginalized ethnic groups; and, if necessary, take special measures to promote equitable access to Program benefits; and (m) consider conflict risks, including distributional equity and cultural sensitivities. 30.29. The environmental and social systems assessment provides a reference that is used to monitor environmental and social systems performance during Program implementation, Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 128 and identifies actions, as needed, to enhance the systems during Program preparation and implementation (the latter are included in the Program’s action plan). The environmental and social risks, and proposed mitigation measures, as appropriate, are inputs to the integrated risk assessment. The assessment includes a review of the arrangements by which Program activities that affect communities will be disclosed, consulted upon, and subject to a grievance redress process. Measures to address consultation, disclosure and grievance should be appropriate to the activities to be supported under the Program. 31.30. Before Program-for-Results appraisal, and as part of the environmental and social system assessment, the task team makes the draft assessment publicly available. The task team subsequently consults with Program stakeholders on the draft assessment. The Bank makes the final assessment publicly available. In addition, the PID, which is made publically available at the concept and appraisal stages, includes information about the environmental and social issues related to the Program. Integrated Risk Assessment 32.31. At different points during the preparation of the Program-for-Results Ffinancing and subsequently during Program implementation, the Bank prepares, with the Borrower’s inputs, an integrated risk assessment to identify the major risks that the Program will not achieve its results. The Bank determines appropriate mitigation measures, and monitors the evolution of risks, the implementation of mitigation measures, and their impact, making adjustments as appropriate. C. Implementation Support 33.32. Signing. After approval of the Ffinancing, the Bank arranges for signing of the legal agreements as soon as the signing requirements are met. If the legal agreements are not signed within 18 months following approval, the Bank normally withdraws the offer of the Ffinancing. Exceptionally, Management may decide to provide the Borrower with additional time to sign. 34.33. Effectiveness. The legal agreements terminate if the conditions for their effectiveness, if any, are not met by the date specified in the agreements. When warranted, Management may decide to extend the effectiveness deadline; normally the deadline is not extended beyond 18 months after the financing approval. When the effectiveness deadline is extended, dated covenants whose dates fall before the new effectiveness deadline become additional conditions of effectiveness. Any decision by Management to declare the legal agreements effective or to extend the effectiveness deadline is taken before the expiration of the effectiveness deadline. Exceptionally, if the legal agreements have terminated for failure to become effective by the effectiveness deadline, Management may decide to reinstate such agreements with the Borrower’s agreement. 35.34. Informing the Executive DirectorsBoard. For Ffinancing approved by the Executive DirectorsBoard, Management informs the Executive DirectorsBoard as part of regular operational reporting of the following: (a) signing delays of more than six months following approval; (b) withdrawal of the Ffinancing offer; (c) effectiveness deadlines; (d) effectiveness delays of more than nine months after approval by the Executive Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 129 DirectorsBoard; (e) legal agreements that terminate for failure to become effective; and (e) terminated legal agreements that have been reinstated. 36.35. Extensions Following Changes in Conditions Prior to Signing or Effectiveness. If an extension of the deadline for signing or effectiveness involves a substantial departure from the conditions under which the Ffinancing was originally approved, the legal agreements are not signed or declared effective until Bank approval of the new conditions is obtained, through a restructuring. 37.36. Borrower’s Role. The Borrower is responsible for implementing the Program, monitoring its progress, evaluating results on completion, and meeting the relevant contractual obligations (including the Program action plan) set out or referred to in the legal agreements with the Bank. These responsibilities include the requirement to maintain agreed fiduciary, environmental and social, and risk management systems, and to deal in a timely and effective manner with failures (individual or systemic) in these areas. 38.37. Bank’s Role. In providing implementation support to the Borrower in accordance with Section III, paragraph 13 of OP 9.00 the Policy, the task team pays particular attention to reviewing the monitoring and verification of the Program’s results and DLIs, and the Borrower’s compliance with its contractual undertakings in the fiduciary, environmental and social, and risk management areas, including those related to the Program’s action plan. The task team regularly assesses the Program, taking into account the integrated risk assessment developed during the preparation stage, uses the risk assessment to respond to and assist the Borrower to respond to pre-identified and new risks, and recommends adjustments to the risk assessment, as may be appropriate. The task team undertakes field visits, liaises as necessary with relevant partners, provides appropriate support to the Borrower, and reviews Program audit reports and progress reports prepared by the Borrower. The task team regularly reports to Management on Program implementation by sharing the aide-mémoires that reflect review of documents, discussions with the Borrower and relevant partners, and visits to Program sites and facilities. 39.38. Program Financial Statements Audits. Unless otherwise agreed by the Bank, The the Borrower is required to submit annual audited Program financial statements after the close of the Borrower’s financial year. Audits need to be carried out by auditors with independence, experience, and capacity acceptable to the Bank, and under terms of reference acceptable to the Bank. The Bank and Borrower agree on, and confirm in the legal agreement, the period for receipt of the annual audit reports. 40. Additional Reviews. Depending on the nature of the Program and the findings of the assessments, the Program’s annual financial statements audit may need to be complemented by other reviews to address certain risks or to strengthen the Program’s audit/inspection arrangements.13 41.39. Cancellation. The Borrower or the Bank may decide to cancel an amount of Ffinancing in accordance with the provisions of the legal agreements. When the Borrower decides to cancel an amount of Ffinancing, and gives notice to the Bank, the cancellation is 13 Such reviews may involve the following: procurement performance measures/indicators; physical inspection assessing the quality of goods, works and services; verification of DLIs; the efficiency and effectiveness of administrative activities; reviews of fraud and corruption “red flags” and complaint handling mechanisms. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 130 effective as of the date of Bank receipt of the request. The Bank does not accept requests for retroactive cancellations. 42.40. If the Bank cancels an amount of Ffinancing, the cancellation is effective as of the date of the Bank’s cancellation notice to the Borrower, except in the case of cancellation of the remaining unwithdrawn balance of the Ffinancing after the Closing Date, in which case the cancellation is effective on the latest of: (a) the Closing Date; (b) the final date for receipt of withdrawal applications by the Bank; or (c) the final date the financing account was charged for a disbursement or credited for a refund. 43.41. Restructuring. If, as part of Program implementation support, the task team identifies, normally with the Borrower’s input, a need to restructure the Program, the restructuring takes place in accordance with Section III, paragraph 18 of OP 9.00 the Policy. 44.42. Additional Financing. Additional Program-for-Results Ffinancing may be provided by the Bank in accordance with Section III, paragraph 20 of OP 9.00 the Policy. Approved additional Program-for-Results Ffinancing is formalized through a separate legal agreement or, if appropriate, through amendments to the original legal agreement(s). 45.43. Fraud and Corruption. Program-for-Results Ffinancing is subject to the ACGs. In providing Program-for-Results Ffinancing, the Bank has the right to investigate allegations of fraud and corruption in the Program and to sanction parties engaged in sanctionable practices under the Program. 46. The ACGs require the Borrower to report complaints and undertake follow-up actions. When allegations of fraud and corruption under the Program are brought to the attention of the Borrower or the Bank, the Bank’s Integrity Vice Presidency (INT) informs and consults with as many relevant parties as is appropriate given the nature of the allegations and the impact that such discussions would have on any related investigations. Task team members report suspected or reported cases of possible fraud and corruption to INT. Fraud and corruption investigations may be launched by INT in response to complaints received by the Bank directly or through the reporting on complaints received by the Borrower. When the Borrower carries out its own investigation into complaints under the Program, the findings by the responsible investigative agency and proposed actions are shared with INT and, as appropriate, other partners. The Bank monitors the progress in the Borrower’s investigation. 47. When allegations of fraud and corruption are substantiated by the Borrower or the Bank, INT may seek administrative sanctions under the Bank’s sanctions procedures against the individual(s) and/or entity(ies) involved, which may result in: (a) temporary suspension or early temporary suspension in connection with an ongoing sanctions proceeding: (b) debarment or other sanctions: (c) cross-debarment as agreed with other international financial institutions, including multilateral development banks: and (d) debarment by the World Bank Group’s General Services Department. Based on its own findings or those provided by the Borrower, INT makes referrals to relevant stakeholders in accordance with its policies and practices. 48. The Borrower must undertake not to allow parties on the Bank’s debarment or suspension lists to be awarded contracts under, or otherwise initiate participation in, the Program during their debarment/suspension period. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 131 49.44. Remedies. The Bank’s legal remedies are specified in the legal agreements. If the task team identifies breaches in the Borrower’s compliance with its contractual obligations, including undertakings in the fiduciary, environmental and social, and risk management areas, including those related to the Program’s action plan, the task team informs Management and makes appropriate recommendations. Management determines whether to exercise legal remedies in accordance with Section III, paragraph 14 of OP 9.00 the Policy. Normally, before exercising its remedies, the Bank provides the Borrower with an opportunity to take timely and appropriate corrective measures acceptable to the Bank. 50.45. Extension of Closing Date. Upon a request from the Borrower, the Bank may decide to extend the Closing Date if: (a) the Program development objectives remain achievable; (b) the performance of the Borrower remains satisfactory, and (c) the Bank and the Borrower agree on actions that will be undertaken by the Borrower to complete the Program. 51.46. Withdrawals after the Closing Date. The Bank may decide, without formally extending the Closing Date, to disburse or approve the use of proceeds of Ffinancing for withdrawal applications received within six months after the Closing Date for DLIs achieved by the Borrower prior to the Closing Date. Exceptionally, upon the Borrower’s request, the Bank may decide to extend the period for receipt of such withdrawal applications. 52.47. Closing the Program Financing Account. The Bank closes the Ffinancing account within two months after the deadline set by the Bank for receipt of withdrawal applications or, if no such additional period is granted, within two months after the Closing Date. Any undisbursed balance of the Ffinancing is cancelled. 53.48. Program-for-Results Financings under Suspension of Disbursements. If a suspension of disbursements is in effect on the Closing Date, any unwithdrawn Ffinancing balance is normally canceled and the Ffinancing account is closed. Exceptionally, Management may decide to authorize a delay in canceling the balance and closing the Ffinancing account if suspension is likely to be lifted imminently and Program and/or country circumstances warrant such a delay. Once the Bank decides to lift the suspension, Management may decide to approve an extension of the Closing Date. 54.49. Program Completion Report. On completion of the Program, the task team prepares an Implementation Completion and Results report (ICR). This covers, among other things, the degree to which the Program objectives and results have been achieved and the overall Program performance, including the achievement of the DLIs. The ICR incorporates the Borrower’s evaluation of the Program and Program -for-Results Ffinancing, as well as of its own performance and the performance of the Bank. Program implementation support and monitoring carried out by the Bank during the implementation period end at completion of the Program. Section IV – Waiver A provision of this Directive may be waived in accordance with Bank Policy, “Operational Policies Waivers”, April 7, 2014, Catalogue No. OPCS5.06-POL.01. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 132 Section V – Effective Date This Policy is effective _________, 2015. Section VI – Issuer The Issuer of this Directive is the Vice President and Head of Network, OPCS. Section VII – Sponsor The Sponsor of this Directive is the Director, Operations Policy and Quality Department. Section VIII – Related Documents 1. Bank Policy, “Program-for-Results”, ____, 2015, Catalogue No. OPCS__. 2. Program-for-Results: Two-Year Review, __2015 3. A New Instrument to Advance Development Effectiveness, Program-for-Results Financing, December 29, 2011 4. Guidelines on Preventing and Combating Fraud and Corruption in Program-for-Results Financing (__ 2015). 5. The World Bank Policy on Access to Information, July 1, 2013. 6. Program-for-Results operations are governed by this Policy, the related Directive, and the following OPs/Policies and BPs/Directive (including any relevant Operational Memoranda and Instructions/Procedures [internal use only]), as applicable: OP 1.00, Poverty Reduction; World Bank Group Directive Country Engagement; OP/BP 2.30, Development Cooperation and Conflict; OP/BP 3.10, Financial Terms and Conditions of IBRD Loans, IBRD Hedging Products, and IDA Credits; OP 4.07, Water Resource Management; OP/BP 4.20, Gender and Development; OP 4.76, Tobacco; OP 7.00, Lending Operations: Choice of Borrower and Contractual Agreements; OP7.20, Security Arrangements; OP /BP 7.30, Dealing with De Facto Governments; OP/BP 7.40, Disputes Over Defaults on External Debt, Expropriation, and Breach of Contract; OP/BP 7.50, Projects on International Waterways; OP/BP 7.60, Projects in Disputed Areas; OP/BP 8.45, Grants; OP/BP 10.20, Global Environmental Facility Operations; OP/BP; OP/BP 14.10, External Debt Reporting and Financial Statements; OP/BP 14.20, Cofinancing; OP/BP 14.40, Trust Funds; BP 17.30, Communications with Executive Directors; and BP 17.55, Inspection Panel. Questions regarding this Directive should be addressed to operations help desk. Bank Directive, “Program-for-Results Financing” | Catalogue Number __________ Page _ of 16 133 ANNEX 7B. TECHNICAL, EDITORIAL, AND SUBSTANTIVE CHANGES Draft Bank Directive, Program-for-Results Financing (currently BP 9.00) Technical, Editorial Reference to Relevant and Substantive Explanatory Notes Paragraph concerned Changes (by topic) Entire document Compliance with In light of the Policy and Procedure Framework Policy, Directive Policy and and Procedure (P&P) (effective January 8, 2014), BP 9.00 is Procedure proposed to be retrofitted to conform with the P&P rules. Framework: Accordingly, all definitions and acronyms are proposed to be listed in Section II of the Directive instead of within the body of the Directive or within footnotes as they were in BP 9.00; if a term was defined or acronym listed in the Policy, they were not defined again in the Directive; defined terms and acronyms are capitalized; undefined terms put in lowercase; references in the text to “OP” would be changed to “Policy” and the term “BP” would be changed to “Directive”; references to other OP/BPs in the Pedigree Box of BP 9.00 are proposed to be listed in Section VIII, Related Documents; and reference to Access to Information Policy is proposed to be in Section VIII, Related Documents. Section III, paragraph 5 Substantive Change As noted in paragraph 93 of this paper explaining the proposed (currently paragraph 5 adjustment to the procurement exclusion, the addition to this in BP 9.00) paragraph explains that a contract is deemed to be “modest” (the term to be used in the Policy) if its estimated monetary value is equal to or less than 25 percent of the estimated total Program expenditures. Section III, paragraph 16 Substantive Change This paragraph links Prior Results to Advances. The linkage has served (currently paragraph 15 as a disincentive for Clients to use the option of Prior Results. Linking in BP 9.00) Prior Results and Advances is an artificial and unnecessary measure that is (i) counter to the flexibility needed by the Client, particularly in IDA countries, to manage their program; and (ii) it is inconsistent with other Bank instruments where project needs are the driver for the advances, and retroactive financing is done independently and subject to a discrete cap. Given the fact that the Client would need to refund the Bank for any Advances received where DLIs are not achieved there is no incentive for Clients to request advances that are not required. And funds are only disbursed for Prior Results if they have been achieved and verified. It is therefore proposed to delete this paragraph. Section III, paragraph Miscellaneous As noted in Table 4(b) which describes the changes to the Policy, the 4(e) and footnote 5 (redundant text; phrase “…governance” and the accompanying footnote are (currently paragraph 4(e) consistency across duplicative and the proposed revisions will ensure consistency of and footnote 5 in BP policies) definitions and thus it is proposed to delete the phrase and the 9.00) accompanying footnote. 134 Technical, Editorial Reference to Relevant and Substantive Explanatory Notes Paragraph concerned Changes (by topic) Section III, footnote 7 Miscellaneous The information in the footnote regarding corporate reviews is (currently footnote 7 in (redundant text; duplicative as such information is currently included in Instructions BP 9.00) consistency across (and will subsequently be issued as Procedures). This level of detail is policies) not appropriate for a Directive and thus it is proposed to delete the footnote. This is a technical nonsubstantive change. Section III, paragraph 10 The information regarding the list of documents to be submitted for and footnote 8 decision reviews is duplicative as it is included in Instructions (and will (currently paragraph 10 subsequently be issued as Procedures). This level of detail is not and footnote 8 in BP appropriate for a Directive and thus it is proposed to delete the 9.00) paragraph and footnote. This is a technical nonsubstantive change. Section III, paragraph 17 The last sentence of the paragraph explains what to include in the (currently paragraph 17 Memorandum of the President (MOP) and is duplicative as such in BP 9.00) information is already in the Access to Information Policy, in addition a template for the MOP is provided to staff. Thus it is proposed to delete the sentence. Section III, paragraph 18 The text regarding disclosure requirements of the PAD is duplicative (currently paragraph of the Access to Information Policy and also Instructions (and will 178in BP 9.00) subsequently be issued as Procedures) and templates. Thus it is proposed to delete the last sentence. Section III, footnote 11 It is proposed to delete, “For more details, see the OPCS Processing (currently footnote 11 in Steps Guidelines and SECPO guidance detailing eSubmission of Board BP 9.00.) documents.” This level of detail not appropriate for the Directive. Technical nonsubstantive change. Section III, paragraph 25 The second sentence in the paragraph which states, “The task team (currently paragraph 25 evaluates the Program fiduciary systems to determine whether they in BP 9.00) provide reasonable assurance that the Program expenditures will be used appropriately to achieve their intended purpose,” is already in the Policy and is repetitive and redundant. The proposal is therefore to delete this. Section III, paragraph It is proposed to delete the phrase “including complaint mechanisms, 26(currently paragraph and how such risks are managed and mitigated.” as it is already in the 26 in BP 9.00) Policy and thus redundant. Section III, paragraph 39 It is proposed to add the following phrase, “Unless otherwise agreed (currently paragraph 39 by the Bank,” to the first sentence so as to be consistent with other in BP 9.00) policies. 135 Technical, Editorial Reference to Relevant and Substantive Explanatory Notes Paragraph concerned Changes (by topic) Section III, paragraphs Miscellaneous These paragraphs repeat/rephrase the text of the ACGs relating to 46-48 (currently (redundant text; responding to allegations of fraud and corruption as well as provisions paragraphs 46-48 in BP consistency across under the Staff Rules which require staff to report fraud and 9.00) policies) corruption allegations to INT. In addition, language regarding INT informing and consulting widely is duplicative of their normal business practice. It is therefore proposed to delete the specific paragraphs. Section III, paragraph Reclassification The paragraph and footnote on “Additional Reviews” is guidance 40 and footnote 13 material. It is proposed to delete the paragraph and the footnote (currently paragraph 40 and incorporate into guidance material. and footnote 13 in BP 9.00) Section III, 17, 18, 35 Miscellaneous It is proposed to change the term Executive Directors to Board. (currently paragraphs (Editorial Changes) 17, 18, 35in BP 9.00) Section III, paragraph It is proposed to include the term “CPF.” 4(a) and 20(b) (currently paragraphs 4(a) and 20 (b) in BP 9.00) Section III, Footnote 12 It is proposed to move the text, “Such a protocol may be (currently footnote 12 in formulated in various forms, for example as a covenant in the legal BP 9.00) agreement between the Bank and borrower, letter or memorandum of understanding, or as part of the minutes of negotiations” from the footnote into the last line of Section III, paragraph 25 (BP 9.00 paragraph 26) Section III, paragraph It is proposed to delete the word “also” from the first sentence; It 26 (currently paragraph is proposed to revise the phrase “The Bank an d Borrower will, 26 in BP 9.00) where required…” to “The Bank and the Borrower may…” 136 ANNEX 8A. GUIDELINES ON PREVENTING AND COMBATING FRAUD AND CORRUPTION IN PROGRAM-FOR-RESULTS FINANCING Dated February 1, 2012 and Revised _____, 2015 Purpose and General Principles 1. These Guidelines address fraud and corruption that may occur in connection with the preparation and implementation of programs financed, in whole or in part, by the International Bank for Reconstruction and Development (IBRD) or the International Development Association (IDA) through Program-for-Results financing. They set out the general principles, requirements, and sanctions applicable to such programs. 2. The Loan Agreement40 providing for the Loan41 governs the legal relationships between the Borrower42 and the Bank43 with respect to the Program44 for which the Loan is made. The responsibility for the implementation of the Program under the Loan Agreement, including the primary responsibility for preventing and combating fraud and corruption, rests with the Borrower. The Bank, for its part, has a fiduciary duty under its Articles of Agreement to “make arrangements to ensure that the proceeds of any loan are used only for the purposes for which the loan was granted, with due attention to considerations of economy and efficiency and without regard to political or other non-economic influences or considerations.”45 These Guidelines constitute an important element of those arrangements and are made applicable to the preparation and implementation of the Program as provided in the Loan Agreement. 23. Recognizing that fraud and corruption leads to wasted resources and undermines development effectiveness, the Bank and the Borrower agree that Aall individuals and entities participating in t h e P programs financed through Program-for-Results financing must observe the highest standard of ethics and,. Sspecifically, that all such persons and entities must take all appropriate measures to prevent and combat fraud and corruption, and refrain from engaging in fraud and corruption, in connection with such the pProgram. In furtherance of these principles and purposes, the Bank and the Borrower further agree and commit to undertaking the actions set out in these Guidelines for the purpose of preventing and combating fraud and corruption in connection with the Program.s. 40 References in these Guidelines to “Loan Agreement” include any Financing Agreement providing for an IDA credit or IDA grant; Trust Fund Grant Agreement or Loan Agreement providing for a recipient-executed trust fund grant or loan in cases where these Guidelines are made applicable to such agreement; and the Program Agreement with a Program Implementing Entity related to any of the above. 41 References to “Loan” or “Loans” include IBRD loans as well as IDA credits and grants, project preparation advances, and recipient-executed trust fund grants or loans for programs to which these Guidelines are made applicable under the agreement providing for such grant and/or loan. These Guidelines do not apply to investment lending (to which separate guidelines apply) or to development policy lending. 42 References in these Guidelines to “Borrower” include the recipient of an IDA credit or grant or of a trust fund grant or loan. 43 References in these Guidelines to the “Bank” include both IBRD and IDA. 44 Reference in these Guidelines to the “Program” means the Program as defined in the Loan Agreement. 45 IBRD Articles of Agreement, Article III, Section 5(b); IDA Articles of Agreement, Article V, Section 1(g). 137 Legal Considerations 3. The Loan Agreement46 providing for a Loan47 governs the legal relationships between the Borrower48 and the Bank49 with respect to the Program50 for which the Loan is made. The responsibility for the implementation of the Program under the Loan Agreement rests with the Borrower. The Bank, for its part, has a fiduciary duty under its Articles of Agreement to “make arrangements to ensure that the proceeds of any loan are used only for the purposes for which the loan was granted, with due attention to considerations of economy and efficiency and without regard to political or other non-economic influences or considerations.”51 These Guidelines constitute an important element of those arrangements and are made applicable to the preparation and implementation of the Program as provided in the Loan Agreement. Definitions of Practices Constituting Fraud and Corruption 44. These Guidelines address the following defined practices in connection with the Program:52 (a) A “corrupt practice” is the offering, giving, receiving, or soliciting, directly or indirectly, of anything of value to influence improperly the actions of another party.53 (b) A “fraudulent practice” is any act or omission, including a misrepresentation, that knowingly or recklessly 54 misleads, or attempts to mislead, a party to obtain a financial or other benefit or to avoid an obligation. 46 References in these Guidelines to “Loan Agreement” include any Guarantee Agreement providing for a guarantee by the Member Country of an IBRD Loan; Financing Agreement providing for an IDA credit or IDA grant; agreement providing for a project preparation advance or Institutional Development Fund (IDF) grant; Trust Fund Grant Agreement or Loan Agreement providing for a recipient-executed trust fund grant or loan in cases where these Guidelines are made applicable to such agreement; and the Program Agreement with a Program Implementing Entity related to any of the above. 47 References to “Loan” or “Loans” include IBRD loans as well as IDA credits and grants, project preparation advances, IDF grants, and recipient-executed trust fund grants or loans for programs to which these Guidelines are made applicable under the agreement providing for such grant and/or loan. These Guidelines do not apply to investment lending (to which separate guidelines apply) or to development policy lending. 48 References in these Guidelines to “Borrower” include the recipient of an IDA credit or grant or of a trust fund grant or loan. In some cases, an IBRD loan may be made to an entity other than the Member Country. In such cases, references in these Guidelines to “Borrower” include the Member Country as Guarantor of the Loan, unless the context requires otherwise. In some cases, the Program, or a part of the Program, is carried out by a Program Implementing Entity with which the Bank has entered into a Program Agreement. In such cases, references in these Guidelines to the “Borrower” include the Program Implementing Entity, as defined in the Loan Agreement. 49 References in these Guidelines to the “Bank” include both IBRD and IDA. 50 Reference in these Guidelines to the “Program” means the Program as defined in the Loan Agreement. 51 IBRD Articles of Agreement, Article III, Section 5(b); IDA Articles of Agreement, Article V, Section 1(g). 52 Unless otherwise specified in the Loan Agreement, whenever these terms are used in the Loan Agreement, including in the applicable General Conditions, they have the meanings set out in paragraph 4 of these Guidelines. 53 Typical examples of corrupt practice include bribery and “kickbacks.” 54 To act “knowingly or recklessly,” the fraudulent actor must either know that the information or impression being conveyed is false, or be recklessly indifferent as to whether it is true or false. Mere inaccuracy in such information or impression, committed through simple negligence, is not enough to constitute fraudulent practice. 138 (c) A “collusive practice” is an arrangement between two or more parties designed to achieve an improper purpose, including to influence improperly the actions of another party. (d) A “coercive practice” is impairing or harming, or threatening to impair or harm, directly or indirectly, any party or the property of the party to influence improperly the actions of a party. (e) An “obstructive practice” is (i) deliberately destroying, falsifying, altering or concealing of evidence material to the investigation or making false statements to investigators in order to materially impede a Bank investigation55 into allegations of a corrupt, fraudulent, coercive, or collusive practice; and/or threatening, harassing or intimidating any party to prevent it from disclosing its knowledge of matters relevant to the investigation or from pursuing the investigation, or (ii) acts intended to materially impede the exercise of the Bank’s contractual rights of audit or access to information. 55. The above practices, as so defined, are sometimes referred to collectively in these Guidelines as “fraud and corruption.” Borrower Actions to Prevent and Combat Fraud and Corruption in Connection with the Program 66. In furtherance of the above-stated purpose and general principles, except as otherwise agreed in writing by the Borrower and the Bank, the Borrower will: (a) takes all appropriate measures to ensure that the Program is carried out in accordance with these Guidelines; (b) takes all appropriate measures to prevent fraud and corruption in connection with the Program, including (but not limited to) adopting and implementing appropriate fiduciary and administrative practices and institutional arrangements to ensure that the proceeds of the Loan are used only for the purposes for which the Loan was granted; (c) promptly informs the Bank of all credible and material allegations or other indications of fraud and corruption in connection with the Program that come to its attention, together with the investigative and other actions that the Borrower proposes to take with respect thereto; (d) unless otherwise agreed by the Borrower and with the Bank with respect to a particular case, and consistent with the applicable law of the Member Country, takes timely and appropriate action to investigate such allegations and indications; 55 As used in the definition of “obstructive practice”, the term “investigation” includes any inquiry undertaken under these Guidelines. 139 reports to the Bank on the actions taken in any such investigation, at such intervals as may be agreed between the Borrower and the Bank; and, promptly upon the completion of any such investigation, reports to the Bank the findings thereof; (e) if the Borrower or the Bank determines that any person or entity has engaged in fraud and corruption in connection with the Program, takes timely and appropriate action, consistent with the applicable law of the member country and satisfactory to the Bank, to remedy or otherwise address the situation and prevent its recurrence; provided that nothing in this sub-paragraph (e) or in sub-paragraph (d) above obligates the Borrower to take action in direct contradiction of the applicable law of the Member Country; (f) cooperates fully with representatives of the Bank in any investigation inquiry conducted by the Bank into allegations or other indications of fraud and corruption in connection with the Program, and takes all appropriate measures to ensure the full cooperation of relevant persons and entities subject to the Borrower’s jurisdiction in such investigationinquiry, including, in each case, allowing the Bank to meet with relevant persons and to inspect all of their relevant accounts, records and other documents and have them audited by or on behalf of the Bank; and (g) ensures that any person or entity debarred or suspended by the Bank is not awarded a contract under or otherwise allowed to participate56 in the Program during the period of such debarment or suspension. Sanctions and Related Actions by the Bank in Cases of Fraud and Corruption 77. In furtherance of the above-stated purpose and general principles, except as otherwise agreed in writing by the Borrower and the Bank, the Bank will: (a) promptly informs the Borrower of all credible and material allegations or other indications of fraud and corruption in connection with the Program that come to its attention, consistent with Bank policies and procedures; (b) in cases where the Bank determines it necessary to do so to fulfill its fiduciary duty, will be permitted mayhave the right to conduct an investigation inquiry into of such allegations or other indications, independently of or in collaboration with the Borrower;; (c) reports to the Borrower on the outcome of any such investigationinquiry; and 56 For purposes of paragraph 6(g), participation does not include the performance under contracts entered into or other engagements began prior to the date of the Loan Agreement 140 (d) have the right tomay sanction57, in accordance with its prevailing sanctions policies and procedures, any individual or entity other than the Member Country58— including (but not limited to) publicly declaring such individual or entity ineligible, either indefinitely or for a stated period of time, to: (i) be awarded a Bank- financed contract; (ii) benefit from a Bank-financed contract, financially or otherwise, for example as a subcontractor; and (iii) otherwise participate in the preparation or implementation of the Program or any other project or program financed, in whole or in part, by the Bank— if at any time the Bank determines that such individual or entity has engaged in fraud and corruption in connection with the Program or any other Bank-financed activity, or is otherwise subject to sanction pursuant to its prevailing policies and procedures.59 8. The Bank also has the right to sanction individuals and entities other than the Member Country if: (a) at any time the Bank determines that such individual or entity has engaged in fraud and corruption in connection with any other Bank-financed activity; (b) another financier with which the Bank has entered into an agreement for the mutual enforcement of debarment decisions has declared such individual or entity ineligible to receive proceeds of financings made by such financier or otherwise to participate in the preparation or implementation of any project financed in whole or in part by such financier as a result of a determination by such financier that the individual or entity has engaged in fraudulent, corrupt, coercive, or collusive practices in connection with the use of the proceeds of a financing made by such financier; (c) the Bank’s General Services Department has found the individual or entity to be non- responsible on the basis of fraud and corruption in connection with World Bank Group corporate procurement pursuant to the Bank’s Vendor Eligibility Policy; or (d) at any time the Bank determines that such individual or entity has violated a material term of the terms and conditions of the Bank’s Voluntary Disclosure Program. Miscellaneous 8. For avoidance of doubt, nothing in these Guidelines is intended to restrict or otherwise affect the Member Country’s sovereign right to investigate, prosecute or take any other action in 57 Sanctions include (but are not limited to) publicly declaring such individual or entity ineligible, either indefinitely or for a stated period of time, to: (i) be awarded a Bank-financed contract; (ii) benefit from a Bank-financed contract, financially or otherwise, for example as a subcontractor; and (iii) otherwise participate in the preparation or implementation of the Program or any other project or program financed, in whole or in part, by the Bank. The Bank may publish the identity of any individual or entity sanctioned under subparagraph 7(d). 58 For purposes of these Guidelines, “Member Country” includes (i) officials and employees of the national government or of any of its political or administrative subdivisions, and (ii) non-autonomous government- owned enterprises. 59 The sanction may, without limitation, also include restitution of any amount of the loan with respect to which sanctionable conduct has occurred. The Bank may publish the identity of any individual or entity declared ineligible under subparagraph 7(c). 141 furtherance of its own laws and regulations. Any inquiries conducted by the Bank pursuant to these Guidelines are administrative in nature, for the purpose of determining compliance with the Bank’s policies, directives and procedures. Inquiries include, but are not limited to, the review of relevant accounts, records and other documents, and interviews with relevant persons. 9. Without prejudice to any provision hereof, in the event that any action to be taken by the Borrower under these Guidelines may conflict with requirements of the applicable laws and regulations of the Member Country, the Bank and the Borrower will consult with a view to identifying and agreeing on alternative actions that will avoid such conflict while ensuring compliance herewith. 109. The provisions of these Guidelines do not limit any other rights, remedies60 or obligations of the Bank or the Borrower under the Loan Agreement or any other document to which the Bank and the Borrower are both parties. 60 The Loan Agreement provides the Bank with certain rights and remedies that it may exercise with respect to the Loan in the event of fraud and corruption in connection with the Program, in the circumstances described therein. 142 ANNEX 8B. SUMMARY OF CHANGES TO ACGS TECHNICAL, EDITORIAL, AND SUBSTANTIVE CHANGES Anticorruption Guidelines (ACGs) for Program-for-Results Financing Technical, Editorial and Reference to Relevant Substantive Changes Explanatory Notes Paragraph Concerned (by topic) Paragraph 3 (was Clarification. It is proposed to add a recitation regarding the harm that paragraph 2) fraud and corruption (F&C) does, together with a joint statement that the Bank and the Borrower agree that program participants must observe the highest standards of ethics, refrain from F&C. This would replace a unilateral statement of Bank policy, emphasizing that the Borrower and the Bank have a common interest in preventing and addressing F&C in the Program. It also addresses one common perception of the current ACGs among borrowers: that it imposes one-sided conditions, contrary to the purported spirit of the PforR instrument. This change and a number of the following are intended to set a different tone and frame the subject matter from a more balanced and cooperative starting point. Paragraph 3 (was Clarification It is proposed to add a specific reference that the Borrower paragraph 2) has primary responsibility for addressing F&C issues, consistent with Borrower’s overall ownership of the Program and with the spirit of the PforR instrument. This was always the implied understanding, but this new drafting makes it explicit. Paragraph 6(e) Clarification It is proposed to add a proviso that Borrower investigative and remedial actions in reaction to F&C should not be in “direct contradiction of the applicable law”, recognizing that, in the overall spirit of the instrument, Borrowers will use national rules and systems to deal with F&C cases. Paragraph 6(f) and Editorial Change A change in nomenclature is proposed, from Bank throughout the “investigation” to “inquiry”. The term “investigation” has document criminal law connotations and led to confusion that the Bank was trying to replace local criminal justice institutions and processes. The term “administrative fact-finding inquiry” is the one that INT itself uses to describe the nature of their investigations in referral and redacted reports. Paragraph 6(f) Editorial Change It is proposed to move the mention of access to books and records, relevant persons from here to paragraph 8, where the nature of Bank inquiries is described. 143 Technical, Editorial and Reference to Relevant Substantive Changes Explanatory Notes Paragraph Concerned (by topic) Paragraph 7(b) Clarification It is proposed to add a proviso that Bank investigations are undertaken when necessary to fulfill the fiduciary duty, clarifying that the purpose of Bank investigations is fiduciary in nature and providing assurance to Borrowers that investigations will not be launched indiscriminately. The provision nevertheless leaves the discretion to decide when an inquiry is required to the Bank. Paragraph 7(b) Editorial Change It is proposed to change the phrase that the Bank will “have the right” to investigation to specify that the Bank “may conduct an inquiry”. While the result is the same, this change in phraseology emphasizes the Bank’s respect for national sovereignty and the Borrower’s primary role in Program implementation, while avoiding any suggestion that inquiries require case-by-case consent. Paragraphs 7(d) and 8 Editorial Change Without changing the substance, the language regarding Bank sanctions is now significantly less detailed. While there was no specific objection to this language from stakeholders, the sheer length of the sanctions language put too much emphasis on the Bank’s role at the (apparent) expense of the Borrower’s. New paragraph 8 Clarification It is proposed to add a new paragraph 8 to clarify that the ACGs do not affect the borrower’s sovereign rights to enforce national/local laws, and that Bank investigations are for administrative fact-finding for the purpose of ascertaining compliance with Bank policies, not a substitute for local authorities. This language also addresses another common concern: the apparent lack of recognition of borrower laws and systems. Underlying these concerns is the fact that F&C is a criminal justice matter in most (if not all) borrowing countries. The Bank’s concerns, on the other hand, in F&C are fiduciary/ administrative in nature. These changes are intended to avoid the conflation of these magisteria. While these issues have been worked out case-by-case, adding explicit language to this effect is aimed at reducing transaction costs going forward. This paragraph also specifies that Bank inquiries include (among other things) review of books and records, as well as interviews with relevant persons. New paragraph 9 Addition It is proposed to add a provision calling for consultations if/when any action required under the ACGs comes into conflict with national laws. This change provides a specific mechanism for discussing and resolving ACG implementation issues that may come up, signaling Bank willingness to be flexible in finding mutually acceptable modalities, while taking care to specify that the objectives remain the same. 144