74652 Approach Paper The World Bank and Public Procurement: An Independent Evaluation September 5, 2012 Background 1. Public expenditure is a key policy tool, and good procurement practices are a major determinant of its effectiveness. On behalf of their citizens, governments typically spend between 5 and 20 percent of their gross domestic product on the procurement of goods, services, and civil works, and a difference between effective and less effective procurement policies that implies savings of even 1 percent of gross domestic product represents a significant part of government budgets.1 In light of its mission of reducing poverty, which in turn hinges on effective development, the World Bank Group has a clear interest in seeing good public procurement practices—ones combining value for money, efficiency, and transparency—prevail universally in its client countries. Good national procurement practices are also essential to the success of the Bank’s newly introduced lending instrument, the Program for Results (P4R) and, in the longer term, offer a natural possibility around which procurement practices can be unified across lending instruments, notably investment projects and development policy operations (DPOs). National public procurement practices in recipient countries have been viewed as a vehicle around which diverse donor practices can be unified. In parallel, partnerships in trade and aid emphasize the need for increased harmonization in global procurement practices.2 The main question for the planned evaluation is, therefore, how effectively the World Bank has helped build well-functioning public procurement systems, essential for effective development. 2. The World Bank Group has also affected public procurement practices in and by its client countries through its policies and procedures for procurement under Bank-financed investment operations. In requiring these procurement policies and procedures, the Bank seeks to ensure that the funds it provides are used for the purpose intended and that they achieve their aim of supporting development as effectively and efficiently as possible. Procurement policies and procedures function as one element of the Bank’s fiduciary safeguards to prevent corruption and ensure the integrity of Bank lending. A second question to be pursued by the evaluation concerns how well the policies and procedures for public procurement under Bank-financed investment projects have fulfilled these aims, and whether changes in these policies and procedures would improve development outcomes. STRENGTHENING NATIONAL PROCUREMENT SYSTEMS THROUGH LENDING AND TECHNICAL ASSISTANCE 3. Bank support to client countries in strengthening national procurement systems has taken several forms.3 The strengthening of national procurement systems in this paper refers to the full set of institutional arrangements—the rules, norms, and procedures, both formal and 1 A Practical Guide to Transforming Procurement Systems, OECD DAC Task Force on Procurement (2011). 2 See, for example, Understanding Public Procurement, Governance brief, Asian Development Bank (2002). 3 OPSOR has undertaken an initial review of procurement capacity building (2011). 1 informal—governing public procurement, as well as the human and other resources and the ways in which they are organized to undertake public procurement transactions. Through its investment lending, the Bank initially supported the establishment of project implementation units for the ring-fenced execution of Bank-financed projects, sometimes complemented by capacity-building support to help ensure client capacity to undertake procurement and to learn by doing. The Bank also undertook a limited number of technical assistance investment loans that focused on strengthening borrower countries’ procurement systems (Table 1). Only six operations, however, had a primary focus (50 percent or more) on strengthening procurement.4 More numerous were a series of Institutional Development Fund (IDF) grants that focused centrally on procurement reform or on overall public sector management, including specific elements related to procurement embedded within this framework. More than 200 IDF grants refer to public expenditure management–related themes, with around 62 over the last decade bearing titles that refer explicitly to procurement (Table 2). Table 1. Investment Lending with Procurement Elements (2002–11) Region (number of projects) Procurement content (%) AFR EAP ECA LCR MNA SAR Total <10 57 19 21 34 5 22 158 10-30 11 2 4 3 1 21 30-50 2 4 2 8 >50 1 1 4 6 Total 71 21 26 41 6 28 193 Source: IEG analysis of World Bank Business Warehouse data. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. Table 2. IDF Grants with Procurement Elements (2002–11) Grants related to public Number of Grants referring explicitly to expenditure, financial management, procurement related procurementb and procurementa IDF grants in a given country Number of Number of Number of countries Number of countries grants grants 1 43 43 28 28 2–4 91 36 34 15 >4 86 12 — — Total 220 91 62 43 Source: IEG analysis of World Bank Business Warehouse data. Note: IDF = Institutional Development Fund. a. All projects under “Theme 27," which consists of public expenditure, financial management, and procurement. 4 Including, prominently, three operations focused entirely on procurement reform, of which two were in Bangladesh (2002, 2007) and one in Bhutan (2008). 2 b. These grants include the word 'Procurement' in the title. 4. Support for strengthening national procurement systems has also been offered through the Bank’s DPOs. DPOs are disbursed directly into countries’ budget systems, based on programs of agreed policy reforms. DPOs that focus on fiscal management or public sector reform sometimes include procurement reform issues in their scope of policy reform. Around 200 DPOs over the past decade have had prior actions that refer explicitly to procurement-related policy reforms (Table 3). DPOs themselves are not bound by Bank procurement procedures, as they are not linked to the purchase of specific goods or services. Countries’ procurement systems are reviewed prior to loan approval, and improvements suggested as a part of loan implementation.5 There was a sharp increase in the share of DPOs in overall Bank lending in response to the 2008–09 global economic crisis. Table 3. Development Policy Operations with Procurement Elements (2002–11) Operations with prior actions relating to public Operations with prior actions referring explicitly to expenditure, financial procurementb management, and Fiscal No. Commitments procurementa year projects ($millions) Average Average share of share of No. Commitments No. Commitments procurement “Theme 27” projects ($millions) projects ($millions) prior actions prior (%) actions (%) 2002 44 9,794 33 7,185 17 4,756 7.4 20.1 2003 45 6,000 30 2,960 15 1,651 9.5 23.5 2004 40 6,132 30 4,172 18 1,671 10.2 25.8 2005 55 6,528 41 3,571 21 2,039 10.6 24.9 2006 50 7,099 37 4,103 22 2,250 9.4 25.9 2007 57 6,280 47 4,637 22 1,680 12.7 29.1 2008 45 6,464 37 5,068 16 1,999 11.6 25.8 2009 67 17,571 48 12,735 28 5,459 12.7 31.3 2010 79 21,056 54 11,554 25 4,584 15.3 35.4 2011 62 9,615 45 5,839 18 3,506 14.4 35.4 Total 544 96,539 402 61,825 202 29,595 11.6 28.2 Source: IEG analysis of World Bank Business Warehouse data. a. All projects with components under 'Theme 27," which consists of public expenditure, financial management, and procurement. b. These DPOs include the word “procurement” in the prior action. 5 Article 19.b; OP 8.60 (Development Policy Lending), World Bank. 3 5. Recently, the Bank announced the introduction of a new results-based lending instrument, the Program-for-Results (P4R). The P4R opens up new opportunities for the use of country systems within the scope of a specific sector or program, based on the strengthening of country procurement capacity.6 The evolution of Bank lending instruments therefore appears to point to the importance of helping countries build their own procurement systems. BUILDING PROCUREMENT CAPACITY THROUGH ANALYTICAL AND ADVISORY WORK 6. In parallel to its support to countries via lending and technical assistance, the Bank has provided advisory services in the form of diagnoses of national procurement systems. In 1999, the Bank introduced a requirement that a Country Procurement Assessment Report (CPAR), based on a standardized approach and methodology, be undertaken on a regular basis in each active borrower country. CPARs were intended to help analyze the existing procurement policies, organization, and procedures of Bank borrowers and to help them develop or modify their systems to (i) increase their capacity to plan, manage, and monitor the procurement process effectively; (ii) improve the accountability, integrity, and transparency of the process and reduce the scope for corruption; and (iii) be consistent with internationally accepted principles and practices.7 Between 2002 and2006, 89 CPARs were undertaken, averaging about 15 per year. From FY05 onward, CPARs ceased to be mandated, though another 26 CPARs (or about 4 per year) have been undertaken since FY06, at the discretion of country departments (Table 4). 7. Although the CPAR declined in use, new diagnostic tools evolved. There was a move to undertake more integrated diagnostics of public procurement and public financial management systems, given their close interlinkage within the wider framework of public financial management and budgetary processes. The Bank’s Public Expenditure and Financial Accountability Secretariat developed a diagnostic tool that included a specific indicator on public procurement. Meanwhile, a new standardized methodology for the diagnosis of procurement capacity was developed at the Organization for Economic Co-operation and Development– Development Assistance Committee (OECD-DAC), known as the Methodology for Assessing Procurement Systems (MAPS). The MAPS developed as a product of the Working Group on Aid Harmonization, in response to the 2005 Paris Declaration for Aid Effectiveness. The Declaration gave prominence to the need for country-level harmonization of aid arrangements of different donors, particularly with regard to procurement. The Bank was a significant participant in the process and contributed to the design of the new product. Introduced in mid-2006, it is estimated to have been applied to some 60 countries; of these, around 22 are reported on the OECD website and a larger number are available in the Bank.8 MAPS exercises are increasingly embedded in recent Bank CPARs, though some are also stand-alone assessments. 6 P4R was approved by the Bank’s Board on January 24, 2012. The Bank’s modernization program has been recently described in the “Update on the Bank’s Business Modernization: Results, Openness, and Accountability Spring 2012”; (IMF World Bank Development Committee April 11, 2012), which puts procurement reform on the agenda for simplification (p. 15). 7 Guidelines for Country Procurement Assessment Reviews, World Bank (2002). 8 Those reported on the website were undertaken with OECD support; however, many countries have now undertaken these themselves, with no OECD involvement. Those available to the public on the OECD-DAC website include Afghanistan, Albania, Botswana, Cameroon, Ghana, Indonesia, Lao PDR, Mongolia, Niger, Paraguay, the Philippines, Rwanda, Senegal, Sierra Leone, Sri Lanka, Tanzania, Vietnam, and the Republic of Yemen. 4 Table 4. Procurement Diagnostics at the World Bank (2002–11) (nos.) Diagnostic type 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Total CPAR 14 25 25 16 5 4 6 2 3 2 102 MAPSa 3 6 9 1 2 1 22 IFA 1 1 1 2 9 14 4 3 2 37 PEFA 6 24 27 26 25 16 4 128 PER 23 20 29 20 19 23 17 18 26 19 214 Total 38 46 55 44 60 74 62 49 49 26 503 Sources: IEG analysis, based on data from the World Bank’s Business Warehouse; PEFA Secretariat; OECD-DAC website. Note: CPAR = Country Procurement Assessment Review; IFA = Integrative Fiduciary Assessment; MAPS = Methodology for Assessment of Procurement Systems; PEFA = Public Expenditure and Financial Accountability; PER = Performance Expenditure Review. a. Includes only those available on the OECD website. However, up to 60 MAPs exercises have been cited as available (World Bank 2012), sometimes undertaken by countries themselves, sometimes embedded within exercises such as recent CPARs. 8. It was expected, under the aid reform agenda, that donor harmonization in procurement would ultimately imply donor convergence on recipient countries’ own systems. In 2008, the Bank launched a pilot program on the use of country systems in procurement as a phased exercise involving 20 country pilots.9 First phase analyses, focused on a diagnosis of participating countries’ existing systems based essentially on the MAPS tool, were undertaken. Fourteen were noted to have made significant progress, but only eight proceeded to the second phase, which involved a comparison of these countries’ systems with the procurement policies and procedures under Bank-financed investment operations. Although four countries/regions were ultimately deemed to have procurement systems deemed adequate for use under Bank investment lending, no projects using country procurement systems were approved under the pilot before its expiration in June 2011.10 PROCUREMENT UNDER BANK-FINANCED INVESTMENT PROJECTS 9. Policies and procedures on procurement under Bank-financed investment operations also seek to support development effectiveness in client countries by ensuring that funds are used for the purpose intended, as efficiently and effectively as possible.11 The policies and procedures governing procurement under investment lending (IL), the principal vehicle for Bank financial support to client countries, are established at the time of loan approval and are intended to ensure that funds loaned by the Bank are used for the objectives envisaged. Investment lending has averaged around 67 percent of new lending by value over most of the last decade, and more than 80 percent of the numbers of new loans. A significant part of this lending, especially contracts subject to the Bank’s International Competitive Bidding process, is 9 “Use of Country Systems in Bank Supported Operations: Proposed Piloting Program,” World Bank (2008); “Piloting Program in the Use of Country Procurement Systems Second Progress Report,” World Bank (2010). 10 These were Brazil (Sao Paulo state), Mauritius, Rwanda, and Senegal. 11 Although the evolution of Bank lending instruments points to the importance of helping countries build their own procurement systems, Bank guidelines on procurement emphasize the fiduciary need to safeguard Bank lending, as required in its Articles of Agreement, to “ensure that the proceeds of any loan are used only for the purposes for which the loan was granted.” 5 subject to prior review and a multistage “no-objection” process by the Bank. Procurement transactions subject to prior review have averaged 53 percent by value over the past decade.12 The International Competitive Bidding process remains fundamental to large contracts under investment lending, once typical of the Bank’s portfolio (Figure 1).13 Figure 1. Procurement Methods at the Bank: Prior Review and ICB (2000-2011) (No. and US$ billion) World Bank Prior Review and International  WB Investment Lending, Prior Review  12.0 Competitive Bidding (ICB) Contracts (Nos.),  35 Contracts and International Competitve  2000­2011 Bidding Contracts (US$B), 2000­2011 Number of contracts (000) 40.0 70 10.0 30 35.0 68 25 30.0 66 8.0 US$ Billion 64 20 25.0 6.0 62 % 20.0 % 15 60 4.0 15.0 10 58 10.0 56 2.0 5 5.0 54 0.0 0 0.0 52 00 01 02 03 04 05 06 07 08 09 10 11 00 01 02 03 04 05 06 07 08 09 10 11 Fiscal Year Fiscal Year Prior Review Contracts Prior Review Contracts ICB Contracts ICB Contracts as % of Prior Review Contracts (Nos.) New IL Commitments ICB Contracts as % of Prior Review Contracts (value) Sources: World Bank Business Warehouse; prior review data. Note: ICB = International Competitive Bidding process. 10. There is a perception among Bank staff that greater concern about the potential for corruption in procurement practices in recent years has resulted in additional layers of risk review in procurement for investment lending.14 The identification of potential procurement related risk has been recently formalized in the Bank, through the Procurement Risk Assessment 12 Upper limit assuming all prior review contracts are wholly Bank financed. 13 See Appendix C. Assuming that all contracts subject to prior review are wholly financed by the Bank, in the absence of disaggregated contract level data. Prior review contracts, as estimated above, declined from over 80 percent of investment lending in terms of value in 2000 to around 40 percent, by value, during the crisis years of FY09-10. IEG estimates that the International Competitive Bidding as a proportion of overall investment lending by value, though still high, may have shown a downward trend from around 50 percent in FY00-02 to around 30 percent in 2011. In terms of numbers of contracts the decline is less significant, averaging 26 percent over the past decade, and ranging from 32-33 percent in FY00-02 to below 20 percent in FY09-11. Looking at goods and civil works alone, the share of competitive bidding in all prior review contracts has averaged around 77 percent over the past decade and has not shown a downward trend (Table C.2). 14 World Bank Country-Level Engagement on Governance and Anticorruption: An Evaluation of the 2007 Strategy and Implementation Plan, IEG (2011). 6 and Management System (P-RAMs), with 11 specific risk indicators (2010).15 The Bank’s increased emphasis on anticorruption over the past decade reflected recognition of its potential negative effects on the achievement of the Bank’s overall development agenda, and its Integrity vice presidency (INT), established in 2001, provides preventive and investigative services to fight corruption. Its link to operational units was strengthened beginning in 2007.16 INT began to undertake reviews of projects where there were allegations of corrupt practices.17 It estimates that more than 95 percent of cases of fraud and corruption referred to INT concern procurement- related issues.18 11. The Bank’s Internal Audit Vice Presidency (IAD) is finalizing a report on selected fiduciary aspects of the Bank’s procurement function. The IAD report examines the extent to which the Bank is discharging its fiduciary responsibilities in the area of procurement, and whether the information systems set up to support it function effectively. Issues covered include the level and rigor of compliance with the prior review process, the quality of the post-review process, and the focus and independence of Independent Procurement Reviews. IAD has also examined the extent to which the newly introduced Procurement Risk Assessment and Management system is utilized.19 SUPPORTING DEVELOPMENT – NEW BUSINESS LINES AND A NEW GLOBAL CONTEXT 12. Given the evolving global context, Bank lending has evolved in directions not envisaged by its present procurement framework. Recent years have seen the introduction of new delivery vehicles, including, for example, an uptake in community-driven development (CDD) projects that aim to give control over planning decisions and investment resources to community groups and local governments.20 They raise particular issues, because of their 15 “Procurement Risk Assessment and Management System. User Guide,” World Bank (2010); updated July 2011, November 2011, January 2012. Its integration into the Bank’s overall risk management framework is envisaged. See “Guidance Note on the Operational Risk Assessment Framework—Risks to Achieving Results,” World Bank (2010). IEG’s IDA Internal Controls: Evaluation of Management’s Remediation Program—Report on Management’s Implementation and IAD’s Review of the Five-Point Action Plan (2010) acknowledged the strengthening of fiduciary controls through the introduction of new risk management tools, but pointed out that the new systems would take time to be effective. 16 Following recommendations of the ‘Volker Panel’ review (World Bank 2007). 17 A corporate strategy for governance and anticorruption (GAC) was developed in 2007, followed by a follow-up report that specifically discusses procurement as an area of focus. INT provides preventive and investigative services to fight corruption, and its links to operational units was strengthened following recommendations of the Volker panel review of INT (World Bank 2007). 18 INT also helps engineer fiduciary controls to mitigate procurement risk, such as its recent list of “red flags” that alert project teams to the potential for fraud and corruption in project procurement, its alert to debarred firms, and its complaints database. 19 IAD's review was conducted on the basis of the analysis of the Bank's internal processes and practices for procurement risk management focusing on three regional vice presidential units. IAD did not interact with borrowers or implementing agencies. 20 See, for example, the CDD Community of Practice Group (http://connect.worldbank.org/explore/SDV/cdd/default.aspx). Concentrated particularly in portfolios such as agriculture, health and education, CDD projects amount to some $2 billion per year and have an outstanding portfolio of around $30 billion. 7 multiplicity of actors, including district level administrations and village committees.21 The Bank has also introduced mechanisms such as Sector-Wide Approaches to help build programs in specific sectors. These involve consensus-building and alignment of implementation arrangements among donors in lower-income countries; whereas in middle-income countries, Bank operations have applied Sector-Wide Approach principles and tools (for example, pooling) to align Bank operations to the client country’s programs and systems. Challenges for Bank procurement policies and procedures have also arisen in public-private partnerships, where the Bank may come in as a gap financier, toward the end of an arrangement where the concession may already have been awarded to a private contractor. It is difficult to retroactively require the application of Bank guidelines, or to reflect certain non-price factors such as the distribution of risk among participants in procurement selections. Questions have also arisen in this context regarding the approach to procurement policies of the Bank and the International Finance Corporation, to the extent that they may both provide funds to similar public-private partnerships. Efforts to accommodate these delivery instruments have been incorporated in the Bank’s procurement guidelines, most recently revised in January 2011, supplemented in some areas by good practice guidance.22 Finally, the Bank is also providing more support in complex areas such as information technology and communications, where its traditional separation of lending and consultancy services is less distinct and where assumptions of buyers’ prior knowledge of the best systems is open to question. 13. Globally, the procurement environment is evolving to incorporate new practices. The Bank’s system of competitive bidding has long underpinned one of its guiding principles, yet there is a growing view outside the Bank that alternative procurement policies may lead to better procurement outcomes. The European Union’s procurement directives, for example, include value for money as a guiding principle, and in the United States, there is increasing emphasis on “competitive dialogue,” in which open exchange of information between buyer and prospective seller(s) is thought to lead to more informed purchase decisions.23 Purchase techniques such as framework agreements and reverse auctions are increasingly being used by some of the Bank’s IBRD clients. New technologies raise possibilities of increased use of e- procurement, which can aid transparency and better price discovery but sometimes lead to questions of platform access. The Bank also points to global changes in the structure of business and trade that have implications for its procurement function, for example, with increasing use of local affiliates by large multinational firms, and with the recognition of the role of procurement in multilateral fora such as the World Trade Organization as well as in regional trade agreements.24 14. A question for the evaluation therefore concerns the extent to which procurement policies and procedures under Bank investment lending have fostered development effectiveness. The Bank’s procurement guidelines refer to four overarching principles: economy and efficiency, equal opportunity to all bidders, encouragement to the development of domestic 21 New guidelines for CDD procurement were posted on the Bank’s intranet on March 27, 2012. 22 A Guidance Note on public-private partnerships was added to the procurement website in September 2010. 23 Debevoise and Yukins (2010). 24 Government Procurement Agreement, World Trade Organization, and Model Procurement Law, UNCITRAL. Many caveats apply, however, in comparing the Bank procurement system to external models. 8 contracting, and transparency in the procurement process. 25 This question will therefore refer to these overarching procurement policy principles. Among other issues, it will examine the extent to which fiduciary and anticorruption concerns may tie up procurement capacity such that this detracts from support for the development of countries’ national procurement systems. It will also review the extent to which they adequately provide for complex and newly emerging aspects of procurement that have arisen. MANAGEMENT PROPOSALS FOR PROCUREMENT REFORM 15. Motivated by the need to respond to a range of issues, Bank management is proposing a major reform of the institution’s procurement policies and procedures. An Initiating Memorandum prepared by Bank management was discussed by the Board on February 29, 2012. Core issues raised in the memorandum include (i) the relevance of the present system, given the changing context of procurement, in terms of global conventions, Bank lending and supplier patterns, and hence the need to incorporate best practice in the procurement area; and (ii) the effectiveness of the system, in terms of helping achieve better overall procurement capacity among borrowers. The memorandum points toward the desirability of a future model of procurement based on risk assessment and decentralization, together with enhanced discharge of the fiduciary mandate. It points out that a risk-based framework that could replace the present compliance-based focus would better adapt to differences among clients. A report is expected in early 2013, with detailed proposals for the implementation of reforms expected a few months later. As part of this exercise, management is preparing a series of background papers. Management also launched an extensive process of consultation and exchange with external partners beginning in mid-2012. Within this overall context, an Independent Evaluation Group (IEG) assessment of Bank procurement policies and procedures would contribute to shaping the Bank’s procurement reform agenda and inform the Bank’s Board at around the same time as management’s review. 16. A recent Board discussion queried how the IEG evaluation would dovetail with the management review. Directors underscored the importance of coordinating the work under the IEG evaluation with that under management review, particularly in terms of consultations with client countries, development partners, and the community of prospective bidders. The value of such consultations, as well as of robust discussion on competing priorities in seeking procurement reform, was also emphasized. In undertaking the evaluation, IEG will seek collaborative engagement with management, particularly in terms of stakeholder consultations. PROCUREMENT ISSUES: OTHER STAKEHOLDERS AND IEG FINDINGS 17. Although the scope of management’s review informs the scope of this IEG evaluation, IEG will also endeavor to cover a broader range of issues regarding Bank procurement policies and procedures that have been raised by other stakeholders (including the Bank’s Board, staff, and external stakeholders) and flagged prominently in past IEG corporate, country and sector evaluations. Findings are indicative even if references to procurement-related factors that affect project or country outcomes are examined, typically, in an incidental way. Looking at IEG’s 54 Country Assistance Evaluations undertaken 25 OP/BP 11.00. 9 between FY02 and FY11, 29 (54 percent) discussed procurement issues; 40 percent of IEG’s corporate and sector evaluations also raise procurement related issues. Significant references to procurement-related issues appear to have increased in recent evaluations (Table 5 and Appendix D). In terms of effectiveness in analytic work and building country capacity, a fifth of sector evaluations and almost two-fifths of country evaluations allude to work undertaken; however, a smaller proportion discusses its successful conclusion. Weak country capacity in procurement is alluded to in more than half of all country evaluations; there are also references to capacity constraints on the Bank side. Issues have also been raised regarding the termination of the recent pilot on the use of country systems and on the extent to which the Bank has helped to achieve donor harmonization in procurement systems, given the trade-offs inherent in harmonization. Several evaluations raise the issue of coordination between procurement reforms and overall public expenditure management, pointing out its value but also constraints to its achievement. 18. Even more evaluations discuss the relevance, effectiveness, and efficiency of the Bank’s procurement systems in terms of the impact on the Bank’s development effectiveness. In terms of effectiveness, prominent mention is made of fiduciary issues related to procurement—48 percent of country evaluations and 30 percent of sector evaluations point to country corruption issues, whereas a smaller but still significant proportion also point to such issues in Bank projects—17–22 percent. Even more prominent are references to efficiency issues—52 percent of sector evaluations and a third of corporate and country evaluations point to delays caused by procurement issues. Several studies have highlighted delays in the procurement process that have been associated with an escalation of project cost—or conversely, economies associated with judiciously designed procurement arrangements. Commentators have also raised questions about the efficiency of the current Bank procurement system, and voices for simplification are strong. Bank regions point to persistent process difficulties, system inefficiencies, unfunded mandates, and, especially, lags in execution. Issues discussed related to relevance refer to the adaptability of procurement guidelines to different contexts such as new instruments, small states, and fragile and conflict states. Other issues have also been raised, including the lack of monitoring and evaluation frameworks in procurement (Table 5 and Appendix D). To some extent, other multilateral development banks (MDBs) also reflect some of these issues, although on a lesser scale.26 A significant area of concern on the Board relates to the potential for a trade-off between a risk-based system that may be better adapted to different countries’ needs, or make greater use of country systems, and the safeguarding of fiduciary risk. The need to not dilute the practice or perception of fiduciary risk mitigation has been emphasized. 26 A review of Asian Development Bank evaluations over the past decade finds references especially to corruption and governance issues in procurement, and irregular practices, as well as to implementation delays attributed especially to limited borrower capacity. There are also references to Asian Development Bank support for capacity development. At the African Development Bank, two of ten recent (2006–11) large-scale reports make some reference to delays associated with procurement; three refer to capacity limitations at the level of staff and two refer to donor harmonization issues. References to procurement in European Bank for Reconstruction and Development and the Inter-American Development Bank evaluations are less frequent; there have been only seven references in 32 major special evaluations from the former bank since 2003. Two mention support to procurement reform. There are two mentions of delays, and two mention the reduction of corruption with good procurement. At the Inter- American Development Bank there has only been one reference to procurement in 17 special evaluations since 2003. 10 Purpose, Objectives, and Audience 19. The proposed evaluation has dual objectives. First, the proposed evaluation will inform the Board, Bank management, and internal and external stakeholders of the extent to which Bank’s support for procurement in client countries has enhanced the Bank’s development effectiveness. The first element will therefore be an examination of the extent to which the Bank strengthened client countries’ own procurement systems, evaluated relative to the objectives of increasing borrower capacity to plan, manage, and monitor the procurement process effectively; improving the accountability, integrity, and transparency of the process and reducing the scope for corruption; and increasing consistency with internationally accepted principles and practices.27 Second, the evaluation will examine the extent to which the procurement system has helped achieve efficiency and development effectiveness in the Bank’s own lending, relative to the overarching principles of the Bank’s own procurement guidelines. The evaluation will be based on the performance of the procurement function over the past decade, which will yield insights to guide further future changes. Table 5. References to Procurement Issues in IEG Evaluations (2002–11) Type of evaluation Corporate Sector CAE/CAS Dimension Procurement-related issues No. % No. % No. % Number of IEG evaluations reviewed in each category FY2002-2011 30 56 54 Total FY02-FY11 12 23 29 Of which, number with evaluative FY 2002-06 3 11 13 findings on procurement FY 2007-11 9 16 16 Is procurement viewed as a problem in effective 10 83 21 91 24 83 Overall question development? Does the evaluation include a recommendation 4 33 5 22 7 24 Procurement recommendations specific to procurement? Was analytic work undertaken? 0 0 5 22 11 38 Analytic work and building country capacity Has analytic work helped to build borrower 1 8 2 9 7 24 capacity? Did the evaluation identify legal/institutional 1 8 4 17 15 52 Country capacity shortcomings in procurement? Does the evaluation identify lack of capacity within 2 17 12 52 12 41 the country? Is there any discussion of inadequate capacity 5 42 8 35 9 31 Bank capacity and organization within the Bank? Has decentralization of Bank staff helped 1 8 1 4 4 14 procurement? Do current procurement processes impede or limit 7 58 2 9 5 17 Donor harmonization donor harmonization? Is there progress toward the use of country 2 17 2 9 3 10 Use of country systems systems, or are there obstacles to their use? Effectiveness of Bank Does the evaluation suggest that present systems 2 17 7 30 14 48 27 Based on the Revised CPAR guidelines (April 2002) that revised the Interim Instructions of June 1998. A key objective of the new procedures was to promote dialog with governments on how to strengthen their public procurement systems. Thus, building country procurement systems and enhancing their efficiency has only relatively recently been clearly articulated as a broad corporate goal of the Bank. 11 Type of evaluation Corporate Sector CAE/CAS Dimension Procurement-related issues No. % No. % No. % procurement systems in the country have corruption problems? Does the evaluation suggest that present systems 2 17 5 22 5 17 in Bank procurement have corruption problems? Fiduciary issues Does the IEG evaluation suggest that procurement 1 8 8 35 13 45 reform is needed for reducing corruption? Efficiency Delaysa Does the IEG evaluation include any specific 4 33 12 52 10 34 mention of delays as a result of procurement? Cost/ Quality Is procurement identified as constraining quality or 1 8 1 4 6 21 cost? Sector-specific procurement Are there procurement issues specific to a 3 25 7 30 10 34 particular sector? Special topics in procurement Does the evaluation discuss procurement in CDD, 1 8 4 17 0 0 PPP or ICT areas? Private sector activity Does the evaluation suggest that procurement 0 0 3 13 3 10 reform is needed for higher participation of PS in World Bank bids? Source: IEG. Note: CAE = Country Assistance Evaluation; CAS = Country Assistance Strategy; CDD = community-driven development; ICT = information and communications technology; PPP = Public Private Partnership a. Delays could be due to shortfalls on the side of either the Bank or the borrower and they are not analyzed separately here. 20. The Bank’s procurement reform agenda involves a huge number of internal as well as external stakeholders. Internal stakeholders comprise Bank management and the fiduciary services anchor, as well as operational (lending) and key anchor units. As illustrated in the Bank’s procurement guidelines, each has a role to play in procurement decisions.28 Stakeholders also encompass the Bank’s Board, in their fiduciary role, as representatives of Bank borrowers as well as their role as providers of goods and services under Bank contracts. Beyond the Bank itself, Bank procurement policies affect a range of international players, beginning with the Bank’s clients and extending to private contractors of goods and services that supply goods and services to Bank clients; other MDBs/international financial institutions (IFIs) and donors; and civil society and participants in the development process. Evaluation Scope, Coverage, and Questions 21. Two overarching questions confront the Bank’s procurement reforms at this juncture: 1. To what extent has the Bank helped its clients develop better procurement capacity and improve their public procurement systems? 2. To what extent does the application of Bank procurement guidelines in its investment lending help support its own development effectiveness objectives, in terms of fostering 28 See Appendix A (Responsibilities Matrix) of the Bank’s OP/BP 11.0, on procurement. 12 economy, efficiency, and transparency in the execution of Bank projects in client countries? 22. These overarching questions form the core of the evaluation. The evaluation will review Bank procurement over the last decade and selectively address aspects of these overall issues confronting the Bank’s procurement function. Its selectivity reflects, first, IEG’s mandate of development effectiveness, and second, the large amount of recent and ongoing analysis on other aspects of procurement-related questions—by Bank management in its extensive and ongoing self-diagnostic and, in terms of fiduciary aspects, by IAD. Select fiduciary aspects are reviewed only to the extent that they affect Bank performance in terms of development effectiveness, and add value to work already undertaken by IAD. IEG will also selectively address questions related to efficiency, insofar as it affects development effectiveness in Bank lending, and explore as needed the balance achieved between conflicting objectives, if this impacts development effectiveness. Based on this framework, and drawing on issues identified by the self-diagnosis, by stakeholders, and in the IEG meta evaluation, the evaluation will examine several detailed questions; the methodology used to answer each evaluation question is described in detail in the Evaluation Design Matrix in Appendix A. I. Developing Client Country Procurement Capacity through Advisory Work, Lending and Non-lending Technical Assistance i. To what extent have Bank country departments supported the strengthening of client procurement systems, through diagnostic and advisory work, and nonlending technical assistance? (Effectiveness) ii. To what extent has the Bank been able to provide support for procurement system reform through its development policy and investment lending platforms? (Effectiveness) II. Integrating Procurement Capacity with overall Public Financial Management iii. Has the Bank been able to integrate advisory work in procurement within a framework of public financial management and efficient public expenditure? (Effectiveness) III. Assisting Global Good Practices in Public Procurement iv. To what extent has the Bank been able to coordinate/harmonize its procurement processes with other MDBs/IFIs/other partners in development in accordance with the global aid effectiveness agenda? (Effectiveness) v. To what extent has the Bank been able to move toward the use of country systems, as envisaged in the aid effectiveness agenda? (Effectiveness) IV. Achieving Transparency and Effectiveness in Bank Lending: Procurement and Fiduciary Risk 13 vi. Have the recent changes in the Bank’s fiduciary environment for procurement, qualitatively improved its overall risk management framework and enhanced its overall development effectiveness? (Effectiveness) vii. Have risk mitigation measures (such as thresholds for prior review), been applied with consistency and efficiency? (Relevance) V. Achieving Economy and Efficiency in Bank Lending: The Procurement Process viii. How efficient/effective have the Bank’s present investment lending procurement systems been, in terms of (i) the extent to which users of the procurement system, at the level of clients and task team leaders, perceive the costs and benefits of the present process, and (ii) the distribution and utilization of Bank procurement staff? (Efficiency) VI. Adapting to Evolving Forms of Lending and Diverse Clients ix. To what extent has the Bank’s present procurement system accommodated the evolving needs of Bank lending (based on a review of procurement issues with regard to (i) CDD projects; (ii) public-private partnerships; and (iii) information and technology projects)? (Relevance) x. To what extent has the Bank’s present procurement system facilitated Bank engagement in fragile and conflict situations? (Relevance) Evaluation Design and Evaluability Assessment 23. The analytic framework refers, as noted above, to broad evaluation criteria:  Effectiveness. The evaluation will assess, first, the effectiveness (in terms of achievement of results) of the Bank’s developmental support to countries for improving their procurement systems and the extent to which the Bank’s lending vehicles or advisory services have been able to make a lasting impact on countries’ procurement functions and performance. Second, the evaluation will selectively review the effect of procurement on the effectiveness of Bank support in its lending operations, in terms of achieving efficiency and economy, equal opportunity, transparency, and support to local development.  Relevance. The evaluation will examine the relevance of the Bank’s current procurement policy, from the point of view of meeting its own objectives;29 supporting new institutional lending directions and delivery mechanisms, and differentiating between clients according to their needs and capacities. 29 Objective principles of Bank procurement policy refer to economy and efficiency, equality of opportunity, and transparency. The Bank has also embraced the perspective of supporting domestic markets, providing development support to client countries, and meeting other Bank institutional objectives, for example, the governance and corruption agenda. 14  Efficiency. IEG will provide evidence on the process efficiency of procurement, for example, in terms of time spent on the procurement cycle, or stages and levels of clearance. This would be based on data on contracts and regional procurement databases maintained by management that include tracking information on procurement process, triangulated with information from interviews and field visits. Field visits will include meetings with country staff, Project Implementation Units, task teams, and Bank procurement staff. If available, information from other MDBs and IFIs will be used as a comparator. DATA SOURCES, SAMPLING STRATEGY, AND ANALYTICAL METHODS 24. A key methodological challenge will be the measurement of effectiveness of Bank interventions for country development. This will be undertaken through the triangulation of information on the following areas: (i) using time series of diagnostic work on the state of procurement in individual countries; (ii) supplementing core diagnostic work with analyses of country progress in procurement described in loan documents; and (iii) seeking inputs from task team leaders and procurement experts in specific areas. Current diagnostic tools such as the CPARs and MAPS exercises will be utilized to the extent available, under (i) above. 25. A range of information sources and methods of analysis will be used in the evaluation. Broadly, data to be collected will comprise the following elements:  Illustrative country case studies.30 A small purposive review of some eight to ten country case studies will be undertaken. Although such a small selection can only be illustrative and not representative, country selection will be based on the principle of regional balance. Countries within regions will be selected based on further criteria: significant and varied interventions for procurement reform and capacity building, the presence of a portfolio of lending in those sectors such as infrastructure, which have large discrete projects, as well as in sectors that rely on decentralized delivery, such as health or education. Field visit consultations will include Bank country management, procurement staff and task team leaders, and clients in the national procurement authority and in ministries executing Bank projects. Issues covered in field visits will include the effectiveness of advisory work, developmental assistance in country procurement capacity through lending, as well as the efficiency and efficacy of the Bank’s lending processes. Case studies will also review the functioning of project implementation units under Bank loans and solicit views from private suppliers to Bank projects as well as civil society organizations involved in procurement. As with surveys, case studies will be used to contribute to the triangulation of evidence collected from portfolio reviews and desk-based analysis.  Staff surveys and structured interviews of select groups of Bank staff. Surveys will be undertaken by staff category, for example, task team leaders of investment lending, task team leaders and team members in fiscal management development policy loans (DPLs), procurement staff/ management at different levels, and potentially country operational advisers. A sampling approach will be adopted for large groups such as task 30 Subject to the availability and experience of an adequate set of consultants. 15 team leaders. The principles underlying sample selection will be regional balance and representativeness of at least two Bank sectors with significantly different procurement processes (for example, large infrastructure projects versus significantly decentralized community development projects).  Review of IEG evaluative evidence on the Bank’s procurement function. IEG will review all evaluations undertaken in the last 10 years that include significant references to the Bank’s procurement function, including corporate evaluations, country and sector evaluations, and project evaluations. This will be undertaken on a complete portfolio basis (Appendix D has extracts).  Reviews of World Bank documents, including: country program documents (Country Assistance Strategies and Country Assistance Strategy Completion Reports), lending- related documents (program and project appraisal documents, Implementation Status and Results Reports), and analytic and advisory work on procurement (CPARs, available MAPS analyses, Public Expenditure and Financial Accountabilities, and informal advisory services). A complete enumeration basis (as opposed to sampling) is envisaged.  Analysis of procurement specific datasets, including, particularly, the form 384 dataset, managerial tracking systems in each region (for example, PROSYS in Africa, similar databases in other regions), post-review information, the complaints database, and so forth. A more in-depth review of specific contracts will also be undertaken, provided information is made available. All regional data made available will be covered.  Analysis of datasets maintained by INT, including fraud and corruption and tracking of “red flag” indicators, with a view to identifying specific areas of weakness in the current procurement system.  Analysis of World Bank procurement rules as well as processes relative to select other IFIs/MDBs/donors. This will be based on desk reviews as well as discussions of respective procurement processes with relevant staff in other agencies. 26. Both complete enumeration and sampling methods will be used. As noted above, although some of the above reviews will be undertaken on a complete enumeration basis, reviewing data from 2002–11, a more selective approach will have to be followed, for example, with regard to reviews of project portfolios and task team leader interviews. In such cases, purposive samples will be constructed that include a spectrum of representative countries and two core sectors. Country selection would include higher- and lower-income countries. Sector coverage will include those that illustrate different procurement processes, for example, a large infrastructure related sector such as transport or energy, as well as social sector lending, such as health or education. 27. The evaluation will employ quantitative as well as qualitative methods. Primary information will be examined through spreadsheet/regression analyses of the Bank’s project portfolio and of survey results that track the current status and recent evolution of the Bank’s procurement policy and processes, to permit the construction of a time-ordered sequence of policy information from existing documentation (such as goals and objectives, advisory support, lending support, and efficiency parameters). To the extent that data permit, statistical analyses will be undertaken that correlate results in specific areas with indicators such as country 16 performance or country governance.31 Qualitative analysis will be undertaken of nonstandardized interview and open-ended survey information (category building, trend identification), followed by synthesis and triangulation of review findings, hypothesis building, and testing. DESIGN STRENGTHS AND LIMITATIONS 28. The strength of the proposed approach is its additionality to the considerable amount of existing and ongoing analysis undertaken by the Bank’s procurement anchor (OPSOR), and also by IAD, and its selectivity, especially with regard to fiduciary aspects. IEG will contribute new analytical material to the key questions confronting the Bank’s procurement reform proposals. A potential disadvantage is the extent to which such a strategy could potentially limit the ability of the evaluation to fully respond to the overarching questions confronting the sector. IEG will attempt to address this by drawing on recent work by IAD, to permit broader conclusions to be drawn, particularly with regard to fiduciary aspects. Another issue in this regard concerns ongoing or recently completed work by OPSOR, much of it relevant to this proposed evaluation. IEG will coordinate with OPSOR and ensure continuous exchange of information. IEG seeks to be kept closely informed of its evolution and findings and to be able to participate in relevant rapidly evolving areas. Management’s work will be acknowledged in the IEG evaluation, although IEG will seek to also answer questions beyond this that so far have not been raised by management. 29. Certain areas will remain beyond the scope of the present evaluation. IEG will not undertake separate and in-depth analysis of issues specific to procurement categories—goods, civil works, and consultants. Although it recognizes that embedding broader public policy issues in public procurement is a growing practice, especially in certain OECD countries, IEG will not be able to include other public good themes in this evaluation. These themes may include additional and quite legitimate public policy objectives, for example, the extent to which procurement should reflect environmental or safeguard principles, support for clean energy, labor standards, and so forth. Although the Bank has struggled to address some of these dimensions in its explorations of new dimensions of procurement policy, at present these areas are led by other Bank guidelines and policies and it is beyond the scope of this evaluation to group these differently.32 Finally, the analysis of select areas of procurement that pose additional challenges (community-driven procurement, information and communications technology procurement, and so forth) will be undertaken, resources and time permitting. Procurement in public and private partnerships will, however, be included, given its pertinence to both the Bank and the International Finance Corporation. Quality Assurance Process 30. The draft report will be peer reviewed to ensure accurateness, credibility, and impartiality of its findings and recommendations. Peer reviewers for the evaluation are 31 As examples, data on procurement thresholds can be correlated with country performance indicators to see whether thresholds are set according to implicit country performance criteria; data on the quality of procurement related advisory work can be correlated with select indicators of improvement in governance, and so forth. 32 OPSOR has prepared background notes and papers in these areas, for example, on benchmarking diversity and green procurement as factors in procurement in different countries (September 2011). 17 Christiaan Poortman (former Vice President, Middle East and North Africa Region, Senior Advisor at Transparency International, and Chair of the Board of CoST [Construction Sector Transparency Initiative]); Mr. Praful Patel (former Vice President, South Asia region, and former Director, Infrastructure, Africa Region); and Dr. Peter Trepte (Senior Fellow in Public Procurement Law, University of Nottingham). An advisory group for the task will also be appointed to provide guidance to the study. The budget for the study is estimated at approximately $970,000. The report will be prepared under the direction of Ali Khadr, Senior Manager. Expected Outputs and Dissemination 31. The primary output of the evaluation will be the report to the Committee on Development Effectiveness (CODE) that will contain the main findings and recommendations, or areas where further work is needed before reform proposals can be elaborated. The report will be submitted to the Board in time to inform the discussion of the Bank’s Phase I report on its procurement reforms, by March 2013. Broad-based dissemination of output to Bank staff responsible for fiduciary aspects as well as staff that use the Bank’s procurement systems in their capacity as team leaders or sector managers is also expected. 32. Ongoing stakeholder interaction will be sought to enhance the evaluation process. Beyond the Board of Directors and Bank staff and management, key stakeholders include member country officials, procurement departments in multilateral and bilateral agencies, and members of the OECD-DAC Working Group on the Aid Reform Agenda, as well as the private sector, for example, International Federation of Consulting Engineers and the International Chamber of Commerce. The Bank will also share its methods and findings with other evaluation agencies. The evaluation will seek to engage stakeholders throughout the evaluation process. IEG will also reach out to stakeholders by engaging in international fora that address procurement issues. The objectives of stakeholder interaction will be to (i) make the evaluative process transparent and participatory by inviting the public to share knowledge and experiences and by keeping stakeholders informed about the evaluation progress; (ii) gather information and qualitative data to triangulate information obtained through other research and information gathering methods; and (iii) develop a core group through which the evaluation recommendations and findings can be channeled when the evaluation is complete. Resources and Timetable 33. IEG will rely considerably on external procurement experts, drawn from other MDBs, and independent consultants, as well as eligible former Bank staff. A team of qualified staff has been identified. Overall, the team will include senior IEG staff and consultants, procurement specialists and research analysts. It will be led by Anjali Kumar, under the management of Ali Khadr, Senior Manager, IEGCC. The evaluation report will be submitted to CODE in May 2013. Milestones of the evaluation are outlined below. 18 Proposed Timeline Item Date IEG one-stop review for the Approach Paper May 22, 2012 Submission of Approach Paper to management/CODE July/August 2012 Comments on the Approach Paper from World Bank August 2012 management Data gathering and analysis May 2012–December 2012 Country visits September 2012–November 2012 First draft February 2013 IEG one-stop review March 2013 Report submitted for internal Bank review March 2013 Report finalized and distributed to CODE May 2013 19 Appendix A. Evaluation Design Matrix Evaluation Information Evaluation subquestions Sources Data collection Data analysis/method question required 1. Introduction: Identifying Issues in the Bank’s Procurement System Building country capacity Meta analysis of IEG (diagnostic work, reform support) evaluative material: Bank capacity issues: corporate evaluations, Background and procurement staff skills; sector / country evaluations introduction. What can delivery mechanism instrument (CAEs and we learn from IEG related issues Significant evaluative CASCRRs); project evaluations abut references to evaluations (sector Portfolio analysis; procurement related Donor coordination issues? procurement related evaluations and task team Spreadsheet analysis, pivot issues that face the Bank Fiduciary issues? Risk issues in World Bank ICRRs); review of leader/staff tables and how has this management? Fraud and projects / country other MDB’s surveys, field visits evolved over time? corruption? assistance evaluative references What have the Bank’s to procurement issues; achievements in this area Efficiency? Time or cost to be supplemented been? associated with procurement later by site visits and processes? TTL/staff questionnaires 2. Providing Development Support to Countries for Improving their Procurement Systems To what extent have Bank country Significant evaluative To what extent has the CASs, CASCR, CAEs. departments supported the references by country Bank helped its clients Triangulation with Spreadsheet analysis, pivot strengthening of client departments to Portfolio analysis, to achieve better public country department tables, qualitative analysis of procurement systems and has this procurement-related field visits procurement and thus interviews in field country visit questionnaires been effectively realized? issues; extent of follow- meet their development visits up to the commitments. goals 20 Evaluation Information Evaluation subquestions Sources Data collection Data analysis/method question required CPARs, MAPS; before To what extent has the Bank Largely available and after comparisons Use of structured provided quality and timely from Bank internal relative to 2008 questionnaires for qualitative procurement-focused advisory Documents. Fiduciary Diagnostic Collection of analysis of diagnostic tools. work? Triangulation with assessment. Meetings relevant documents Structured and sequential country department with informed analysis to trace progress interviews in field procurement staff over time. visits DPLs: review of Portfolio reviews; procurement relevant Use of structured country case studies. actions in a sample of questionnaires for qualitative To what extent has the Bank Largely available select DPLs analysis of procurement supported procurement reforms from Bank internal Investment lending: related prior actions and in its policy based DPOs and Documents. Review Downloading of (selective review of triggers in DPOs. Structured through its TA loans? of TA/IDF loans. relevant documents portfolio in field visits): analysis of procurement Triangulation with from Bank systems value of PIUs, reform supported in IL or country department procurement issues in TAL operations. . Structured interviews in field ISRs; TALs; IDF grants, and sequential analysis to visits. other NLTA trace progress over time. Has the Bank been able Use of structured PEFA documents-- to integrate advisory PEFA analysis, questionnaires for qualitative Analysis of application Collection of work in procurement Governance Funds analysis of diagnostic tools within Bank’s PFM relevant within a framework of and IDF analysis. Comparisons of results on framework. Meetings documents; public financial Triangulation with PEFA with governance with informed analysis of use of management and country department indicators, CPIA, and so procurement staff governance and efficient public interviews in field forth. Structured and IDF facilities expenditure? visits sequential analysis to trace progress over time. 21 Evaluation Information Evaluation subquestions Sources Data collection Data analysis/method question required To what extent has the Bank been able to move Structured analysis of UCS case studies, towards the use of Documents to be Use of structured UCS country case phases I and II, country systems? What provided by questionnaires for qualitative studies maintained by explains the Bank’s OPSOR analysis of diagnostic tools OPSOR limited move in this direction? To what extent is the Bank able to coordinate Analysis of outputs of /harmonize its Working Groups under procurement processes OECD-DAC; with other MDBs/IFIs/ Review of other Structured other partners in IFIs/MDBs as well as Information from interviews with development? Has the major donor partners OECD-DAC and members of the aid Qualitative methods Bank taken a lead role in (DFID, CIDA/USAID, from other MDBs harmonization international policy and so forth) in terms of working groups dialogue concerning their procurement public procurement in processes developing countries? 3. Achieving Transparency in Bank Lending and Reducing Fiduciary Risk P-RAMs assessments to assess their quality and Have the changes in the Bank’s integration into the Does the design of the fiduciary environment for Downloads of Bank’s overall risk P-RAMS data from present procurement procurement, introduced in the relevant internal Spreadsheet analysis, management framework. OPSOR and project system help the Bank to past 2-3 years been implemented and external qualitative analysis. portals; INT data meet its fiduciary and integrated into its overall risk datasets INT data for analysis of obligations, and ensure management framework? procurement aspects of that Bank funds are INT cases spent for the purposes Data on thresholds for OPSOR data, internal they were intended (to Have risk mitigation measures prior review and for Bank data, for Downloads of be coordinated with Statistical analysis including (such as its thresholds for prior different forms of example, on relevant internal IAD)? regression analysis; data review), been consistently and procurement, in regions governance measures, and external quality permitting. efficiently applied? and the center. external measures of datasets governance 22 Evaluation Information Evaluation subquestions Sources Data collection Data analysis/method question required 4. Achieving Economy and Efficiency in the Bank Lending How effective has the OPSOR and Bank Requests from Statistical and spreadsheet procurement process been in regional procurement Data on contract procurement staff analysis - comparability of terms of review and decision data bases; sample How efficient and processing at regions and data across regions to be making? How effective have its data on contract effective have the anchor reviewed. outcomes been? processing. Bank’s present How do users of the Banks’ procurement processes Qualitative procurement systems (task team been, in terms of the use information as well leaders in the Bank, country Interview data with Structured field visits; Spreadsheet analysis as well of staff time and as scorings and clients, and private sector TTLs, clients and staff interviews and as qualitative analysis of resources? ratings through contractors) perceive the suppliers surveys responses. field visits and efficiency of the Bank’s questionnaires. procurement system? 5. Adapting to Evolving Forms of Lending and Diverse Clients 1. CDDs (decentralized procurement to entities outside the government); 2. PPPs (procurement in To what extent does the Bank’s a framework that Has the Bank’s present Portfolio data on present procurement system includes private partners procurement system projects in each area; Review of accommodate the needs of new and commercial allowed it to adapt to the ISRs, country case documents, and different instruments/delivery practices) – examination evolution in its lending studies; interviews Structured Qualitative analysis mechanisms of Bank lending? of coordination with IFC and in its country with experts in the interviews, field engagement? Bank and with task visits. 3. ICT procurement team leaders (large and complex integrated systems, buyer/supplier knowledge); large service components 23 Evaluation Information Evaluation subquestions Sources Data collection Data analysis/method question required To what extent does the Bank’s present procurement system allow Information on adequate differentiation of procurement related Country and project Review of approach between different issues in Bank lending to information in the documents, countries? In particular, Are fragile states; Qualitative analysis Bank (CAEs, ISRs, Structured present systems adapted to the comparisons with structured interviews) interviews special needs of fragile and practices by other conflict affected states? donors 24 Appendix B. Evaluation of the Bank’s Procurement System--Draft Report Outline I. Introduction i. The Evolution of the Bank’s procurement system ii. The Current Debate—Key Issues Confronting the Bank’s Procurement System iii. Identification of issues to be reviewed: IEG’s Meta Evaluation II. Providing Development Support for Procurement: AAA, Nonlending Technical Assistance (NLTA), and Lending i. Bank Support to Procurement Systems—Country Programs ii. Bank Support to Procurement Systems—Analytic and Advisory work (CPARs, MAPs, IFAs); NLTA/IDF and other grants iii. Bank Support to Procurement Systems—DPLs/technical assistance loans Conditionality relevant to procurement reform Effectiveness of DPLs as an instrument of procurement policy reform Effectiveness of DPLs as an instrument of procurement policy reform III. Integrating Procurement Capacity with overall Public Financial Management i. The PEFA instrument; other diagnostics used by PREM/PF ii. Integration: Public financial management and procurement IV. Supporting Procurement Harmonization and the Use of Country Systems i. Coordination and harmonization of procurement processes with other MDBs/IFIs/ other partners in development in accordance with the global aid effectiveness agenda ii. The Use of Country Systems experiment—A structured analysis iii. New Bank instruments—P4R and implications for Use of Country Systems iv. The Contributions of the Bank in the Aid Reform Agenda a. The Bank’s systems and other MDBs b. The Bank’s procurement systems and other donors V. Achieving Development Effectiveness in Bank Lending: Fiduciary Risk i. Evolution of the overall framework for risk mitigation—Guidelines and practice ii. Application of risk mitigation measures—Quality, consistency, effectiveness iii. Integration of procurement risk management in overall risk management framework iv. Follow up to identified risks VI. Achieving Economy and Efficiency in Bank Lending i. Efficiency of the present control environment in terms of client/ staff time— Estimates of contract processing ii. Efficiency—Perceptions of staff and clients VII. Adapting to Evolving Lending and Diverse Clients i. Incorporating new instruments/delivery mechanisms: a. CDD b. PPP c. ICT ii. Incorporating different client profiles a. Bank procurement in fragile situations b. Bank procurement in middle-income countries/low-income countries VIII. Conclusions and Recommendations 27 Appendix C. Bank Procurement Systems – Prior Review and International Competitive Bidding Table C.1. World Bank Lending, Prior Review and International Competitive Bidding Contracts: (goods , civil works, and services) (2000–11) Value of  Prior  ICB  No. of ICB  All new   No. of  ICB   IL new   Prior   No. of  Review  Contracts  Prior  ICB  Contracts  commit­ Prior  ICB  Contracts  commit Review  ICB  Contracts  as % of  Review  Contracts  as % of  FY  ments  Review  Contracts  as % of  ments    Contracts  Contracts as % of new  new  Contracts  as % of  Prior  (ILs+DPLs)  Contracts  (US$B)  Prior  (US$B)  (US$B)  (000)  commit­ commit­ as % of ILa  ILa   Review  (US$B)  (000)  Review  ments  ments  contracts  contracts  2000  15.3  10.2  8.2  8.5  5.2  2.8  54  34  81  51  63  33  2001  17.3  11.5  7.6  7.9  5.2  2.5  44  30  66  46  69  32  2002  19.5  9.7  7.6  8.1  5.0  2.6  39  25  78  51  65  32  2003  18.5  12.5  7.8  8.3  4.8  2.4  42  26  62  38  62  29  2004  20.1  13.9  8.9  8.4  5.1  2.4  44  26  64  37  58  29  2005  21.9  15.3  9.5  8.3  6.0  2.1  43  28  62  40  64  26  2006  23.6  16.3  7.9  7.6  5.2  1.8  34  22  49  32  65  24  2007  24.5  18.3  9.8  7.5  6.7  2.0  40  27  54  37  68  27  2008  24.7  18.1  11.9  7.6  8.0  2.0  48  32  66  44  67  26  2009  46.4  28.1  10.9  8.8  7.1  2.0  23  15  39  25  66  22  2010  58.5  35.5  13.3  11.1  8.8  2.1  23  15  37  25  67  19  2011  42.6  31.1  13.7  10.6  9.4  1.9  32  22  44  30  69  18  Total  332.9  220.3  117.0  102.8  76.6  26.7  35  23  53  35  65  26  Source: World Bank Business Warehouse. Note: ICB Contract data includes only contracts subject to World Bank prior review. ICB = International Competitive Bidding; IL = investment lending. a. The value of contracts entered in the Form 384 represents the total commitment against a contract and not the actual portion of the contract eligible for Bank disbursement. While broad directional trends in the ratios of these contracts to IL are relevant, the Bank financed portion of the contracts will be lower for all years. Due to the absence of contract accounting at the Bank’s controllers, it is not possible to determine disbursement of the Bank-funded portion of an ICB contract.     Table C.2.  World Bank Prior Review & ICB Contract Data, (goods , civil works and  services) (2000–11)  Prior  Value of  No. of all  ICB  No. of ICB  All new   Prior  Review  Prior  ICB   IL new  Prior  ICB  No. of ICB  Contracts  ICB  Contracts  commit­ Review  Contracts  Review  Contracts  commit­ reviewed  Contracts  (G&CW)  as % of  Contracts  as % of  FY  ments   Contracts  as % of  Contracts  as % of  ments  (G&CW)  (G&CW)  contracts   new  as % of  Prior  (ILs+DPLs)  (G&CW)  new  as % of   Prior   (US$B)  contracts  (US$B)  (000)  commit­ ILa   Review  (US$B)  (US$B)  commit­ IL   a Review  (000)  ments  contracts  ments  contracts  2000  15.3  10.2  6.6  5.0  5.2  2.8  43  34  65  51  79  56  2001  17.3  11.5  6.5  4.6  5.2  2.5  38  30  56  46  81  55  2002  19.5  9.7  6.4  5.0  5.0  2.6  33  25  66  51  77  51  2003  18.5  12.5  6.5  4.8  4.8  2.4  35  26  52  38  74  50  2004  20.1  13.9  7.3  4.7  5.1  2.4  36  26  52  37  70  51  2005  21.9  15.3  8.0  4.5  6.0  2.1  36  28  52  40  76  47  2006  23.6  16.3  6.7  3.8  5.2  1.8  28  22  41  32  77  48  2007  24.5  18.3  8.3  4.1  6.7  2.0  34  27  46  37  81  51  2008  24.7  18.1  10.3  3.9  8.0  2.0  42  32  57  44  77  51  2009  46.4  28.1  9.5  4.5  7.1  2.0  21  15  34  25  75  43  2010  58.5  35.5  11.4  5.0  8.8  2.1  19  15  32  25  78  42  2011  42.6  31.1  12.2  4.6  9.4  1.9  29  22  39  30  77  42  Total  332.9  220.3  99.8  54.5  76.6  26.7  30  23  45  35  77  49  Source: World Bank Business Warehouse. Note: ICB Contract data includes only contracts subject to World Bank prior review. ICB = International Competitive Bidding; IL = investment lending. a. The value of contracts entered in the Form 384 represents the total commitment against a contract and not the actual portion of the contract eligible for Bank disbursement. While broad directional trends in the ratios of these contracts to IL are relevant, the Bank financed portion of the contracts will be lower for all years. Due to the absence of contract accounting at the Bank’s controllers, it is not possible to determine disbursement of the Bank-funded portion of an ICB contract.       29 Table C.3. World Bank Lending: DPLs and Investment Lending (new commitments, 2002–11) Loan commitment values  Proportion of IL projects  Number of loans     ($B)  %  Invest­ Invest­ Dev policy  Dev policy  By number  Approval FY  ment  ment  By value  lending  lending  of loans  lending  lending  2002  51  192  9.8  9.7  50  79  2003  50  209  6.0  12.5  67  81  2004  43  213  6.2  13.9  69  83  2005  56  236  6.6  15.3  70  81  2006  53  244  7.3  16.3  69  82  2007  57  261  6.3  18.3  74  82  2008  47  272  6.6  18.1  73  85  2009  71  256  18.4  28.1  60  78  2010  85  299  23.0  35.5  61  78  2011  69  315  11.6  31.1  73  82  Total  582  2497  101.7  198.7  66  81  Average  excluding  FY09­10  53.25  242.75  7.5  16.9  68  82  Source: World Bank Business Warehouse. Note: IL = investment lending. a. The value of contracts entered in the Form 384 represents the total commitment against a contract and not the actual portion of the contract eligible for Bank disbursement. While broad directional trends in the ratios of these contracts to IL are relevant, the Bank financed portion of the contracts will be lower for all years. Due to the absence of contract accounting at the Bank’s controllers, it is not possible to determine disbursement of the Bank-funded portion of an ICB contract. Table C.4: World Bank Disbursements, Prior Review and International Competitive Bidding Contracts (goods, civil works, and services), 2000–11  Prior  Total  Prior review  Prior review  ICB  IL  ICB contracts as  ICB contracts as  review  disbursements  contracts as %  contracts as % of  FY  contracts  disbursements   % of IL  % of total  contracts  (ILs+DPLs)  of IL  total  (US$B)  (US$B)  disbursementsa   disbursements  (US$B)  (US$B)  disbursementsa  disbursements  2000  8.2  5.2  18.4  12.6  41  65  28  45  2001  7.6  5.2  17.1  11.4  46  67  31  44  2002  7.6  5.0  17.8  11.0  45  69  28  43  2003  7.8  4.8  18.9  10.6  45  74  25  41  2004  8.9  5.1  16.9  10.9  47  81  30  53  2005  9.5  6.0  18.6  12.3  49  77  33  51  2006  7.9  5.2  20.7  12.8  40  62  25  38  2007  9.8  6.7  19.2  12.9  52  76  35  51  2008  11.9  8.0  19.6  13.3  60  89  41  61  2009  10.9  7.1  27.6  16.7  43  65  26  39  2010  13.3  8.8  40.1  19.5  45  68  22  33  2011  13.7  9.4  32.1  19.6  48  70  29  43  Total  117.0  76.6  266.7  163.5  47  72  29  44  Source: World Bank Business Warehouse. Note: ICB Contract data include only contracts subject to World Bank prior review. DPL = development policy lending; ICB = International Competitive Bidding; IL = investment lending. a. The value of contracts entered in the Form 384 represents the total commitment against a contract and not the actual portion of the contract eligible for Bank disbursement. While broad directional trends in the ratios of these contracts to IL are relevant, the Bank financed portion of the contracts will be lower for all years. Due to the absence of contract accounting at the Bank’s controllers, it is not possible to determine disbursement of the Bank-funded portion of an ICB contract. 31 Appendix D. IEG Meta Evaluation of Procurement Issues in the World Bank—Corporate Evaluations with Major References to Procurement IEG’s Review of IDA Internal Controls: An Evaluation of Management's Assessment and the IAD Review (2009) notes progress in shifting towards an integrated system of risk management: “Finally, strengthening and building country systems in these key areas has been, and increasingly is, an important focus of the Bank’s institution-building, governance and fiduciary support work in its client countries. Indeed, over 50 percent of DPLs in FY07 (up from an average of 24 percent in 1990–94) included conditionality on public sector governance, including in the areas of procurement, financial management, and budget transparency. While it is not a new area of emphasis, the importance of this work under the GAC strategy, combined with the work underway to implement the recently approved country systems pilot in procurement area, provide an added emphasis to the Bank’s efforts to strengthen the effectiveness, efficiency, reliability and transparency of control systems and institutions in client countries with the help of all of the Bank’s existing tools and instruments.” World Bank Country Level Engagement on Governance & Anticorruption (2012): The evaluation finds that “the Bank has often achieved better results on governance and anti- corruption through sector reforms in banking, energy and other utilities, and on procurement, when government ownership was present than it has through attempts at wider public sector reforms and visible anticorruption measures, such as the establishment of anti-corruption commissions.” The Country Financial Accountability Assessments and Country Procurement Assessment Reports: How Effective Are World Bank Fiduciary Diagnostics? (2008) highlighted the fragmented treatment of overall fiduciary risk by the Bank in client countries – it was dealt by two sector boards and three units – when there was an essential need to approach these issues in an integrated manner. It recommended that fiduciary instruments use an integrated risk analytical framework that includes a common approach to defining and addressing issues related to “fiduciary risks”. World Bank Progress in Harmonization and Alignment in Low-Income Countries (2011) notes that “Progress in the use of country financial management and procurement systems is constrained by inadequate capacity, weaknesses in public financial management and procurement systems, and the Bank’s fiduciary obligations. Even where country capacity has been built in financial management and procurement, capacity is hard to retain because of weak incentives, including low civil service salaries, which have led to loss of skilled staff. These constraints highlight the need for continued efforts in building capacities in financial management and procurement, together with greater government commitment to public financial management reforms and to establishing an incentive system to attract and retain the needed skills in the context of wider civil service reforms.” Appendix E. References ADB (Asian Development Bank). 2002. “Understanding Public Procurement.” Governance Brief, Asian Development Bank. Debevoise, E.W. & Yukins, C. 2010. “Assessing the World Bank’s Proposed Revision of Its Procurement Guidelines,” The Government Contractor, Vol. 52, No. 21, May 2010. Independent Evaluation Group (IEG). 1977. “The Role and Use of Consultants in Bank Group Projects.” Report no. 1824, Washington, DC: World Bank. ———. 1981. “An Interim Report on Procurement Issues in Bank-Financed Projects.” Report no. 3557, Washington, DC: World Bank. ———. 1990. “A Review of Procurement in Policy-Based Lending.” Report no. 8341, Washington, DC: World Bank. ———. 2002. India Country Assistance Evaluation. Washington, DC: World Bank ———. 2006. Debt Relief for the Poorest—An Evaluation Update of the HIPC Initiative. Washington, DC: World Bank. ———. 2006. Hazards of Nature, Risks to Development – An IEG Evaluation of World Bank Assistance for Natural Disasters. Washington, DC: World Bank. ———. 2006. Small States: Making the Most of Development Assistance – A Synthesis of World Bank Evaluation Findings. Washington, DC: World Bank. ———. 2007. The Development Potential of Regional Programs – An Evaluation of World Bank Support of Multicountry Operations. Washington, DC: World Bank. ———. 2007. Water Management in Agriculture- Ten Years of World Bank Assistance, 1994-2004. Washington, DC: World Bank. ———. 2008. Country Financial Accountability Assessments and Country Procurement Assessment Reports: How Effective are World Bank Fiduciary Diagnostics? Washington, DC: World Bank. ———. 2008. Public Sector Reform: What Works and Why? An IEG Evaluation of World Bank Support. Washington, DC: World Bank. ———. 2008. The World Bank in Indonesia 1999-2006 - Country Assistance Evaluation. Washington, DC: World Bank ———. 2009. Bangladesh Country Assistance Evaluation 2001-2008. Washington, DC: World Bank ———. 2009. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition and Population- An Evaluation of World Bank Group Support since 1997. Washington, DC: World Bank ———. 2009. The World Bank Group Guarantee Instruments 1990-2007 –An Independent Evaluation. Washington, DC: World Bank ———. 2010. Cambodia – An IEG Country Assistance Evaluation 1999-2006. Washington, DC: World Bank ———. 2010. Climate Change and the World Bank Group. Phase II: The Challenge of Low-Carbon Development. Washington, DC: World Bank. ———. 2010. IDA Internal Controls: Evaluation of Management’s Remediation Program--Report on Management’s Implementation and IAD’s Review of the Five-Point Action Plan. Washington, DC: World Bank. 33 ———. 2010. Mozambique Country Program Evaluation. Washington, DC: World Bank ———. 2010. The World Bank in Nigeria 1998-2007 - Country Assistance Evaluation. Washington, DC: World Bank ———. 2011. Capturing Technology for Development – An IEG Evaluation of World Bank Group Activities in Information and Communication Technologies. Washington, DC: World Bank. ———. 2011. Trust Fund Support for Development – An Evaluation of the World Bank’s Trust Fund Portfolio. Washington, DC: World Bank. ———. 2011. World Bank Country-Level Engagement on Governance and Anticorruption: An Evaluation of the 2007 Strategy & Implementation Plan. Washington, DC: World Bank. ———. 2011. World Bank Progress in Harmonization and Alignment in Low-Income Countries. Washington, DC: World Bank Organization for Economic Co-operation and Development (OECD-DAC). 2011. A Practical Guide for Transforming Procurement Systems. Paris: OECD. ———. 2011. International Engagement in Fragile States: Can’t We Do Better?” OECD, Paris. ———. 2006. “Methodology for Assessment of National Procurement Systems.” OECD, Paris. United Nations Commission on International Trade Law (UNCITRAL) Secretariat.(1994, 2011) “UNCITRAL Model Law on Public Procurement.” UNCITRAL, Vienna, Austria. Volcker, Paul, Gustavo Gaviria, John Githongo, Ben W. Heineman Jr., Walter Van Gerven, and Sir John Vereker. 2007. Independent Panel Review of the World Bank Group Department of Institutional Integrity. Washington, DC: World Bank. World Bank. 1997. “The World Bank Procurement Function- Adjusting to Emerging Needs.” SecM97- 854, World Bank, Washington, DC. ———. 2002. Guidelines for Country Procurement Assessment Reviews. Washington, DC: World Bank. ———. 2005. “Meeting the Challenge of Africa’s Development: A World Bank Group Action Plan,” Africa Region Department, World Bank, Washington, DC. ———. 2007. “Strengthening World Bank Group Engagement on Governance and Anti-Corruption.” March 21, 2007, World Bank, Washington, DC. ———. 2008. “Use of Country Procurement Systems in Bank-supported Operations: Proposed Piloting Program,” R008-36/3, World Bank, Washington, DC. ———. 2010. “Guidance Note on the Operational Risk Assessment Framework (ORAF) – Risks to Achieving Results.” Operations Policy and Country Services, World Bank, Washington, DC. ———. 2010. “Piloting Program in the Use of Country Procurement Systems—Second Progress Report.” Washington, DC: World Bank. ———. 2010. “Procurement Risk Assessment & Management System User Guide.” OPSOR, May 2010 Updated July 2011, November 2011, January 2012.World Bank, Washington, DC. ———. 2011. “Strengthening World Bank Engagement on Governance and Anti-Corruption: Procurement Companion Piece.” World Bank, Washington, DC. ———. 2011. “Diversity Benchmarking Study in Selected Countries” OPSOR, Mimeo, September. ———. 2011. “Green Procurement Policy Framework in Selected Countries” OPSOR, Mimeo, September. ———. 2012. “Applying the Principles for Good International Engagement in Fragile States and Situations to Strengthen and Transform Public Procurement Systems.” Background note, OPSOR, World Bank, Washington, DC. ———. 2012. (February) “The Bank's Procurement Policies and Procedures: Initiating Discussion and Approach Papers - Proposed Review.”Report submitted to the Audit Committee and to CODE, World Bank, Washington, DC. ———. 2012. “Guidance Note for Design and Management of Procurement Responsibilities in Community- Driven Development Projects.” World Bank, Washington, DC. ———. 2012 (April). “Investment Lending Reform: Modernizing and Consolidating Operational Policies and Procedures.” OPCS, April 26, Washington, DC. ———. 2012. (February) “Virtual VP Review of the Procurement Policy Review Proposal--An Approach Paper.” World Bank, Washington, DC. ———. 2012 (April). “Update on the Bank’s Business Modernization: Results, Openness, and Accountability Spring 2012,” DC2012-0005 April 11, Washington, DC. World Trade Organization. 1996. “Agreement on Government Procurement (GPA).” World Trade Organization, Geneva. 35