Report No: ACS19338 Europe and Central Asia Enhancing Public Spending Transparency and Performance E-Procurement Forum Proceedings Report June 30, 2016 GGO03 EUROPE AND CENTRAL ASIA Document of the World Bank Table of Contents Introduction .................................................................................................................................................. 3 The reason to convene: ECA’s opportunity of better public resource management............................. 3 Knowledge Exchange .................................................................................................................................... 4 Country systems and experiences ........................................................................................................... 4 The role and experience of development partners................................................................................. 4 The vendors of E-GP solutions ................................................................................................................. 4 Indicators ...................................................................................................................................................... 5 Organic indicators as ranked by the discussion groups .......................................................................... 5 Challenges ................................................................................................................................................... 10 The level of E-GP adoption ..................................................................................................................... 10 The need for a holistic coverage of the system (360 degrees) ............................................................. 10 Need for close coordination between key actors involved .................................................................. 11 Way Forward .............................................................................................................................................. 11 Setting up a web-based data collection tool ......................................................................................... 11 Data collection for setting the baselines ............................................................................................... 12 Wider agreement on a set of common indicators................................................................................. 12 Piloting organic indicators ...................................................................................................................... 12 Annual review: results and improvements ........................................................................................... 12 Annexes....................................................................................................................................................... 12 Introduction Organized by the World Bank and co-sponsored by the European Bank for Reconstruction and Development (EBRD), the “E-Procurement Forum: Enhancing Public Spending� was conceived to facilitate the sharing of country experiences in public procurement performance measurement. It also aimed to develop and agree on a common set of performance indicators and to take action to set a baseline for all countries against the agreed indicators and their annual updates. The specific objectives of the Forum, which is expected to be an ongoing event, are aligned with the World Bank’s strategy: “Promoting E-Government Procurement (E-GP) – Strategy and Role of the World Bank in Europe and Central Asia (ECA) Countries,� published by the Bank in June 2015 (E-GP Strategy—see Annex I). The welcoming remarks of the representatives of the Austrian Government, the European Commission, the World Bank, and EBRD set the tone for the working sessions, conveying consistent messages in favor of strengthening knowledge and practice exchange to enable countries to move faster toward adopting E-Procurement (E-GP) and performance-oriented systems. Forum participants were able to learn from the on-the-ground experiences of participating countries, international development partners, and the commercial suppliers of E-GP–related tools and training. In addition, participants succeeded in developing a set of performance indicators deemed most critical to the implementation of procurement performance assessment systems. The reason to convene: ECA’s opportunity of better public resource management The promotion of E-GP in ECA countries fits within the Bank’s broader goal of good governance enhancement, which calls for the more sustainable management of public resources, including those delivered through current public procurement systems. A positive shift from a strictly legal compliance-oriented approach to a more performance-driven model is taking place in many countries worldwide, as public procurement policy makers and practitioners come to the conclusion that compliance per se, without pursuing better performance levels, does not serve the ultimate goal of providing the best service to citizens. Although E-GP has long been on the agenda of many countries and international organizations, there is currently an opportunity to emphasize the quantitative measurement of procurement performance and thus make public procurement more accountable. The scarcity of public budgetary resources in most countries, coupled with the increasing importance of making citizens aware of how their money is being spent (governance and procurement objectives), has led the World Bank, EBRD, and other multilateral development banks (MDBs), international financial institutions (IFIs), and development partners to put more focus on the way they can support countries to better measure the performance of their own public procurement systems. There is a broad consensus on the benefits countries can expect from collecting and using E-GP data, including data generated through the tools already in use on the E-GP adoption/uptake and the performance of the entire public procurement system. The availability, timely collection, and rigorous treatment and analysis of quantitative data allow for an assessment of whether the key objectives of both good governance and efficient public procurement have been fulfilled. When promptly and accurately collected and treated, the data sets will function as indicators that will significantly improve monitoring and accountability, as well as the quality, quantity, and analysis of available information. Overall, the use of good and reliable data and indicators will enable an improved decision-making process that will potentially translate into better public services and a higher level of citizen satisfaction. Nevertheless, many policy makers, regulators, and practitioners point out that data are often not easy to collect, or worse, sometimes simply do not exist because procurement activities are carried out through many different systems that are not linked or integrated in any way. There is also a recurrent criticism of the variety of entities that keep demanding the same information from different angles and perspectives without any harmonized set of commonly agreed indicators that would make the task of public procurers in charge of reporting information much less cumbersome. Document of the World Bank In view of the positive feedback of Forum participants and the challenges envisaged by the modernization of public procurement in the coming years, it is expected that the Forum will serve as a permanent platform for continuous knowledge exchange between public procurers in ECA countries, both by convening on a yearly basis and by maintaining the communication channel open through the web. The Forum was held in Vienna and involved 85 participants from 30 countries and nine partner organizations. All participating country contributions to the Forum are available on the Forum website: http://www.worldbank.org/en/events/2015/10/20/e-procurement-forum-enhancing-public-spending. Knowledge Exchange The Forum provided a unique opportunity to exchange knowledge at three different levels: Country systems and experiences All participating countries contributed information on their current public procurement system architecture and the key performance indicators on which they have already collected data. Taking into account the number of countries and the time constraints of a three-day event, the following 12 submissions were selected and vetted for presentation at the Forum and triggered very useful discussions: - Austria (Federal Procurement Agency) - Bangladesh (Central Procurement Technical Unit) - Bulgaria (Public Procurement Agency) - Czech Republic (experience of Zindex) - FYR Macedonia (Public Procurement Monitoring and E-Procurement Department) - Georgia (State Procurement Agency) - Italy (Consip - Concessionaria Servizi Informativi Pubblici) - Kosovo (Public Procurement Regulatory Commission) - Portugal (Impic – Ionsitituto dos Mercados Públicos do Imobiliário e da Construção) - Russian Federation (Ministry of Economic Development) - Serbia (Public Procurement Office) - Slovakia (experience of Transparency International) - Ukraine (Ministry of Economic Development and Trade – Division of Public Procurement Regulation Department) However, since all of the submissions invaluably contributed to the practice community, they are all published in full on the Forum website (see link above) in English and Russian and on the community of practice website (https://collaboration.worldbank.org/groups/e-procurement). In addition, a summary of each submission is available in Annex II of this document. The role and experience of development partners The views and experience of development partners and international organizations were also shared through the high-quality inputs of EBRD, the European Commission, the Islamic Development Bank (ISDB), the Millennium Challenge Corporation (MCC), the Organisation for Economic Co-operation and Development (OECD), the Support for Improvement in Governance and Management Initiative (SIGMA), the United Nations Commission On International Trade Law (UNCITRAL), and the World Bank. These organizations are committed to supporting member country efforts to reform their public procurement systems. To that end, they urged the countries to combine the implementation of E-GP with the adoption of more efficiency-oriented policies and practices that will be made possible through the substantial public procurement data that will be generated. The vendors of E-GP solutions Companies that have developed and are offering E-GP-related solutions and tools were invited to showcase their product and service portfolios. The attendance rate for participants at the vendor tables was nearly total, reflecting the high interest triggered by the initiative as well as the format and layout of the exhibition/meeting room. Instead of the standard exhibition booths, each vendor participated in a roundtable discussion, at which participants were able to raise questions, inquire about available solutions and tools that could potentially meet their needs, and follow live demonstrations and simulations. It should also be noted that the Forum agenda was designed to allow participants enough time to connect and exchange information and viewpoints through bilateral and/or informal discussions. In this way, the Forum generated an overall impression that the community of public procurement policy makers and practitioners is strengthening in the ECA region as a result of a growing exchange of experience, a genuine interest in benchmarking success cases, and the proactive search for common tools that could be used to evaluate and improve the performance of the national systems. The fact that some ECA countries are already European Union (EU) members and others are on their way toward accession as candidates or potential candidates may offer a particularly interesting motivation toward accelerating the pace of reforms. In fact, the new EU public procurement strategy and rules, under implementation since the adoption of the 2014 Directives, provide an external stimulus for modernizing the country systems. Among other policy directions, the EU’s reform of pu blic procurement has been generally designed to increase efficiency and promote value-for-money–oriented solutions and provisions, representing a significant evolution from a system that had been much more focused on transparency. In addition, a formal, mandated deadline of October 18, 2018 has been set for all EU member states to switch to a fully-fledged E-GP-based system, including e-tendering for all public contracting authorities. Indicators The World Bank’s E-GP strategy has been designed and should evolve with a view to “(i) strengthening public procurement systems in ECA countries by promoting further adoption of E-GP, (ii) developing e-GP capacity and sharing relevant experience, and (iii) building capacity in using electronic data to measure public procurement performance� (see “Promoting E-Government Procurement (E-GP)� in Annex I). The strategy is strongly focused on the quantitative measurement of performance. After proposing a methodology to set up a harmonized measurement system in each country as well as at a regional level, the strategy illustrates the potential of a quantitative-driven approach by attaching, on an indicative basis, a set of indicators and the possible algorithms needed to calculate them. (These can be found in Annex III-A, “Overview of Public Procurement Key Performance Indicators (Indicative List)� and III-B, “Overview of Public Procurement Key Performance Algorithms (Indicative List)� to the E -GP strategy document.) However, any set of indicators aimed at forming the basis of a measurement system needs to be implemented in a manner that takes into account the policy goals of the system’s beneficiaries as well as the system’s current level of functioning. Moreover, it is important to secure the participation of as many of the system’s stakeholders as possible in order to collect their views and understand their interests; participation and ownership will ensure that the entire system is perceived as trustworthy and as a beneficial working tool for all stakeholders within the procurement community. The 2015 E-Procurement Forum should be regarded as an activity conceived precisely with that aim: to learn from those who are involved in policy making, regulatory functions, or operations about the most relevant indicators, based on their needs, that should be used to measure system performance. The Forum in fact embodied this participative approach to building a measurement system. Organic indicators as ranked by the discussion groups Organic indicators resulted from a simple process that benefited considerably from the agenda’s structure, the quality and clarity of the presentations, and the fact that the audience consisted of public procurement experts. A simple two-step methodology was put in place in order to obtain, in the most spontaneous way, each participant’s views on the list of priority indicators. As a first step, each panel moderator identified five or six indicators—based on the presentations delivered and the dialogue with peer panelists—that could be considered the most relevant for beginning the measurement/collection process in the short term and grouped them under one of four categories. The second step consisted of inviting all discussion groups (tables A to L) to examine the list of indicators (the sum of the inputs collected from the moderators of sessions two–seven) and select two indicators per category. No country- or organization-based representation mechanisms were used in this process and random seating allowed for an intermingling of participants in terms of countries and/or affiliation, ensuring that the results reflected the individual opinions of every roundtable participant. Some discussion groups/tables selected the most relevant indicators per category, while others ranked them by session (two for each). The results were not compromised, as there already were a number of overlaps due to the fact that some indicators capture the same reality but are formulated in different ways (for example, some refer to different levels of analysis, i.e., indicator and sub-indicator). The work of enhancing the formulation of organic indicators and streamlining their categorization will be a key activity of the ongoing forum process and will benefit from the practical experience of the data collection and analysis. The table below shows the full list of indicators as suggested by each session moderator and the voting of all participants as a result of the discussion groups organized by table: VOTING BY PARTICIPANTS OVER THE INDICATORS PROVIDED BY PANEL MODERATORS DISCUSSION GROUPS (TABLES A to L) # INDICATOR CATEGORY A B C D E F G H I J L Total % Donor/Partner presentations 2.1. Level of competition (number of bidders) EFFICIENCY X X X X X X X X 8 7.48% SESSION 2 2.2. Transaction costs EFFICIENCY X X 2 1.87% 2.3. Price overruns/schedule and time overruns QUALITY EFFICIENCY 0 0.00% 2.4. Success ratio/cancellation EFFICIENCY X X X X X X X X X 9 8.41% Ratio for red flags (complaints could be merged/combined - % 2.5. TRANSPARENCY X X 2 1.87% of successful complaints) # INDICATOR CATEGORY A B C D E F G H I J L Total % Ratio of tenders dropped against tender documents handed out 3.1. COMPETITIVENESS 0 0.00% Country presentations (sold out) SESSION 3 3.2. % of contracts signed within the validity period EFFICIENCY X X 2 1.87% 3.3. Procurement value passed through e-platforms E-GP ADOPTION X X X X X X X X 8 7.48% 3.4. Savings (in unit prices) per CPV code EFFICIENCY X 1 0.93% 3.5. Average length of process by type of procurement procedure EFFICIENCY X X X X 4 3.74% 3.6. % deviation contract price vs. final actual price EFFICIENCY X X X 3 2.80% # INDICATOR CATEGORY A B C D E F G H I J L Total % EFFICIENCY Country presentations 4.1. Cancelled contracts X X X X 4 3.74% GOVERNANCE SESSION 4 Bidder participation - average bid count in one sector vs. 4.2. COMPETITIVENESS X X X 3 2.80% national Winner concentration - supplier rankings via value and volume COMPETITIVENESS 4.3. X X 2 1.87% per year. GOVERNANCE 4.4. Prices of categories over time and bidder participation COMPETITIVENESS 0.00% 4.5. Professionalization of the PP officials and practitioners ADOPTION X X X X X X 6 5.61% # INDICATOR CATEGORY A B C D E F G H I J L Total % Country presentations Centralized vs. decentralized procurement in the country 5.1. EFFICIENCY X X 2 1.87% SESSION 5 (number and value) 5.2. Savings in works, goods, and services (number and value) EFFICIENCY X X X X 4 3.74% Competition: number of valid bids vs. number of valid 5.3. COMPETITIVENESS X 1 0.93% international bids Open procedures with notice vs. closed procedures by 5.4. COMPETITIVENESS X X X X 4 3.74% invitation 5.5. Open access to procurement information and decisions TRANSPARENCY X X X X X X X X X 9 8.41% # INDICATOR CATEGORY A B C D E F G H I J L Total % 6.1. Up-take of e-Procurement ADOPTION OF E-GP X X X 3 2.80% Donor/Partner presentations 6.2. Ratio direct vs. competitive procurement methods COMPETITIVENESS X X X 3 2.80% Number of bids and diversity of supplier base/repeated wins COMPETITIVENESS SESSION 6 6.3. X 1 0.93% (collusion) TRANSPARENCY How many international bidders come to country-based 6.4. COMPETITIVENESS X X X X 4 3.74% procedures How do price estimates relate to contract award and price paid at EFFICIENCY 6.5. X 1 0.93% the end TRANSPARENCY 6.6. Award time (efficiency) and unsucceful procurements EFFICIENCY X 1 0.93% EFFICIENCY 6.7. Number of complaints vs. successful complaints X X X X 4 3.74% TRANSPARENCY # INDICATOR CATEGORY A B C D E F G H I J L Total % 7.1. Total number of bids vs. total number of competitive tenders COMPETITIVENESS X X 2 1.87% Length of procedures from publication of tender documents to 7.2. EFFICIENCY X X X 3 2.80% Country presentations conclusion of contract SESSION 7 Share of contracts concluded with SMEs in total value/number 7.3. COMPETITIVENESS X X 2 1.87% vs. total public contracts Share of concluded contracts below the thresholds for small 7.4. value procurements that are not published in the public TRANSPARENCY X X 2 1.87% procurement portal % of contracts awarded through competitive procedures vs. 7.5. TRANSPARENCY X X X 3 2.80% total contracts % of cancelled procedures vs. total number of published COMPETITIVENESS 7.6. X X X X 4 3.74% procedures GOVERNANCE TOTAL VOTES 107 100.00% The results from the Forum’s discussion groups provided an indication of how useful a properly organized and implemented measurement system will be for policy makers and practitioners. The request that only the two most relevant indicators be selected per category or, as happened within some discussion groups, per session had the merit of requiring a more detailed assessment behind each indicator. The graph below shows the top five indicators (out of 34) that together represent more than a third (37.39 percent) of total votes cast, thus allowing for the identification of the five key concerns of public procurers in the forum: (i) Success ratio/cancellation of procedures before awarding a contract (ii) Open access to procurement information and decisions (iii) Level of competition (number of bidders) (iv) Procurement value passed through e-platforms (v) Professionalization of public procurement officials and practitioners Top 5 indicators Professionalizati on of the PP officials and Success ratio/ practitioners cancellation [PERCENTAGE] [PERCENTAGE] Procurement value passed through e-platforms [PERCENTAGE] [CATEGORY [CATEGORY NAME] NAME] [PERCENTAGE] [PERCENTAGE] Participants in all discussion groups/tables were also asked to qualify the selected indicators under the various categories. The graph below shows the results: Ranking by category Quality/ [CATEGORY NAME] Efficiency [CATEGORY NAME] [VALUE] [VALUE] [VALUE] [CATEGORY NAME] [VALUE] [CATEGORY NAME] [CATEGORY NAME] [VALUE] [VALUE] [CATEGORY NAME] [VALUE] [CATEGORY NAME] [CATEGORY NAME] [VALUE] [VALUE] The above list of categories can be streamlined in order to reduce overlaps and increase consistency throughout the measurement system. The suggested solution is to merge the categories that are now overlapping as follows: “Efficiency/Governance,� “Efficiency/Transparency,� and “Quality/Efficiency� could merge into the broader category of “Efficiency,� while “Competitiveness/Governance� and “Competitiveness/Transparency� could merge into “Competitiveness.� As a result of this aggregation, the categories would show the following results: - Efficiency (36.45 + 3.74 + 4.67 percent = 44.86 percent) - Competitiveness (0.93 + 5.61 + 17.76 percent = 24.30 percent) - Adoption (15.89 percent) - Transparency (14.95 percent) Transparency Adoption 14.95% 15.89% Competitivene Efficiency ss 44.86% 24.30% Aggregated categories After aggregating the indicators under the above four categories, efficiency is clearly the most relevant, representing almost half of the participants’ expressed concerns. It should also be noted that competitiveness represents a quarter of total votes cast, which, when compared to the much lower scores of transparency and adoption (of E-GP), confirms that the audience is in line with the current policy trend—there is a clear shift from transparency toward efficiency and competitiveness. Adoption should thus be regarded as instrumental to reaching the other goals, which explains its slightly low figure. This category will likely tend to decrease over time as the adoption programs deploy. Based on the aggregated categories above, the following seven indicators will be suggested as a basis for the initial performance measurement system (the savings indicator included as the sum of indicators 3.4, 3.6, 5.2, and 6.5 in the table above): Category Indicator % How to measure Percentage of procedures without Success ratio/cancellation 8.41% cancellation Percentage of savings from award Efficiency price vs. cost estimate; Savings 8.40% Percentage of savings from final price after contract completion vs. award price Percentage of procurement opportunities advertised on a portal Open access to procurement Transparency 8.41% with free access; information and decisions Percentage of awarded contracts published on a portal with free access Procurement value passed Percentage of procurement value 7.48% through e-platforms with online transactions Adoption Professionalization of the PP Percentage of certified E-GP users in 5.61% officials and practitioners contracting authorities Level of competition 7.48% Average number of bidders Competitiveness Open procedures with notice Percentage of number of contracts vs. closed procedures by 3.74% procured competitively invitation Challenges A number of challenges sprang from the presentations and ensuing discussions. The following are highlighted as the most critical to the adoption of performance-oriented procurement in the near future: The level of E-GP adoption The availability of reliable and real-time performance indicators depends largely on the extent of the adoption of E-GP in each country, as well as the possibility of using data mining and business intelligence tools to transform huge amounts of raw data into meaningful information that could be used for decision making at both the policy and operational levels. There is no doubt that all GP will eventually become E-GP; it is only a question of time. The challenge is to guarantee that the necessary political drive, capacity building, and change management are deployed in order to accelerate the pace of this reform. The need for a holistic coverage of the system (360 degrees) Even in countries where E-GP solutions are already in operation, the whole public procurement “universe� is not covered in every instance. The result is that in some cases, the E-GP–generated data do not reflect all types and methods of procurement, nor do they capture all the activity of every contracting entity. In fact, to be meaningful, most indicators require basic holistic statistical information. For example, to be adequately calculated, the indicator “Procurement value passed through e -platforms� versus total value of awarded contracts requires that the total awarded value, without exception, be known. Several participants in the Forum mentioned the difficulties often faced in tracking and recording every single public procurement operation and contract. Often times, some types of contracts escape statistical oversight, for example, some call-off contracts awarded under a framework agreement or contracts awarded by state-owned companies. Thus the challenge is how to capture as much of the entire public procurement structure as possible in a holistic manner so that policy makers, regulators, and procurers can be afforded a 360 degree vision of the entire system. Need for close coordination between key actors involved In view of the need to adopt a set of common key performance indicators that can meet the needs of policy makers, regulators, and procurers without making their data-related tasks too cumbersome, it is critically important that MDBs, IFIs, development partners, and the relevant government authorities work together to move forward on this task. Way Forward Based on the list of organic indicators that were collected after two rounds of discussions, the first promoted by each session moderator with peer panelists and the second conducted at all nine tables of participants, a simple measurement system can be set up for the short term. As noted above, one of the key objectives of the Forum was to reach an agreement on a common set of indicators that could be used by ECA countries to measure their procurement systems. Having fulfilled that objective, it is now critical to deploy a straightforward action plan aimed at setting the baseline for every ECA country. Although the concrete measures will have to be discussed and agreed upon on a bilateral basis with each country authority in charge of providing the relevant data, the following could be envisaged as steps toward a common rollout during 2016: Setting up a web-based data collection tool The World Bank will create a web-based tool enabling countries to upload data relating to the agreed list of indicators. Although not yet integrated or connected with any national information and communications technology (ICT) system or tool, it should nevertheless allow for an easy collection of data provided by each country/agency in as many electronic formats as possible. The Bank’s tool is first and foremost expected to set up a very basic process based on (i) the input of data by the accredited country representatives, followed by a quick validation in order to ensure compliance with the protocol for collecting data or (ii) the transfer of the data in a predefined file format for input into the web-based tool by the entity in charge of its administration. Data collection for setting the baselines It is important to set the baseline (year zero) for the organic indicators agreed in the workshop and to pilot a basic measurement system. At this stage, data will be collected from the various national authorities who participated in the workshop or by the relevant national agency designated for this purpose. Wider agreement on a set of common indicators As already foreseen in the E-GP strategy, the highest priority should be given to joining forces with the coordinating efforts of EBRD, EU, OECD, the World Trade Organization (WTO), and UNCITRAL, with the aim of designing and developing a common system of performance indicators (E-GP adoption and performance indicators). In addition, the national authorities and agencies in charge of managing public procurement should be consulted on a bilateral basis before further developments of the measurement system can be proposed, adopted, and implemented. This measurement system should be simple, as automated as possible (which is why the strategy links the measurement system so closely to E-GP adoption), reliable, and relevant (if coupled, as suggested, with analytical features, such as independent reports, opinions, an outlook on trends, etc.). Piloting organic indicators Once the data have been collected and baselines set, piloting will start in collaboration with the national entities) in charge of providing the information. A permanent dialogue between the Bank and the national entities) will be maintained in order to discuss details and validate information. The process of fine-tuning the formulation and categorization of the organic indicators will start during the piloting process. This will ensure that the key performance indicators are formulated in such a way as to capture the target information and allow for a meaningful data comparison, both along a predefined time series and on a regional level. Annual review: results and improvements It was agreed that the Forum should gather on an annual basis in order to provide the public procurement community—ECA countries, MDBs, IFIs, development partners, and E-GP solution providers/vendors—a regular update on the progress made on E-GP adoption and the setup and real use of measurement systems based on the agreed key performance indicators. Without prejudice regarding the inclusion of any other items of interest, the standard/recurrent agenda of the annual forum should include the presentation and discussion of an Annual Report on the World Bank (ECA Region) Public Procurement Measurement System (ECA – PPMS), comprising the following sections: (i) a review of the indicators in use (the formulation of indicators and the algorithms used for calculating them), (ii) a classification of the categories of indicators, (iii) an assessment of the improvement of data collection and treatment, procedures, and protocols, (iv) an appraisal of the major shortcomings detected and corrective measures envisaged, and (v) an annual Activity Plan (from forum to forum). Annexes  ANNEX I – E-GP-Strategy of the World Bank in Europe and Central Asia countries, June 2015  ANNEX II – Summary of all Country Submissions  ANNEX III – List of Participants  ANNEX IV – Forum Agenda Standard Disclaimer: This volume is a product of the staff of the International Bank for Reconstruction and Development/ The World Bank. The findings, interpretations, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Copyright Statement: The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/ The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, http://www.copyright.com/. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org.