38751 IDA15 IDA's Performance-Based Allocation System: Options for Simplifying the Formula and Reducing Volatility International Development Association Resource Mobilization (FRM) February 2007 List of Abbreviations and Acronyms ARPP Annual Review of Portfolio Performance CPIA Country Policy and Institutional Assessment CPIAA-C Country Policy and Institutional Assessment, Clusters A-C CPIAD Country Policy and Institutional Assessment, Cluster D CPR Country Performance Rating FY Fiscal Year GOV Governance Rating GNI Gross National Income IDA International Development Association ISR Implementation Status Report MDB Multilateral Development Bank PBA Performance Based Allocation PCEF Post Conflict Enhancement Factor PIP Portfolio Improvement Program POP Population PORT Portfolio Performance Rating QAG Quality Assurance Group Table of Contents Executive Summary................................................................................................................. ....i I. Introduction ........................................................................................................................1 II. IDA's PBA System..............................................................................................................1 III. Options for Simplifying the PBA Formula..........................................................................4 A. Background....................................................................................................................4 B. Setting the Stage for Simplification ...............................................................................5 C. Proposed Options for Simplification..............................................................................7 D. Comparing the Options and the Way Forward ............................................................11 IV. Volatility of Country Performance Ratings .........................................................................16 A. Using Actual Problem Projects Only..........................................................................18 B. Averaging Actual Problem Projects............................................................................18 C. Changing the Conversion Scale..................................................................................19 V. Questions for Deputies...........................................................................................................21 List of Charts Chart 1: IDA Country Performance Rating .................................................................................2 Chart 2: Comparing Ratings and Allocations Under Option I.....................................................8 Chart 3: Comparing Ratings and Allocations Under Option II .................................................10 Chart 4: The Linear and Geometric Options Closely Track Benchmark Allocations ..............12 Chart 5: Minor Differences between Options I and II...............................................................15 Chart 6: Steps for Calculating the Current Portfolio Performance Ratings ..............................17 List of Tables Table 1: Resource Allocation Formulas ......................................................................................4 Table 2: Two Options for Simplifying the PBA Formula ...........................................................7 Table 3: Predicting Changes in Allocations with Option I..........................................................9 Table 4: Predicting Changes in Allocations with Option II.......................................................11 Table 5: Summary Table Comparing Options with the Benchmark..........................................12 Table 6: Different Country Performance Rating Scales Generate the Same Shares .................13 Table 7: Reducing Portfolio Performance Rating Volatility ....................................................20 Annex 1: Overview of IDA's Performance-Based Allocation System .....................................22 Annex 2: Steps for Calculating Country Performance Ratings ................................................26 Annex 3: Methodology for Calculating the Effective Weight of Governance..........................29 Annex 4: Comparison of Benchmark Formula with Options I and II .......................................31 Executive Summary 1. Purpose. Each year, IDA allocates the bulk of its resources to partner countries using the Performance-Based Allocation (PBA) system. This PBA system, which balances country performance and needs, has evolved over the past few replenishment cycles. Over time, however, the system has become more complex and volatile. At the IDA14 Mid-Term Review, Deputies requested management to "... simplify the allocation formula and reduce unwarranted ratings volatility."1 This paper responds to that request. 2. Simplification. The options for simplification focus on the country performance ratings, a major component of the PBA system. At the IDA14 Mid-Term Review meeting, Deputies provided two guiding principles for simplification. First, they requested that simplified options retain the weight of governance similar to the current formula. Second, they asked that the new options track closely the allocations from the current formula to minimize disruptions at the country level. With this guidance in mind, two options are proposed ­ a geometric formula (Option I) and an additive formula (Option II). Current Formula Country performance rating = (0.8 *CPIA + 0.2* PORT) * (GOV/3.5)1.5 IDA country allocation = f (Country performance rating , Population1.0, GNI/capita-0.125) 2.0 Where, GOV is the average rating of six governance criteria, 5 of which are drawn from CPIA (cluster D)2 and the procurement rating drawn from the Annual Review of Portfolio Performance. Option I Country performance rating = (CPIA A-C) 0.24* (CPIA ) 0.68 D * (PORT) 0.08 IDA country allocation = f (Country performance rating 5.0, Population1.0, GNI/capita-0.125) Option II Country Performance Rating = (0.24 * CPIA A-C + 0.68 * CPIA + 0.08 * PORT) D IDA country allocation = f (Country performance rating 5.0, Population1.0, GNI/capita-0.125) 3. A comparison of the options shows that: (i) both Options I and II make the formula much more transparent compared to the current formula, with Option II being the 1 IDA (2006). "Chairman's Summary: IDA14 Mid-Term Review Meeting." 2 CPIA refers to the Country Policy and Institutional Assessment ratings, which has four clusters A-D. Clusters A-C assess a country's socio-economic and structural policies while Cluster D assesses public sector management and institutions. -ii- simpler of the two; (ii) in Option I, proportional changes matter whereas in Option II absolute changes matter. Theoretically, this could have interesting policy implications over time. Option I provides larger proportional changes in allocations at lower levels of performance than at the top. Option II treats countries the same way at all levels of performance. In practice, however, a closer examination of underlying data shows that these differences are small; and (iii) Option I leads to closer harmonization with the Asian Development Fund. 4. The proposed options make it much easier for country teams to explain the country performance ratings formula to their counterparts in partner countries. The impact of each component of the country performance rating is now very clear. However, while the coefficients/exponents derived in this simplification exercise help minimize the differences in allocations between the current formula and the proposed options, they are still not intuitively easy to comprehend. Some further minor simplifications, such as rounding the coefficients/exponents could make the formula not just mathematically simpler, but also intuitively easier for country teams to explain to their counterparts in partner countries. 5. Volatility. Unwarranted volatility in country performance ratings ­ and therefore allocations ­ comes from the way the portfolio performance ratings are computed. To mitigate this, three technical fixes are suggested to modify the calculation of portfolio performance ratings. These include: (i) using actual problem projects instead of actual plus potential problem projects; (ii) using the average data on actual problem projects at the end of four quarters rather than an end-year snapshot; and (iii) revising the conversion scale used to convert the percent of projects at risk into a rating. 6. Next steps. The paper ends with three questions for the Deputies. · Would Deputies support simplification of the PBA formula along the lines suggested in Options I or II? · Would Deputies be supportive of simplifying the formula further by rounding the exponents/coefficient to make the formula more intuitively easy to explain? · Do Deputies support management proposals to lower unwarranted volatility in the country performance ratings through changes in the way the portfolio performance ratings are calculated? Specifically, do they support using: (i) information on actual problems projects only instead of actual plus potential problem projects, (ii) averaging end-quarter actual-problem-project data, and (iii) the new conversion scale? IDA's Performance-Based Allocation System: Options for Simplifying the Formula and Reducing Volatility I. Introduction 1. Purpose of the paper. In a discussion of IDA's Performance Based Allocation (PBA) system at the IDA14 Mid-Term Review held in November 2006, Deputies supported management's proposal to "...simplify the allocation formula and reduce unwarranted ratings volatility."3 Complexity in the current formula used to calculate country performance ratings is due to the double counting of governance elements and the structure of the formula. Unwarranted volatility in country performance ratings4 is mainly due to the portfolio performance ratings.5 This paper proposes options for simplifying the formula used to calculate the country performance ratings and for reducing their excessive volatility. 2. Timing of the decisions. Regarding options for simplifying the formula used to calculate the country performance ratings, management aims to reach closure at the second IDA15 Replenishment meeting scheduled for June 2007. On the proposals to reduce unwarranted volatility due to the portfolio performance ratings, management would like to reach a decision at the time of the Paris meeting in March 2007, so that the changes can be applied to the FY08 IDA allocation exercise. 3. Layout of the paper. This paper is structured as follows. Section II provides background for the paper with an overview of IDA's PBA system. Section III explores options to simplify the formula used to calculate the country performance ratings. Section IV suggests ways of reducing volatility of country performance ratings by modifying the calculation of the portfolio performance ratings. Section V concludes with some questions for the Deputies on the way forward. Annex 1 describes the PBA system in detail. Annex 2 describes the steps involved in calculating country performance ratings in detail. Annex 3 outlines the steps to calculate the effective weight of governance in the current country performance rating. Annex 4 shows detailed comparative statistics on the two proposed options for simplification. II. IDA's PBA System6 4. The PBA system. IDA allocates the bulk of its resources to partner countries using the PBA system. 7 As background to the paper, this section provides a short description of the main components of the PBA system. 3 IDA (2006). "Chairman's Summary: IDA14 Mid-Term Review Meeting." 4 IDA country performance ratings are the main determinants of IDA allocations along with population and, to a lesser extent, GNI per capita. 5 Portfolio performance ratings, a component of IDA country performance ratings, measure the health of IDA's active portfolio in each country. 6 This section is drawn from IDA (2006), "IDA's Performance Based Allocation System: a review of the governance factor." -2- 5. The PBA system and the country performance ratings. In the PBA system, the country performance ratings play a major role. Chart 1 shows the main components of the country performance ratings ­ the Country Policy and Institutional Assessment (CPIA), the portfolio performance ratings and the governance factor. Each year, Bank country teams prepare CPIA ratings8 for their respective countries, which assess the quality9 of a country's present policy and institutional framework. Portfolio performance ratings, constructed from the Bank's Annual Review of Portfolio Performance (ARPP), reflect the percentage of IDA-funded projects at risk in each country. The governance factor, calculated using parts of the CPIA and portfolio ratings, aims to provide extra weight to public sector management and institutions in a country to ensure that IDA resources are being used well. The CPIA, portfolio performance ratings and governance factor feed into the calculation of the country performance ratings, all of which are now publicly disclosed by IDA. Chart 1: IDA Country Performance Rating Country Policy and Institutional Portfolio Performance 80% 20% Assessment (CPIA) Rating (from ARPP) Weighted Average 5 governance 1 governance related related indicator indicators from from ARPP the CPIA Governance Factor IDA Country Performance Rating Source: Based on Annex 1 of the IDA14 Deputies' Report 7 An estimated 62 percent of IDA14 resources will be allocated using the PBA formula. In addition, around 20 percent of IDA14 resources are estimated to go to capped-blend countries (India, Indonesia and Pakistan). Around 10 percent of resources are estimated to go to post-conflict countries while another 8 percent will go for special purposes agreed upon during the replenishment discussions. 8 The CPIA is initially prepared by Bank country teams, after which it is subject to an extensive Bank- wide review. 9 "Quality" refers to how conducive that framework is to fostering poverty reduction, sustainable growth, and effective use of development assistance. -3- 6. The equation for calculating the country performance rating for each country is shown below. The weighted average of the CPIA rating (80 percent) and the portfolio performance rating (20 percent) is multiplied by the governance factor. Equation (1): Country performance rating formula Country performance rating = (0.8 * CPIA + 0.2 * portfolio performance rating) * governance factor 7. The governance factor. As seen in equation (1), the governance factor is applied as a multiplier. It is made up of six criteria,10 of which five are drawn from Cluster D11 of the CPIA ratings and the sixth from the ARPP. The governance factor for each country is calculated as follows: Equation (2): The governance factor Governance Factor = (average rating of 6 governance criteria / 3.5)1.5 The average rating is divided by 3.5, which is the mid-point of the CPIA scale, and then raised to an exponent of 1.5, forming the governance factor. So for governance scores above 3.5, the rating is increased while for scores below 3.5, it is decreased. 8. Balancing country performance and needs. In addition to the country performance ratings calculated as described above in equation 1, IDA also uses measures of population and GNI per capita in allocating resources. Equation 3 shows that a country's performance is the dominant determinant of IDA allocations ­ a score twice as high would result in four times the allocation, other things remaining constant. Population also affects allocations significantly ­ the relationship is linear with the population term, whereby a higher population results in a proportionately increased allocation. In addition, while all IDA countries are poor, there is an additional modest bias towards countries with lower GNI per capita. Finally, each country also receives a base allocation of SDR 1.1 million per annum, which favors the small countries. Equation (3): PBA formula IDA country allocation per annum = base allocation + f (Country performance rating 2.0, Population1.0, GNI/capita-0.125) 10 The governance factor in IDA14 includes one less criterion than in IDA13. The omitted criterion is the result of the consolidation of the CPIA questionnaire in 2004, which reduced the number of criteria in the CPIA from 20 to 16. 11 Cluster D is the final cluster of the CPIA ratings (see Annex 1, Box A1.1), which measure public financial management and institutions. The other 3 clusters, A-C, rate socio-economic and structural polices. -4- III. Options for Simplifying the PBA Formula A. Background 9. Criteria for allocating aid. Most Multilateral Development Banks (MDBs) have adopted "rules-based" methods for allocating scarce concessional resources to partner countries. The formulas used by the African Development Fund, the Asian Development Fund and IDA to allocate aid are shown in Table 1. In their aid allocation formulas, the three MDBs use similar components to strike a balance between the performance of a country and its needs, with the former as the main determinant of allocations. All three formulas capture country performance through the CPIA ratings, portfolio performance ratings and governance. Country needs are largely captured by including measures of population and GNI per capita in the formula. Table 1: Resource Allocation Formulas African [(0.7*CPIA + 0.3*PORT) * (GOV/3.5) * PCEF] 2.0* Pop * GNI -0.125 p.c Development Fund Performance Needs Asian [ES_CPIA 0.7* PORT 0.3* GOV] 2.0* Pop0.6 * GNI -0.25 p.c Development Fund Performance Needs International 2.0 -0.125 Development [(0.8*CPIA + 0.2*PORT) * (GOV/3.5)1.5] * Pop * GNI p.c Association (IDA) Performance Needs Notes: 1. CPIA: Country Policy and Institutional Assessment; ES_CPIA is CPIA clusters A-C; PORT: Portfolio performance ratings; GOV: Governance rating; Pop: Population; GNI p.c: GNI per capita; and PCEF: Post-Conflict Enhancement Factor, which is equal to 1 when it is not a post conflict country. 2. IDA and African Development Fund have base allocations. 10. Important differences. Despite using similar components, the three MDBs combine them in slightly different ways in their formulas, with interesting implications. For instance, because of the way the components are combined, the weights of governance differ across the MDBs ­ 59 percent for the African Development Fund, 50 percent for the Asian Development Fund, and 68 percent for IDA.12 In contrast to IDA and the African Development Fund, the Asian Development Fund does not double count governance. Moreover, the dispersion of country performance ratings also varies across the MDBs, allowing for differences in the way good performers are distinguished from the weaker ones. In addition, the Asian Development Fund uses a smaller exponent on population compared to IDA and the African Development Fund, thus favoring smaller 12 This effective weight of governance of 68 percent is derived from an econometric exercise and is slightly different from the 66 percent (obtained as shown in Annex 3) cited in earlier papers on PBA system. -5- countries. Instead, IDA and the African Development Bank have base allocations that favor small countries. Finally, the African Development Fund includes a Post-Conflict Enhancement Factor (PCEF) in the PBA formula to direct more resources to countries emerging from conflict. IDA directs resources to post-conflict countries using the Post- Conflict Progress Indicators, although the term is not directly built into the PBA formula. 11. IDA's PBA formula has become complicated. As with the other MDBs, IDA's PBA formula has evolved over time taking into account new analytical insights and donor priorities. Reflecting this, IDA's formula to calculate country performance ratings has been amended five times during the IDA11-13 period. While such changes were made to increase the efficient use of IDA's resources, they also made the formula more complex over time. As a result, it is now difficult to disentangle the impact of each component on allocations at the country level. 12. Simplification is necessary. Each year, billions of dollars are channeled to countries using the PBA formula. Since 2006, IDA has been at the forefront in enhancing the transparency of its resource allocation system through public disclosure of the ratings that feed into the PBA formula. However, given the complexity of the formula, the way in which the ratings affect the final allocations is not readily evident. The lack of transparency in the current formula comes from two main sources: (i) the double counting of governance and (ii) the structure of the formula, which includes a combination of additive and multiplicative/geometric element.13 Simplifying the formula would enhance its transparency making it easier for partner countries and country teams to better understand what drives changes in their allocations. 13. Guiding principles for simplification. While recognizing the need for simplification at the IDA14 Mid-Term Review, participants expressed their preference at the same time for : · Retaining the "weight given to governance"14 as in the current formula. · Maintaining, to the extent possible, the current allocations per capita and differentiation between the stronger and weaker performers. These guiding principles are adhered to in designing the options for simplifying the formula. B. Setting the Stage for Simplification 14. How to simplify the formula. Given the guiding principles described above, this section explores options for simplifying the current formula. This is mainly achieved by: 13 Once the performance-based allocations are determined using the formula described in Section II, further steps involve grants-related discounts and the Multilateral Debt Relief Initiative (MDRI) netting-out mechanism. These two additional steps add to the complexity of the allocation exercise. The focus in this section is only on the PBA formula. 14 IDA (2006). "Chairman's Summary: IDA14 Mid-Term Review Meeting." -6- · Changing the structure of the formula. Retaining only an additive or a geometric structure in a formula makes interpretation much easier. Take, for instance, a linear formula such as country performance ratings = x*CPIA + y*PORT + z*GOV. An improvement in CPIA of w would always increase country performance ratings by w*x. In a geometric formula, country performance ratings = CPIAx * PORTy * GOVz, a w percent increase in CPIA can be approximated to an x*w percent increase in country performance ratings. The current formula includes both additive and multiplicative/geometric elements so that the final effect on country performance ratings cannot be readily discerned. It would, therefore, be advisable to retain either the additive or geometric structure but not combine them. · Eliminating double counting. The double counting of governance makes it difficult to isolate the impact of changes in governance alone on the country performance ratings. Cluster D of the CPIA (see Annex 1) enters the country performance ratings formula twice ­ in the CPIA ratings (in a linear way) and in the governance factor (in a multiplicative/exponential way). In addition, procurement ratings enter into the portfolio performance ratings as well as in the governance factor.15 A way to simplify it would be to account for governance only once, while adjusting its exponent/coefficient within the formula to maintain a similar overall weight. So cluster D should be counted just once and dropping potential problem projects from the calculation of portfolio performance ratings (to be discussed later, see Section IV), would imply that the procurement rating will also have to be excluded from the calculation of the governance factor.16 15. Getting started. To get started, it is necessary to establish a benchmark to compare the proposed options and to establish the weights of each of the components of the formula. · Benchmark formula. An appropriate benchmark has to be set up to compare the options. For this, the current formula is used after incorporating the revised portfolio performance ratings described later in Section IV and dropping the procurement rating from the governance factor. · Weights of other components. The proposed simplification options try to retain not just the current weight of governance but also the relative impacts of the other two components ­ CPIA A-C and portfolio performance ratings. Using the benchmark formula, these relative weights are estimated by regressing the 15 The double counting of the procurement rating is complicated. For a project to count as a potential problem project, three or more "flags" need to be raised. If one of these three flags happens to be the procurement flag, then it gets double counted ­ once in the portfolio performance ratings and then again in the governance factor. But it is possible that for a project, only one "flag" ­ the procurement flag is raised. In that case, it enters into the governance factor but not in the portfolio performance ratings ­ therefore there is no double counting. 16 Governance experts within the World Bank are exploring the possibility of highlighting more explicitly the quality of procurement practices in the appropriate criteria within the CPIA governance cluster. -7- logarithm of country performance ratings on the logarithms of CPIA A-C , portfolio performance ratings (using changes described later in Section IV), and cluster D. The regression yields the following coefficients: 0.6 for CPIA A-C, 0.2 for portfolio, and 1.7 for the CPIA . D C. Proposed Options for Simplification 16. Table 2 shows the current formula, the benchmark formula and two options for simplification. Table 2: Two Options for Simplifying the PBA Formula Country performance rating formula Exponent applied to the country performance rating Current Formula (0.8*CPIA + 0.2*PORT old) * (GOV old /3.5)1.5 2 Benchmark formula (0.8*CPIA + 0.2*PORT) * (GOV/3.5)1.5 2 Option I (CPIA A-C)0.24* (CPIA )0.68* (PORT)0.08 D 5 Option II 0.24*CPIA A-C + 0.68*CPIA + 0.08*PORT D 5 Notes: 1/ CPIA A-C is the average score of CPIA clusters A, B, and C and CPIA is the average score of CPIA cluster D. D 2/ GOV oldis the average of the CPIA cluster D and the procurement rating. 3/ PORT old does not include changes proposed in Section IV. 4/ Benchmark formula is different from the current formula because it uses portfolio performance ratings which were re- calculated using the proposed changes in Section IV. Also, since potential problem projects are no longer considered, the procurement flag is also dropped from the governance factor. 17. Option I: a geometric formula. One way of simplifying the formula is to multiply the components and raise them to exponents that reflect their respective weights (derived as shown in paragraph 15). This specification is similar to the one used by the Asian Development Fund. Country performance rating = (CPIA A-C) 0.6 * (CPIA ) 1.7 D * (PORT) 0.2 IDA country allocation = f (Country performance rating 2.0 , Population1.0, GNI/capita-0.125) 18. Relative rankings are important. CPIA A-C is raised to an exponent of 0.6, portfolio performance rating to 0.2 and CPIA (governance) to 1.7 to retain the effective D weights as in the benchmark formula. Using these exponents causes the country performance ratings to range from 1 to 84,17 well beyond the range of current country performance ratings (0.15 - 13.5). Although the numerical values of country performance ratings generated by this formula are different from those generated by the benchmark formula, the resulting allocations per capita are very similar because the relative country rankings are preserved. However, the exponents used in this formula limit its transparency since they add up to 2.5 and interpreting the weight of each 17 If the country scores 6 on CPIA A-C, 4.5 on portfolio, and 6 on CPIA , then the country performance D rating would be around 84. -8- individual component requires an additional step. For example, the weight of CPIA D (governance) has to be calculated as (1.7 / (1.7+0.6+0.2)) = 68 percent. 19. Rescaling the exponents. To further enhance the formula's transparency, the exponents can be rescaled to sum up to one. Not only does this make interpretation easy, it also has some attractive mathematical properties.18 The proposed new exponents of 0.24, 0.08 and 0.68 for CPIAA-C, portfolio performance ratings, and CPIA respectively D now reflect the "weights" or "shares" of the components. Option I: Country performance rating = (CPIA A-C) 0.24 * (CPIA ) 0.68 D * (PORT) 0.08 20. Obtaining the same dispersion in allocations. The rescaled exponents now limit the range of country performance ratings from 1 ­ 6, while in the benchmark formula the ratings range from 0.15 - 13.5. The same underlying data would therefore generate country performance ratings that are less dispersed (Annex 4), resulting in less differentiation between the strong and weak performers. To obtain the same dispersion of allocations as in the benchmark formula, the exponent in country performance ratings has to be increased from 2 to 5. This higher exponent minimizes the sum of squared differences between the per capita allocations that emerge from the benchmark formula and those that emerge from Option I. Chart 2 compares the ratings and allocations that emerge from the benchmark and Option I formulas. IDA country allocation = f (Country performance rating 5.0, Population1.0, GNI/capita-0.125) Chart 2: Comparing Ratings and Allocations under Option I Comparing IDA Country Performance Ratings Comparing Allocations per capita 6.0 16.0 14.0 5.0 12.0 4.0 10.0 I I noti 3.0 onit 8.0 Op Op 6.0 2.0 4.0 1.0 2.0 0.0 0.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 Benchmark Benchmark 18 This geometric weighted formula in which weights sum up to one makes the function linearly homogenous and therefore, it exhibits constant returns to scale. By Euler's theorem, the exponent of each component represents the relative share of that component in the total country performance rating. For instance, the relative share of governance can be obtained as [CPIAD* (CPR/CPIAD)]/CPR. Looking at it another way, the exponent of each component can be interpreted as the partial elasticity of the country performance rating with respect to that component. This is because the share of each component is equivalent to the expression (CPR/CPIAD)/(CPIAD/CPR). -9- 21. Easier to interpret and predict changes. A logarithmic transformation of the formula above yields the elasticities of the country performance rating with respect to each of its underlying components. For instance, a 1 percent increase in the CPIA D element can readily be interpreted as causing a 0.68 percent increase in country performance rating. A similar increase in CPIA A-C and portfolio ratings would cause an increase in country performance ratings by 0.24 and 0.08 percent respectively. 22. It is now easier to predict changes in country performance ratings and allocations on the basis of changes in underlying components. The general formula for deriving the approximate change in country performance rating from changes in its components is: % CPR = (%CPIAA-C * WCPIA cluster A-C) + (%PORT * WPORT) + (%CPIA * WCPIA D cluster D) Where, CPR is the country performance rating and Wi are the respective exponents of the components. The expected change in allocations per capita (other things remaining constant19) can be approximated as follows: % Allocation per capita = % CPR * 5 23. A numerical example. Table 3 predicts the change in allocations due to changes in the underlying components. The percentage changes in ratings (7 percent for CPIA , D 17 percent for CPIA A-C and 17 percent for portfolio) are multiplied by the exponents to yield the expected changes in country performance ratings. This change in country performance ratings can then be used to calculate the expected approximate change in allocations per capita by multiplying them by 5. Table 3: Predicting Changes in Allocations with Option I Assumed Percentage Weighted Component Rating Component Change in Percentage Weight Time (t) Time (t+1) Rating Change* (i) (ii) (iii)=(i)*(ii) CPIA (D) 3.00 3.20 6.7% 0.68 4.5% CPIA (A-C) 3.00 3.50 16.7% 0.24 4.0% Portfolio 3.00 3.50 16.7% 0.08 1.3% Country Performance Rating 3.00 3.30 1.00 10% CPR Exponent 5.00 Expected Change in IDA Allocation * (10%*5.00) 50% assumed 3.00 IDA Allocation (SDR) (3.00*(1+50%)) 4.50 Notes: 1/ Assumes no change in population and GNI per capita between t and t+1. * Linear approximations. 19 Factors that could affect the country allocations include changes in the IDA envelope as well as changes in performance of other countries. -10- 24. Option II: a linear formula. Another way of simplifying the current formula is to make it linear and eliminate the double counting of governance. Option II: CPR = (0.24 * CPIA A-C+ 0.68 * CPIA + 0.08 * PORT), where CPR is D the country performance rating. 25. A more transparent formula. In this formula, the CPIA rating consists of only clusters A-C while cluster D enters only once as the governance element of the formula. The coefficients of the different components of the country performance ratings formula reflect their weights within the formula (as derived in paragraph 15). They are rescaled to sum up to one and therefore the country performance ratings fall within the same range from 1- 6 as the CPIA. CPIA gets a coefficient of 0.68 to retain the approximate D effective weight that it has in the benchmark formula. For the same reason, CPIA A-C and portfolio performance ratings receive coefficients of 0.24 and 0.08 respectively. 26. Obtaining the same dispersion in allocations. This linear formula lowers the dispersion of country performance ratings compared to the benchmark formula from 0.97 to 0.48 (see Annex 4). This is because the rescaled coefficients keep the range of country performance ratings between 1 - 6 while in the benchmark formula, the ratings range from 0.15 - 13.5 because of the exponential governance factor. The same underlying data generates country performance ratings that are more bunched in the linear formula and therefore there is less differentiation between the strong and weak performers. To obtain the same dispersion of allocations as in the benchmark formula the exponent in country performance ratings has to be increased from 2 to 5. This higher exponent minimizes the sum of squared differences between the per capita allocations that emerge from the benchmark formula and those that emerge from Option II. Chart 3 below compares the country performance ratings and allocations per capita under Option II against the benchmark formula. IDA country allocation = f (CPR 5.0, Population1.0, GNI/capita-0.125) Chart 3: Comparing Ratings and Allocations under Option II Comparing IDA Country Performance Ratings Comparing Allocations per capita 6.0 16.0 14.0 5.0 12.0 4.0 II II 10.0 onit 3.0 onit 8.0 Op Op 6.0 2.0 4.0 1.0 2.0 0.0 0.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 Benchmark Benchmark -11- 27. Easier to interpret and predict changes. Given its simpler additive structure, the coefficients of the components can now be interpreted as weights of the individual components. Moreover, it is now easier to predict changes in country performance ratings and allocations on the basis of changes in underlying components. The general formula for deriving the change in country performance rating from changes in its components is: CPR = (CPIAA-C* WCPIA clusters A-C) + (PORT * WPORT) + (CPIA * WCPIA D cluster D ,) Where, CPR is the country performance rating and W represents the weight of each component. The expected change in allocations per capita (all other things constant) can be approximated as follows: % Allocation per capita = % CPR * 5 28. A numerical example. For example, Table 4 predicts the change in country performance rating due to changes in the underlying components. The expected change in country performance rating can be calculated by multiplying the changes in the underlying components (0.2 for CPIA , 0.5 for CPIA D A-C and 0.5 for portfolio performance ratings) with the respective coefficients. The change in per capita allocations can be then approximated by multiplying the expected change in country performance rating by 5. Table 4: Predicting Changes in Allocations with Option II Assumed Absolute Component Rating Component Weighted Change Change in Weight in Rating Time (t) Time (t+1) Rating (i) (ii) (iii)=(i)*(ii) CPIA (D) 3.00 3.20 0.20 0.68 0.14 CPIA (A-C) 3.00 3.50 0.50 0.24 0.12 Portfolio 3.00 3.50 0.50 0.08 0.04 Country Performance Rating 3.00 3.30 1.00 0.30 Percentage Change in CPR Rating (0.30/3.00) 10% CPR Exponent 5.00 Expected Change in IDA Allocation* (10%*5.00) 50% assumed 3.00 IDA Allocation (SDR) (3.00*(1+50%)) 4.50 Note: Assumes no change in population and GNI per capita between t and t+1. * Linear approximations D. Comparing the Options and the Way Forward 29. Table 5 summarizes the comparisons among the options and the benchmark formula. Both options proposed for simplification retain key desirable features of the formula (weight of governance and dispersion of ratings) and closely track per capita allocations that emerge from the benchmark formula to minimize disruptions at the -12- country level (chart 4). Before comparing the options, a couple of interesting observations are in order. Table 5: Summary Table Comparing Options with the Benchmark Benchmark formula Option I Option II Double counting of governance Yes No No Both additive and multiplicative elements in the formula Yes No No Procurement rating No No No New proposed portfolio ratings Yes Yes Yes Rating dispersion 0.97 0.48 0.48 Rating volatility 0.17 0.08 0.07 Allocation per capita per annum dispersion 3.63 3.63 3.62 Allocation per capita per annum volatility 0.44 0.45 0.43 Ratio of first to last quintile 5.37 5.40 5.34 (Effective) weight of governance 68.0% 68.0% 68.0% Chart 4: The Linear and Geometric Options Closely Track Benchmark Allocations 16.0 14.0 12.0 ) DRS( 10.0 pitaac 8.0 per s cation 6.0 Allo 4.0 2.0 0.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 Normalized IDA Country Performance Ratings Benchmark Linear Geometric · First, similar allocations per capita are maintained. Although the ratings that emerge from the benchmark formula and the options differ, they generate the same allocations per capita. This is because once the replenishment size is fixed, it becomes a zero sum game and what matters is the relative position of each -13- country. The numerical values of the country performance ratings are transformed into "shares" and then multiplied by the envelope to get the country allocations (taking into account population and GNI per capita). So although the options yield different numerical ratings, they generate the same relative positioning (and therefore shares) of countries as in the benchmark formula. Table 6 illustrates how the same underlying components (CPIA A-C , CPIA and D portfolio performance ratings) generate different ratings but same shares and therefore per capita allocations. Table 6: Different Country Performance Rating Scales Generate the Same Shares Benchmark formula Option I Option II Ratings (Ratings)^2 Shares Ratings (Ratings)^5 Shares Ratings (Ratings)^5 Shares Country 1 1.27 1.60 7.0% 2.33 68.02 7.0% 2.34 69.56 7.0% Country 2 1.61 2.58 11.3% 2.56 109.54 11.3% 2.57 112.03 11.3% Country 3 2.00 3.99 17.5% 2.79 169.25 17.5% 2.80 173.09 17.5% Country 4 2.44 5.95 26.2% 3.02 252.54 26.2% 3.04 258.27 26.2% Country 5 2.94 8.62 37.9% 3.26 365.81 37.9% 3.27 374.11 37.9% Sum 22.74 965.15 987.07 Note: Does not take population and GNI per capita into consideration. Assumes same underlying data for each formula. · Second, the balance between performance and needs is maintained. In the benchmark formula, country performance ratings are more dispersed because of the governance factor. In contrast, ratings are less dispersed in Options I and II. Therefore, raising the exponent on country performance ratings from 2 to 5 in the proposed options results in similar shares and therefore similar levels of allocations per capita as the benchmark. Even with the stress on performance in the benchmark formula, 41 percent of the resources go to the bottom third of the countries ranked by GNI per capita and approximately 66 percent of resources flow to the bottom half of IDA countries. Options I and II do not alter this flow of resources to the poorest countries despite their apparent greater stress on performance due to an exponent of 5 on country performance ratings in the PBA formula. · The flow of resources to the poorest countries is made possible (apart from the use of population and GNI per capita elements in the PBA formula) by capping allocations to India, Indonesia and Pakistan at levels well below what they would get using the PBA formula. This helps steer approximately 50 percent (instead of 22 percent if there were no caps) of IDA resources flow to Africa. Also helping in this regard are the special post-conflict allocations ­ mainly to countries in Africa ­ which ensure that resources flow to countries where there are special needs. In addition, a large share of the top-up allocations for regional projects also goes to Africa. So while a larger exponent on performance in IDA's allocation formula (in benchmark formula as well as the options) suggests its primacy in determining allocations, a number of additional -14- allocation rules agreed during the replenishment cycles help IDA balance its focus on country performance with needs. 30. Choosing between options. Both Options I and II are constructed so that they closely mirror allocations generated by the benchmark formula. A comparison of the two options shows that the differences between them are minor. · Easier to explain. Both Options I and II simplify the current formula and make it easier for the country teams to explain it to their counterparts in partner countries. Of the two, Option II may be slightly easier to explain because of its simple additive structure, where the coefficients clearly show the impact of each individual component on the country performance rating. Option I involves use of concepts like elasticities, which may be slightly more difficult to explain to non-technical audiences. · Absolute versus proportional changes ­ a policy choice? In Option I, proportional changes matter whereas in Option II absolute changes matter. Theoretically, this could have interesting policy implications over time. Option I provides larger proportional changes in allocations at lower levels of performance than at the top. Option II treats countries the same way at all levels of performance. For example, in Option I, an absolute change of 1 in CPIA D will result in a higher percentage change in country performance rating when CPIA changes from 2 to 3 (50 percent increase) than when it changes from 5 to D 6 (20 percent increase). In Option II, an absolute change of 1 in CPIA will D always cause a change of 0.68 in country performance rating, regardless of whether the governance rating changes from 2 to 3 or from 5 to 6. Minor differences in practice. The actual values of CPIAD, CPIA A-C and portfolio performance ratings fall within the range of 2 to 4.5. As shown in the chart 5 below, in this relevant range, the differences in allocations generated by Options I and II are minor and would therefore not have significant policy implications. · Harmonization. Option I would lead to greater harmonization with the Asian Development Fund, which currently uses a geometric formula. -15- Chart 5: Minor Differences Between Options I and II CPIA (A-C) Governance (Cluster D) Option II 5.50 Actual CPIA range 5.50 Option II Actual Governance range 4.50 4.50 pc 3.50 apcp 3.50 Option I SDR 2.50 2.50 Option I DRS 1.50 1.50 0.50 0.50 0.50 1.50 2.50 3.50 4.50 5.50 6.50 0.50 1.50 2.50 3.50 4.50 5.50 6.50 CPIA (A-C) Governance (Cluster D) Portfolio 5.50 Actual Portfolio range 4.50 Option II pacp 3.50 2.50 SDR 1.50 Option I 0.50 0.50 1.50 2.50 3.50 4.50 5.50 6.50 Portfolio 31. Suggested way forward. The current simplification of the formula is guided by factors described in paragraph 13. The coefficients/exponents that emerge from the simplification exercise track the benchmark allocations per capita. Among the components, governance (CPIA ) receives the highest weight because analytical work D shows that it is important for development performance of a country and it mitigates fiduciary risks to aid funds. To simplify the formula further, one could envisage slight recalibrations of weights to make it intuitively easier for country teams to explain to partner countries. 32. For instance, it is difficult to explain in Option I why CPIA A-C has an exponent of 0.24 or why portfolio has an exponent of 0.08. The exponent on CPIA A-C could be rounded to 0.3 and the exponent on portfolio performance ratings to 0.1. This would bring down the weight of CPIA slightly to 0.6. The message would still be that D governance is the dominant factor, which is why it receives twice the weight as the remaining CPIA clusters. Because the CPIA is highly correlated with CPIA 20 D A-C, this would not sharply alter the flow of resources other than in those exceptional cases where the correlation is not as strong. 20 See footnote 20 in IDA (2006), "IDA's Performance Based Allocation System: a review of the governance factor." -16- IV. Volatility of Country Performance Ratings 33. Volatility in country performance ratings is driven mainly by the way the portfolio performance ratings are computed. 34. How are portfolio performance ratings currently computed? Portfolio performance ratings, which reflect the health of IDA's active portfolio, enter into the calculation of the IDA country performance ratings with a weight of 20 percent (Section II and Annex 1). To calculate the portfolio performance ratings in a country (Annex 2, Step 2), all projects-at-risk are identified at the end of each calendar year.21 Projects at risk consist of actual problem projects as well as potential problem projects.22 The percentage of projects at risk in each country is corrected for age23 and is then converted into portfolio performance ratings using a conversion scale (chart 6). 35. Implementation experience. Experience during the IDA14 period shows two main problems related to portfolio performance ratings. The first relates to the realism of these ratings and the second relates to their volatility. · Realism of portfolio performance ratings. The FY06 Annual Report on Portfolio Performance points out that under-reporting of risks by staff has become a problem. The Report points out that staff may have the incentive to under-report problem projects because of the link with IDA allocations. · Volatility of portfolio performance ratings. A particularly thorny problem revealed during IDA14 has been that the portfolio performance ratings are 21 This information is obtained from the Implementation Status Reports (ISR) prepared by the task team leaders of projects. The ISR is the Bank's tool for internal reporting by task teams to managers/senior management on the implementation performance and prospective outcomes of operations. The first ISR is due within 12 months of the Board approval of the project or earlier in accordance with regional or unit requirements. While the Bank-wide minimum reporting standard is annual, the frequency of updating ISRs is project-specific. 22 "Projects at risk" are those which are at risk of not meeting their development objectives. They are the sum of actual problem projects and potential problem projects. Actual problem projects are those that are rated in the ISRs as "marginally unsatisfactory" or worse on Implementation Progress and/or where Development Objectives are rated as not likely to be achieved. Potential problem projects are those that, although rated as moderately satisfactory in both Implementation Performance and Development Objectives, are affected by factors likely to bring about eventual unsatisfactory outcomes. These potential problem projects are identified by criteria ("flags") that take into account not only various aspects of actual implementation experience, but also other relevant factors such as economic management and past portfolio performance in the country. Specifically, potential problem projects are identified as projects exhibiting three or more of the 11 flags listed in Annex 2 for investment projects (and of seven flags for Development Policy Operations). 23 During IDA12, an "age correction" was introduced in the calculation of portfolio performance ratings for countries with "young" portfolios with an average age of less than 3.1 years. Experience had shown that countries with young portfolios had fewer projects at risk and therefore received higher portfolio performance ratings. These higher ratings did not reflect better portfolio performance but just that the problems had not yet surfaced. Analysis also showed that the percentage of projects at risk decreased by five percent for each year that a country's portfolio is younger than the average age of the entire IDA portfolio. Therefore, a deterioration factor was introduced to correct for this (Annex 2). -17- volatile, which in turn increases volatility of IDA allocations. Analysis shows that unwarranted volatility in portfolio performance ratings comes from two main sources: the conversion scale and the use of information from one point in time at the end of each calendar year. Chart 6: Steps for Calculating the Current Portfolio Performance Ratings Actual Problem Projects Potential Problem Projects Step 1 (Year End Data) (Year End Data) Calculating Percent of Projects at Risk Projects at Risk (Percent of Total Portfolio) Is the average portfolio age less than Step 2 Yes 3.1 years? No Applying the Age Correction Factor Age Correction Factor Age Correction Factor is Applied does not Apply Conversion Scale Step 3 Converting Percent of Projects at Risk to a Rating Portfolio Rating 36. Measures to enhance realism and lower volatility. Three measures are examined to enhance the realism of portfolio performance ratings and to lower unwarranted volatility. These relate to A. Measuring projects at risk by including just the actual problem projects and leaving out the potential problem projects. B. Calculating the percent of projects at risk by averaging the information at the end of four quarters instead of using information at just one point in time at the end of each calendar year. C. Changing the conversion scale to make it more consistent with the range of actual CPIA ratings. -18- Each of these measures is discussed in detail below. A. Using Actual Problem Projects Only 37. Realism of ratings. Portfolio performance ratings include both actual and potential problem projects to provide a comprehensive picture of the health of the portfolio. However, as pointed out in the FY06 Annual Report on Portfolio Performance, the inclusion of potential problem projects may have led to under-reporting of potential problems by staff and consequently, to "optimism" in the ratings. Therefore, switching to the use of actual problem projects only to calculate the portfolio performance ratings instead of using actual plus potential problem projects would enhance the reliability of ratings. This is because actual problem projects are more easily identified than potential problem projects based on risk flags.24 38. Marginal impact. An examination of the data shows that dropping potential problem projects has only a marginal impact on portfolio performance ratings ­ so the change will not prove very disruptive. In the past three years, only 10-12 countries have reported potential problem projects in the ISRs. It is only in these countries that the proposed switch to actual problem projects would result in a modest increase in portfolio performance ratings. 39. Therefore, management recommends abandoning the use of potential problem projects from the calculation of the portfolio performance ratings. This will enhance the quality of the portfolio performance ratings in the PBA formula, while restoring the potential problem projects concept as an early warning instrument for Bank managers. B. Averaging Actual Problem Projects 40. Using end-year information. To calculate portfolio performance ratings, projects-at-risk data are obtained at one point in time ­ at the end of each calendar year. For example, the FY06 allocation exercise used the projects-at-risk data as of December 31, 2004. Using such a snapshot of information has three drawbacks. First, it may not be representative of the average health of the portfolio during the year, especially if the ISRs are updated just before or after the cut-off date. Second, it is quite possible for staff to "clean-up" the ISRs if they know the point in time when the projects-at-risk data are being picked up for calculation of portfolio performance ratings. Third, using a snapshot is likely to lead to more volatile project-at-risk data than using an average for the whole year. Instead, a simulation using an average of end-quarter data on actual problem projects shows a reduction in the volatility of the portfolio performance ratings. 24 In addition, (a) the revised realism Index linking actual problem projects to IEG's recent outcomes performance will also provide incentives for staff to rate projects more realistically. This is because the Index is periodically scrutinized by the regional management and the peer pressure as well as increased oversight will make it difficult for country teams to under-report actual problem projects; and (b) financial incentives are provided in some regions to increase candor in reporting problem projects. For instance, the Africa Region provides additional funding (to finance additional administrative costs) for intensive supervision of problem projects from an escrow fund of $1.1 million. -19- 41. Using the IDA14 conversion table and quarterly average of actual problem projects data, simulated results (for the IDA13 period) show a drop in the standard deviation from 0.87 to 0.63. Therefore, management recommends using actual problem project data at the end of each quarter during a year instead of using information at the end of each calendar year. C. Changing the conversion scale. 42. Changes in the conversion scale. The conversion scale that translates a country's percentage of projects at risk to portfolio performance ratings (Annex 2) was recalibrated several times in the past. The last such recalibration to the conversion scale was made at the start of the IDA14 period. The motivation for the recalibrations was to align the portfolio performance ratings with the CPIA ratings.25 However, the conversion scale became a source of unwarranted volatility. Experience in FY06 and FY07 ­ the first two years of IDA14 ­ showed that a small change in the percentage of projects at risk resulted in a sharp change in the portfolio performance ratings at the top end of the scale.26 For example, if one project in a country with ten IDA projects becomes classified as at risk, it reduces the portfolio performance score by around 40 percent from 6.0 to 3.5. 43. Discussions with the regions. Against the backdrop of increased volatility in portfolio performance ratings observed during the IDA14 period, several discussions were held with the regional operational advisors. Apart from the consideration of better aligning the portfolio performance ratings with the CPIA ratings, regional advisors underscored the need to recognize that development financing by IDA, as lender of last resort, is inherently risky and therefore, one would expect at least a modest percentage of projects to be at risk. Accordingly, they felt that countries should not be "penalized" disproportionately for small deteriorations in portfolio performance. For instance, they suggested that for the first 15 percent of projects at risk the ratings should ideally not fall by more than 1.0 (and conversely, should not rise by more than 1.0 in the case of an improvement from 15 percent to 0 percent). The Regional advisors also recommended anchoring the score of more than 50 percent projects at risk at the "unsatisfactory" 2.0 - 2.5 level, which is in line with the institutional norm when a "Portfolio Improvement Program" (PIP)27 is called for. 44. Revising the conversion scale. With these considerations in mind, a revised conversion scale is proposed to convert the percent of actual problem projects to a portfolio performance rating. The proposed conversion scale aligns portfolio 25 See footnote 59 of Additions to IDA Resources: Fourteenth Replenishment, Annex 1, IDA, March 10, 2005. 26 This excessive volatility (for countries moving from a perfect portfolio to at least one project at risk or for countries moving from a few projects at risk to a perfect portfolio) was offset by management in FY06 and FY07 through the use of moving averages. 27 PIP countries are those designated for intensive portfolio monitoring and supervision, normally with 50 or more percent of projects at risk and/or 33 or more percent commitments at risk, with more than eight active projects and and/or $250 million in commitments (see Annual Report on Portfolio Performance, FY 2004, World Bank, January 31, 2005, Annex 3, page 44). -20- performance ratings with CPIA ratings and is in line with the advice of the regional operational advisors to lower volatility. Alignment of portfolio performance ratings with the CPIA ratings was done by matching the percentage distributions of countries at each CPIA rating. This has the effect of transforming the skewed distribution of actual problem projects into a distribution of portfolio ratings that matches that of the CPIA. This proposed conversion scale based on a percentile matching technique may have to be periodically revisited (perhaps once every replenishment period) to reflect significant changes in the distribution of CPIA ratings. 45. Simulating the use of the new conversion scale for the IDA13 period shows that the average volatility of portfolio performance ratings declines from 0.63 to 0.31. Management, therefore, recommends the adoption of the proposed conversion scale to reduce volatility in the portfolio performance ratings. Table 7: Reducing Portfolio Performance Rating Volatility Conversion Table Underlying Data Volatility (standard deviation) Current IDA14 Conversion Table Annual Projects at Risk 0.87 Current IDA14 Conversion Table Annual Actual Problem 0.88 Projects Current IDA14 Conversion Table Quarterly Actual Problem 0.63 Projects Proposed IDA15 Conversion Table Quarterly Actual Problem 0.31 Projects Note: Simulations using data from the IDA13 period. 46. Summing up. The central focus of using portfolio performance scores has been to ensure strong management of IDA resources to achieve intended results. Within that framework, adopting the three proposed steps ­ using actual problem projects only, using quarterly data, and using the new conversion scale ­ would greatly reduce the volatility of a country's IDA performance rating, and would thus allow the underlying trend in country portfolio performance to emerge more clearly. Simulation results show that combining all three steps (actual problem projects, quarterly data and new conversion scale) would reduce the standard deviation of ratings by around 64 percent, from 0.87 to 0.31 (Table 7). The collateral effect would be to mitigate undue pressure on the portfolio rating. While the objective of the proposed changes is to reduce noise, the one-time impact of introducing them in individual country ratings, and therefore on their allocations, could be in either direction.28 A simulation using IDA13 data shows that the average shock due to the introduction of the revised portfolio performance ratings is around 10 percent. Given that the changes may affect some countries more than others, 28 For instance, (i) using just the actual problem projects means that the 10-12 countries that report potential problem projects would see their portfolio performance ratings go up while there will be no changes for other countries; (ii) introducing quarterly data instead of end-year information also causes slight changes in the percent of projects at risk, and (iii) using the new conversion scale implies that the disproportionate benefit received by countries that went from 1 project at risk to no projects at risk under the IDA14 conversion scale would no longer receive this benefit. -21- management intends to apply a two-year phase in (FY08-09) of the new portfolio performance ratings in case of a significant impact on any particular country. Section V. Questions for Deputies · Would Deputies support simplification of the PBA formula along the lines suggested in Options I or II? Option I Country performance rating = (CPIA A-C) 0.24* (CPIA ) 0.68 D * (PORT) 0.08 IDA country allocation = f (Country performance rating 5.0, Population1.0, GNI/capita-0.125) Option II Country Performance Rating = (0.24 * CPIA A-C + 0.68 * CPIA + 0.08 * PORT) D IDA country allocation = f (Country performance rating 5.0, Population1.0, GNI/capita-0.125) · Would Deputies be supportive of simplifying the formula further by rounding up the exponents/coefficient to make the formula more intuitively easy to explain? · Do Deputies support management proposals to lower unwarranted volatility in the country performance ratings through changes in the way the portfolio performance ratings are calculated? Specifically, do they support using: (i) information on actual problems projects only instead of actual plus potential problem projects, (ii) averaging end-quarter actual-problem-project data, and (iii) the new conversion scale? -22- Annex 1: Overview of IDA's Performance-Based Allocation System 1. Purpose of the annex. This annex provides an overview of the PBA system and highlights the enhancements that were agreed upon during the IDA14 deliberations.29 Box A1.1: CPIA Criteria A. Economic Management 1. Macroeconomic Management 2. Fiscal Policy 3. Debt Policy B. Structural Policies 4. Trade 5. Financial Sector 6. Business Regulatory Environment C. Policies for Social Inclusion 7. Gender Equality 8. Equity of Public Resource Use 9. Building Human Resources 10. Social Protection and Labor 11. Policies and Institutions for Environmental Sustainability D. Public Sector Management and Institutions 12. Property Rights and Rule-based Governance 13. Quality of Budgetary and Financial Management 14. Efficiency of Revenue Mobilization 15. Quality of Public Administration 16. Transparency, Accountability, and Corruption in the Public Sector 2. Country Policy and Institutional Assessment (CPIA). The performance of IDA countries is assessed annually using the CPIA. The CPIA assesses each IDA country's present policy and institutional framework for fostering poverty reduction, sustainable growth and ability to effectively use development assistance. The system has evolved over time and now comprises 16 criteria grouped in four equally weighted clusters: (i) economic management; (ii) structural policies; (iii) policies for social inclusion and equity; and (iv) public sector management and institutions (Box A1.1).30 To ensure that ratings are consistent with performance within and across regions, (i) detailed questions and definitions are provided to country teams for each of the six rating levels for each of the 16 criteria; and (ii) an institution-wide process of rating and vetting a dozen `benchmark' countries is carried out to anchor the ratings in all IDA regions. This is followed by a process of institutional review of all country ratings before they are finalized. 3. Annual Review of Portfolio Performance (ARPP). The CPIA underpins IDA's country performance ratings but is not its only determinant. To capture the important dimension of quality of development project and program management, the Bank's 29 IDA (2005), "Additions to IDA Resources: Fourteenth Replenishment. Working together to Achieve the Millennium Development Goals." 30 For the CPIA 2004 Questionnaire, see www.worldbank.org/ida/.pdf. There used to be 20 criteria; the reduction to 16 was based on a recommendation by the External Panel of Experts. -23- Annual Report on Portfolio Performance (ARPP) is used to determine a rating for each country's implementation performance. The portfolio ratings are based on the percentage of IDA funded projects in the country that are considered at risk. These percentages are translated into 1-6 scores with the help of a conversion table.31 4. Governance factor. The governance factor is derived from the five criteria in the CPIA's governance ­ or public sector management and institutions ­ cluster D, plus the three-year moving average of the procurement flag that is an element of the ARPP.32 The average score of these six governance criteria is divided by 3.5, the mid-point of the 1-6 range, and an exponential of 1.5 is applied to this ratio:33 Governance Factor = (average governance rating / 3.5)1.5 The country's overall rating is then multiplied by this factor, resulting in an increase or decrease of the overall IDA country performance ratings. 5. Country performance ratings. A weighted average rating is calculated of the CPIA (80 percent) and the portfolio rating (20 percent). In the second step, this composite rating is multiplied by the "governance factor" to produce the country's IDA country performance ratings. 6. Balancing country performance and needs. IDA's resources are allocated on the basis of the IDA country performance ratings (ensuring that good performers get a higher share of IDA's available resources), population, and GNI per capita:34 IDA Country Allocation = f( Country Performance Ratings , 2.0 Pop1.0,GNI/Cap.-0.125 ) The formula shows that the country's policy and institutional performance is the dominant determinant (twice as high a score results in four times the allocation), while population also affects it significantly (the relationship is linear: a higher population results in a proportionally increased allocation). Finally, there is a modest bias in favor of the IDA eligible countries with a lower GNI per capita.35 31 The Conversion Table was amended during IDA14 to assure: (i) consistency with the new CPIA which now has definitions for each of the rating levels from 1-6; and (ii) alignment of the average ARPP ratings with that of the CPIA ratings by reducing the ARPP conversion scores. 32 The moving average was introduced in 2004 to reduce unwarranted volatility of this governance indicator. 33 Participants decided to maintain the 1.5 exponential of the governance factor. At the time of the mid- term review they will reflect on how this has affected the allocations at the country level, especially in the case of the strongest and weakest performing countries. 34 For a more detailed specification of the IDA allocation formula. 35 Poverty is heavily weighted in the allocation, since normally IDA eligibility requires that the country's per capita GNI is below the annually reviewed IDA operational cut off-level, which as of July 1, 2004, stands at $895. -24- 7. IDA allocations and Country Assistance Strategies (CAS). The allocation norm establishes the financial resources available for each IDA country for a given year, and gives an indication of resource availability in the following year. The terms in effect for the allocation will be determined in accordance with the grant allocation system for IDA14. Country allocations vary over time with changes in a country's performance, as well as with changes in other countries' performance, changes in eligibility for IDA resources and for IDA grants, and overall IDA resource availability. The allocation norm is the basis for the financing scenarios set out in CASs or Transitional Support Strategies (TSSs). 8. Minimum allocation. In addition to their performance-based allocations, all countries are allotted a basic allocation of SDR1.1 million per annum. In terms of per capita allocations, this benefits small states. 9. Exceptions to the PBA system. There are some considerations that merit exceptions to the allocation norms. · First, capped blend countries with access, or potential access, to IBRD receive less than their allocation norms due to their broader financing options. · Second, countries emerging from severe conflict can, under certain conditions, be provided with additional resources in support of their recovery and in recognition of a period of exceptional need.36 The special post-conflict allocations may be provided for up to four years, plus three years of phase down to the performance-based norm.37 If such countries have large and protracted arrears to multilateral creditors, they may also be eligible for grant support in the pre-arrears clearance period.38 · Third, in cases where the existing allocation would not allow for a sufficient response, additional allocations may be provided to IDA countries in the aftermath of major natural disasters. · Fourth, additional allocations may be provided on a one time basis to countries that are in the process of re-engaging with IDA after a prolonged period of inactivity on the basis of a strong transition plan with concerted 36 See Aid delivery in conflict afflicted IDA countries: The Role of the Bank (November 2004). Annex on IDA's approach for allocating aid to post-conflict countries (p. 16-20). 37 The duration of exceptional eligibility was lengthened following the review of the post-conflict allocation system at the IDA13 Mid-Term Review. At the same time, the post-conflict allocation norms were somewhat lowered so as to leave the overall extra post-conflict allocations unchanged. See IDA's Performance-Based Allocation System: Current and Emerging Issues, IDA/R2003-0203, October 24, 2003. 38 This provision was first included in IDA12, and was kept unchanged in IDA13. The use of these grants is expected to be very limited, and must comply with the policy framework approved by IDA's Executive Directors in July 2001 for the provision of these grants. -25- donor support.39 This exception will only be used after all other options have been exhausted, and its use is expected to be very limited in the IDA14 period. The level of resources made available will be less than what is typically provided under the post-conflict allocation system, and the duration of exceptional allocations will not exceed two years, with one further year's support possible subject to strong performance. The rationale for using this exception will be clearly presented in the TSS. · Fifth, there is a special provision for selected regional integration projects. As a continuation of this pilot program which was started in IDA13, IDA14 envisages up to SDR375 million per annum of such projects, whereby SDR125 million is to be contributed from the participating countries' IDA allocations, and the balance from the special provision. 39 While these cases are rare, there are circumstances such as the recent Haiti transition where countries require exceptional assistance due to severe insecurity and the partial or total collapse of the state, but that are not eligible for IDA post-conflict assistance. -26- Annex 2: Steps for Calculating Country Performance Ratings 1. Country performance ratings play a major role in IDA's PBA system. This Annex shows the steps involved in calculating these ratings. Step 1: Take aggregate CPIA ratings. For example, assume that Country X has an aggregate CPIA rating of 3.42. Step 2: To calculate portfolio performance ratings. · Take the number of actual and potential projects at risk in Country X's portfolio. o Actual projects at risk are those for which Implementation Progress (IP) is unsatisfactory or Development Objectives (DO) are not likely to be achieved40. o Potential problem projects are those that, although rated as satisfactory for both IP and DO, are affected by factors likely to bring about an eventual unsatisfactory outcome. These projects are identified by criteria or "flags". For investment operations, potential problem projects are those that exhibit three or more of the following 11 risk flags in the last Implementation Status Report (ISR) and three out of 6 risk flags (flags that apply are marked by an asterix) for Development Policy Operations41. Legal covenants Safeguards Counterpart funds Monitoring and evaluation* Financial management Procurement Project management* Long-term risk* Effectiveness delay* Disbursement delay* Country record* 40 The DO rating, which is closest in concept to the measure of final evaluated project outcome in Implementation Completion Reports, is the likelihood of attaining the objectives set in the Staff Appraisal Report/Project Appraisal Document (SAR/PAD) (as formally revised during implementation). The IP rating is based on an overall judgment of implementation performance in relation to the benchmarks in the SAR/PAD. The DO rating takes into account not only implementation progress, but also other factors, such as inappropriate design, unforeseeable adverse economic and financial developments, price fluctuations of project outputs and changes in government policy. 41 IDA's portfolio tracking actually involves twelve criteria. In addition to the eleven criteria listed here, it also includes a flag for the country environment. Since the latter is measured by the CPIA score (which already plays a major role in IDA's performance based allocation system), it is eliminated in this application, in order to avoid double counting. -27- o Sometimes, the regional operations manager can overrule the ratings done by task team leaders and decide that the project is at risk. In this case, the golden flag is raised and the project is considered to be at risk. Assume that country X has 2 projects at risk. · Divide this by the total number of projects in the portfolio of country X. If country X has 23 projects in the portfolio, then it has 2/23 = 9 percent of projects at risk. · Then an age correction is made to the percent of projects at risk. If the portfolio is young and has an average age of less than 3.1years, then an age correction of 5% is applied.42 Let us assume that portfolio of country X has an average age of 3.7 years. So an age correction does not apply. · So country X has 9% of projects at risk. This is converted to a rating using the scale shown in Table A2.1. Therefore, the corresponding portfolio rating for country X using the following scale is 3.5. Table A2.1: Conversion Scale Percent of Projects at Risk Rating 0% 6.0 1% 5.5 2% 5.0 3-4% 4.5 5-6% 4.0 7-10% 3.5 11-15% 3.0 16-32% 2.5 33-60% 2.0 61-99% 1.5 100% 1.0 Step 3: To calculate the governance factor. · Take the aggregate rating for cluster D of the CPIA. For country X, assume these ratings are: 3.0, 3.0, 3.0, 3.0, and 2.5 for questions Q12-16 respectively. · Calculate procurement ratings in the following manner: o Add total number of projects for which the procurement flag is raised for each IDA country. The procurement flag measures quality, reliability and transparency of procurement administration for a project. There are six possible ratings ­ highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory and highly unsatisfactory. When a project scores moderately 42 Assume that a portfolio has an average age of 1.8 years and 0% projects at risk. The age modification is applied in the following manner: 0+(3.1-1.8)*5% = 6% -28- unsatisfactory or less, the procurement flag is raised. Example: Country X has 3 projects for which the procurement flag has been raised. o Divide by total number of projects in the portfolio to get the percent of projects in the portfolio for which there are procurement problems. Suppose Country X has 23 projects in its portfolio. Therefore percent of projects for which there are procurement concerns are 3/23 = 13% o Convert this percentage into a rating by using the following conversion scale. 13 % of projects at risk correspond to a score of 3.5 for Country X using the procurement rating conversion scale Table A2.2. Table A2.2: Conversion Scale to Obtain Procurement Ratings Percent of Projects Rating 0% 5.0 1-5% 4.5 6-10% 4.0 11-15% 3.5 16-20% 3.0 21-25% 2.5 >=25% 2.0 o Use the newly computed rating to calculate a 3-year moving average of the procurement rating to obtain the final procurement rating. Suppose country X's procurement ratings for the past two years are 3.5 and 2.5. Calculating the 3-year moving average using the newly computed procurement rating (3.5+3.5+2.5) gives the new procurement rating of 3.17, which is then used in the governance factor. · Take average of CPIA cluster D and the procurement rating. This is (3.0+3.0+ 3.0+3.0+2.5+3.17)/6 = 2.9 · Now, the governance factor is calculated as (2.9 / 3.5)1.5 Step 4: To calculate the country performance rating. · Country performance rating = (80%*CPIA + 20%Portfolio rating)*(governance factor) For country X, Country performance rating = [(0.8*3.42) + (0.2*3.5)]* (2.9 / 3.5)1.5 = 2.65 -29- Annex 3: Methodology for Calculating the Effective Weight of Governance 1. The relative importance of governance in the country performance rating has to reflect that governance appears in: (i) five of the 16 criteria in the new CPIA (cluster D, with a weight of 25 percent); (ii) one of the 11 criteria ("flags") of the ARPP; and (iii) the governance factor (derived from the average of these five CPIA governance criteria plus the procurement flag drawn from the ARPP).43 2. The calculation involves six steps: · Step 1a. Start from a neutral score for all criteria of 3.5 - resulting in a 3.5 country performance rating. Calculate the percentage increase in the country performance rating resulting from a 10 percent increase in the rating of one CPIA governance criterion. · Step 1b. Repeat this calculation, but now for a 10 percent increase in the rating of the one ARPP governance criterion. · Step 1c. Calculate the weighted average x, taking into account that: (i) in the case of the governance criteria five come from the CPIA and one from the ARPP; and (ii)that the CPIA is weighted 80 percent, and the ARPP 20 percent. · Step 2a. Starting from a neutral score44 for all criteria of 3.5, calculate the percentage increase in the country performance rating resulting from a 10 percent drop in the rating of one non-governance CPIA criterion. · Step 2b. Repeat this calculation, but now for a 10 percent increase in the rating of the one ARPP non-governance criterion. · Step 2c. Calculate the weighted average y, taking into account that: (i) in the case of the non-governance criteria 11 come from the CPIA and 10 from the ARPP; and (ii) that the CPIA is weighted 80 percent, and the ARPP 20 percent. · Step 3. Calculate the ratio x/y showing how much more impact a governance rating change has than a non-governance rating change (about six times, see below). · Step 4. Repeat steps 1-3 for rating increases of 20 percent, 30 percent, 40 percent, and 50 percent. · Step 5. Calculate the average of these ratios, z. · Step 6. Calculate the Effective Governance Weight (EGW) in the country performance rating calculation according to the formula specified below that reflects: (i) the extra impact (z) that governance criteria ratings have on the final country performance rating45, (ii) the number of those heavy-weighted criteria 43 Governance factor = (average governance rating / 3.5)1. 5 44 The score of 3.5 is half way between the minimum score of 1 and the maximum score of 6. 45 Note that to reflect this extra impact, governance criteria get credited with a weight of z, and non- governance criteria with a weight of 1. -30- among the CPIA and ARPP criteria; and (iii) the CPIA and ARPP weights of 80 percent and 20 percent, respectively. EGW = 0.8 * ((5*z / (5*z +11*1)) + 0.2 * ((1*z / (1*z + 10*1)) 3. Table A3.1 shows the ratio results which form the basis for the EGW calculation for the current governance factor with the new CPIA with 16 criteria. It shows that on average changes in governance criteria have 6.07 times as much impact on the overall ICP rating as changes in non-governance criteria. Entering this result into the formula gives an EGW of 66 percent. EGW = 0.8 * ((5*6.07 /(5*6.07 +11*1)) + 0.2 * ((1*6.07 / (1*6.07 + 10*1)) = 0.66 Table A3.1: Country Performance Rating Changes as a Result of Governance and Non-Governance Rating Changes Increase in Resulting Increase in Resulting Ratio governance increase in non- increase in (X/Y) Percentage criterion the country governance country increase rating performance criterion performance rating (X) rating rating (Y) 0% 3.50 0.0% 3.50 0.00% 10% 3.60 2.9% 3.52 0.48% 5.99 20% 3.70 5.8% 3.53 0.96% 6.03 30% 3.81 8.8% 3.55 1.44% 6.07 40% 3.91 11.8% 3.57 1.93% 6.11 50% 4.02 14.8% 3.58 2.41% 6.15 Average 6.07 -31- Annex 4 Table A4.1: Comparison of Benchmark Formula with Options I and II Benchmark Option I Option II formula Dispersion of country performance ratings 0.97 0.48 0.48 Volatility of country performance ratings (std. 0.17 0.08 0.07 deviation) Dispersion of per capita allocations per annum 3.63 3.63 3.62 Volatility of per capita allocations per annum (std. 0.44 0.45 0.43 deviation) Ratio of top to bottom quintiles 5.37 5.40 5.34 (Effective) weight of governance 68 68 68 Minimum country performance rating 0.28 1.25 1.33 Maximum country performance rating 6.56 4.50 4.50 Minimum per capita allocation per annum 0.93 0.91 0.99 Maximum per capita allocation per annum 13.99 13.99 13.99