38851 IDA15 SELECTIVITY AND PERFORMANCE: IDA'S COUNTRY ASSESSMENT AND DEVELOPMENT EFFECTIVENESS International Development Association Development Economics, Office of the Chief Economist (DECVP) February 2007 Abbreviations and Acronyms CPIA Country Policy and Institutional Assessment CPR Country Performance Ratings ECA Europe and Central Asia GDP Gross Domestic Product HDI Human Development Index IEG Independent Evaluation Group PBA Performance Based Allocation TABLE OF CONTENTS Executive Summary................................................................................................................. ....i I. Introduction ........................................................................................................................1 II. Towards a Model Linking Policies and Institutions to Outcomes ......................................3 III. Country Assessments and the Human Development Index (HDI)......................................8 IV. Patterns for Other Indicators..............................................................................................12 A. Under 5-Mortality.....................................................................................................12 B. Immunization Rate....................................................................................................13 C. GDP per capita..........................................................................................................14 V. Endogeneity of the CPIA ­ does it matter? .......................................................................16 VI. Extending the Analysis to the Normalized CPR................................................................17 VII. Country Ratings and Indicators of Project Performance ...................................................20 VIII. Conclusion .........................................................................................................................23 Annexes: Annex 1: Sample of Countries....................................................................................................27 Annex 2: Distributions of the Averages and Changes of CPIA During the Two Decades ........29 Annex 3: HDI and Other Indicators: Regression Results...........................................................31 Annex 4: CPR Regression Results..............................................................................................33 Annex 5: CPIA Ratings and Project Performance......................................................................35 Figures: Figure 1: Trends in Human Development Index: 1975-04..........................................................9 Figure 2: HDI Adjusted Change and Composite Policy Indicator, 1995-04.............................10 Figure 3: HDI Paths Over Time.................................................................................................11 Figure 4: Mortality Rate and Policy Composite, 1995-04.........................................................13 Figure 5: Immunization Rate and Policy Composite, 1995-04..................................................14 Figure 6: GDP per capita Growth Rate and Policy Composite, 1995-04..................................15 Figure 7: CPIA Annual Change and GDP per capita Annual Growth, 1985-04.......................17 Figure 8: CPR Vs CPIA, Average 1995-04...............................................................................18 Figure 9: CPR Vs CPIA, 1985-94 .............................................................................................18 Figure 10: IDA Commitments and CPR....................................................................................20 Figure 11: CPIA Average Scores and Project Ratings ..............................................................22 Figure 12: CPR Average Scores and Project Ratings................................................................22 Executive Summary 1. In a discussion of IDA's Performance-Based Allocation (PBA) system at the IDA14 Mid-Term Review held in November 2006, Deputies asked management to review how it relates to "development results"1 in IDA countries. This paper responds to that request. 2. Since IDA's PBA system rests mainly on CPIA ratings, this paper examines their link with country level outcomes ­ the Human Development Index (HDI), under-5 mortality rate, GDP growth and the Immunization rate. The paper extends this analysis to Country Performance Ratings (CPRs), which directly feed into IDA's performance- based allocations. In addition, the paper examines the link between IDA project performance and CPIA ratings. Finally, the paper also considers how the results bear on the debate on "needs versus performance." 3. The model used to examine these links includes both average level of CPIA/CPR and their changes over a decade as explanatory variables for development outcomes. In addition, the model uses dummies for Africa (to account for difficult initial conditions) and another for countries with high HIV/AIDS prevalence rates. Finally, the model includes an initial value of the development indicator to allow for possible convergence effects. 4. The main conclusion of this paper is that a country that sustains a high CPIA/CPR over several decades will open up a sizeable HDI advantage over a country with a low CPIA/CPR assuming that all other factors remain equal. Specifically, in the long run, the HDI will converge to a higher level if a high CPIA/CPR is sustained. Similar conclusions hold for some other indicators of development, including per capita growth. 5. A second conclusion is that, even though development is a long-run process, countries can shift between long-run development paths relatively quickly. When the quality of policies and institutions undergoes a major change, countries can see substantial changes in outcomes over decades. In terms of the HDI, strong improvers advance at twice the average rate and poor performers see static, or declining, development outcomes. Post-conflict countries can thus experience rapid recovery; while countries caught in a downward political and economic spiral with worsening policies will see large deteriorations over periods of a decade. 6. A third result is the confirmation that African countries face a much more difficult development challenge than others. Adjusting for policies and institutions as well as for the prevalence of HIV/AIDS, they have progressed at half the speed of other countries over the last two decades in terms of the HDI, or even less, and their gains in reducing under-5 mortality are also smaller. However, the slower rate of progress in Africa as compared with other IDA countries appears to have moderated during the later decade suggesting perhaps the beginning of a catch-up process. HDI ratings have also been 1 IDA (2006). "Chairman's Summary: IDA14 Mid-Term Review Meeting." - ii - severely affected by HIV/AIDS, mainly through its impact on life expectancy. High- prevalence countries have been thrown back at least four decades on the HDI scale. However, AIDS has appeared as a less significant determinant of the other development outcomes covered in this paper. 7. Project-level data suggest a similar story. Higher CPIA/CPR scores at the initial stages increase the likelihood of success, as measured by IEG ratings. So does a further improvement in the CPIA/CPR during project implementation. Project performance is therefore sensitive to both initial conditions and to changes in country performance during the active period of the project, confirming the importance of stability for achieving expected results. Achieving a high rating on projects is more difficult in Africa than in other regions assuming that all other factors remain equal. 8. The Africa and HIV/AIDS effects highlight the debate on "needs versus performance" and the complexity involved in moving to "results-based" aid. Making extra assistance conditional on outcome indicators has considerable appeal, but it is essential to understand whether good performance reflects extra effort or exogenous factors, such as favorable demographic trends, better geography, rapidly-growing neighbors, exceptional terms of trade shocks, or others. Performance rating can be seen as an effort to separate out the impact of such exogenous factors on performance. A forward-looking assessment would increase aid to Africa beyond the level suggested by current ratings or performance on outcomes, because of the greater challenge ahead. IDA allocations can be interpreted in this light with caps on large Asian countries freeing-up more resources for African countries that appear to face more difficult development challenges. Similarly, the base allocation can be interpreted as an effort to compensate for the higher vulnerability of small countries. Even allowing for the effect of caps and post-conflict-allocations, IDA allocations have remained performance-based. 9. How do these results bear on the use of performance-based allocations by IDA? They support the use of indicators along the lines of the CPIA/CPR, estimated on a reasonably consistent basis over long time periods, to provide an effectiveness-based anchor for the system of development assistance, including the potential value of such indicators in facilitating an open dialog on development progress. In terms of aid allocation, the PBA system is preferable to arbitrary, politically driven, allocations in at least three respects. · First, it provides a check on excessive aid allocations to poorly-performing countries, reserving resources for others. · Second, use of the PBA system backed up by stable IDA resources improves the stability and predictability of resource flows where this is most needed ­ to those countries maintaining stable performance. · Third, performance ratings provide a standard to help separate out exogenous factors that make development more or less challenging in different countries and regions. The adjustments to the PBA system (through capping and post-conflict allocations) take into account these exogenous factors that affect long-run development. Selectivity and Performance: IDA's Country Assessment and Development Effectiveness I. Introduction 1. In a discussion of IDA's Performance-Based Allocation (PBA) system at the IDA14 Mid-Term Review held in November 2006, Deputies asked management to review how it relates to "development results"2 in IDA countries. Specifically, they requested that management examine whether the PBA system directs resources to countries where results are being achieved. This paper responds to that request. 2. As a major pillar of the multilateral core of the global development architecture, IDA plays a central role in two dimensions: resource flows and policy dialog. These two dimensions come together in the form of IDA's Performance Based Allocation (PBA) system. The Country Performance Rating (CPR), the performance measure used to allocate IDA resources through the PBA, in turn, rests heavily on the World Bank's annual assessments of development policies and institutions, the Country Policy and Institutional Assessment (CPIA) ratings. This paper considers the relationship between CPIA/CPR and various indicators of development progress, as well as project performance. It also considers some associated issues such as the implications of possible endogeneity of the ratings and the tensions between performance-based and results-based allocation. 3. These issues should be considered within the framework of what is sometimes termed the "New Partnership Model" of development assistance. This model, which emphasizes country-driven development supported by (rather than led by) donors, has a number of implications. It places greater weight on overall development effectiveness, encompassing the use of all resources, relative to trying to attribute progress to the effectiveness of aid resources alone. While donors will continue to have legitimate fiduciary concerns over the use of their own resources, the quality of country systems becomes more important than previously. There is, correspondingly, greater emphasis on contribution to shared goals than on trying to attribute achievements to particular partners. 4. The logical consequence of this approach is an associated focus on "selectivity" which is more compatible with country ownership, and with the recognition that a "one size fits all" approach will not necessarily be effective in all cases. Selectivity, in turn raises the need for some indicator of development effectiveness that is not overly determined by a rigid set of policies, and its use to guide the allocation of financial support. 5. Section II considers the nature of the CPIA (which is the main input into the CPR) in greater depth, and develops a model to assess the statistical relationship between country ratings and development outcomes. The model takes a 20-year view because 2 IDA (2006). "Chairman's Summary: IDA14 Mid-Term Review Meeting." - 2 - development is a long-run process, and both policies and outcomes are subject to measurement errors which can overshadow short-run changes. These factors must be considered in the specification of any potential relationship between performance indicators and development. 6. Section III applies this model to the relationship between CPIA ratings and the Human Development Index (HDI), which combines data on incomes, life expectancy and several measures of education into the broadest overall index of development outcomes in use. We consider the set of IDA-eligible countries (excluding ECA countries, very small countries, and countries with inadequate data), and estimate relationships over decade-long averages for two periods, 1985-94 and 1995-04. Even though the CPIA index has changed slowly over the period, the results suggest a systematic relationship, and also that certain other factors, such as location in Africa and the rate of HIV/AIDS infection, are important determinants of countries' HDI levels. We also consider how the results bear on the debate on "needs versus performance," including suggestions that have been made for modifying the PBA system by placing a greater direct emphasis on results. 7. Section IV applies the same model to three other development indicators ­ under- five mortality rates, immunization rates and growth of GDP per capita. The first two indicators are especially interesting because the HDI only includes life expectancy as a health indicator. The results are shown to be broadly similar, and this suggests a reasonably dependable longer-run relationship between aggregate development indicators and the CPIA. 8. Assessments along the lines of the CPIA are not likely to be truly exogenous relative to development outcomes. This is considered in Section V, together with the question of whether endogeneity matters. We conclude that, while it is difficult to assess the degree of endogeneity, analysts do not appear to mechanically extrapolate from outcomes into ratings. 9. Section VI reports results of substituting the normalized CPR index for the CPIA in some of these relationships. The formula relating the CPR to the CPIA has changed over time, making the former a less stable measure than the latter. However, results are quite similar, largely because both the governance cluster of the CPIA (cluster D) and portfolio performance tend to be correlated with the overall CPIA. This suggests that allocating aid in line with the CPR will tend to shape the pattern towards countries performing better on outcome measures over the long run. 10. Section VII considers the relationship between measures of project performance and the CPIA and the normalized CPR. Both the volume and nature of projects will, of course, be partly endogenous to the policies and governance of the recipient country, and this would be expected to impart a bias towards reducing the significance of any linkage.3 3 For example, budget support might be expected to be more effective in a country with good governance and fiscal controls than in a badly governed country, but it is less likely to be implemented in the latter case. - 3 - Like several other studies of the relationship between measures of country and project performance, we find that a higher CPIA is associated with a higher probability that a project is judged as successful according to three measures of project quality; we also find that both initial conditions and changes in performance during the project's implementation affect the rating. These findings extend also to the normalized CPR. 11. Section VIII concludes. 12. Before proceeding, it may be useful to note what this paper does not seek to do. It does not try to show that the CPIA/CPR is a perfect indicator of development effectiveness or to estimate the effectiveness of aid. It also does not try to separate out a strictly causal interpretation running from the performance measures to development outcomes, even though some simple tests, which suggest that analysts do not mechanically extrapolate from observed outcomes to the CPIA ratings, lend comfort to a somewhat causal interpretation. II. Towards a Model Linking Policies and Institutions to Outcomes 13. Development is a long-run process, with many factors intervening between policy actions, institutional change, and country-level outcomes. To specify a parsimonious model relating CPIA ratings and development outcomes without resorting to a very large number of trial-and-error regressions, it is helpful to consider the likely causal chains involved.4 14. The CPIA is a broad indicator of the quality of development policies and institutions, as assessed by World Bank staff. From its early years, when it emphasized macroeconomic policies, the index has evolved well beyond the "Washington Consensus" to take into account a wider range of policy areas and to place more emphasis on the effectiveness of implementation. It is evidence-based rather than a projection of good intentions. Because ratings are not adjusted relative to exogenous conditions or any baseline measurement of country "capacity" it cannot be considered as a normative indicator of "development effort."5 Because of the weight placed on institutional capacity and implementation, especially in more recent years, upward changes in the CPIA tend to be slow though catastrophic deterioration in policies can result in somewhat more rapid downward revisions. 15. The CPIA index changed scale in 1998, and have therefore been renormalized. For the country sample the average rating over 1995-04 was 3.3 on a 1­6 scale. Except for countries at the extremes, few countries change their ratings by more than 1 point over a few years but substantial changes over a 5-10 year period are not unusual. A strong performer will usually take about a decade to transit from a middling ratio of 3 to a strong rating of 4. Even cases of sharper decline will generally extend over a few years, 4 Many studies have examined the relationship between policies and growth; for a recent assessment for Africa see Ndulu (2006). 5 Thus, CPIA ratings can be reasonably high for some countries despite severe donor frustration over their failure to do better, or low for some disadvantaged countries making a serious effort. - 4 - as deteriorating governance and macroeconomic management causes a gradual wilting away of the capacity needed to deliver a range of public services. Changes are negatively correlated with initial values. If a strong performer experiences a major change it is inevitably in a downward direction whereas a weak performer has nowhere to go but up. Although many countries maintain stable performance over long periods, this makes it difficult to use past values to predict the CPIA many periods ahead. 16. In the previous decade 1985-95, CPIA ratings tended to be more volatile, and the spread between higher and lower performers was considerably wider. The reason for this difference is not clear; but it may be because there was less emphasis on slowly-changing institutions and implementation capacity in the earlier period. However, the difference in the distributions is important, as it affects the interpretation of the magnitudes of regression coefficients ­ those for the earlier period are likely to be smaller in magnitude. 17. Like other indicators, estimated ratings will be subject to measurement error. On the 1-6 scale, the standard error of a CPIA rating has been estimated at around 0.24. Small changes in ratings will therefore not provide a reliable guide to actual changes, and in most cases several years will be needed to discern a trend with reasonable certainty because errors are likely to dominate small changes.6 Development outcomes too are subject to appreciable measurement error. On some outcomes, administrative data is unreliable and survey-based data are only collected infrequently if at all. Particularly in the many countries with low statistical capacity, short-run changes in outcomes are likely to be modeled rather than to reflect the situation on the ground.7 18. The CPIA is unlikely to be exogenous relative to development outcomes. In certain areas, such as macroeconomic management and trade, it is relatively straightforward to base ratings on policy indicators. Examples could include assessing macroeconomic management by the budget deficit relative to GDP or trade restrictiveness by the average level of tariffs, adjusted to take into account non-tariff barriers. However, this is useful only in cases where there is strong a priori evidence of the general effectiveness of the policy prescriptions. These areas are few, and even there the cross-country application of uniform policy prescriptions can be contentious.8 In addition, uni-dimensional indicators are practical for few policy areas. Most require more subtle and multi-faceted indicators ­ even trade reform, for example, is not simply a matter of reducing tariffs but may include the implementation of customs reforms to facilitate the cross-border flow of goods. Multi-dimensional indicators raise the problem of determining the appropriate weights on different components of policy, to which there is no simple answer.9 In some areas, such as the delivery of health services or social 6 Gelb, Ngo and Ye (2005) 7 World Bank DECDG, 2006 8 For example, while most would agree that very large budget deficits not financed by grants are undesirable, there are different views on the benefits from lowering already low inflation. Similarly, while there may be a consensus that severe anti-export bias is desirable, economists differ on the value of having very low tariffs. Most analysts would probably agree that the weight that trade policy should receive in overall policies could differ for very small and very large countries. 9 For example, countries are commonly judged on the number of HIPC tracking indicators for budget and financial management that attain certain standards. Yet if the integrity of the cycle of budget - 5 - protection, there is no clear "model" that can be applied to countries spanning a wide range of development to serve as a reference standard. A content analysis of the criteria and sub-indicators of the CPIA suggests that only about one third of the 16 CPIA criteria can be assessed largely using quantifiable policy indicators. 19. In many performance areas of the CPIA, there is therefore no alternative to subjective assessments of the quality of development policies and their implementation. These can be either expert-based or survey based, and they will certainly draw on a range of service delivery indicators, including specific public actions identified as critical to reform. But they will inevitably be influenced by the perceptions of development outcomes based on available data against the backdrop of: resource availability, the performance of comparators and exogenous factors. 20. Ratings are therefore likely to be somewhat endogenous to outcomes. They may respond with a lag as evidence on school enrolments, educational achievements and changes in infant and maternal mortality becomes available to influence the assessments of skeptical analysts. This does not undermine the usefulness of the CPIA as a judgment on the effectiveness of countries' policies and institutions ­ good analysts will use whatever information is available to them. However, it does complicate lag structure, since some aspects of the CPIA might lag outcomes whereas other components probably lead outcomes. 21. Since the CPR is largely a function of the CPIA, the remarks made about the latter will generally hold for the former. CPR and CPIA ratings were the same before 1998. At the beginning of the 1990s, the CPIA ratings included three components: short-term economic management, long-term economic management, and poverty alleviation policies. Countries were rated in each of these three components on a scale of 1 to 5.10 IDA portfolio performance was added to performance assessment in the 1993 CPIA exercise and was assigned a 20 percent weight in the CPIA rating.11 22. In 1998, the CPIA was reconfigured to provide an agreed set of Bank-wide criteria for promoting growth and poverty reduction. From 1998 onwards, the CPIA consisted of 20 criteria, each assigned equal weight--the number was recently trimmed back to 16. The ratings scale was changed from 1­5 to 1­6. Two major changes are worth mentioning: formulation, execution, reporting and audit is only as strong as its weakest link, this suggests rating by the level of the weakest performance area. The optimal strategy for reform will differ depending on the weighing system used. 10 "Short- term management includes consideration of monetary, fiscal, exchange rate, and pricing policies. For the long term, consideration is given to structural policies ­ including external and domestic trade regimes, private sector development, tax and financial sector policies, governance, and natural resource management. Poverty alleviation includes policies promoting the delivery of social services, and the reduction of biases against the agricultural terms of trade and the demand for labor." ("IDA Allocations Revisited," IDA10 Discussion Paper 2, Annex , page 20) 11 This weight was then reduced to 10 percent in 1995 and then to 7 percent in 1996 and 1997. - 6 - · first, the IDA portfolio performance rating was moved from CPIA into the allocation phase in the PBA system; · second, from 1998 onwards, the CPIA has included five criteria on governance, with a combined weight of 30 percent, reflecting the importance attached to governance. 23. Due to these two important changes, the CPR has been diverging from the CPIA ratings since 1998.12 To make up the CPR from 1998 onwards, the CPIA was given an 80 percent weight, the portfolio performance rating was assigned a 20 percent weight, and the "governance discount" was finally applied.13 The use of the governance discount, though, was criticized for putting upward pressures on the ratings at the cut-off point and imparting a punitive bias. To address these concerns, the governance discount was replaced in 2001 by a "governance factor." The governance factor is calculated by dividing the average governance rating (average of the ratings on the six governance criteria) by 3.5 and applying an exponent of 1.5 to this ratio. Under the new CPIA with 16 different criteria, the methodology used to build CPR remains the same, although the governance factor now refers to only six criteria (five governance-related CPIA criteria plus the procurement practices criterion included in the Annual Report on Portfolio Performance rating). 24. Systematic relationships between CPIA/CPR and development outcomes can only be expected to be apparent in the fairly long run. The likely magnitude of errors and (both forward and backward) lags between policy indicators and outcomes suggests that a period of about a decade might be needed to show effects. The estimates below are therefore made using CPIA/CPR scores and development outcomes averaged over 1995- 04, and 1985-95. These are reported separately because the CPIA/CPR has evolved over the period.14 25. The use of long-term averaging raises a difficulty, because a given average can reflect very different circumstances during the period. An average rating of 3.3 can accurately reflect an extended period of middling performance. But it can also result when countries start out from strong ratings and decline precipitously (Zimbabwe, Cote D'Ivoire) or where they start from a baseline of low capability and consistently strengthen performance (Rwanda, earlier Uganda). Depending on lags and speed of adjustment, development outcomes can evolve differently in these cases. Rapidly adjusting variables will move from one equilibrium to another along with changes in 12 Therefore, it is not surprising that we find near perfect correlation between CPIA ratings and CPR prior to FY 99 (which refers to the 1997 CPIA exercise). 13 The governance discount was activated when a borrower scores "highly unsatisfactory" in three or more of the seven governance criteria (the six CPIA governance criteria and the Annual Report on Portfolio Performance's procurement criterion). The governance discount reduced the overall performance rating by one-third. 14 Similar estimates have been made for 5 year averages. The sign pattern of coefficients usually conforms to the decade estimates, but as expected they tend to be murkier because of short-term noise in the series, and are not reported here. Estimates have also been made for the period 1985-04 pooling decade averages, with broadly similar results. - 7 - policies whereas slowly adjusting ones will still be transitioning towards their long-run equilibria even if the level of the CPIA is quite stable during the decade. Capturing these trend dynamics is vital. 26. The model below therefore includes both the average level of the CPIA/CPR and its change over the decade as right-hand-side variables. In addition, three control variables are potentially important. · First, an Africa dummy, to allow for possible factors (such as risk, the burden of diseases such as malaria, high dependency ratios, the greater prevalence of landlocked countries, economic sparseness and fragmentation, adverse neighborhood effects and other geographic features that contribute to high transport costs) that may not be captured in the CPIA ratings but are likely to affect both growth and other development outcomes;15 · Second, an indicator of the prevalence of HIV/AIDS. AIDS has had a devastating impact on several aspects of human development in high-prevalence countries, and can be considered as a form of exogenous shock. Both threshold dummies and the level of incidence in 2005 have been used; the results below report the second specification. All of the six countries with very high prevalence (above 10 percent) are in Africa. · Third, the initial value of the development indicator is included to allow for possible convergence effects. Quantitatively larger gains in literacy, mortality and many other development indicators are possible when starting off from a low base than when levels are already high.16 These are not the only controls that might be relevant. Oil exporting countries have experienced particularly large terms of trade gains in the last several years, and several of them, especially in Africa, have also seen an upsurge in oil sector development. Such shocks can influence certain development outcomes. 27. The model used is then: (Development outcome) = a1*(CPIA or CPR average) + a2*(CPIA or CPR) + a3*(Africa dummy) + a4*(HIV/AIDS incidence) + a5*(initial development outcome) where is the change over the decade. The expected sign pattern for the model is: a1 and a2 positive; a3, a4 and a5 negative. 15 For studies discussing many such factors, see Teunissen and Ackerman (2005) 16 For example, median performance in reducing under-5 mortality has been about 3 per year when mortality rates are 300 per thousand but only 1 per year when they are 30 per thousand (Eifert and Gelb (2005)). - 8 - 28. The sample of countries is set out in Annex 1, distinguishing them according to whether their average CPIA and the change in their ratings over the decade are respectively above or below the average. The sample covers all IDA-eligible countries as of 1980, including China, which was IDA-eligible until 1999 and Egypt which was eligible until 1981 and then again until 1999. Countries with populations below one million were excluded; so were countries in Europe and Central Asia and those which had been part of the Soviet Union prior to its demise. These countries have faced the problem of system change rather than that of development; their indicators would therefore not be expected to evolve in the usual way. Countries lacking adequate data were also excluded.17 The distributions of CPIA averages and changes over the two decades are shown in Annex 2. III. Country Assessments and the Human Development Index (HDI) 29. The HDI produced by the UNDP is an average of three indexes, each of which is a ratio of the national level of achievement to the "maximum" attainable level. The three indexes measure: life expectancy; real GDP per capita (in PPP $); and educational attainment, with the last being a combination of literacy and enrolment data.18 The HDI is scaled to the range from 0 to 1. On average, the sample of IDA countries has seen its mean HDI increase by about 0.032 per decade. This means that a typical country will take about 130 years to transit from the low level of the HDI of about 0.3 to the reasonably high level of 0.7. As shown in Figure 1, African countries lag other countries. 30. Estimates of the relationship between the CPIA and HDI are cross-country, using averages over two decades: 1995-04 and 1985-95. Table 3.1 in Annex 3 shows the estimated relationship for the period 1995-04. All signs are as expected: · Both the average CPIA and its change are significant correlates of the increase in the HDI at the 99 percent level of confidence; · The coefficient on the Africa dummy is significantly negative, as is that on the control for the incidence of HIV/AIDS; · The initial level of HDI is also significantly negative, indicating strong convergence ­ the countries that started out at lower levels increased their HDI more than those starting off from a higher level. 17 Excluding China and Egypt would raise a problem of adverse selection for the sample. ECA countries experienced distinctive trajectories, with measured GDP first declining substantially with reforms then recovering; some also experienced large declines in life expectancy (World Bank (1996)) 18 In the HDI calculation, minima and maxima are defined for each index: 25 years to 85 years for life expectancy, $100 and $40,000 for real GDP per capita, and 0-100% for the four categories combined in the educational attainment index (Educational Attainment = 2*(adult literacy) + (combined primary, secondary, and tertiary enrollment ratios). The index is defined as (actual ­ minimum) / (maximum ­ minimum). The overall HDI is then a simple average of the Life Expectancy, Real GDP, and Educational Attainment Indexes. - 9 - Figure 1: Trends in the Human Development Index: 1975-04 HDI 1975-2004 (Africa vs. Non-Africa) 0.7 t) 0.6 bes=1 st,row 0.5 0=( 0.4 ex Non-Africa Average Indtne Africa Average mp 0.3 Develo 0.2 na m Hu0.1 0 1975 1980 1985 1990 1995 2000 2004 31. The first interesting result is the magnitude of the Africa dummy. Correcting for policy, HIV/AIDS and initial HDI African countries progress only about half as fast as non-African countries; gaining only 0.021 over the decade. Non-African IDA countries would transition from an HDI of 0.3 to 0.7 in about 90 years; African countries would take over 150 years to do so.19 32. Having a high HIV/AIDS prevalence rate has a dramatic impact on the HDI. Correcting for other factors, a prevalence rate of 20 percent corresponds to a loss of 0.08 relative to a country not affected by HIV. For a typical African country this is equivalent to the loss of four decades of progress. 33. The coefficient on the initial HDI index suggests that convergence is quite strong. Consider two countries F and G, the former with low initial HDI of 0.30 and the latter with a middling level of 0.50. Correcting for other factors, the change in the HDI index over the decade in country F would be 0.024 above the change in country G. In the long run, the HDI converges to a value that depends on the long-run value of the CPIA.20 34. Policies and institutional quality have a strong relationship with the increase in the HDI over the decade. Consider three archetype countries A, B and C. Country A ­ the "sustained reformer" ­ sees its overall CPIA improve from 3 to 4 during the decade. 19 The negative coefficient on the Africa dummy does not simply reflect the greater prevalence of landlocked countries; it remains highly significant when a landlocked dummy is introduced. In principle it could also result from mis-measurement in the CPIA, but the overestimation for Africa would need to be very large, over 1 on the 1-6 scale. Regional cross-checking and comparisons of the CPIA with other indices suggests that such an over-estimate is not plausible. 20 With high levels of the CPIA the level to which the HDI converges can be more than one which is greater than the maximum possible value. For countries approaching higher CPIA levels we expect the HDI to asymptote at 1; however, our sample includes only IDA countries at the lower range of the HDI. There will clearly be additional factors constraining the increase in the HDI as countries move towards very high levels. - 10 - Country B holds its CPIA steady and equal to 3. Country C ­ in a "downward spiral"- sees its CPIA fall from 3 to 2. Correcting for other factors, the HDI would increase by a further 0.031 in country A ­ this represents a doubling of progress for the average country. Country C would see no progress at all, with the deterioration in policies and institutions canceling out normal trend gains. 35. Long-run CPIA Levels are also important. The performance difference between two other archetype countries H and K, the first sustaining the CPIA at 2 and the second holding it at 4 over the decade is also very large at 0.050, more than the average rate of progress for the sample. 36. Figure 2 shows the change in the HDI index adjusted for controls, relative to a composite policy variable constructed by weighting the average CPIA and its change over the decade by the estimated coefficients. Figure 2: HDI Adjusted Change and Composite Policy Indicator, 1995-04 e genc .2 er onvc and VIH 5 .1 a,cirfA orf edt .1 adjus hangec 5 .0 HDI .04 .06 .08 .1 .12 Institutional variable (composite of CPIA average and CPIA change) IDA countries Fitted line 37. These results are illustrated in Figure 3, where trajectories of the HDI are depicted for two countries. Assuming that they start out from the same point, OA is the path for the first country, which sustains a CPIA rating of 3; OB is for the second, which sustains a rating of 4. Over time, the HDI scores approach their limits, which are higher for the stronger performer. Suppose at time T the first country improves its policies, moving its CPIA rating from 3 to 4 over the period of a decade and then sustains the rating at 4. Its - 11 - HDI index moves along the path OCD, increasing quite rapidly during the decade both because of the trend effect and because the average is increasing, and thereafter following the trajectory for CPIA = 4. The model therefore combines countries' movements along long-run trend paths in HDI with the possibility of shifting relatively rapidly between paths if policies and institutions change substantially, as they have in a number of countries. Figure 3: HDI Paths Over Time HDI Paths Over Time B CPIA=4 D HDI CPIA 3->4 A CPIA=3 C 0 38. Similar regressions for the decade 1985-95 are shown in Table 3.1 (Annex 3). The pattern is the same, with the coefficient on Africa dummy showing that the period was a "lost decade" for African development.21 39. These results suggest quite a strong statistical relationship between both CPIA levels and changes and the HDI composite development outcome, over extended periods. But they also show the importance of other factors. The additional challenges faced by African countries halves progress as measured by the HDI. A high prevalence of HIV/AIDS has also had a highly negative effect, setting back the level of development, as measured by the HDI, by some four decades. 40. The results are also relevant for the debates on "needs versus performance" and "results-based aid." Some have suggested that, as a substitute for policy selectivity, or perhaps as a supplement to it, allocations might be determined more directly by results. Kanbur (2005) observes that no uniform measure of policy performance can be expected to apply to all countries, and suggests that, in addition to a CPIA-type indicator, outcome 21 The parameter estimates for both the average and the change of the CPIA are smaller but their magnitude is partly offset by the greater variance in the CPIA and the change in the CPIA in the earlier period. - 12 - measures might receive a modest positive weighting. This would encourage countries to deviate from "orthodox" prescriptions if home-grown approaches are able to produce better results. 41. The complexity of this question is suggested by comparing Kanbur's proposal with that of Wood (2006) who notes that different countries might progress towards the MDGs at different speeds even with similar policies, because of exogenous factors. Resource allocations to move the less advantaged countries towards the MDGs will therefore favor the lagging countries, conditional on equal scores on the policy indicator. This means that with a greater emphasis on anticipated needs, current outcomes will be weighted negatively. 42. These suggestions are different because different factors are assumed to be responsible when outcomes diverge between countries with similar measured policies. In the case of IDA, actual allocations diverge from PBA allocations in three cases: caps on allocations to large countries, exceptional financing to post-conflict countries and base allocations to all countries. As noted by Wood, capping the large Asian countries is consistent with a view that long-run development is more challenging in Africa. Without caps on the allocations for India, Pakistan, Indonesia, and China before 1999, allocations to African countries would represent only some 22 percent of IDA instead of their current share of 50 percent. African countries therefore receive allocations more than double those warranted by size and policy performance alone, presumably to compensate for their greater perceived development challenges. This is supported by the regressions above which show that African IDA countries derive considerably smaller gains in outcome indicators relative to other IDA countries, at given levels of policies and institutions.22 Base allocations to all countries benefit the small countries disproportionately and could similarly be interpreted as compensating for greater vulnerability. IV. Patterns for Other Indicators 43. The same model is now applied to three further development indicators: changes in the under 5 mortality rate, changes in the immunization rate, and growth of GDP per head in constant 2000 US Dollars. Even though immunization is an intermediate indicator, the first two are of particular interest because the HDI includes only life expectancy, which is itself a "reduced form" encompassing many possible health-related factors. A. Under-5 Mortality 44. The results of estimating the model to the under-5 mortality rate for 1995-04 are shown in Annex 3, Table 3.2. The sign pattern is as predicted, allowing for the fact that progress is reflected in a reduction in the mortality rate: 22 If outcomes were positively included in allocation formulae in mechanical ways, countries with high HIV/AIDS incidence would also receive lower allocations. - 13 - · The average of the CPIA is significant but the change is less so; · However, considering again the three countries A, B and C, we find that the difference in their policy performance corresponds to a divergence in the reduction in mortality rates of 23.7 per thousand over the decade. · African countries have experienced lower achievements than other countries. · Convergence is significant, because it is easier to see large gains in reducing mortality from high levels. Figure 4 shows the scatter of declines in the mortality rates against the composite policy variable, adjusting for controls. · The HIV/AIDS coefficient is of expected sign but not significant. Figure 4: Mortality rate and Policy Composite, 1995-04 000) 40 1, per( edt 20 jus ad 0 hangec e 0 atry -2 alitt 0 or m -4 5- 0 Under -6 20 30 40 50 60 Institutional variable (composite of CPIA average and CPIA change) IDA countries Fitted line 45. The regression for the earlier decade is similar (Annex 3, Table 3.2). Coefficients of both the average and the change of the CPIA are significant at the 90 percent level. The Africa dummy is very large, indicating that during this period declines in mortality in Africa lagged those in other countries by 30 per thousand, assuming comparable policies and starting points. B. Immunization Rate 46. Applying the model to the immunization rate produces broadly similar results (Annex 3, Table 3.3). The difference between countries A and C translates into a - 14 - divergence of immunization rates of 18 percent over the period.23 The scatter of adjusted immunization rate versus the policy composite is shown in Figure 5. Over the earlier decade, the CPIA change is significant and the CPIA average less so (Annex 3, Table 3.3). Other variables are not significant. Figure 5: Immunization rate and Policy Composite, 1995-04 ) %( 80 olsr cont orf 60 edt adjus 40 angehc e atr 20 onita uniz m 0 mI 20 25 30 35 40 Institutional variable (composite of CPIA average and CPIA change) IDA countries Fitted line C. GDP Per Capita 47. When the model is applied to the growth of GDP per head for the later decade, both the CPIA level and change are significant. The growth rate of the "sustained reformer" country A exceeds that of "downward spiral" country C by 3.4 percent or, over a decade, a difference in GDP per capita of about 35 percent. The Africa dummy indicates a growth handicap of roughly 1.6 percent (Annex 3, Table 3.4). HIV/AIDS slows growth but is not significant. Figure 6 shows the scattergram of adjusted growth rates against the policy composite. 23 However, it needs to be kept in mind that the change of CPIA is not significant at the 90 percent level. - 15 - Figure 6: GDP per capita growth rate and Policy Composite, 1995-04 ) %( 10 olsr ontc orf edt 5 adjus h owt gr a apitc per 0 PDG 3 4 5 6 Institutional variable (composite of CPIA average and CPIA change) IDA countries Fitted line 48. Oil exporters benefited from exceptionally favorable price trends in the latter part of the period, and several also from accelerated oil sector development. When included, an oil country dummy24 proved to be both large (3.48) and significant at the 99 percent level. Inclusion also increased the level and significance of the CPIA. Excluding the oil exporters altogether further increases the parameter estimate for the average CPIA ­ a one point increase is associated with over 3 percent higher growth (Annex 3, Table 3.4) as well as accentuates the negative coefficient of the Africa dummy. 49. Results for the earlier period are similar. Both the average and change in the CPIA are significantly related to growth at the 95 percent level. Growth rates in countries A and C diverge by 3.2 percent. The coefficient on the Africa dummy is strongly negative, indicating a penalty of 2.1 percent, larger than in the second period (Annex 3, Table 3.4).25 24 Oil exporters are defined as the countries in our sample whose average oil exports/total exports ratio over the decade was above 30 percent. This corresponds to Angola, Cameroon, DRC, Egypt, Nigeria, Sudan and Yemen for the most recent decade, and Angola, Bolivia, DRC, Egypt, Indonesia, Nigeria and Yemen for the earlier period. 25 As for the most recent decade, we considered the oil factor in the regression. Once the oil dummy is included, both the change and the average of CPIA remain significant at the 95 percent level, although the magnitude of their parameter estimates decreases a little. The oil dummy is significant at the 95 percent level and high (2.46). Excluding the oil countries makes the parameter estimates for both the - 16 - 50. The results for the individual development indicators confirm the general form of the relationship developed in Section III and estimated for the CPIA and the HDI. They suggest that, over the last two decades both the level and trend in the CPIA have been quite strongly associated with development performance. Although development is a long-run process, countries making major gains in policies and institutions can experience quick wins. Rapid progress is possible in lagging countries sustaining good, and improving, policies. On the other hand, countries with deteriorating policies and institutions can see substantial losses over a period of several years. The results also provide a reminder of how much Africa has lagged behind other regions and that even improved outcomes after 1995 have not resulted in Africa closing the development gap. They also indicate the very high cost, in terms of certain human development outcomes, of HIV/AIDS. V. Endogeneity of the CPIA ­ Does it matter? 51. In their review of studies on the relationship between institutions and growth, Pande and Udry (2005) conclude that cross-country regressions have provided compelling evidence of a causal link between a cluster of "good" institutions and long- run growth, but that they are unable to either disentangle the effects of specific institutional channels on growth or to understand the impact of institutional change on growth.26 This conclusion is relevant to the relationship between the CPIA and the wider set of outcomes discussed here ­ as noted above, it would be surprising if the CPIA was not endogenous to outcomes because analysts will use whatever information is available, including on outcomes, to set ratings. But it is difficult to test rigorously for the degree of endogeneity. 52. Testing is possible for "naïve" endogeneity however, where this is defined as analysts responding in a mechanistic way to outcomes, including those impacted by external factors or exogenous shocks. Two such cases noted above are the HIV/AIDS pandemic, which has severely deteriorated development outcomes for several countries and the positive growth impact of high energy prices on oil exporters. Tests for changes in the CPIA show some response to these developments, but in neither case is this significant. Indeed, once Zimbabwe is excluded, the average change in CPIA ratings for high HIV/AIDS countries is a little higher than for other countries. 53. A further test is to examine the responsiveness of ratings to short-run GDP growth rates which are known to be highly variable and often driven by weather, terms of trade or other shocks in poor countries. Data from 1985 to 2005 show that the most recent annual GDP growth rate available at the time of a rating has low predictive power for the change in the CPIA (Figure 7). Extending the lag to two years produces even less predictive power ­ and a negative relationship. average and the change of CPIA decrease only slightly, suggesting that the results are reasonably robust. 26 This is essentially because the number of variables available as instruments is limited and because they tend to be derived from persistent features of a country's institutional environment, such as its colonial past. - 17 - Figure 7: CPIA annual change and GDP per capita annual growth, 1985-04 2 1 e hangc 0 annual A CPI -1 -2 -40 -20 0 20 GDP per capita growth rate (%) IDA countries Fitted line 54. There is therefore little evidence that CPIA ratings have responded in a "naïve" way to shocks that influence development outcomes. This suggests that, while the CPIA may reflect available information on outcomes, for example, in making its assessment of service delivery, analysts do not mechanically infer policy judgments from outcome trends, and that they do try to separate out exogenous factors from policy factors. This provides some comfort to the use of the ratings as an indicator, possibly partly reflecting outcomes, but based on independent judgments on policies and institutions VI. Extending the Analysis to the Normalized CPR We now extend the analysis to the CPR, as used each year to allocate IDA. The formula relating the CPR to the CPIA has changed several times, so that the actual series used does not relate in a consistent way to the CPIA. The two series are closely related, however, as shown in Figures 8 and 9. The CPR is normalized to between 0 and 6 each year, with the extremes being the lowest and highest ratings.27 27 The CPR was normalized using the formula: 6* (CPR ­ min CPR) / (max CPR ­ min CPR). - 18 - Figure 8: CPR vs CPIA, average 1995-04 4 4 200 3 5- 199 e ag erva 2 R CP 1 2 2.5 3 3.5 4 CPIA average 1995-2004 Figure 9: CPR vs CPIA, average 1985-94 5 4 4 199 5- 198 3 e ag erva R 2 CP 1 1 2 3 4 5 CPIA average 1985-1994 - 19 - 55. The CPR performs in much the same way as the CPIA in the above equations. Annex 4 Table 4.1 shows regression results for the HDI index over the most recent decade, and for the previous decade. The lack of significance for the latter is due to the use of a smaller sample due to fewer countries having CPR.28 Annex 4 also summarizes the results of regressions for the individual indicators, using CPR instead of the CPIA. The results are broadly similar to those we had found using the CPIA, except, again, because of the use of a smaller sample of countries for CPR regressions.29 56. These results suggest that if IDA allocates resources to countries according to the CPR it will tend to steer assistance to countries more likely to achieve favorable development outcomes in the long run. Actual commitments have diverged from performance allocations for several classes of countries. Figure 10 shows IDA actual commitments to the sample of countries over the last decade relative to the average CPR, distinguishing "capped" countries and countries which benefited from post-conflict allocations. Across all countries, there is no clear relationship between the CPR and actual commitments, but excluding the countries which received post-conflict allocations and those that were capped, the relationship is a reasonably close one ­ the average CPR rating between 1995-04 is a significant correlate of the average IDA actual commitment per capita during the same period at the 95 percent level of confidence, as illustrated by the steeper line in Figure 10.30 28 When restricting our original HDI regression (first decade, using CPIA) to the sample of countries for which CPR is available, the CPIA average and its change over the period are not significant correlates of the change of HDI at the 90% level of confidence anymore. 29 There is some difference in the relative orders of magnitude of the parameter estimates for the average and change of the rating system, with the CPR effect being stronger in changes and the CPIA effect stronger in levels. 30 Part of the remaining dispersion is because other criteria, such as the minimum allocation and income per head, are not factored in - 20 - Figure 10: IDA commitments and CPR )RDS( 15 4002- 12 9951 erage 9 av ta, apicr 6 pe on ati 3 oclal A ID 0 0 .5 1 1.5 2 2.5 3 3.5 4 4.5 CPR average 1995-2004 IDA countries Caps Post-conflict Fitted line (all) Fitted line (excluding caps and post-conflict countries) VII. Country Ratings and Indicators of Project Performance 57. Since the early 1990s, several papers have found significant relationships between country ratings and project performance,31 some estimating this on a cross-country and some on a project-by-project basis. Here, we draw on a sample of 4370 projects evaluated for all countries between 1980 and 2004 by the World Bank's Independent Evaluation Group (IEG). Our results distinguish between the IDA countries in our sample and the rest of the countries, and also include an IDA dummy to distinguish between IBRD and IDA projects in blend countries. The average duration of projects was 6.37 years, and ranged from a minimum of 1 year to a maximum of 15 years. Project evaluations are typically performed during the final year of the project or the following year. However, they have occasionally been done up to three years before the end of the project or more than six years afterward. Although we expect little effect from outliers given the constrained scale on which projects can be evaluated we disregard the few projects with evaluations more than seven years after the project end as outliers. 31 An early study, linking project performance to governance indicators is Isham, Kaufmann and Pritchett (1992). - 21 - 58. IEG ratings encompass three metrics: · Outcome: the extent to which the project's major relevant objectives were achieved, or are expected to be achieved efficiently (on a scale of 0 to 5); · Sustainability: the resilience to risk of net benefits flows over time (on a scale of 0 to 4); · Institutional Development Impact: the extent to which a project improves the ability of a country or region to make more efficient equitable and sustainable use of its human, financial and natural resources (on a scale of 0 to 3). As shown in Table 5.1 in Annex 5, the three ratings are correlated across projects, suggesting that results found for one dimension of performance will resemble results found for the others. 59. Regression results, reported in Annex 5, suggest the following conclusions:32 · CPIA ratings are significant determinants of project performance as measured by all three metrics. If the initial CPIA is significant, so are the average CPIA and the change in the CPIA during the time of project implementation; · The results are similar for the sample of countries used above, and for other countries; · The relationship between policy and project success is quite powerful. For example, a one unit higher average CPIA during the period of implementation (which encompasses both level and trend effects) is associated with a doubling of the odds of having the highest outcome for a project vs. having a lower outcome;33 · The likely impact of exogenous factors in Africa (as captured in the Africa dummy) remains consistently negative and statistically significant for our sample; · Including quality of supervision reduces in some cases the magnitude and significance of the Africa effect, but it does not substantially change the effects of policy. 60. Results using the CPR reach similar conclusions. It should be noted that the CPR formula includes a measure of project portfolio performance so that the average CPR during the active period of a project may not be independent of the performance of that project; however, with many active projects in a typical country portfolio, the loss of independence is likely to be small.34 32 Following Guillaumont and Laajaj (2006) we use ordered logit regressions. 33 This result refers to a specification using our sample of IDA countries with the average and the change of CPIA on the right-hand side. Annex 5 presents more details. 34 Country performance ratings include portfolio ratings that are assessments made by country teams of the active IDA portfolio. The dependent variable is an independent IEG rating of projects that have been completed. To the extent that the former are a good predictor of the latter, there may be some endogeneity in the average relationship over the active period. ] - 22 - 61. Figure 11 summarizes the results of one relationship, between the average CPIA during the period of implementation of the project and project ratings for outcomes. As the country rating improves, the probability of receiving a low rating falls and of achieving a high rating rises. At a CPIA of 5, about 70 percent of projects receive a rating of 4, and only 10 percent receive a rating of 1; the ratios are almost reversed at a CPIA of 1. The bar chart shows the expected value of the outcome at each CPIA. Figure 12 shows the corresponding figure for the CPR. Figure 11: CPIA Average Scores and Project Ratings 80 5 4.5 70 4 60 ) 3.5 Expected outcome %( (RHS) e 50 e p(outcome=0) m 3 m tco co p(outcome=1) oufo out 40 2.5 p(outcome=2) yit edt ec p(outcome=3) abil 2 30 xpE obrP p(outcome=4) 1.5 p(outcome=5) 20 1 10 0.5 0 0 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 CPIA average during the project Figure 12: CPR Average Scores and Project Ratings 80 5 4.5 70 4 60 ) 3.5 (% e Expected outcome 50 e moctuofo 3 (RHS) omct p(outcome=0) ou 40 2.5 p(outcome=1) y edtc ilitbaborP p(outcome=2) 2 30 pexE p(outcome=3) 1.5 p(outcome=4) 20 1 p(outcome=5) 10 0.5 0 0 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 CPR average during the project 62. These conclusions should be considered as tentative, because of the initial nature of the work. But they confirm the conclusions of other studies, and provide strong support to the proposition that policy and institutional quality matter for project performance, both at the time projects are initiated and considering changes over the period of implementation. - 23 - VIII. Conclusion 63. The main conclusion of this paper is that country ratings of IDA-eligible countries have been quite strongly associated with a range of development indicators and outcomes. The relationships, estimated using decade averages, are fairly long-term, and involve both the average level of the CPIA/CPR over periods of a decade and its trend change. All else equal, a country that sustains a high CPIA/CPR over several decades will open up a sizeable HDI advantage over a country with a low CPIA/CPR. In the particular formulation used, in the long run the HDI index will converge to a higher level if a high CPIA/CPR is sustained. Similar conclusions hold for some other indicators of development, including GDP per capita growth. Over a decade, high CPIA/CPR countries will see a considerable increase in GDP/head relative to low CPIA/CPR countries. 64. A second conclusion is that, even though development is a long-run process, countries can shift between long-run development paths relatively quickly. When the quality of policies and institutions undergoes a major change countries can see substantial changes in outcomes over periods of a decade. In terms of the HDI, strong improvers advance at twice the average rate and poor performers see static, or declining, development outcomes. Post-conflict countries can thus experience rapid recovery; while countries caught in a downward political and economic spiral with worsening policies will see large deteriorations over periods of a decade. 65. Changes in CPIA or CPR ratings cannot be well predicted over an extended period from the historical, experienced-based, ratings alone. This highlights the importance of having a good understanding of the social and political processes in the countries concerned to complement the information contained in the ratings, and of strengthening governance in the broader sense to improve the sustainability of good policies. 66. Given the very broad nature of the CPIA and the CPR and the difficulty of assessing many ratings in a fully "objective" manner that does not draw on evidence of outcomes, such indicators are unlikely to be fully exogenous. Indeed, it would be surprising if they were, since analysts should use all available information, including outcomes, in making their assessments. This makes it difficult to interpret these results in a strictly causal sense. However, tests for "naïve" endogeneity show that analysts do not extrapolate mechanically from outcomes to CPIA ratings. This is useful, because it suggests that the ratings are reasonably effective in separating out the impact of external factors, including shocks, that influence outcomes. 67. A third result is the confirmation that African countries face a much more difficult development challenge than others. Adjusting for policies and institutions, and also for the prevalence of HIV/AIDS, in terms of the HDI they have progressed at half the speed of other countries over the last two decades, or even less, and their gains in reducing under-5 mortality are also smaller. However, the slower rate of progress in Africa as compared with other IDA countries appears to have moderated during the later decade - 24 - suggesting perhaps the beginning of a catch-up process. HDI ratings have also been severely impacted by HIV/AIDS, mainly by the impact on life expectancy. High- prevalence countries have been thrown back at least four decades on the HDI scale. However, AIDS has appeared as a less significant determinant of the other development outcomes covered. 68. Project-level data suggest a similar story. Higher CPIA scores at the initial stages increase the likelihood of success, as measured by IEG ratings. So does a further improvement in the CPIA during project implementation. Project performance is therefore sensitive to both initial conditions and to changes in country performance during the active period of the project, confirming the importance of performance for achieving expected results. Similar results hold for the historical normalized CPR. All else equal, the probability of achieving a high rating is also lower for projects in Africa than in other regions. 69. The Africa and HIV/AIDS effects highlight the debate on "needs versus performance" and the complexity involved in moving to "results-based" aid. Making extra assistance conditional on outcome indicators has considerable appeal, but it is essential to understand whether good performance reflects extra effort (in ways not captured by the CPIA/CPR, which may not entirely fit the circumstances of a particular country) or exogenous factors, such as favorable demographic trends, better geography, rapidly-growing neighbors, exceptional terms of trade shocks, or others. Performance rating can be seen as an effort to separate out the impact of such exogenous factors on performance. A forward-looking assessment would increase aid to Africa beyond the level suggested by current ratings or performance on outcomes, because of the greater challenge ahead. IDA allocations can be interpreted in this light, with caps on large Asian countries freeing-up more resources for African countries that appear to face a more difficult development challenges; the minimum allocation can be interpreted as an effort to compensate for the higher vulnerability of small countries. After allowing for the effect of caps and post-conflict-allocations, IDA allocations have indeed been performance-based. 70. How do these results bear on the use of performance-based allocation by IDA and the role of IDA in the context of the overall aid architecture? They do not, of course, suggest that the CPIA or CPR, as they currently stand, are necessarily the ideal indicators. Indeed, because formulae have evolved over time, the statistical relationships cover a range of specifications. However, they support the use of indicators along the lines of the CPIA/CPR, estimated on a reasonably consistent basis over long time periods, to provide an effectiveness-based anchor for the system of development assistance, including the potential value of such indicators in facilitating an open and inclusive dialog on development progress. In terms of aid allocation, the PBA system is preferable to arbitrary, politically driven, allocations in at least three respects. · First, it provides a check on excessive aid allocations to poorly-performing countries, reserving resources for others. - 25 - · Second, use of the PBA system backed up by stable IDA resources improves the stability and predictability of resource flows where this is most needed ­ to those countries maintaining stable performance. · Third, performance ratings provide a standard to help separate out exogenous factors that make development more or less challenging to different countries and regions. The adjustments to the PBA system (through capping and post-conflict allocations) take into account these exogenous factors that affect long-run development. - 26 - References Benno, Ndulu J. (2006). "Ramping up African Growth: Lessons from Five Decades of Growth Experience." Economic Affairs, 26(4): 5-11, December. Chauvet, Lisa, Paul Collier, and Andreas Fuster (2006). "Supervision and Project Performance: A Principal-Agent Approach." Working paper draft, 2006. Dollar, David and Victoria Levin (2004). "The Increasing Selectivity of Foreign Aid, 1984-2002," Policy Research Working Paper 3299, The World Bank. Gelb, Alan, Brian Ngo, and XiaoYe (2004). "Implementing Performance-Based Aid in Africa." World Bank Africa Region Working Paper 77 (Washington). Guillaumont, Patrick and Lisa Chauvet (2006). "Aid and Performances: A Reassessment." Journal of Development Studies 37(6): 66-92. Guillaumont, Patrick and Rachid Laajaj. 2006. "When instability increases the effectiveness of aid projects." World Bank Policy Research Working Paper 4034, October. Isham, Jonathan, Danny Kaufmann and Lant Pritchett (1992), "Civil Liberties, Democracy and the Performance of Government Projects" World Bank Economic Review Volume 11, No 2 Kanbur, Ravi (2005). "Reforming the Formula: A Modest Proposal for Introducing Development Outcomes in IDA Allocation Procedures." SAGA Working Paper, January. Pande, Rohini and Chris Udry (2005), "Institutions and Development: A View From Below" Economic Growth Center Discussion Paper no 928, Yale University. Teunissen Jan Joost and Aki Akkerman (eds) ( 2005), Africa in the World Economy: The National, Regional and International Challenges. Fondad, The Hague. Wood, Adrian (2006). "Looking Ahead Optimally in Allocating Aid." SSRN Working Paper Series, August. http://ssrn.com/abstract=927405 World Bank (2006). "Statistical Capacity Improvement in IDA Countries: Progress Report," prepared by the Development Data Group (DECDG), May 16. World Bank (1996). World Development Report: From Plan to Market. Oxford University Press. - 27 - Annex 1: Sample of countries The tables below show, for each decade, the countries in the sample according to their CPIA average and CPIA change along the decade. · For the period 1995-04 CPIA above mean CPIA below mean CPIA change Burkina Faso Angola above mean Egypt Burundi Ethiopia Cameroon Lesotho Congo, DRC Madagascar Congo, Republic Malawi Liberia Mongolia Niger Mozambique Nigeria Senegal Rwanda Tanzania Sierra Leone Uganda Somalia Yemen Sudan Bangladesh Cambodia CPIA change Central African below mean Benin Republic Bolivia Chad China Guinea Cote d'Ivoire Guinea-Bissau Gambia Haiti Ghana Laos Honduras Togo India Zimbabwe Indonesia Kenya Mali Mauritania Nepal Nicaragua Pakistan Sri Lanka Vietnam Zambia - 28 - · For the period 1985-94 CPIA above mean CPIA below mean CPIA change above mean Bangladesh Benin Central African Bolivia Republic Burkina Faso Chad Gambia Congo, Republic Mauritania Egypt Nepal Ethiopia Sri Lanka Guinea Tanzania Honduras Vietnam Laos Lesotho Mozambique Nicaragua Sierra Leone Uganda CPIA change below mean Burundi Cameroon China Congo, DRC Cote d'Ivoire Haiti Ghana Liberia Guinea-Bissau Myanmar India Niger Indonesia Nigeria Kenya Somalia Madagascar Sudan Malawi Zambia Mali Pakistan Rwanda Senegal Togo Zimbabwe - 29 - Annex 2: Distributions of the averages and changes of CPIA during the two decades CPIA average, 1995-04 1 .8 yit .6 Dens .4 .2 0 1 1.5 2 2.5 3 3.5 4 4.5 5 CPIA CPIA change, 1995-04 .8 .6 y nsit .4 De .2 0 -4 -3.5 -3 -2.5 -2 -1.5 -1 -.5 0 .5 1 1.5 2 2.5 3 3.5 4 CPIA change - 30 - CPIA average, 1985-94 .8 .6 yit .4 Dens .2 0 1 1.5 2 2.5 3 3.5 4 4.5 5 CPIA CPIA change, 1985-94 .5 .4 yit .3 Dens .2 .1 0 -4 -3.5 -3 -2.5 -2 -1.5 -1 -.5 0 .5 1 1.5 2 2.5 3 3.5 4 CPIA change - 31 - Annex 3: HDI and other Indicators: Regression Results Table 3.1: HDI and CPIA Dependent variable HDI change 1995-04 HDI change 1985-95 CPIA average 0.026 *** 0.015 * CPIA change 0.018 *** 0.0089 * Africa dummy -0.028 ** -0.047 *** HIV prevalence rate -0.0042 *** Initial HDI -0.12 *** -0.12 ** Constant 0.041 0.062 R-squared 0.67 0.40 Observations 48 41 Table 3.2: Under-5 mortality and CPIA Dependent variable Under-5 mortality rate Under-5 mortality rate change 1995-04 change 1985-95 CPIA average -14.11 ** -6.77 * CPIA change -4.80 -4.96 ** Africa dummy 18.60 ** 30.38 *** HIV prevalence rate 0.63 Initial mortality rate -0.17 *** -0.19 *** Constant 39.66* 15.42 R-squared 0.31 0.53 Observations 49 42 - 32 - Table 3.3: Immunization rate and CPIA Dependent variable Immunization rate Immunization rate change 1995-04 change 1985-95 CPIA average 10.45 ** 5.16 CPIA change 3.78 5.56 ** Africa dummy 1.75 -8.76 HIV prevalence rate -0.35 Initial immunization rate -0.60 *** -0.49 *** Constant 15.87 32.32 ** R-squared 0.53 0.42 Observations 47 43 Table 3.4: GDP per capita growth rate and CPIA GDP per capita growth GDP per capita growth Dependent variable rate rate GDP per capita growth change 1995-04 change 1995-04 rate change 1985-95 Oil countries excluded CPIA average 1.50 ** 3.70 *** 1.90 *** CPIA change 0.95 * 0.64 0.66 ** Africa dummy -1.55 * -2.97 *** -2.11 *** HIV prevalence rate -0.016 0.049 Initial growth rate -0.00052 -0.0054 *** -0.0027 ** Constant -1.60 -7.0075 *** -3.84 * R-squared 0.24 0.58 0.50 Observations 51 44 44 - 33 - Annex 4: CPR Regression Results Table 4.1: HDI and CPR Dependent variable HDI change 1995-04 HDI change 1985-95 CPR average 0.012 ** 0.011 CPR change 0.052 ** 0.039 Africa dummy -0.041 *** -0.045 *** HIV prevalence rate -0.0035 *** Initial HDI -0.19 *** -0.095 Constant 0.13 *** 0.071 * R-squared 0.71 0.37 Observations 45 33 Table 4.2: Under-5 mortality and CPR Dependent variable Under-5 mortality rate Under-5 mortality rate change 1995-04 change 1985-95 CPR average -8.24 ** -3.08 CPR change -17.40 -17.98 Africa dummy 19.05 ** 32.07 *** HIV prevalence rate 0.56 Initial mortality rate -0.17 *** -0.17 *** Constant 16.53 -5.79 R-squared 0.31 0.48 Observations 48 35 - 34 - Table 4.3: Immunization rate and CPR Dependent variable Immunization rate Immunization rate change 1995-04 change 1985-95 CPR average 6.33 ** 7.39 ** CPR change 13.78 32.69 *** Africa dummy 0.46 0.39 HIV prevalence rate -0.28 Initial immunization rate -0.62 *** -0.51 *** Constant 34.58 *** 22.03 * R-squared 0.54 0.46 Observations 46 35 Table 4.4: GDP per capita growth rate and CPR GDP per capita growth GDP per capita growth Dependent variable rate rate GDP per capita growth change 1995-04 change 1995-04 rate change 1985-95 Oil countries excluded CPR average 0.72 * 2.02 *** 1.31 *** CPR change 3.10 * 0.21 3.02 ** Africa dummy -1.68 ** -2.81 *** -2.82 *** HIV prevalence rate 0.00058 0.061 Initial growth rate -0.0017 -0.0059 *** -0.0050 * Constant 1.62 -0.35 0.14 R-squared 0.24 0.58 0.47 Observations 48 42 35 - 35 - Annex 5: CPIA Ratings and Project Performance 1. Given the fairly high correlation between the three metrics used in the IEG study, we only show the results of regressions using project outcome as the dependent variable. Table 5.1: Correlation between the three different metrics of project performance Outcome Sustainability Institutional Development Impact Outcome 1 Sustainability 0.6775 1 Institutional Development Impact 0.6084 0.5594 1 2. Table 5.2 shows the results of ordered logit regressions using the following explanatory variables: · overall CPIA at the beginning of the implementation of the project; · average of the overall CPIA during the project; · change of the CPIA during the project; · Africa dummy; · IDA dummy (equal to 1 if the country in which the project is implemented is IDA-eligible). Table 5.2: Project performance and CPIA ratings Dependent variable: outcome Same sample of Rest of the countries All countries of the project countries as before Specification (1) (2) (3) (4) (5) (6) CPIA initial value 0.694 *** 0.976 *** 0.782 *** CPIA average 0.702 *** 0.886 *** 0.747 *** CPIA change 0.608 *** 0.234 *** 0.82 *** 0.294 *** 0.694 *** 0.272 *** Africa dummy -0.355 *** -0.375 *** -0.356 * -0.355 * -0.358 *** -0.391 *** IDA dummy 0.13 0.129 0.334 ** 0.274 ** 0.158 ** 0.13 * Observations 2565 2524 2063 2035 4628 4559 3. Note: All estimations include sectors dummies and annual dummies, the latter referring to the last year of the project. 4. Table 5.2 suggests that, in addition to the information content of the initial CPIA, there is a substantial impact of large variations in the CPIA during the project on the probability that the latter will be judged as successful. - 36 - 5. Since regressions use ordered logit, the parameters estimates need to be interpreted appropriately. Taking equation 2, (which covers our sample of IDA countries, and uses both the CPIA average and change during the length of the project), the odds-ratio OR can be defined as: P outcome < 5 CPIA = c +1 ) ) OR = P (outcome = 5 CPIA = c +1 ( P (outcome = 5 CPIA = c ) ) P outcome < 5 CPIA = c ( 6. Where CPIA is the average of the CPIA during the length of the project, c is a positive number between 1 and 6, and 5 is the highest level of outcome for a given project. The odds-ratio is equal to: OR = exp(0.702) 2.02 7. For a project in a given country A, OR can be interpreted as the ratio of the odds of having an outcome S = 5 (the highest rating) relative to having a rating less than 5 if country A has a CPIA rating of 4 throughout the project period, relative to the corresponding ratio if the CPIA is only three. The OR of 2 holds for any two other CPIA ratios with one unit difference. 8. Another way to interpret the results is by looking at marginal effects. Below are marginal effects relative to the outcome of the project for our sample of IDA countries. A one unit increase in the average CPIA score changes the probability of having the outcome mentioned in the top row by the percentages below. 9. The first three rows refer to the three previous specifications using the average and the change of CPIA on the right-hand side, while the fourth specification (EX) refers to the comparison between the outcome of the project in a country having its average CPIA during the length of the project at the 25th percentile (here CPIA = 3.28) and in a country having its average CPIA at the 75th percentile (here CPIA = 4). Table 5.3: Marginal effects of a one-unit increase of the CPIA on project outcome Project outcome/ Highly Unsatisfactory Moderately Moderately Satisfactory Highly Specification unsatisfactory Unsatisfactory Satisfactory Satisfactory (2) -1.1% -11.8% -1.7% -2.7% +15.7% +1.8% (4) -0.6% -12.4% -3.4% -4.5% +19.6% +1.6% (6) -1.1% -11.7% -1.7% -2.7% +15.4% +1.8% (EX) -0.8% -8.5% -1.2% -1.9% +11.3% +1.3% - 37 - 10. Similarly, Table 5.4 below shows the marginal effect of a one-unit increase in the variation of the CPIA during the length of the project. Table 5.4: Impact of a one unit increase in the variation of the CPIA during the project Project outcome/ Highly Unsatisfactory Moderately Moderately Satisfactory Highly Specification unsatisfactory Unsatisfactory Satisfactory Satisfactory Using the average of -0.3% -3.9% -0.5% -0.9% 5.2% 0.6% CPIA Using the initial value -1.04% -10.31% -1.51% -2.39% 13.68% 1.58% of CPIA 11. Interestingly, when the supervision rating is included, the Africa dummy is significant at the 90 percent level of confidence only, whereas it was significant at the 99 percent level without the supervision rating. Since the African dummy is negative, this finding suggests that there are some components of negative project performance that may not be attributable to the African characteristic of a project, but are captured by characteristics of project supervision as assessed by IEG. This could warrant further examination. 12. The results of this section therefore provide considerable support to the proposition that levels and changes in the CPIA are statistically related to project performance during implementation. Part of this effect can be captured by the CPIA rating at the outset of the project, which can be used as an indicator of potential success. The remainder, however, reflects the impact of changes in the CPIA during implementation which is also shown to be powerful.