-_-_~~~~w __ _ _ _ --.. 4 i ; POLICY RESEARCH WORKING Pt PER 1496 Reviving Project Appraisal How can the World Bank improve the economic at the W orld Bank analysis of its projects? By shifting from precise rate of return calculatiorns to a Shantayanan D)evarajan broader examination of the Lyn Squire rationale for public provision Sethaput Suthiwaarz'-Narue put and focusing on three critical aspects: the counterfactual private sector response, the fiscal impact, and the fungibility of lending. The World Bank Policy Research Department Public Economics Division and Office of the Director August 1995 icy RFSEARCH WORKING PAPER 1496 Summary findings [)evarajan, SqLlire, and Suthiwart-Narueput focus on two * Fiscal impact. Applying the private sector broad questions: counterfactual would lead the Bank to undertake projects * What is the proper role for project evaluation in with a reasonable case for public intervention, such as today's world, where countries have reduced major basic infrastructure, primary education, and rural health. economic distortions and are reconsidering the role of These projects typically share the characteristic that costs the state? are borne by the public sector while benefits are enjoyed * Besides project evaluation, how else can economic by the private sector. But in the absence of analvsis ensure high-quality projects? nondistortionary, lump sum taxes, there is likely to be a The authors argue for a shift in the emphasis of project positive marginal cost of taxation and a premium on evaluation away from a concern with precise rate of public income. Since the Bank has not used such a return calculations to a broader examination of the premium and treats public costs and private benefits rationale for public provision. In this context, three areas equally, it has systematically overestimated the net critical for proper project appraisal are the benefits of these projects. counterfactual private sector supply response, the fiscal * Fungi bility of lending. Project-specific appraisal can impact, and the fungibility of lending. at best assess only the rate of return and the acceptability I Countterfactual private sector supply respontse. Any of the project being appraised. rhis limitation is type of cost-benefit analysis - be it in the public or the problematic because the project might have been private sector - requires the project evaluator to specify undertaken even without Bank financing. If that is the the counterfactual: what would the world have looked case, the Bank is actually financing some other project- like in the absence of the project? Since World Bank one nlot subject to appraisal by the Bank - that would projects are public sector projects, the relevant not have been in the investment program without Bank counterfactual involves assessing what the private sector financing. This problem arises because financial resources would have otherwise provided, and the relevant are fungible to some extent. One way to alleviate this magnitude for evaluation purposes is the net concern is to conduct public expenditure reviews before contribution of the public project. Failure to consider embarking on the appraisal and financing of specific explicitly the private sector counterfactual during projects. Furthermore, financing a portion of the evaluation biases the lending mix of the Bank away from government's sectoral investment program may be more projects with strong public good characteristics toward effective than project-specific lending. projects with private good characteristics. This paper - a joint product of the Public Economics Division and the Office of the Director, Policy Research Department - is part of a larger effort in the department to improve the allocation of public expenditures in developing countries. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Cynthia Bernardo, room NI 0-053, telephone 202-473-7699, fax 202-522-1154, Internet address prdpe@worldbank.org. AUgust 1995. (29 pages) The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about| development issues. An objective of the series is to get the findings out quickly, euen if the presentations are less than fully polished. The papers carry the names of the authors and should be used and cited accordingly. The findings, interpretations, and conclusions are the authors' own and should not be attribuited to the World Bank, its Executive Board of Directors, or any of its mnember countries. Produced by the Policy Research Dissemination Center REVIVING PROJECT APPRAISAL AT THE WORLD BANK SHANTAYANAN DEVARAJAN LYN SQUIRE SETHAPUT SUTHIWART-NARUEPUT Policy Research Department The World Bank The authors wish to thank Mark Baird, Pedro Belli, Jeffrey Hammer, and Anandarup Ray for their insightful comments on an earlier draft. REVIVING PROJECT APPRAISAL AT THE WORLD BANK "We have found that the extent to which Isocial cost-benefit analysis is] used and [lhasi real influence is not great, even in the World Bank. " Little and Mirrlees, 1991 Given this assessment of current practice by Little and Mirrlees, what steps, if any, should the World Bank take to put its appraisal of projects on a firmer analytical footing? In addressing this question we steer clear of the finer points of project appraisal -- how to measure externalities, how to estimate shadow prices -- and focus instead on two broad questions: 0 What is the proper role for project evaluation in today's world where countries have reduced major economic distortions and are reconsidering the role of the state? We argue that this change in circumstances calls for a shift in the emphasis of project evaluation away from a concern with the precision of rate-of-return calculations to a broader examination of the rationale for and merit of public-sector provision. The second question also moves away from a focus on rate-of-return calculations: * Besides project evaluation, how else can economic analysis be used to ensure high-quality projects? Here we suggest that pre-appraisal, sectoral analysis and especially reviews of sectoral public expenditure programns can contribute substantially to the quality of projects approved by the World Bank. We address these two questions in Sections II and III of the paper respectively. We begin in Section I with an assessment of the conclusion drawn by Little 2 and Mirriees regarding the current state of project evaluation in the World Bank. Section IV offers some concluding thoughts. I. THE CURRENT STATE OF PROJEcT EVALUATION AT THE WORLD BANK Before turning to a more qualitative assessment, we examine the proportion of projets ,subiect to evaluation and show how this has changed over time. For this we need to categorize projects according to their year of approval, that is, when the evaluation, if any, was done. Unfortunately, information on whether a project appraisal includes a rate- of-return calculation or not only becomes available on a systematic basis when the project is reviewed by the Bank's Operations EvalU!tion Department at the time of completion (that is, when the project loan is fully disbursed). Thus, the number of projects for which we have information falls off dramatically after 1985 -- from 114 in 1985 to only 5 by 1989. We therefore confune the analysis to the 15-year period from 1970-85. For obvious reasons, we also confine the analysis to investment projects -- that is, we exclude all Structural and Sectoral Adjustment Operations, and all Technical Assistance Operations. Figure 1 indicates a significant drop in the percentage of World Bank investment projects subject to the scrutiny of a rate-of-return calculation -- from almost 70 percent in 1970 to only slightly more than 40 percent in 1985. Most of the decline occurred in the 1980s. Taking 1980 as the swing year, the average share of projects subject to economic analysis fell from almost 70 percent in the preceding five years to about 50 percent in the subsequent five years. The trend is confirmed by the fact that in 1991, 51 % of investment projects had rate of return calculations (World Bank, 1991). 3 Figure 1:Projectswlth IRR. 1970-85 Percent of Total Number of Inven t ment Projects 80% 250 70% 200 60% 50% 150s Pmrcenta 40% - u~*n*. of 30%t 20% 10% 5 0% 0 At least thiree developments could explain this phenomenon: Sectoral Shifts. Traditionally, the World Bank has not calculated rates of return for projects in the social or financial sectors. An increase in the number of projects in these sectors could therefore explain the declinling use of project analysis. However, the data do not support this view (see Figure 2). Sectors where rates of return are usually calculated such as agriculture, energy, industry, telecommunications, and tranLsport maintained (and in fact increased) their share in total projects approved in that period. What has happened is a within-sector decline in the use of economic analysis especially in agriculture, energy, industry, and transport (see Figure 3). Post- 1985, sectoal shifts may have become more important as more of our 4 Figure 2:Sectcat compowon ot Projcs Percent of Projects In Sectors Wlth v. WdlSou IR 1 00% 70% 00% ID) I 40%~ 30% 20% 0% ig7 iķ Y ae lending has been allocated to the social sectors and the environment -- almost 20 percent of lending in 1994 compared with less than 8 percent in 1985. * Quality Control. Prior to 1982 the World Bank had a fairly powerful Central Projects Department that set standards for project appraisal and reviewed the quality of appraisal reports. After 1982 the function of quality control was devolved to regional vice presidencies which were themselves the producers of the appraisal reports. The incentives implicit in this organizational structure were unlikely to encourage the use of rigorous appraisal techniques. At focus groups on project evaluation, staff identified several reasons for the poor quality of economic analysis, 5 Figure 3: AWrcultLra. Enrgy, Trnn qn and War 3 onawon Sector Percont of Projects wlU IRR 100% 40%} including (1) pressure to deliver a given amount of lending within strict deadlines, and (2) the lack of anyone to lobby for the importance of economic analysis (Thieme, 1995). * Other Demands. The decade of the 1980s saw the emergence of structural adjustment operations as an important and high-visibility activity for the World Bank. Between 1980-82 and 1986-88 the share of these operations in total lending increased from 7 to 24 percent. As a result, some of the Bank's best economists became absorbed in macroeconomic matters and 6 attention to the details of investment projects declined. Over time, the number of recruits trained in project analysis also declined. While obviously judgmental, it appears from the above that shifts in the sectoral composition of investment lending have contributed little to the declining use of project analysis and that the answer probably lies more in changes in the degree of central quality control and the emergence of other demands on the time of economists. Interestingly, these new demands were aimed at removing the very distortions that shadow prices were intended to correct for. Furthermore, as we show later, a less distortionary policy environment is associated with an increased probability of project success, perhaps reducing the need for careful cost-benefit analysis. Little and Mirrlees appear to be correct therefore in their view that the extent to which project analysis is used at the World Bank "is not great." But what about the quality of the analysis when it is used? Judgements on this point are difficult to make precisely because there is no mechanism of central quality control that could be expected to monitor performance on a consistent basis. Nevertheless, two comments can be made, one of which is specific to the approach advocated by Little and Mirrlees while the other is more general. The specific point concerns the use of distributional weights and the marginal cost of public funds. The evidence is clear: other than in a few experimental exercises distributional weights and the marginal cost of public funds have not been part of project analysis in the World Bank. Before asking why, it is important to clarify a key difference between distributional weights and the marginal cost of public funds. The need for distributional weights arises because of a concern with distribution whereas the need for 7 the marginal cost of funds arises because of a concern with the efficiency of resource allocation. To elaborate, imagine that in making its decisions about public investments the government decides to treat a marginal dollar accruing to each and every member of society as being equally valuable. This by itself is sufficient justification for the analyst to ignore distribution weights. Marginal benefits accruing to different members of society can be aggregated because they all have the same value. This decision notwithstanding, one might still want to incorporate the marginal cost of public funds ;hrough a premium on public relative to private income. Assume for example that at the margin public income is generated through a distortionary importTariff. Any project that makes a claim on public income therefore elicits a further increase in import tariffs. The increased tariffs will almost certainly impose a cost through its impact on the allocation of resources, which ought to be captured in the project appraisal. Despite the conceptual difference between the marginal cost of funds and distributional weights, the former was widely seen as just another element of the system of distribution weights and arguments for dropping distributional weights came to apply to the marginal cost of funds as well. Why did the World Bank drop distributional weights (and by association the marginal cost of funds)? At least three reasons can be advanced: * Subjectivity. The use of distributional weights requires a judgement about the value of marginal increases in income to different members of society which staff were extremely reluctant to make, even though they recognized that the use of equal weights for everybody was itself a value judgement. * Practicality. Many staff held the view that the actual task of identifying who actually benefitted from a project was overwhelming. 8 * Efficiencv. A more compelling argument took the view that, while govemrnment should clearly be concerned about the distribution of income, project selection was not the most efficient instrument for pursuing this goal. This argument would claim that it is much more important for example to ensure that the broad allocation of public expenditure to different spending categories is more or less right than to worry about who benefits from every single project. These arguments together with the disappearance of central quality control and the emergence of new demands on the time oteconomists were sufficient to ensure that distributional weights were never seriously used in the World Bank. And as noted above the marginal cost of funds suffered a similar fate even though none of the arguments applied directly to the marginal cost of funds, which has a different conceptual rationale. We will return to this point later. With distributional weights and the marginal cost of funds abandoned. we might characterize what remains of project evaluation at the World Bank. therefore, as the border price rule, discounting, and sporadic use of a standard conversion factor. sectoral conversion factors being rarely if ever used. Given this more limited scope, how good is the Bank's project analysis? A general assessment of quality is possible thanks to a special review that was undertaken of the 92 investment operations for which rates of return were calculated and that were approved by the World Bank's Board in 1991. The review tried to assess projects according to whether the underlying economic analysis was consistent with the calculations embodied in the rate of return. While obviously subjective, this approach rated the quality of project analysis as "good or better" for 55 9 percent of the projects, as "marginal or acceptable" for a further 29 percent, and as "poor" for the remaining 15 percent (World Bank, 1991). That the ratings had some content is suggested by a follow-up study four years later which looked at the correlation between the rankings described above and the actual performance of the projects (World Bank, 1995). During implementation, supervision reports routinely provide a judgement on the likelihood that a project will be a failure. The probability that a project would be considered a failure dropped significantly as the quality of economic analysis at appraisal rose -- if the project analysis was rated good or better the probability of failure was judgedT0 be less than 5 percent whereas it rose to more than 30 percent if the analysis was rated poor. Although this exercise is subject to criticisms of omitted variable bias (e.g., the quality of analysis could indicate a competent project manager), it is the first attempt that we know of to provide some assessment of the value of good project appraisal. The same study also updated the 1991 assessment of quality with the following results. Of the 112 projects approved in calendar year 1993 and subject to cost-benefit analysis, 20 percent were rated good or better, 42 percent were rated average or acceptable, 25 percent were rated barely acceptable or marginal, and 13 percent were rated as poor, an outcome not unlike that of the 1991 study. Given the previous tinding about the percentage of projects subject to rate-of-return analysis, and the current finding that little more than half of the appraisals were considered good or better, we conclude that only around 25 percent of projects are approved on the basis of satisfactory rate of return analysis. 10 II. PRoJEcT EVALUATION FOR THE 1990S Given the disappointing assessment of current practice, how can we improve matters? Two steps seem worth pursuing -- a more rigorous assessment of the underlying rationale for public sector intervention, and a reconsideration of the role of the marginal cost of funds. Both arise from the changed circumstances in which project evaluation is to be practiced. We first describe these changes and then move to the specifics of our two proposals. Changed Circumstances. When Little and Mirrlees wrote their book governments were expanding public investment rapidly and7huch of that investment was in industry and related sectors. Indeed the first version of their book published in 1969 was titled Manual of Industnal Project Analysis. At that time major distortions arising from trade policy and exchange rate policy were characteristic of most developing economies (see for example Little, Scitovsky, and Scott, 1970). It is not surprising therefore that an important focus for Little and Mirrlees was on techniques -- especially the use of border prices and conversion factors -- that addressed these concerns. Moving into the second half of the 1990s, these concerns remain but in a much reduced form. Many countries are now grappling with the privatization of public enterprises rather than with new investment, and reforms of trade policy and exchange rate systems have reduced the distortions of most concern to Little and Mirrlees. In these circumstances, it may well be that the modest attention paid by the World Bank to sectoral conversion factors is a sensible allocation of the time of economists. On the other hand, more attention should be paid to the basic question of whether a project ought to be in the public sector or not. As we have noted, the composition of the World Bank's portfolio has 11 shifted to more emphasis on basic infrastructure, social services, and environmental projects for which the rationale for public-sector provision is strong. But, there are still many projects entering the portfolio that could well be shifted to the private sector. At the same time, the importance of macroeconomic stability is now widely accepted as a prerequisite for sustained development. And a prudent fiscal policy is central to macroeconomic stability. It follows that projects that place a burden on the budget have to be reviewed with care. Fiscal balance will require that the budgetary costs of these projects be recouped through some other tax instrument, which in turn will introduce distortionary costs somewhere in the econorn9. These costs ought to be included in the evaluation of the project which was of course precisely the original rationale for the marginal cost of public funds. Moreover, as the composition of projects shifts in favor of those with a clear public sector rationale, we can expect that an increasing proportion of projects will be characterized by public costs and private benefits (see below). In these circumstances, the need for a careful assessment of budgetary impact and. in turn, of the distortionary cost of alternative sources of revenue will become greater. Public-Private Choice. These changes -- greater scrutiny of what types of activities ought to be undertaken by the public sector and the emergence of a more rational exchange rate and tariff structure -- render the few aspects of the Little-Mirrlees approach that are practiced at the World Bank less important. Indeed, instead of a concern with the calculation of the rate of return for a public-sector project, interest is now focussed on whether a project ought to be in the public sector at all. However, the principles underlying Little-Mirrlees, not to mention other manuals on project evaluation (UNIDO [19721, Squire and van der Tak (19751), can be applied to the changed circumstances and 12 new set of questions about the appropriate role of government. In particular, all the approaches to cost-benefit analysis require the project evaluator to specify the counterfactual: what would the world have looked like in the absence of the project? This principle is relevant whenever there is a private sector alternative to public provision. To illustrate, assume that the government is contemplating an investment which produces private goods -- a shoe factory, say. One possible judgement regarding the counterfactual is that, in the absence of the public sector project, nothing would have happened. In this case, the analyst should focus on a comparison of the costs incurred by the project and the benefits it is expected(to yield. The use of border prices and conversion factors would obviously be relevant. Assuming that the evaluation points to a positive net present value (NPV), the project would be accepted. But the appropriate counterfactual might be that the private sector would have produced the shoes anyway (assuming the enterprise is profitable). In this case, the relevant magnitude is the net contribution of the public shoe factory. The NPV of the public sector project over and above that of the private sector project (evaluated at shadow prices) may well be zero.' While the shoe factory case may seem obvious, there are many World Bank projects where a private sector alternative appears feasible and yet no private-sector counterfactual was specified. A quick survey of the titles of projects approved by the World Bank's Board of Directors indicates that only 13 percent could be said to have an obvious public sector rationale while many others seemed to have a strong private-sector characteristic (see Table I for examples). Of course, the titles of projects are not an adequate basis for arriving at a definitive judgement, but it can reasonably be concluded that there is a need to consider the private-sector counterfactual more systematically. 13 Furthermore, the 1995 report notes that "In many cases, it was not possible to discern the with and without project situation on the basis of the information provided" in the appraisal report quite apart from whether a private sector alternative was considered (World Bank, 1995, p. 15). Table 1: Sample of IBRD and IDA Projects with Clear v Unclear Public Component: 1994 Org. Country Project Title Amount Projects with Clear Public Component SM IBRD Algeria Emergency Desert Locust Control Project 30.0 Brazil AIDS and STD Control 160.0 China Shanghai Environment 160.0 Indonesia Dame Safety Project 55.0 Jamaica Tax Administration Reform 13.2 Mexico Second Primary Education 412.0 Morocco Environrnental Management 6.0 Peru Basic Health and Nutrition 34.0 IDA Chad Health and Safe Motherhood 18.5 Gambia Capacity Building for Environmental Management 2.6 India Maharashtra Erergency Earthquake Rehab 246.0 Uganda Sexually Transmnitted Infections Project 50.0 Projects with Unclear Public Component IBRD Bulgaria Water Companies Restructuring and Modernization 98.0 China Telecomnunications Project 250.0 India Container Transport Logistics Project 94.0 Korea Financial Intermediation Project 100.0 Malaysia Second Rubber Industry Smaliholders Dev. Auth. 70.0 Papua New Guinea Petroleum Exploration and Development Tech. Asst. 11.0 Philippines Leyte-Luzon Geothermal Project 113.0 Romania Industrial Development Project 175.0 Tunisia Private Investment Credit Project 50.0 IDA Madagascar Petroleum Sector Reform Project 51.9 Mozambique Gas Engineering Project 30.0 Uganda Cotton Subsector Development Project 14.0 Total 1994 IBRD and IDA Loans and Credits 20,836.0 IBRD and IDA Projects with Clear Public Component 2,747.3 Percentage of Total Loans and Credits with Clear Public Component 13.2% To be sure, specifying the private-sector counterfactual is not always straightforward. Yet, some principles can be invoked. First, if the project is producing a private good and is profitable at market prices so that there is good reason to believe the 14 private sector will undertake it, then the difference between the factual and counterfactual should be zero. In this case there is no advantage to having the project in the public sector and the point can probably be established without a serious evaluation.2 Second, at the other extreme we have the case of pure public goods. Here there is no prospect of private provision and hence no need to worry about a private-sector counterfactual. But pure public goods -- defense, for example -- are relatively rare. And, third we have the case of market failure arising from externalities, indivisibilities, information failures, etc. Here there may be a case for government intervention, but the private-sector counterfactual will not necessarily be zero. Since this is the case which is likely to be most relevant to World Bank projects, we consider it in more detail. When we have a market failure arising from say a positive externality, the private sector will provide some of the good, but not the socially optimal amount. In Figure 4, which is drawn for non-tradeables, the positive externality is captured by the marginal social benefit (MSB) curve lying above the marginal private benefit (MPB) curve. An example would be secondary education, which many would claim has significant positive externalities, although private secondary schools exist alongside public ones in many countries. In Figure 4, the private sector would provide up to q* of the good on its own. The socially optimal amount, however, is q**. (In the shoe factory example referred to earlier, the MSB and MPB curves coincide, so that q* = q**.) In evaluating projects in sectors such as these, the analyst needs to establish two points. First, that public provision results in a greater supply of the good than would have occunred with just private sector provision (that is, overall supply should exceed q*). If this were not the case, public provision would simply be crowding out private providers. 15 Figure 4: Public Provision and the Prvate Sector Counterfactual MC' MSB P* ~ ~ ~ ~ ] , , , ~~~MPB q q' q* The private-sector counterfactual would be equal to the project (assuming costs are the sarme) and the NPV of the public sector project would be zero. Second, we should ensure that the project does not result in total supply of the good beyond q** (that is, beyond the point where marginal costs exceed marginal social benefits). Thus, the relevant range for public provision in this sector is between q* and q**. We deal with the former (q*) here because it bears on the issue of the private sector counter factual and confront the latter (q**) in our discussion of the marginal cost of funds. Determining q*, an unobserved counterfactual, is difficult but it may be feasible in many cases to arrive at an estimate using information on willingness-to-pay. Appraisal of the Leyte-Luzon Geothermal Project in the Philippines, a US$ 1.3 billion project, illustrates the idea. The economic analysis of this project includes an estimate of consumer willingiess-to-pay based on current bulk energy tariffs in the Luzon system. In turn, this 16 allows an assessment of what the private sector would have provided and determines the minimum level of public sector supply. And as we shall see next it also has important implications for tariff policy.' Revisiting the Marginal Cost of Funds. The marginal cost of funds highlights the need to ensure that public sector projects attempt to recover as much of their costs as possible from the private sector (i.e., that an appropriate pricing strategy accompany the project's implementation). In Figure 5, we draw again on the Geothermal Project in the Philippines to show how a project's internal rate of return (IRR) changes with different pricing policies and different levels of the mltginal cost of funds. We calculate the IRR of the project for different values of the marginal cost of funds and different pricing policies. If the electricity is sold at its market value and/or there is no premium on public income (i.e., the marginal cost of funds is one), the IRR for the project is 10.5 percent. But any other combination of pricing policy and premium would result in a substantially lower IRR. In the extreme case, when the electricity is sold at 40 percent of the market price, and the premium is 100 percent (i.e., the marginal cost of funds is two), the IRR drops to 4.3 percent, or by 60 percent. In value terms (at a 10% discount rate), the project's NPV drops from over $29 million to -$645 million. In the case of this power project it is reasonable to expect that full cost recovery is the appropriate goal, the rationale for public provision presumably being the size and indivisible nature of the project which make private provision difficult. Thus, the premium may not be especially critical except as a reminder that less than full cost recovery imposes distortionary costs elsewhere in the economy. But, for many of those projects with a reasonable case for public intervention -- basic infrastructure, primary 17 Fige 5: Leyte-Lunn Geothwmxi Pmject IRR by MagXil Cost of Fux and Tarif (Base Case IRR =10.5%) 120DO20 18 2 0 _ ~1. '6 12nfvPce 10 N'brVmi Cof of rnk education, rural health, and so on -- full cost recovery is not likely to be a desirable goal. Thus, these projects will share a common characteristic: costs are borne by the public sector, benefits are enjoyed by the private sector. If there is a premium on public income (which is almost always the case, since governmnents resort to distortionary taxation for revenue), then the net present value of these projects will be lower than otherwise calculated. Put another way, since the World Bankc has not used a premium (that is public costs and private benefits are treated equally), it has systematically overestimated the NPV of these projects (see Squire, 1989). 18 Consider again Figure 4. To the extent that there is a premium, the fact that the government suffers a fiscal cost (and hence a real cost implied by the need to generate revenue through some other means) causes the overall cost of the project to increase, shifting the MC curve upward to MC'. The optimal supply will now be at a level below q**, call it q**'. What is the relationship between this optimal supply level and the decision to accept or reject the project'? We would reject projects in the interval (q**',q**) even if they recovered costs up to the private sector's willingness to pay. Of course, the analyst will never know the exact value of the externality. It is, however, possible to estimate the cost assdcSated with raising revenue from the least- distortionary tax instrument. Estimates for developed countries range from 32 to 47 percent in the U.S. to 120 percent in Sweden. Browning [1987] discusses four key factors that interact to determine the marginal welfare cost and account for the wide variation in estimates. These include the elasticity of labor supply; the marginal tax rate; how (balanced-budget) government spending affects actual labor earnings;' and the progressivity of the change in the tax structure that produces the incremental tax revenue.' In developing countries, the marginal cost of funds is likely to be even higher to the extent that these countries have access to a more limited set of tax instruments (e.g., trade taxes in low-income countries) which are highly distorting. On the other hand, if as some believe, substitution elasticities are lower in developing countries, the deadweight loss from taxation will also be lower. Even in the absence of country-specific estimates, we suggest that it is still worth showing the net impact of the project on the government's budget and conducting sensitivity analysis for a plausible range of the marginal cost of funds. 19 III. IMPROVING PROJECT QUALITY INDIRECTLY Focussing on rates of return or the immediate economic analysis of projects is one means and presumably an important one of improving the quality of projects. But other approaches may be as or more important. HIere we examine two possibilities. The first is the positive counterpoint to the earlier observation that a concern with macroeconomic and structural reforms diluted the time and effort devoted to project evaluation. The consequent improvement in policy may have affected project performance positively. And the second emphasizes the importance of sector studies, the existence of which may improve the quality of future projects in tharsector. In the following we explore both possibilities, offering some empirical evidence in each case. The Policy Environment. To investigate the relationship between rates of return and the policy environment, Kaufmann (1991) looked at 1,200 World Bank projects in 58 developing countries. He linked the reestimated rate of return with various indices of policy-related distortions. The broad pattem is shown in Table 2: the lower the distortion, the higher the project's rate of return. This pattern is confirmed by his econometric estimation (which further controlled for country-specific factors, external shocks, etc.). For most of the policy distortions, a large change in the index is associated with a 3 to 7 percentage point difference in the rate of return. Furthermore, he showed that countries which moved from high to low distortions significantly improved the rate of return to their projects. 20 Table 2: Economic Policies and the Performance of Projects Percent Unsatisfactory Policy Distortion Average ERR (Social Sector Projects) Trade Restrictions Highly restrictive 13.20 28.10 Nonrestrictive 19.00 0.00 Exchange Rate Highly overvalued 8.20 37.00 Slightly overvalued 17.70 17.40 Real Interest Rate Negative 15.00 29.40 Positive 17.30 17.50 Fiscal Deficit High 13.40 29.30 Low 17.80 11.30 Price Distortions of Tradeables High distonions 15.80 25.70 Low distortions 17.40 14.80 Source: Kaufmann (1991); Kaufmann and Wang (199S). These results are both troubling and reassuring. They are troubling insofar as project evaluation techniques are in principle supposed to incorporate existing policy distortions. Had the shadow prices for these projects been properly calculated, and appropriate project selection criteria followed, there should be no relationship between policies and rates of return. However, as we have seen, these techniques were used only on a limited basis. What these results indicate is that the "costs" of not calculating the proper shadow prices are greater the more distorted is the economy. As such this confirns the appropriateness of the original Little and Mirrlees focus on key distortions and, by the same token, our suggestions to reconsider the emphasis in a changed world. At the same time, the Kaufmann results are reassuring in that the World Bank's increased emphasis on 21 policy reform (possibly at the expense of better project analysis) may have yielded an unintended benefit: better returns on projects. Kaufmann and Wang (1995) undertook a similar analysis for social-sector projects, where instead of rates of return (which are not usually calculated for projects in these sectors), they use an ex post measure of the project's "success". Again, they find that the severity of the various distortions has a significant effect on project success with the percentage of unsatisfactory projects typically doubling as one moves from a low-distortion to a high distortion environment (see the second column in Table 2). Sectoral Anathsis. The World Bank rdctinely carries out economic and sector work (ESW) on its client countries. Some recent econometric work (Schneider, 1995) shows that the number of ESW reports on a country or sector contributes significantly to project success in that country/sector., The percentage of successful projects (as determined by the Bank's Operations Evaluation Department at project completion) rises from 69 to 100 when the number of ESW reports increases from zero to four or more (see Figure 6). Schneider's econometric results confirm that this relationship is statistically significant. Controlling for other factors, he concludes that one ESW report in a particular country and sector increases the probability of a project's success by nine percentage points, while two reports would increase it by sixteen points. 22 Figue 6: E aoneumc anI Sector Wuik (ESV au PIject Surcess Pew= of Suxessfui Prvjcts* _ ~~~~100 77%75 0 1 2 3 4orw. or f EW Repot in CauhySegrinu 3 ym PfcIt Peda e For 431 F.jewaiWpm PCR raqs ov i 9 94 Som: Sdno*r. sw Sia Wu*andRPAsoeu OrV 'Tabk I. Included in the Bank's ESW are reviews of public expenditures in particular sectors or the country as a whole. The reviews can be. and in some cases have been, used to inform project selection in the country or sector. These sectoral reviews are important because project-specific appraisal reports can (at best) only assess the rate of return or acceptability of the project being appraised. This is problematical for two reasons both of which are especially important for multilateral lending agencies. First, the project being appraised may have been undertaken without external linancing from a multilateral agency. In this case, the external funding is actually financing some ui.hzr project which without the financing would not have been in the investment program. This problem arises because financial resources are fungible at least to some extent. It is unlikely that the 23 projects evaluated by the Bank are those marginal projects which would otherwise have not occurred. For the 99 projects it evaluated in 1993, the Operations Evaluation Department found an average economic rate of return of 21% (World Bank, 1994).This average rate of return appears too high to indicate marginal projects. And second, even if the project would not have been in the investment program without external funding, there is no guarantee that it is the best of all those projects not currently in the program because of lack of financing. Yet the latter is the relevant question if the World Bank is to maximize the development impact of its lending. The above argument calls for sector-wlde reviews prior to project-specific appraisal and financing and perhaps explains an irnportant part of the observed link between sectoral analysis and subsequent project success. To illustrate consider two examples of sectoral public expenditure reviews (PERs). The Agricultural Expenditure Review of India (Pradhan and Pillai-Essex, 1993) examined several agricultural programs as projects and calculated their economic rate of return. It concluded that two programs -- a fertilizer subsidy and a crop production scheme -- had a zero rate of return because there was no justification for public provision of these goods (the benefits accrued directly to the individual farmers). Yet, the bulk of central-government expenditures were going to these two schemes. The sLudy also pointed to the high rates of return in groundwater irrigation and extension services, recommending a reorientation of public expenditures in that direction. Similarly, an Infrastructure Public Expenditure Review of Peru (Humplick and Paterson, 1994) calculated the economic rate of return of expanding each of the major road links in Peru. The results showed a large variation in the IRR. Using a cutoff rate of 12 24 percent, the report was able to recommend dropping several road expansion projects, leading to a $275 million reduction in the government's road investment program. These reviews actually go beyond setting a good foundation for subsequent project appraisal -- they also improve the overall quality of the sectoral investment program. Moreover, if the result is a satisfactory program, then the specific project financed and appraised in depth by the World Bank becomes le.i important. Rather, the Bank could have the greatest development impact by associating liself withl that project where its knowledge and technical expertise is likely to be of most value. Alternatively, the World Bank could finance a time-slice of an agreed stctoral expenditure program, an option that is receiving greater attention in the Bank. Furthermore, sectoral expenditure reviews can shed light on the two elements which we have identified as crucial for project evaluation in the 1990s, namely, the private-sector counterfactual and the marginal cost of funds. In reviewinc the government's expenditure program within a sector, a sectoral expenditure review should identify areas where the private sector is providing, or can provide, the goods and scrvices in question. For instance, a review of public expenditures in the health sector of Malaysia (World Bank, 1992) noted that 62 percent of spending by the Ministry of Health was on medical care (mostly private goods) and only 23 percent was on public heaith while the latter had a higher marginal impact on health status.7 While sectoral expenditure reviews would not calculate the premium on public income, PERs that cover the entire goverrunent budget could in principle do so, insofar as they evaluate the government's budget constraint as part of the macroeconomic framework. At the very least, PERs should 25 identify those countries where the gains from applying the marginal cost of funds in project evaluation -- or the losses from not doing so -- are the greatest. IV. CONCLUSION To some extent, the changed circumstances noted above vindicate the relatively limited attention the World Bank has paid to project appraisal. At the same time those changed circumstances have pointed to new priorities and raised old concerns that are not adequately addressed at present. In two areas we conclude that continuation of current practice with in one case some tightening on quality seems appropriate: * Distributional Weights. The view ttmt other instruments, especially the broad allocation of public expenditures and tax policy, are able to influence the distribution of income more efficiently than project selection, coupled with the practical difficulties of identifying income-specific project beneficiaries leads to the conclusion that it is not worth revisiting the issue of distributional weights. Shadow Prices. The use of border prices, a discount rate, and a standard conversion factor should be routine practice. Given the Kaufmann results, this point is especially important where economy-wide distortions remain significant. In addition, where the yes-no decision proves sensitive to the value of the standard conversion factor, further disaggregation -- sectoral conversion factors -- would be required. More importantly, we suggest on the basis of this review that there are three areas where the World Bank should think seriously about improving its analysis in the interests of both its development impact and the quality of its portfolio: 26 * Public versus Private. In as much as the World Bank is still financing projects that appear to be producing private goods, a greater effort should be made to assess the feasibility of private sector alternatives, including those that would prevail under regulatory or price (tax-cum-subsidy) policies. And even where there is a clear rationale for public provision, care should be taken to ensure that public provision results in a greater (that is, closer to optimal) supply than would be the case with just private provision. * Marginal Cost of Funds. Because the World Bank is likely to be financing an increasing number of projects that aM characterized by public costs and private benefits, the idea of incorporating the marginal cost of public funds in project appraisal should be reconsidered. Ideally, this should entail estimation of the marginal cost of funds at least in countries where the World Bank is investing in several projects or where it is likely to be especially large -- a task that could be most appropriately undertaken by the Research Department. Finally, for all projects, whether it is possible to calculate a rate of return or not, and whether a country-specific estimate of the marginal cost of funds is available or not, the analyst should measure the impact of the project on the budget and explain what forms of cost recovery have been considered. * Public Expenditure Reviews. Wherever possible, the World Bank should allocate resources to reviews of sectoral expenditure programs before embarking on the appraisal and financing of specific projects. Indeed, if a sectoral review has been undertaken, financing a time-slice of the program may be the most effective use of World Bank funds. 27 If history is any guide, proposals such as these are unlikely to make much headway without a change in the structure of internal incentives. At a minirnum, a reinstatement of a central projects unit to both provide assistance to project analysts and exert a degree of quality control would seem to be required. ENDNOTES 1. Furthermore, if there is any added distortion from the public sector's need to raise funds, the return to the public investment will be negative. 2. If the good is private but is not currently produced privately, then one would need to compare public provision with salc of project description (e.g., blueprints) to the private sector. 3. Alternatively, it may be argued that the current level of private-sector provision represents the profit-maximizing amount given the costs imposc by, inter alia, government-induced distortions. Then the appropriate comparison should be between the net benefits of the project and removing those distortions. 4. If the government provides a service that taxpayers would have otherwise purchased on their own, then the spending would be a perfect substitute for disposable income. By contrast. if marginal government spending provides no benefits to taxpayers, there is an income effect from the balanced budget spending operation that counters the substitution effect (Browning, p. 18). 5. The more progressive the tax change, the greater the marginal welfare cost. Raising additional revenue by increasing the rates of progressive taxes (e.g., federal income taxes) implies large increases in marginal tax rates (Browning, p. 19). 6. The study reviewed 431 projects for which Project Completion Reports assigned ratings during the period July 1992 - October 1994. 7. Regression analysis of infant mortality on various types of public medical expenditures showed that expenditures on safe water and immunization had a much higher impact than expenditures on public-provided doctors. 28 REFERENCES Browning, Edgar K. 1987. "On The Marginal Welfare Cost of Taxation." The American Economic Review 77(1): 11-23. Humplick Frannie, and William Paterson. 1994. 'Peru: Infrastructure Public Expenditure Review." World Bank, Latin America and the Caribbean Region, Washington, D.C. Little, I.M.D., Scitovsky, T. and Maurice Scott. 1970. Industry and Trade in Some Developing Countnes: A Comparative Study. Development Centre of Organization for Economic Cooperation and Development, Oxford: Oxford University Press. Little, I.M.D. and J.A. Mirrlees. 1991. "Project Appraisal and Planning Twenty Years On." In Proceedings of the World Bank Annual Conference on Development Economics 1990, World Bank, WasllThgton, D.C. Kaufmann, Daniel. 1991. "Determinants of the Productivity of Projects in Developing Countries: Evidence from 1,200 Projects." Background Paper for World Development Report 1991. Kaufmann, Daniel and Yan Wang. 1995. "Macroeconomic Policies and Project Performance in the Social Sectors: A Model of Human Capital Production and Evidence from LDCs." World Bank, Washington, D.C. Pradhan, Sanjay and Jeeva Peruma Pillai-Essex. 1993. "India: Public Expenditure Review Sector Report III: Agriculture." World Bank, South Asia Region, Washington, D.C. Schneider, Robert. 1995. "Economic Sector Work and Results on the Ground." World Bank, Washington, D.C. Squire, Lyn and Herman G. van der Tak. 1975. Economic Analysis of Projects. World Bank, Baltimore: Johns Hopkins University Press. Squire, Lyn. 1989. "Project Evaluation in Theory and Practice." In Chenery, Hollis and T.N. Srinivasan. eds.. Handbook of Development Fconomics Vol. II. New York: Elsevier Science Publishing Company. Thieme, Alfred. 1995. "Summary: Needs Assessment of Staff with Regard to Investment Project Evaluation." PMDTR, World Bank, Washington, D.C. UNIDO. 1972. Guidelinesfor Project Evaluation. United Nations, New York. 29 World Bank. 1991. "Economic Analysis of Projects: Towards a Results-Oriented Approach to Evaluation (ECON Report)." World Bank, Washington, D.C. World Bank. 1992. "Malaysia: Fiscal Reform for Stable Growth." Report No. 10120- MA, Asia Region. World Bank, Washington, D.C. World Bank. 1994. Annual Review of Evaluation Results 1993. Report No. 13794. Operations Evaluation Department, World Bank, Washington, D.C. World Bank. 1995. "A Review of the Quality of Economic Analysis in Staff Appraisal Reports for Projects Approved in 1993." Operations Evaluation Department, World Bank, Washington, D.C. Policy Research Working Paper Series Contact Title Author Date for paper WPS1478 Promoting Growth in Sri Lanka: Sadiq Ahmed June 1995 A. Bhalla Lessons from East Asia Priya Ranjan 82168 WPS1479 Is There a Commercial Case for Panayotis N. Varangis June 1995 J. Jacobson Tropical Timber Certification? Rachel Crossley 33710 Carlos A. Primo Braga WPS1480 Debt as a Control Device in Herbert L. Baer June 1995 G. Evans Transitional Economies: The Cheryl W. Gray 85783 Experiences of Hungary and Poland WPS1481 Corporate Control in Central Europe Peter Dittus June 1995 G. Evans and Russia: Should Banks Own Stephen Prowse 85783 Shares? WPS1482 A Measure of Stock Market Robert A. Korajczyk June 1995 P. Sintim-Aboagye Integration for Developed and 38526 Emerging Markets WPS1483 Costa Rican Pension System: Asli Demirguc-Kunt June 1995 P. Sintim-Aboagye Options for Reform Anita Schwarz 38526 WPS1484 The Uruguay Round and South Asia: Nader Majd July 1995 J. Ngaine An Overview of the Impact and 37947 Opportunities WPS1485 Aggregate Agricultural Supply Maurice Schiff July 1995 J. Ngaine Response in Developing Countries: Claudio E. Montenegro 37947 A Survey of Selected Issues WPS1486 The Emerging Legal Framework for Pham van Thuyet July 1995 G. Evans Private Sector Development in 85783 Viet Nam's Transitional Economy WPS1487 Decomposing Social Indicators Using Benu Bidani July 1995 P. Sader Distributional Data Martin Ravallion 33902 WPS1488 Estimating the World at Work Deon Filmer July 1995 M. Geller 31393 WPS1489 Educational Attainment in Developing Vinod Ahuja July 1995 M. Geller Countries: New Estimates and Deon Filmer 31393 Projections Disaggregated by Gender WPS1490 Trade Reform Design as a Signal to Eric Bond July 1995 A. Estache Foreign Investors: Lessons for Steve Chiu 81442 Economies in Transition Antonio Estache Policy Research Working Paper Series Contact Title Author Date for paper WPS1491 Equilibrium Incentives for Adopting Peter W. Kennedy August 1995 E. Schaper Cleaner Technology Under Emis:sions Benoit Laplante 33457 Pricing WPS1492 Trade Policies, Macroeconomic Sarath Rajapatirana August 1995 J. Troncoso Adjustment. and Manufactured 37826 Exports: The Latin American Experience WPS1493 Migration and the Skill Composition Ramon Lopez August 1995 J. Ngaine of the Labor Force: The Impact Maurice Schiff 37947 of Trade Liberalization in Developing Countries WPS1494 Adjustment and Poverly in Mexican Ram6n L6pez August 1995 J. Ngaine Agriculture: How Farmers' Wealth John Nash 37947 Affects Supply Response Julie Stanton WPS1495 Raising Household Energy Prices Caroline L. Freund August 1995 G. Langton in Poland: Who Gains? Who Loses9 Christine I. Wallich 38392 WPS1496 Reviving Project Appraisal at the Shantayanan Devarajan August 1995 C. Bernardo World Bank Lyn Squire 37699 Sethaput Suthiwart-Narueput WPS1497 Public Choices between Lifesaving Maureen L. Cropper August 1995 A. Maranon Programs: How Inportant are Lives Uma Subramanian 39074 Saved?