ICRR 12650 Report Number : ICRR12650 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/25/2007 PROJ ID : P057570 Appraisal Actual Project Name : Bhutan - Urban US$M ): Project Costs (US$M): 10.80 10.35 Development Project Country : Bhutan Loan/ US$M ): Loan /Credit (US$M): 10.80 10.35 Sector Board : UD US$M): Cofinancing (US$M ): Sector (s): General water sanitation and flood protection sector (57%) Other social services (14%) Roads and highways (14%) Sub-national government administration (11%) Central government administration (4%) Theme (s): Municipal governance and institution building (23% - P) Pollution management and environmental health (22% - P) Environmental policies and institutions (22% - P) Other urban development (22% - P) Decentralization (11% - S) L/C Number : C3310 Board Approval Date : 12/21/1999 Partners involved : Closing Date : 12/31/2005 06/30/2006 Evaluator : Panel Reviewer : Group Manager : Group : Roy Gilbert Ronald S. Parker Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: Overall: (i) to improve the quality of life through enhancement of the level and coverage of basic urban services in selected secondary towns of Bhutan Specific: (ii) to strengthen the financial and institutional capacity of the responsible authorities to deliver such services; and (iii) to finance high priority investments in such services for which there is broad public support . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): a) Institutional Development (appraisal cost: US$1.6 m./ actual cost: 1.7 m.), incl: (i) TA to strengthen the Urban Housing Development Division (UHDD) in the areas of project management and monitoring, operation and maintenance, and urban planning; (ii) TA and training for UHDD to develop urban development and environmental guidelines for solid waste management . b) Urban Development (appraisal cost US$9.2 m./ actual cost: US$8.6), incl: in all 10 towns (i) installation and upgrading of water supply and distribution systems; (ii) provision of solid waste collection and septic tank de -sludging equipment and modernization of landfills; and in 2-8 towns (iii) miscellaneous works such as footpaths, street lighting, access roads and storm water drains, market places, and rehabilitation of public toilets . - The project towns in Bhutan were : Bumthang, Duksum, Lhuentse, Paro, Rangjung, Tingtibi, Trashiyangtse, Trongsa, Wangdue and Zhemgang . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The IDA credit, that financed 100 percent of project costs, was fully disbursed . Even though it had been 92 percent disbursed by the original closing date, a six month extension was agreed to allow the completion of several key sub-projects that were not finished by that date . This was the Bank's first urban development operation in Bhutan . 3. Relevance of Objectives & Design: The overall objective was modestly relevant in reflecting the Borrower's policy of balanced regional development and decentralization, and environmentally sustainable development . The project's objectives were also consistent with the 1999 CAS that called for the Bank to support Bhutan's own preferences of focusing upon high priority social and physical infrastructure . The objectives remain relevant to the most recent 2005 CAS, and its emphasis upon better access to infrastructure and improved public sector management, especially through monitoring and evaluation. Relevance would have been even higher had the project objectives and design explicitly addressed poverty reduction, Borrower and CAS priorities throughout the time of this project . But being the first Bank-financed urban development operation in Bhutan, the design of the project was kept simple and flexible, while still focusing upon components that would help achieve the project objectives . 4. Achievement of Objectives (Efficacy): (i) modest achievement : The project improved the quality of life in ten towns throughout Bhutan, through improvements to water supply and other urban infrastructure in the ten participating towns . But service coverage fell far short of targets. Although a 2006 household survey revealed that 73-83 percent of respondents in three towns considered that the quality of the water supply had improved, the ICR correctly points out the weakness of this evidence, given that similar levels of satisfaction were reported by respondents living in towns not benefited by project water investments. But, for the first time in Bhutan, urban development projects were executed by local private contractors. Many sub-projects were on a scale that local governments called "dzongkhags" in Bhutan had never experienced before. Mechanisms of local community participation were strengthened, giving beneficiaries a voice in sub-project planning and implementation. (ii) modest achievement : Though small in scale, project TA reinforced by on -the-job learning helped strengthen the financial and institutional capacity of the responsible local dzongkhags to deliver urban water and solid waste services in particular. But much more still needs to be done in strengthening oversight and monitoring and evaluation systems, and well as building solid financial systems for cost recovery . (iii) modest achievement : The project was able to deliver a range of priority investments on the scale anticipated at appraisal. A beneficiary survey showed particularly strong public approval of improvements in roads and water supply services, but less so with footpaths and garbage collection services . 5. Efficiency (not applicable to DPLs): The improvement of actual performance over appraisal estimates is greater than the five percentage points increase in the ERR, but the coverage is less . This is because the ICR estimate was (correctly) more conservative in attributing tourism spending benefits to the project and in avoiding double counting of benefits that it (correctly) critiqued in the appraisal. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 20.7% 85% ICR estimate Yes 25.8% 46% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project did achieve its objectives, but reached fewer beneficiaries than intended, and significant capacity shortcomings remain. Also, the project lost some relevance for not explicitly focusing upon poverty reduction, an important government priority. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR provides a tentative assessment of risk, given many unknowns at the time of implementation completion . For instance, operations and maintenance systems have been put in place, but their effectiveness is still untested . Cost recovery mechanisms for water also remain untried . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: There were serious shortcomings in project preparation that overestimated local implementation capacity and had not made clear at start-up which sub-components were to be financed . The weak capacity of local government dzongkhags, to administer procurement, for instance, was not factored into the project design . Nor was the lack of monitoring capacity at the central level . The ICR reports that with too many widely spread and inaccessible towns participating, the project was not well designed for being difficult to supervise . But the more distant towns are likely to be the poorest and least served, precisely the ones that a poverty focused strategy should try to reach. Knowing this at the design stage, and rather than pulling out of such towns, the Bank could have done more to ensure the project had sufficient human and financial resources for adequate project oversight even in the most remote towns . Even given these constraints, Bank performance improved during supervision, even though Bhutan's difficult terrain proved challenging for the many supervision missions fielded . at-Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Local government dzongkhags often submitted ineligible claims for reimbursements, sometimes failing to obtain contractor invoices for this purpose . Periodic bulk submissions of delayed requests meant that there were insufficient funds in the Special Account to reimburse the claims made . But, through the project, there was improved interaction between central and local governments on the environmental screening of urban investments in Bhutan. Financial reporting during implementation was weak . It was made worse by frequent breakdowns--caused by lack of technical support in Bhutan --of the computerized financial management system that had been customized for the project . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: A lot of attention went into the design of M&E at appraisal, mostly in tracking the delivery of project components . Nine performance indicators, measuring the shares of populations with specific benefits, were formulated . They included, for example, the percent of households with access to safe water supply, the percent of solid waste collected and disposed of properly, as well as the length of access roads built and maintained . But baseline data at the project start up was not collected . Scant attention was paid to implementing M&E . Nor was data on the performance indicators collected, partly because there were too many . For that reason, the number of performance indicators was reduced at mid-term and explicit targets set for them . The ICR reports the results of just four indicators, with mixed results that, overall, point to the modest achievement of the objectives . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately Central government oversight and M&E Satisfactory for the project was weak. Quality of ICR : Exemplary NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: To intensify project engagement with local governments, urban development projects should deploy sufficient human and financial resources and adequate reporting mechanisms to ensure that they can bring their support even to remote towns--among the poorest and the most in need --where access may be difficult . Especially when implementation is decentralized, responsibilities at each level should be made very clear and their assignment should take into account capacity constraints at the local level especially . Even when implemented by local governments, urban development projects should aim to strengthen the monitoring and evaluation capabilities of central government institutions that have to guide and oversee decentralized units. The design of urban development projects should ensure that they are coherent with the prevailing cultural norms. In Bhutan, for instance, this meant relaxing a traditional grid layout for streets in new housing areas, that was anathema to many Bhutanese, even though a such a layout may lead to a efficient land use . 14. Assessment Recommended? Yes No Why? To take the evaluation forward using some ICR material as baseline data, something that was not possible for ICR to do, given the absence of baseline data for project start up . To provide inputs to future IEG work on municipal development. An assessment would also allow IEG to update the assessment of the risks to the development outcomes in an under -evaluated country. 15. Comments on Quality of ICR: This is a very good ICR indeed . It describes very well what was implemented under the project and provides good summaries, in tabular form, of what was done in each project town, what was spent and what was delivered by the project. It is frank about the project's shortcomings, as well as the data limitations that constrain the evaluation . Through its own assessment, it demonstrates how good methodological insights can come from self -evaluation. For instance, it recognizes that the significance of the positive beneficiary feedback about improved water supply in project cities is undermined by similar positive feedback coming from cities that had not been served by the project . It also clearly differentiates indicators and measures of inputs from indicators and measures of outcomes . In addition, it provides an informative critique of the risks of double counting and lack of attribution in assessing benefits for economic rate of return estimates . The ICR also includes a good analysis of the beneficiary survey results . Finally, its coverage is enriched by a well prepared borrower ICR that brings additional assessments of what was achieved at the level of particular towns. a.Quality of ICR Rating : Exemplary