ICRR 13079 Report Number : ICRR13079 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/16/2010 PROJ ID : P047762 Appraisal Actual Project Name : Rural Water Supply US$M ): Project Costs (US$M): 27.70 31.15 And Sanitation Project Country : Tanzania Loan /Credit (US$M Loan/ ): US$M): 26.00 30.09 Sector Board : WAT US$M ): Cofinancing (US$M): 0.70 0.90 Sector (s): Sanitation (40%) Water supply (40%) Sub-national government administration (10%) Health (5%) Central government administration (5%) Theme (s): Rural services and infrastructure (33% - P) Decentralization (17% - S) Participation and civic engagement (17% - S) HIV/AIDS (17% - S) Other financial and private sector development (16% - S) L/C Number : C3623 Board Approval Date : 03/26/2002 Partners involved : Beneficiaries Closing Date : 06/30/2006 06/30/2008 Evaluator : Panel Reviewer : Group Manager : Group : George T. K. Pitman Fernando Manibog IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: The PAD version states the objectives are : "to ensure access to improved and sustainable water and sanitation services in rural communities in Tanzania ." The DCA states this is "to assist the Borrower in providing improved and sustainable water and sanitation services in rural communities ." This review uses the DCA version because it is more realistic. Thus IEG recognizes two objectives : (a) to assist provision of improved water and sanitation services in rural communities; and (b) to assist provision of sustainable water and sanitation services in rural communities . This was to be accomplished through the implementation of the new Rural Water Supply and Sanitation (RWSS) sector policy and the preparation of a National RWSS program . There were 3 key indicators at appraisal : 1. Decentralized and demand-based district implementation model validated and operational in up to 12 districts; 2. Water supply and sanitation conditions improved in about 250 communities located in up to 12 Project districts; 3. National RWSS Program and Strategy endorsed by key stakeholders . The primary beneficiaries of the Project at appraisal were to be about 650,000 people living in about 250 rural communities and small towns of less than 15,000 in 12 districts. Women and children who typically spent long hours fetching water of doubtful quality would be the prime beneficiaries of the Project . At Mid-Term Review (MTR) both the number of rural communities and the initial number of target beneficiaries were scaled down to 133 and to 550,000 respectively. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components (or Key Conditions in the case of DPLs, as appropriate): 1. Establishment of the district implementation model . Estimated at US$ 1.20 million, actual cost US$ 2.01 million . This component was planned to start in the three districts of Rufiji (Coastal Region), Mpwapwa (Dodoma region), and Kilosa (Morogoro Region) and gradually expanding to about 12 out of Tanzania's 113 districts. Activities included: (a) support to the establishment and strengthening of operational capacity (equipment support and training) of District Water and Sanitation Teams (DWSTs) to prepare district RWSS plans and appraise RWSS sub-projects proposed by communities; (b) support to the establishment of district RWSS Funds (a conditional grant system) to finance the construction of new RWSS schemes, the rehabilitation and expansion of existing systems and possibly other water related small scale infrastructure investments, and (c) assistance to communities in the proper management and operation of these systems through appropriate hygiene education as well as promoting HIV/AIDS mitigation and prevention activities . 2. Construction of RWSS community sub -projects . Estimated at US$ 21. 21 .78 million . 21 .30 million, actual cost US$ 21. This was planned to give grants to about 250 communities (at MTR this was revised downward to 133) located in 12 districts to improve the WSS service; this would include the construction or the rehabilitation of existing dug or drilled wells, boreholes equipped with handpumps or powered pumps, spring tapping, piped systems, household latrines and community sanitation facilities . DWSTs would employ facilitation service providers (NGOs) for community facilitation and extension services and technical services providers (engineering consultants) for sub-project design and construction supervision . This component was designed to support the set up of a network of private retail outlets for the distribution of spare parts for handpumps and of certified installation and maintenance handpumps agents in the districts . It also included an "innovation window" for exploring community contracting where possible, or technologies and management arrangements untested in Tanzania. 3. Institutional strengthening and develop ment of a National RWSS Program . Estimated at US$ 3.55 million, actual cost US$ 3.40 million . This included: (a) support to stakeholder consultative process on the strategy for scaling up towards a National RWSS Program; (b) technical assistance to the preparation of the National RWSS Program; (c) development of a management information system (MIS) for tracking lessons learned from the proposed Project; (d) assistance to the restructuring of RWSS sector institutions within the Ministry of Water and Livestock Development (MWLD) and capacity-building of key stakeholders (MWLD, NGOs, consultants, contractors, suppliers of equipment and service providers ). It covered dedicated RWSS sector capacity building support to IDA-financed Tanzania Social Action Fund (TASAF) project districts; and (e) assistance to MWLD for overall management of the Project . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The Operations Portal records total disbursement as US$ 30.09 million. The ICR annex 1(b) shows US$28.5 million. Project closing was extended twice for a total of 2 years: the first for18-months to over come delays in community uptake and because the more complex mechanized piped water supplies required more time to build : the second for 6 months to complete project activities . At project closing US$0.86 million was cancelled. 3. Relevance of Objectives & Design: Overall relevance is rated substantial . Objectives . Relevance is rated high . The project was a key component of the government's 2002 National Water Policy that supported realization of the targets for WSS in Tanzania's Vision 2025 and the National Strategy for Growth and Poverty Reduction . The National Water Policy’s core principles (efficiency, subsidiarity and integration) endorsed decentralized delivery of water and sanitation services in accord with the Local Government Reform Program and the development of the private sector in the water and sanitation services . Improved water supply and sanitation, along with health and education, were the second priority of the Bank's 2000 CAS. The objectives were relevant to the Joint Assistance Strategy for Tanzania (JAST, 2006) that provides the national medium-term framework for managing development cooperation between the government and its development partners. JAST places a high priority on provision adequate water and sanitation services, recognizing that an additional 11.2 million people have to have access to clean and safe water to meet government targets by 2010. And this will have to be increased by 13.4 million to reach the targets of Vision 2025. Design . Relevance is rated substantial . The pure public sector delivery model for RWSS in Tanzania prevalent until the early-1990s failed to deliver, compounded by too many actors in the sector . The move from a 'free water policy" in the period 1967-1991 to one in which either the MWLD or communities operated schemes did not provide a solution for sustainability either . The model adopted by the project - a decentralized demand-driven approach building on lessons gained from the Bank's experience in other countries - replicated successful experience elsewhere and was relevant provided that institutional weaknesses were addressed . At the agency level the reform of MWLD to become the coordinating organization for sector investments, and for facilitating the involvement of the private sector and NGOs as service delivery educators and enhancers was appropriate . The formation of DWSTs under the responsibility of District Councils to prepare district WSS plans and appraise RWSS presented by the Communities that took the lead in selecting, designing and implementing RWSS replicated the Bank's experience elsewhere (via the Bank's RWSS Thematic Group) and Tanzanian NGO experience. However, design underestimated the time needed to build local capacity at all levels and the effect of poor governance. Similarly the intent to utilize the private sector and NGOs was laudable but overestimated available local capacity. The two-stage approach that started with a pilot in three Districts was highly relevant given these constraints - but too rushed. 4. Achievement of Objectives (Efficacy): Ex-post national-scale institutional outcomes and output data covers the whole project . Outcome data below national level relies on a ex-post beneficiary assessment at 26 randomly selected sample sites (out of 121 or equivalent to a 21% sample). At each site financial, technical and institutional achievements were obtained through semi-structured interviews of key -informants. Objective (a) provide improved water and sanitation services in rural communities . Efficacy is rated substantial when total beneficiaries are counted . Outcomes : Water supply : The total number of beneficiaries 521,630 was close to the revised target of 550,000 people. Increased water supply coverage was reported for 8 districts covering 243,385 people or 47% of all beneficiaries. Incremental coverage ranged from 3% in Kiteto district to 27% in Rufiji district. The population weighted average coverage increased from 44% to 56%. Data from the 3 remaining districts were not presented in the ICR. 121 communities (revised MTR target was 133) and 3 of the 4 targeted towns installed improved water and sanitation facilities. 70% of 17 communities surveyed stated that RWSS led to time savings and productivity increases . Time savings varied from 0.75 to 4.3 hours a day per household (ICR Annex 5 Table 5.6). Savings accrued primarily from shorter travel distances . A small sample (17 water points) in 5 districts found 14 or 82% of beneficiaries were satisfied with the taste, smell and color of water supplies . There are no data on per capita water consumption . It was expected to increase above the 10 liters/capita/day found at appraisal. Sanitation and hygiene : Increased sanitation coverage was reported in 7 districts covering 192,155 people or 37% of all beneficiaries. Incremental improvements ranged from 5% (Singida and Iramba districts) to 21% (Rufiji district). Overall the population weighted average coverage increased from 57% to 69%. Data from the 4 remaining districts were not presented in the ICR . The is no information or data on behavioral changes regarding hygiene or sanitation practices or the project's effect on the incidence of water -bourne diseases. Outputs : Facilitation Service Providers (FSPs comprising district and regional NGOs and technical service providers) were established and engaged to provide 12 months capacity-building and technical support to stimulate community demand and assist community planning for RWSS . In practice FSPs had much less than 12 months - sometimes only days - to work with communities. The 3 pilot districts were successful in the RWSS program although it took far longer to implement than planned . Expansion to the other 9 districts was problematic primarily due to local capacity constraints and poor governance (ICR Annex 5 sections 3 and 4) and these problems, allied with more expensive technical choices of water supply occasioned by the difficult hydrogeological conditions, caused the number of schemes to be reduced by 52% at MTR. Water supply . 2,457 water points were completed (target 2,497). This included 140 boreholes (of which 117 were fitted with motorized pumps); 154 shallow wells (141 were fitted with hand pumps); 187 water storage tanks (target 213); 734 km of pipes (target 816); and 72 cattle troughs (target 74). Disbursement pressure at the district level meant that some schemes started civil works before water was found. Mechanized boreholes that served 411,371 people (or 80% of all beneficiaries) had an overall 12% failure rate in finding sustainable water supplies . In one district failure rates due to poor hydrogeological conditions were as high as 4 out of 6 holes drilled and this lowered confidence and significantly pushed up costs . Sanitation . 147 communities received training and awareness -raising of better hygiene and sanitation behaviors; school and community sanitation clubs were established and 220 demonstration latrines were constructed. Household sanitation marketing . Over 540 artisans were trained in the manufacture and installation of hygienic sanitation slabs (SanPlats) and 30 SanPlat centers established . of the 2,829 slabs built, 1,615 (57%) were sold and 1,208 (43%) were installed. Private sector engagement . Two contractors were given 3-year contracts to establish a supply -chain for pumps and spare parts in 3 Business units in 3 districts. Zonal depots were established and certified technicians were trained and appointed . These units are only partially effective (ICR page 25, para 10). In addition 71 other non-government service providers were active (this included 4 consulting firms; 4 water utility contractors; 13 drilling companies; 4 NGOs offering facilitation services and many local small works contractors). In addition 10 consultants/service providers were engaged to implement 10 subprojects (compared with the appraisal target of 30) under the Innovation Window. Objective (b) provide sustainable water and sanitation services in rural communities . Efficacy is rated modest . Outcomes : Communities did not buy into the project at the level expected : Actual community contributions were 3% of capital costs vs the 5% mandated at appraisal. Main reasons for shortfalls were given as poverty, inadequate levels of institutional support to build local capacity (ICR Annex 5, para 3.2 -3.4), technical problems such as finding reliable water supplies, and poor governance. Communities adopted 2 water charging schemes for water supplies : flat-rate (32% of the ex-post sample) and "pay-as-you-fetch" (68% of the ex-post sample). But only 40% of communities maintain scheme accounts of revenue and expenditure data . Communities were not able to fully manage the whole project cycle and subsequent operations and maintenance - a critical risk identified at appraisal (PAD, page 11). Sustainability of water supplies : About 86% of randomly sampled water supply schemes were functioning (ICR page 36, para 4.1). About 10% of the 19 working schemes in the sample were capable of covering full operation and maintenance costs - and these were 2 pumped schemes managed by commercial operators . The appraisal target was that 100% of O&M would be financed by communities . In the overall sample 62% of water supply schemes covered operating costs . It was much better for the "pay-as-you-fetch" schemes (78%) that were mostly piped supplies than the flat -rate schemes (29%) that tended to be shallow wells. "Problems associated with poor financial management and embezzlement of funds were reported as prevalent in all districts where revenue was being collected, and frustration over such incidences seems to be an important cause of low revenue collection ." (ICR Annex 5 section 4.2). District service providers for maintenance did not always to meet demand or engage in a coordinated way (particularly in remote areas). In the best areas response time was measured in days; in the worst it was up to 3 months. A stakeholder workshop strongly endorsed the use of private operators but noted: "low private sector capacity in many places means that a gradual approach is likely to be necessary and should entail substantial supervision and support " (ICR page 43). Sustainability of sanitation is unknown . There is no outcome evidence presented in the ICR . Outputs : Institutional reform : This was substantially completed at the central level . The demand-based implementation model was adopted and established with varying success in all 12 districts. The model was adopted in the 2007 National Water Sector Development Strategy for RWSS . A National RWSS program was formulated and is operational in most districts . Service delivery . The MOW was restructured to allow decentralized service -delivery. DWSTs were created and are operational led by District Water Engineers (ICR Annex 1, section 2). While the ICR states they are operational in all 12 districts, data are presented for only 6 districts (ICR Annex Table 5.1). While the DWSTs are effective in some districts (e.g. Mpwapwa) the survey found that they need to expand their attention to building better local water institutions to ensure good governance because corruption is undermining financial sustainability . Since October 2007 the Ministry of Water (MOW) transferred earmarked sanitation funds to districts and 2 ministry staff look after community sanitation and hygiene . The ICR states this is insufficient and that hygiene and sanitation are not yet integrated into national planning and budgeting (ICR, page 23, para 5). Village water committees were formed but were of very uneven effectiveness (ICR Annex 5, para 3.4). Contracts for innovative approaches to RWSS generally failed to produce worthwhile innovations . Overall efficacy is rated modest . 5. Efficiency (not applicable to DPLs): Economic rates of return were calculated at appraisal and on completion . On the basis of 70% of time saving and additional water produced the ICR estimated the ERR to range between 16% for shallow wells with hand pumps; 18% for mechanized deep tubewells and piped distribution systems; and 49% for gravitational schemes. While the assumptions are stated, no data are presented to show how the ERR was derived . Despite this, investments were not very equitable. Unit investment costs ranged from a low of US$ 2.5 per capita served with hand pump schemes to a high of US$105 for the deep tubewell piped supply option . The TTL stated that highly variable unit costs reflected the difficult hydrogeological conditions, not extravagant equipment choice . Average per capita investment costs (US$ 40) were at the ceiling agree at appraisal (US$40/capita) ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal % 4.3% ICR estimate % 70% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: On the basis of substantial relevance, modest efficacy and substantial efficiency a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: There is insufficient funding to cover maintenance which will become critical given that communities preferred the technically more expensive and sophisticated option . As the ICR notes (page 38) "problems associated with poor financial management and embezzlement of funds were reported as prevalent in all districts where revenue was being collected, and frustration over such incidences seems to be an important cause of low revenue collection ." In addition it was noted in the ISR six months before project closure (December 2007): "Discussions with district leaders and communities during field visits as well as on past missions continue to confirm that communities whose subprojects are still under construction do not have adequate arrangements and structures for post -construction management. The focus continues to be on delivering infrastructure for which good systems exist but which virtually collapse after commissioning." a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Quality at entry . Project design was bold and innovative . It clearly incorporated many lessons from other countries and cultures that had shown promise in developing demand -responsive provision of RWSS. The two-stage design was sensible . At the highest level the policy and institutional reforms were in harmony with the sector norms. However, while appraisal of government and agency institutions was good, this does not seem to be the case for the district and community levels . Overall appraisal was too optimistic about the availability and development of local capacity to facilitate a demand -led approach and undertake timely water source identification, procurement and disbursement . Appraisal identified these risks as modest but they became substantial. Design paid inadequate attention to outcome indicators . Supervision . Slow disbursement and unexpected demand for the more expensive technical options required radical restructuring of targets at MTR and this effectively delayed the project by 2 years. In retrospect it may have been more sensible to have reduced the credit and roll -out of the demand-driven model more slowly to match the pace of capacity -building at district and community level . Government, however, was strongly against downsizing the project as they believed hardware delivery was of fundamental importance .While M&E activities proceeded slowly the supervision team has to be given full marks for its initiative to undertake a well -designed beneficiary survey to inform the ICR process . Supervision reports were very thorough and contained substantial supporting documentation. at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government . They fully bought into the policy and required sector reforms and these became effective . The government's commitment has secured the US$ 390 million multi-donor follow-on RWSS project. In practice, however, the desire to increase service coverage at the expense of local -level capacity building after MTR raises a question about government's commitment to a new model for service delivery . Implementing Agency . Although much was achieved, reorganization posed considerable challenges particularly at the district level and in coordinating and engaging with the private sector and NGOs on capacity -building and provision of essential O&M services . Because of these issues there was sometimes problems with effectiveness of service providers ("communities generally complained that the presence of Facilitation Service Providers was too brief (less than a day) to be effective in convincing people to contribute " ICR page 36), and project phasing (e.g. "some mentioned the unfortunate effects of having to start civil works before a source had been identified " ICR page 35). Counterpart funding transfer to districts was frequently delayed . Capacity-building to support RWSS and on governance remains incomplete as does attention to systemic M&E . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design . The PAD makes only passing reference to the requirements and design of an M&E system and a MIS and design was left to the Borrower's Project Management team in the field after project inception . Given that component 3(c) specifically states the project will develop "a management information system (MIS) for tracking lessons learned" this is a major shortcoming. Implementation . Templates for routine input and output technical and financial monitoring were developed and incorporated in the Project Operations Manual . The system was installed in head office and 9 of 12 districts. Data input was slow and this limited the utility of M&E . Resort was made to ad hoc measures that successfully bridge the gap to full computerization. Very little, if any, attention was given to monitoring outcomes and impacts except at the end of the project. As noted above (sections 4 and 8), the Bank's supervision team produced a well -design survey to evaluate the impact and outcomes of the project 8 months before project closing . Utilization . The MIS system produced partial data "thus limiting the full potential of the MIS in effective planning and M&E." (ICR, page 6). In contrast, the ICR beneficiary survey provides very valuable insight into project achievements and challenges. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards . Environmental safeguards (B) applied to scheme appraisal. An Environmental Assessment and Social Assessment was conducted for the 3 pilot districts. According to the DCA all sub-projects had to undergo environmental screening. On this basis an Environmental Management Plan was produced for the whole project and it included an EA checklist for screening proposed RWSS schemes . Supervision reports reported safeguard issues very well and thoroughly. Fiduciary . Although the routine government audits were unqualified, the Bank's Comptroller's Office issue a special opinion that there were several questionable or unsupported items to the total sum of about US$ 153,000 because of poor bookkeeping. This was still being resolved in March 2009. The ICR ( para 2.4.3) notes: " Project funds transferred to districts were largely accounted for ." Bank requirements are that project funds should be fully accounted for. 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Institutions and capacity -building to Satisfactory ensure sustainability did not mature as planned. Risk to Development Moderate Significant Weak local level institutions, poor Outcome : governance, corruption and embezzlement threaten financial sustainability. Bank Performance : Satisfactory Moderately Insufficient appraisal of local level Satisfactory capacity and buy-in and not enough attention to the time and resources required to build local capacity and institutions. Borrower Performance : Satisfactory Moderately According to the ICR harmonized Satisfactory guidelines if one of the ratings is moderately satisfactory, overall performance cannot be better then moderately satisfactory. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Demand -responsive projects require considerable attention during appraisal to beneficiaries and their institutions . This is to ensure that they are able to exercise their governance, organizational, management and financial responsibilities. Where there are shortfalls in capacity remedies should be addressed in project design. It is essential to allow sufficient time for capacity -building. Multiple service providers for design and technical implementation of RWSS projects may create coordination problems . It is better to have a single intermediary institution between government and communities that has responsibility for coordinating design and implementation . IEG notes in their assessment of the Nepal RWSS project that NGOs performed this role very satisfactorily (Report 44368 of 06/23/2008). 14. Assessment Recommended? Yes No Why? This project was innovative but ran into unforeseen problems in transferring the Bank's experience from other regions to Africa. A more detailed assessment may throw light on the problems and solutions and produce useful lessons. 15. Comments on Quality of ICR: A very candid and quite well-written account of the project . It could have been more objective about the appraisal problems. In contrast it was very open about the damaging impact of poor governance as revealed by the beneficiary survey. It is not clear what the status is of the 3 districts for which few data are presented . There are a number of minor errors. Annex 1 shows total project costs were US$ 30.02 million and financing was US$31.15 million; actual costs were US$30.09 million according to the operations dashboard and US$ 0.86 was cancelled. The PPF fee is missing. Table 3.2 shows monetary sums as US$, but they are TShs . No currency is shown for Table 5.2 and 5.5. Paragraph 3.3.2 is in TShs, not US$. It would have been helpful to have given more detail on the ERR calculation . a.Quality of ICR Rating : Satisfactory