46357 ExternalEvaluation of the Water and REPORT Sanitation Program for the period 1999 to 2003 External Evaluation of the Water and Sanitation Program for the period 1999 to 2003 Report submitted on June 2004 to theWater and Sanitation Program Council ITAD~Water 12 English Business Park, English Close, Hove, BN3 7EE, United Kingdom in association with The Water, Engineering and Development Centre (WEDC) Loughborough, United Kingdom CONTENTS GLOSSARY AND LIST OF ABBREVIATIONS 4 EXECUTIVE SUMMARY 8 WSP COUNCIL DECISIONS 13 1 INTRODUCTION 16 2 EVALUATION FRAMEWORK 17 3 THE PROGRAM FROM FY 1999 TO FY 2003 19 3.1 The Program's Objectives and Field of Work 3.2 WSP's Focus Countries 3.3 Main Activities in the Evaluation Period 3.4 Institutional Arrangements and Governance 3.5 Funding and Finance 3.6 Previous Evaluations 4 EVALUATION OF THE FIVEYEARS FY 1999 TO FY 2003 26 4.1 Introduction to this Chapter 4.2 Strategic and Planning Issues 4.2.1 Strategy and the Definition of the Program's Role 4.2.2 Focus and Balance 4.2.3 Planning, Monitoring, Internal Evaluation 4.3 Evaluation against Our Framework 4.3.1 Relevance 4.3.2 Effectiveness 4.3.3 Efficiency 4.3.4 Sustainability and Replicability 4.3.5 Outcomes and Impact of Program Interventions 4.4 Evaluation against other Aspects 4.4.1 Institutional and Governance Arrangements 4.4.2 Responsiveness to Opportunity 4.4.3 Finance and Resources 4.4.4 Relationships between Actors 4.4.5 Cross-Cutting Issues 4.4.6 Knowledge, Communication, and Publications 4.4.7 Staffing and Staff Skills 4.4.8 Languages and Sub-Regions 4.4.9 Response to Evaluations and Self-Evaluation 4.5 WSP's Comparative Advantage 5 FUTURE CONTEXT AND WORKING ENVIRONMENT 50 5.1 Trends in the Sector 5.2 Needs and Demands of WSP's Partners and Clients 6 RECOMMENDATIONS 52 6.1 Continuing with the Program's Work 6.2 Finance and Resources 6.3 Institutional Arrangements and Governance 6.4 Objectives and Strategy 6.5 Staffing and Staff Skills 6.6 WSP's Ways of Working 6.7 Relationships with Other Global Organizations Appendix 1 Terminology for this Evaluation 57 Appendix 2 Evolution of WSP's Planning and Monitoring Methods, 1998-2003 59 Appendix 3 Previous Evaluations 61 Appendix 4 Finance 65 Appendix 5 People Met and Consulted 66 Appendix 6 Documents Seen and Consulted 70 Appendix 7 Terms of Reference for this Evaluation 77 Annexes concerning WSP's four regions (each having its own contents list and appendices): Annex AF Africa Region Annex EAP East Asia and Pacific Region Annex LAC Latin America and Caribbean Region Annex SA South Asia Region Annexes available on request GLOSSARY AND LIST OF ABBREVIATIONS AA or AAA Analytical and Advisory Activities (a type of funding by theWorld Bank) ACDI Agencia Canadiense de Desarollo Internacional (= CIDA) ADB Asian Development Bank AF AfricaRegion APL Adaptable Program Loan (a multi-phase lending instrument of theWorld Bank) APPJ Proyecto Alimentación de Agua para Pueblos Jovenes (a project for water supply to new settlements in Lima, Peru) ASCID Agencia Sueca de Cooperación International para el Desarollo (= SIDA) AusAID Australian Agency for International Development BNWP Bank-NetherlandsWaterPartnership BP Business Plan BP-CY 1998 WSP's Business Plan For CalendarYear 1998 (planning changed to fiscal years in 1999) BP-FY... WSP's Business Plan for FiscalYear... CARE Cooperative for American Relief Everywhere (an international NGO) CAS Country Assistance Strategy (World Bank) CEPIS/PAHO Pan-American Center for Sanitary Engineering and Environmental Sciences CDD Community-Driven Development CIDA Canadian International Development Agency condominial This name (derived from Portuguese and Spanish terms) is given to a set of methods for approach designing, constructing, and operating WSS services in urban and peri-urban areas. The infrastructureforasuitablegroupofadjacenthouseholdsorplotsisdesignedasaunit,with asingleconnectiontothemainsystem(sewersorwatermains).Withintheunitthenetwork isoftenroutedacrossplotsorunderpavements(sidewalks)insteadofunderroads,yielding economies in the length, depth, and diameter of pipes. Secondly, social and engineering work is integrated: communities are involved from the planning stage onwards, and are giveninformedchoices.Oftenhygieneeducation,and/orhelpwithimprovementofsanitary facilities inside dwellings, are included in the package. CY CalendarYear Danida Royal Danish Ministry of Foreign Affairs DDA Demand-Driven Approach DFID Department for International Development (UK Development Agency) DGDC Directorate-General for Development Cooperation (Belgian Development Agency) DRA Demand-Responsive Approach EAP East Asia Pacific Region ESA External Support Agencies ESW Economic and SectorWork (World Bank) EU European Union FIS = SIF, in Spanish-speaking countries FY FiscalYear 4 External Evaluation Report Go... Government of... GTZ Deutsche Gesellschaft fürTechnische Zusammenarbeit (German Development Agency) GWP GlobalWaterPartnership HASWAS Hygiene and Sanitation in Water Supply (Lao PDR) HIPC Highly Indebted Poor Country IDA International Development Association IBRD International Bank for Reconstruction and Development (the World Bank) IO Internal Order (a job code in theWorld Bank's monitoring system) IRC Resource Center forWater Supply and Sanitation, The Netherlands KfW KreditanstaltfürWiederaufbau(Germanaidorganization) LAC Latin America and Caribbean Region M&E Monitoring and Evaluation MDB Multi-lateral Development Bank MDG(s) Millennium Development Goal(s) MoH Ministry of Health MoU MemorandumofUnderstanding MPA MethodologyforParticipatoryAssessment(resultingfromthePLAinitiative;fordetailssee Section 4.4.5) n/a not applicable NGO Non-Governmental Organization O&M Operations and Maintenance ODI Overseas Development Institute OED Operations Evaluation Department (World Bank) PAC Program Advisory Committee or, in Peru, a certain Peruvian urbanWSS project PEAP Poverty Eradication Action Plan PLA Participatory Learning and Action Initiative (a WSP-IRA initiative in 1998-2000; for details see Section 4.4.5) PPIAF Public-PrivateInfrastructureAdvisoryFacility PPP Public-PrivatePartnership PPPL Proyecto Piloto De Pequeñas Localidades, Peru (pilot project for small towns) PRAGUAS RWSS project of the Government of Ecuador PROAGUA W&S program in Chiclayo, Peru, of the German Technical Cooperation ­ GTZ PRONASAR Programa Nacional de Agua y Saneamiento Rural (RWSS project of the Government of Peru, covering small towns as well as rural areas) PROREDES Programa de Redes Secondarias (de SEDAPAL); program of secondary networks, of SEDAPAL in Lima PROMESAL ProyectodeMejoramientoSanitarioparaareasmarginalesdeLima.Projecttoimprovethe sewerage systems in peri-urban Lima, implemented by SEDAPAL, funded by World Bank and JBIC PROSABAR Programa de Saneamiento Básico Rural (Bolivia) PRS Poverty Reduction Strategy PRSC Poverty Reduction Support Credit External Evaluation Report 5 PRSP Poverty Reduction Strategy Paper PSP Private Sector Participation PSSP PrivateSectorServiceProvider RAC Regional Advisory Committee (ofWSP; see Section 3.4 for description) RBM Results-BasedManagement RRAS-CA Red Regional de Agua y Saneamiento de Centro America (regionalWSS network) RWS RuralWater Supply RWSS Rural Water Supply and Sanitation S&H Sanitation and Hygiene SA South Asia SANBASUR Proyecto de Saneamiento Básico en la Sierra Sur (Cusco, Peru) SDC Swiss Agency for Development and Cooperation SEDAPAL Servicios de Agua Potable y Alcantarillado, Lima (PotableWater and Sewerage Service) SIDA Swedish International Development Agency SIF Social Investment Fund (usually a government-run funding instrument for channelling considerable investments into pro-poor interventions, often linked with debt relief arrangements) SSIP Small-Scale Independent Provider SSPSS Small-Scale Providers of Sanitation Services ST Small Towns SWAp Sector-Wide Approach (World Bank) TA Technical Assistance TF Trust Fund: aWorld Bank mechanism whereby funds from bilateral donors are channelled toWSP regions and projects ToR Terms of Reference; theToR for this evaluation were issued byWSP in October 2003 as part oftheWorldBank'sRequestforProposalsdatedJuly2003 UES Urban Environmental Sanitation UNDP United Nations Development Program UNICEF United Nations Children's Fund UWSS Urban Water Supply and Sanitation WB WorldBank WHO World Health Organization WPA Work Plan Agreement WRM Water Resource Management WSP Water and Sanitation Program WSP-ESA Water and Sanitation Program-East and Southern Africa WSP-WCA Water and Sanitation Program-West and Central Africa WSS Water Supply and Sanitation WSSCC Water Supply and Sanitation Collaborative Council WSTF WaterServicesTrustFund WTP Willingness-to-Pay WUP WaterUtilityPartnership 6 External Evaluation Report External Evaluation Report 7 EXECUTIVE SUMMARY The Water and Sanitation Program (WSP) is an providing such services, both organizational international partnership, set up in 1978, whose methods and technology; declared aim is `to help poor people gain sustained Capacitybuildingtostrengthentheinstitutions, access to improved water supply and sanitation'. In of all sizes and kinds, which undertake the bulk of late2003,theWSPmanagement,assecretariattothe the service provision work; and WSPCouncil,appointedITAD~Water,inassociation with WEDC, to carry out an external evaluation of Guidingandinfluencingthedesignofwaterand WSP'sworkinthefivefiscalyears,1999to2003,andto sanitationprojects tobefundedandimplemented makerecommendationsforthefuture.Thisisthefinal by others, which gives the Program valuable evaluation report, issued in June 2004. This main leverage to stimulate large-scale outcomes with report,supportedbyfourannexes,describesworkin limitedresources. WSP'sregions,eachwithitsownexecutivesummary.1 All these ways of working are applied in three types ofsituations:rural,smalltowns,andurban(usually Evaluation of WSP's Work in peri-urban), though many aspects cut across this FY 1999to2003 pragmatic classification. Our overall finding is that WSP was effective and efficientintheserviceofvaluableobjectivesduring WSP focuses on a limited number of countries to the review period. Various changes made during make good use of limited funds. The geographic the five years have improved the Program's focussharpenedduringthefiveyears,reducingfrom governance and performance. 27to22countries,whicharenowdistributedamong the Program's regions as follows: TheevaluationanalyzedtheProgram'sworkinterms Africa Region (AF): 9countries of the extent to which its own activities and outputs achieved outcomes and lasting impacts that were EastAsia-Pacific beyond its control. As a partnership organization Region (EAP): 5countries workinginadifficultfield,wheremanydevelopment initiatives in the past have proved ineffective or Latin America & unsustainable, most of the Program's work involves Caribbean Region (LAC): 5countries changing attitudes, spreading knowledge and South Asia Region (SA): 3 countries, understanding, and demonstrating the merits of including India appropriate ways of providing services in particular whose large size and culturesandsituations.Theachievementofoutcomes federalstructuremake to the benefit of poor people depends on the actions it a special case; ofpartiesotherthanWSPitself.TheProgramisactive several Indian in supporting the development of appropriate States are involved policies and effective institutions in the public and private sectors, and organizational arrangements Finance and Resources where they can work well together. TheProgram'sannualexpenditureaveragedUS$ 13 millionduringthelastfouryearsevaluated,ranging WSP's work is organized in four main areas that from 12 to 15 million. Global management costs interact to strengthen each other: were kept down to about nine percent of total Seeking to guide and influence the policies of expenditure.The current expenditure rate remains governments, national and local, on water and ataboutUS$ 14million.Assuredfundingforthenext sanitation service provision for poor people; three years is about US$ 12.5 million per year, or The generation, articulation, filtering, and up to US$ 16 million, with some funds that are propagationofknowledge aboutimprovedwaysof currently under negotiation. Most donors tend to 1These four annexes can be viewed and printed from www.wsp.org 8 External Evaluation Report commit funds for only about three years ahead, declared core objective, namely helping its clients which is unfortunate because most of WSP's and partners to achieve sustainable access to WSS interventionsneedperiodsoftheorderoffiveyears forpoorpeople.Thefocushasbeensharpenedover ormoretoachievetheirfullpotential.Roughlytwo- the evaluation period, both geographically and thirds of the future funding is tied by donors to thematically. Focusing on the poor is a complex particular countries, themes or projects. WSP's matter, and specific pro-poor interventions are not abilitytorespondtoemergingopportunities,which necessarily always the best mean of achieving the it generally does well, is constrained by the limited objectives;WSPmustcontinuetoadoptabroadand proportion of core or untied funding. Such funds thoughtful approach. The continuity and hence are also valuable for smoothing out fluctuations in effectiveness of WSP's focus would be helped if its fundingforparticularregions,whichotherwisecan donors could find ways to plan any change of lead to loss of momentum and sometimes loss of prioritiesseveralyearsahead. experiencedstaff.AfundingcrisisintheLACregion in 2001 did have such consequences, but the Planning and monitoring were weak in the earlier operation was kept going at the initiative of SDC yearsofourevaluationperiod,withseveralchanges and with the help of the World Bank; that region's intheformatofBusinessPlans,butstepsweretaken funding has now stabilized again. Funds are and they have now been much improved. There managed under the World Bank's procedures; this remains a need to consolidate cost-effective ways evaluation does not include financial auditing, but of combining coherent, outcome-focused, multi- the accountability appears satisfactory. year planning with creativity and adequate flexibility, and to achieve consistent application of These resources currently enable WSP to deploy such planning and monitoring procedures over all about 80 staff, of which less than 10 percent are in regions and all projects. the headquarters group. The Program also shares someadministrativestafftimeatheadquarters,and Impact makes good use of interns and consultants. TheProgram'sworkhasevidentlyachievedrelevant outcomes in diverse situations, with considerable Given the effectiveness ofWSP and the large global impactalreadyobservedandmoreconsideredlikely need for sustainableWSS, there is scope forWSP to to follow. Some interventions have been less usefullyexpandthescaleofoperations;thatscaleis successful than others on these criteria, but that is effectively limited by donor funding and not by the to be expected in a difficult field and for an Program's inherent capacity. We consider that the institution that is often leading the way rather than current management structure, with four regions, following proven recipes. couldfruitfullyhandleanincreaseinturnoverofup to 50 percent. This would, of course, depend on Institutional and Governance Arrangements donor funding, and on being able to attract extra The Program is an international partnership staff of the necessary caliber and experience. An reporting to a Council and administered by the expansion much beyond that level might require WorldBank.Itworksthroughfourfairlyautonomous somerestructuring. regional offices and a small headquarters, mostly located in or near World Bank offices. We consider Strategy, Planning, Monitoring, that these fundamental institutional and and Resource Management governance arrangements are satisfactory, and for We consider that, at the start of the five-year practical purposes the best available, but that the evaluation period, there was an underlying lack of details could be improved and clarified. In clarity about ends and means, especially in WSP's particular the degree to which WSP is in practice written statements, and subsequent formulations controlled by the World Bank should be discussed havenoteffectivelyreplacedthosestatements.Ata by the Council and an agreed statement should practical level, the consequences were not very replacetheambiguousprovisionsoftheCharter.We severe because the Program's implied objectives consider that the provision for the Council to be were usually understood by its relatively small automatically and always chaired by a World Bank number of key staff. In our view, the Program has officialcouldwithadvantagebechanged.Thisrefers thus succeeded in maintaining a good focus on its not to presiding at meetings but to the role of the External Evaluation Report 9 Chair between the annual Council meetings when intellectually difficult and challenging, and no one it is required to provide leadership and resolve should expect perfection. Inevitably in conflicts,andtorepresenttheWSPCininteractions development work, some interventions fail, or withotherstakeholders. succeed only partly. It appears to us that a more self-questioning attitude on WSP's part, together Interactions with Other Actors in the WSS Sector with a less defensive attitude to constructive These interactions are, of course, crucial for a criticism, would strengthen rather than weaken relatively small organization operating as a its effectiveness. partnership and relying on others to generate large-scale impacts. WSP has many important WSP's Comparative Advantage relationships with other sector actors, and most of We conclude that the Program has significant themareverysatisfactory.Therearesometensions, comparative advantages in relation to other not necessarily of WSP's making, which could be organizations working in its field or overlapping to alleviated by good communication and by a greater or lesser extent with its functions. Its clear thinking and discussion about shared and partnership style enables it to collaborate and separate objectives. complement those other organizations where appropriate.Asanalyzedinthisreport'sSection 4.5, Knowledge, Communication, and Publications there are several types of work, such as advocacy, WSP's knowledge-related work is important and policyadvice,learningorknowledgemanagement, fruitful as a complement to its other activities, and which are also covered by others. But WSP's global is mostly well done. Quality and appropriateness reach, valuable contacts, long track record, high of publications are usually good, though inevitably professional capacity, and ability to maintain variableinadifficultfieldwhereallstaffareinvolved, continuing real-time contact with key decision- includingthelessexperiencedones.Standardshave makers all give it particular advantage. improved in the course of the evaluation period. Recommendations Staffing and Staff Skills We expect the current trends to continue towards WSPhasanexcellentcoreofstaffwithrelevantskills increasing decentralization in relevant countries, andexperience,buttheyarespreadquitethinlyand andtowardsanincreaseintheneedforsustainable slightlyweakenedbyrecentlosses.ButtheProgram's WSS serving poor people in urban and peri-urban work poses big challenges, and we see the need for areas without the reduction of the needs in rural continuing regular review of future staffing, areas and small towns. Decentralization will place grounded in a very clear understanding of WSP's heavydemandsonlocalgovernmentbodies,which special objectives and its comparative advantage often have little relevant experience. The relative to other organizations. Millennium Development Goals are an important Corporate Culture and expression of collective will, though as loosely Response to Commentaries definednumericaltargetsforaccesstoservicesthey The Program's work was the subject of many arenotusefulforplanningandresourceallocation. evaluations during the review period, and it WSP'sworkisandremainshighlyrelevanttoseveral responded with several beneficial changes. We of the MDGs and their associated targets. notice that the Program's staff tends to respond to Our principal recommendation is that WSP any criticism, even if mixed with much praise, in a shouldcontinuewithitsvaluablework,andshould sensitiveanddefensiveway.Thisreflectstheculture seekasignificantfundingincreaseforthenextfive oftheWorldBanktowhichWSPstaffarecontinually to 10 years. exposed,andalsotheirdedication,professionalism, and pride in their work. It may also reflect a desire By `significant increase' we mean something of the to achieve measurable targets, though we consider order of 30 to 60 percent, rather than 10 percent or that too much emphasis on such targets tends to 100 percent.This does not refer to any quantitative underminethevalueofa`learningorganization'.But analysis; it merely reflects our views of the sector's this sensitivity seems to us unnecessary and not needs, of the Program's comparative advantages, entirely healthy. The Program works at the leading and of the scale of work that could be handled edge of a field which is professionally and withoutmajorrestructuringoftheProgram.Thenext 10 External Evaluation Report fiveto10yearswillbecrucialforthesector,notleast h) WSP should seek to clarify its strategy by in relation to the MDGs, whose time-scale runs to sharpening the definition of its objectives, 2015. To attract a funding increase of this order, emphasizing the need to address institutional WSP would, of course, have to persuade existing failureandineffectiveness. donors to give more, and/or persuade new donors to start giving. i) WSPshouldbothplanandmonitoritsresource use with a multi-year perspective, rather than Our more detailed recommendations in Chapter 6 just within fiscal years as it has tended to do suggest actions by WSP and its Council that we until recently. consider would help to do this.We do not prejudge howeasyordifficulttheseactionsmaybe.Themain j) Donors should seek ways to commit funds points, not in order of importance or urgency, are: several years in advance, typically five years. a) The Program should remain a partnership k) WSP should make systematic and transparent organizationdirectedbyitsmainstakeholders arrangements for coping with the fact that representedonitsCouncil,whilebeinghosted donors often do not commit funds more than and administered by the World Bank, and three years in advance. operating under World Bank financial and l) The Program and the donors together should accounting systems. seek ways of achieving a joint flexibility b) Within that structure, the Council should in resource allocation by increasing the consider taking on a more active role than at proportion of untied, partially-tied or present, through a delegated executive core funding. committee or otherwise, in the direction and m) WSPshouldseekwaystoimprovetheflexibility control of the Program. of resource allocation in the course of each c) TheCouncilshoulddebatetheextenttowhich project's life. WSP's thematic and technical work ought to n) WSP should continue with regular reviews of becontrolledbytheWorldBank,andifpossible the mix of staff skills, and seek ways of replace the ambiguous provisions of the providing key staff with a degree of security Charter with a clear and agreed statement. of employment over more than about d) TheCouncilshouldconsiderchangingitsrules threeyears. so that it elects its Chair from time to time o) Whendesigningitsownprojects,theProgram instead of being chaired automatically by an should strengthen its efforts to identify and official of theWorld Bank. spell out intended outcomes and impacts, e) The Council, the WSP, and the World Bank rather than concentrating on outputs, and should all take conscious steps to ensure that should link this to its internal monitoring by otherpartiesunderstandthedistinctnatureof finding simple ways to record the perceived the program and do not perceive it as a achievement or otherwise of those outcomes. subservient agent or arm of theWorld Bank. p) When a WSP project is being designed, this f) WSPcentralmanagementshouldmaintainits careful formulation of the intended outcomes present balance between controlling the andimpactsshouldleadtoidentificationofthe regions and allowing them more autonomy, otheractorswhoseattitudesandbehaviorneed and the regions should respect that balance to change, and then to an analysis of their and comply with the necessary central constraints, habits, motives, and priorities. disciplineregardingplanning,monitoring,and q) When asked to help others to design an resourcemanagement. investment project, WSP should take care g) Awayshouldbesoughtforreportingtheviews always to offer and apply its own distinctive oftheRegionalAdvisoryCouncilstotheannual experience and judgement, rather than Council meetings. currently fashionable solutions. External Evaluation Report 11 r) In view of the political sensitivity of to foster the work of sub-regions such as institutional reform, WSP should further West Africa; donors should consider strengthen its efforts to build up close and helping with the costs of translation and trusting relationships both with donors/ interpretation. funders and with government officials and local politicians. v) WSP staff should have enough confidence in their own caliber and achievements to adopt s) WSP should consider using a wider range of amoreself-questioningculture,togetherwith media, and should more often share a less defensive attitude to constructive the distribution of publications with criticism. other organizations. WSP should reconsider how it works with other t) WSP should seek more ways to exchange organizations on the globalWSS and related fields, information between its regions. buildingonitsreviewofobjectivesandstrategy,and u) WSP should continue and strengthen its on the comparative advantages of different efforts to overcome language barriers and organizations. 12 External Evaluation Report WSPCOUNCILDECISIONS WSP Council Decisions of May 28, 2004, management presented its response to the on the Response to the External evaluation, and in particular to the Evaluation and Adoption of recommendations set out in the draft evaluation Recommendations report'sExecutiveSummary(whichwerealmostthe A draft version of this report was presented to the same as those of this final report). The table below WSP Council at its meeting in Berlin on May 27, records the recommendations, the management 2004, and discussed. On May 28, the WSP responses, and the Council decisions. Evaluator Recommendation Program Response Council Response The Council should consider changing No change recommended. No change. its rules so that it elects its Chair from time to time instead of being chaired automatically by an official of theWorld Bank. The Council,WSP, and theWorld Bank Agreed. Clear instructions to be Agree withWSP should all take conscious steps to issued byWSP management to response. ensure that other parties understand the Programstaffclarifyingdistinct distinct nature of the Program and do natureofProgram not perceive it as a subservient agent or [by August 1, 2004]. arm of theWorld Bank. WSP central management should Agreed. Recognizing the maintain its present balance between regional nature ofWSP, controlling the regions and allowing the global identity of them more autonomy, and the regions WSPshouldbe should respect that balance and comply maintained and with the necessary central discipline strengthened. regarding planning, monitoring, and resourcemanagement. A way should be sought for reporting the Agreed. RAC reports/MoUs to be AgreewithProgram views of the Regional Advisory Councils distributed to Council response. to the annual Council meetings. [by August 1, 2004]. WSP should seek to clarify its strategy Agreed.WSP will issue a revised NewProgramManager by sharpening the definition of its statement clarifying its (PM) to issue revised objectives, emphasizing the need objectives [by January 1, 2005]. statement to reflect to address institutional failure MDG priorities by and ineffectiveness. January1,2005. WSP should both plan and monitor its Agreed. As acknowledged in Agreed. Council would resourceusewithamulti-year report, in FY04WSP began like to see modifications perspective, rather than just within planning on a multi-year basis, reflected in the BP fiscal years as it has tended to and will continue to improve forFY05. do until recently. the process. External Evaluation Report 13 Evaluator Recommendation Program Response Council Response Donors should seek ways to commit WSP will develop a funding Agreed. Chair would like funds several years in advance, typically strategy and continue to engage to involve finance sub- fiveyears. with donors on an individual committee (Dutch, basisforfuturefunding. SDC, DFID, Danida) in Draft funding strategy sent to this process. donors by December 1, 2004. WSP should make systematic and Agreed. As above, to be reflected Agree withWSP transparent arrangements for coping in the funding strategy. response. with the fact that donors often do not commit funds more than three years in advance. The Program and the donors together Agreed. As above, to be reflected Agree withWSP should seek ways of achieving a joint in the funding strategy. response. flexibility in resource allocation by increasing the proportion of untied, partially-tied or core funding. WSP should seek ways to improve WSP already reviews and revises Agree withWSP flexibility of resource allocation in the allocations of resources at mid- response. course of each project's life. year and start of following year. StartinginFY05,WSPwillconduct the review at the project level. WSP should continue with regular WSP and WSS Sector Board Council asks Bank to reviews of the mix of staff skills, and seek alreadyreviewstaffskills takeadvantageof ways of providing key staff with a degree periodically. By nature of its existing and new of security of employment over more funding,WSP can only issue employment conditions thanaboutthreeyears. fixed-term contracts. Subject within the Bank to to performance, these maximize security of are extended. employment. When designing its own projects, the Agreed. As acknowledged in the Agree. Council urges Program should strengthenitsefforts to report, from FY04 onwardsWSP greateremphasis identify and spell out intended hasbegunoutcome/impact- on outcomes. outcomes and impacts, rather than based planning, and will concentrating on outputs, and should introducesemi-annual find simple ways to systematically project-level monitoring in FY05. record the perceived achievement or otherwise of those outcomes. When aWSP project is being designed, Agreed.The outcome indicators Agree. Council urges this careful formulation of the intended WSP develops already take into greateremphasis outcomes and impacts should lead to account the expected change in on outcomes. identification of the other actors whose behavior of the client.These will attitudes and behavior need to change, befurtherstrengthened. and then to an analysis of their constraints, habits, motives, and priorities. 14 External Evaluation Report Evaluator Recommendation Program Response Council Response Whenaskedtohelpotherstodesignan Depending on the context and Council agrees with investment project,WSP should take drawing on its global knowledge Program,andurgesWSP care always to offer and apply its own and field experience, based on to bring its comparative distinctive experience and client demand, WSP provides advantagetobear, judgement,ratherthancurrently appropriate TA. promoting innovation fashionablesolutions. and new ideas. In view of the political sensitivity of Agreed.WSP's permanent field Council agrees with institutionalreform,WSPshouldfurther presence and its role as a neutral Program,andurgesWSP strengthen its efforts to build up close broker has enabled it to build to bring its comparative and trusting relationships both with trusting relationships with sector advantage to bear and to donors/funders and with government actors and provide real time collaborate with UN officials and local politicians. technical assistance on a system and other continuous basis. Continuous partners. efforts will be made to strengthen partnerships. WSP should consider using a wider Agreed.WSP has begun using a Council agrees with range of media, and should more often range of media and will Programresponse. share the distribution of publications integrate the approach into its with other organizations. forthcoming communication strategy [by November 1, 2004]. WSP should seek more ways to Agreed.WSP regions already have Council agrees with exchange information between cross-regional projects. RTLs will Program response, and its regions. take initiative to promote would like to see this moreeffectivecross-regional item reflected in agenda exchanges. One of the main at future meetings. Chair responsibilities of the new global andWSP management thematic coordinator will requested to include in be promoting cross-regional the result agreement of exchanges. the PM and the RTL specificcross-regional work in global and regional work programs. WSPstaffshouldhaveenough WSP welcomes constructive Council commends the confidence in their own caliber and criticism and healthy debate. quality of theWSP staff. achievements to adopt a more self-questioningculture,together with a less defensive attitude to constructive criticism. WSP should reconsider how it works Agreed.WSP's modus operandi is Council urges new PM with other organizations on the global working in partnerships. The to take lead in revising WSS and related fields, building on its Program will revisit its existing the strategy. review of objectives and strategy, and on partnership strategy and Council reiterates the the comparative advantages of revise it as necessary importance it sees in the differentorganizations. [by January 1, 2005]. coordination between WSP and other lead actors in the sector, such as WSSCC/GWP. External Evaluation Report 15 1 INTRODUCTION The Water and Sanitation Program (WSP) is an This evaluation has been guided by our discussions international partnership, set up in 1978, whose with many stakeholders and WSP staff in the four declared aim is`to help poor people gain sustained regionsandtheheadquarters,andbyresponsesfrom access to improved water supply and sanitation'. others to whom we sent a questionnaire. Names of Though it started as a series of separate projects bothgroupsarelistedinAppendix 5.Ithasalsobeen supported by the UNDP and implemented by the guided by our study of many documents, listed in World Bank, theWSP has now evolved into a global Appendix 6.Theevaluationremainsanindependent partnership, active in some 30 countries and and external one, and all the conclusions and financially supported mainly by bi-lateral recommendations are our own. development agencies. Its governance was redefined in a`Program Charter' in 2001, which set We acknowledge with thanks the assistance of the up a Water and Sanitation Program Council and WSP management in Washington and the regions, made provision for that Council to commission and of all the WSP staff who have helped us with regular reviews of the performance of the WSP. In finding and collating information and with making October 2003 theWSP management, as secretariat relevant contacts. We also thank the many totheWSPCouncil,issuedTermsofReference(ToR) stakeholders in various countries who have given for an External Evaluation of WSP's operations in timetomeetusandhaverespondedtoourquestions the five-year period, July 1998 to June 2003, with orshowedusprojectsimplementedwithorbyWSP. recommendationsforthefuture.InDecember2003 This main report presents the evaluation at global theBritishfirmITAD~Water,inassociationwiththe level, for all regions and the center. Supported by Water, Engineering and Development Center appendices,ithasfivechapters:Chapter 2explains (WEDC),wasappointedtocarryouttheevaluation. the framework of this evaluation, and Chapter 3 WorkbeganonJanuary12, 2004andwasscheduled describesWSP's work in the region. The evaluation to be completed by July 2004. The evaluation was of the five-year period is then presented in conditioned by its budget and ToR. Only one Chapter 4, which covers the 28 topics required by memberoftheevaluationteamvisitedeachofWSP's theToR.Chapter 5summarizesexpectedtrendsand fourregions,andoneofthesevisitswasdelayeduntil needs in the next few years, and recommendations April. The team is referred to in this report as `we'. for the future are in Chapter 6. Regional details are Thisisthefinaloverallevaluationreport.Itisissued presentedinfourannexes,eachofwhichhasitsown in late June 2004. executive summary and appendices. 16 External Evaluation Report 2 EVALUATION FRAMEWORK In order to give our evaluation a clear framework department's attitudes or behavior are internal to and nomenclature, Table 2.1 defines a hierarchy of that project and are therefore not outcomes of planning and evaluation levels, from inputs to such projects. impacts.Itiswordedinageneralwaytorefertoany sort of project relative to its implementing agency, In this evaluation we are dealing with the work and by which we mean the party who controls the activitiesofWSP,asanimplementingagencyofwhat activities which use inputs to produce outputs. it and its subcontractors do. If WSP, and any (Ourtableusesthesametermsasthe`Results-based subcontractorsunderitscontrol,performactivities management (RBM) framework' which was and produces outputs which change the attitudes prepared for WSP in 2003,2 but defines them and behavior of other parties, those changes are more closely.3) outcomes of WSP's work. This holds even if the changed party is the implementing agency of a The important distinction, emphasized in the related and wider development project: such right-hand column of this table, is between things changes are outcomes of WSP's work though only that an implementing agency can control, through outputsofthatwiderproject;weareevaluatingWSP, itsresources,activities,andoutputs,andthingsthat not the wider projects. it cannot control, though it often seeks to influence them. The identification of outcomes as distinct For example, suppose that a government from outputs therefore depends on the identity of department responsible for developing rural water the implementing agency. In many development supply schemes observes that most of these projects the implementing agency is a government schemes are not sustainable and the users are not department,soanyproject-inducedchangesinthat satisfied; the infrastructure breaks down and the Table 2.1: Hierarchy of Planning and Evaluation Levels Hierarchy of Levels Explanation Control Distinction Impacts Aggregated effects in the long term (several years at least) and Outside the onawidescale(wholecountiesorregions),whichcanbepartly direct control attributed to the implementing agency's work. ofthe implementing Outcomes Behavioralchangesbytargetgroups,orotherexternaldesired responses to the implementing agency's outputs, in the short agency. term and on a local scale. Outputs Deliverables, products or services which stimulate desiredoutcomes. Within the direct control Activities DiscretedefineditemsinworkplansorBusinessPlans,carried ofthe outbytheimplementingagency,orbyothersunderitscontrol. implementing Inputs Resources, such as money, staff or time, controlled by the agency. implementing agency. 2Results-based management framework for WSP; Report from the initial phase; Per O. Bastoe, March 2003. 3Some versions of RBM terminology use `outcomes' as a wider term, including within it our `impacts' under the name `final outcomes'. It is widely recognized that a particular RBM framework needs to be designed for each specific organization, especially one that works by partnerships (as emphasized in the undated UNDP document, `Introduction to Results-Based Management; RBM in UNDP: Overview and General Principles', which was developed in cooperation with SIDA). External Evaluation Report 17 users do not have the skills and financial resources increase the likelihood of other parties changing torepairandmaintainit.ThedepartmentasksWSP their priorities and methods. to help, and after analyzing the situation WSP suggests adding new steps to the implementation Appendix1describeshowweusefivefurtherterms process for future schemes (for example, specified in ourToR, defining the following specific participatory needs assessment, design guided by aspects for our evaluation: users' informed choice, setting up effective user relevance groups for maintenance). WSP persuades the effectiveness government people of the merit of these steps and efficiency trainsgovernmentstafftousethem.Laterschemes sustainabilityandreplicability are implemented in this new way and prove responsiveness sustainable. For the government department, as strengthofinstitutions implementing agency for the water supply As a partnership organization, WSP works with schemes, the new steps are activities, the better- many other organizations in various ways. It designedschemesandtheircapableusergroupsare distinguishesbetweendifferentsortsofrelationship outputs,andthesustainablewatersuppliesenjoyed by the use of the following terms, which are not by the users are outcomes. For WSP, as executing preciselydefinedbutwhichhelptoindicatethesort agency of the advisory and training function of relationship in particular cases4: embedded in this process, the analysis and persuasionareactivities,theworkshopsandtraining UltimateClients:PoorWSSconsumersservedby manualsareoutputs,andthemoreenlightenedand WSP's direct clients. successfulnatureofthegovernmentdepartmentis anoutcome.Thesustainablewatersuppliesenjoyed Direct Clients: Primarily refers to developing by the users is another, more indirect outcome country governments (especially with respect to which,alongwiththefinalimpact(sustainedwater country work), but also refers to private sector supply over many years and beyond the location of service providers, semi-autonomous utilities, etc. the original project), is common to both, the In addition, WSP loosely uses this term to refer to government department and WSP. knowledge clients, such as NGOs and strategic partners, whenWSP provides information to them. ForWSP's interventions or projects, the distinction betweenthingswithinoroutside`thedirectcontrol Strategic Partners: These are a wide range of ofWSP' is a useful one, but it must be remembered agencies with whom WSP collaborates in some that activities and outputs canbedesignedtomake activities, such as WSSCC, UNICEF, UN Agencies, outcomes and impacts more likely to follow (that is, WaterAid,WUP, and many NGOs. to be more effective). When an intervention is well designed in this way, certain matters are in effect Financing Partners: Organizations who support shiftedfromoutsideitsfieldofaction(assumptions) WSP financially have a special stakeholder to inside it (activities and outputs). This is often relationship to the Program. Most of them channel done by including persuasion and deliberate the funds through the World Bank's Trust Fund attitude-changing work among WSP's activities, to mechanism. 4Most of the text of these working definitions was provided by WSP in February 2004. 18 External Evaluation Report 3 THE PROGRAM FROM FY 1999 TO FY 2003 3.1 The Program's Objectives and strategy document, whose main statements and Field of Work categories are summarized inTable 3.1. WSP'sworkwasdefinedoverthefive-yearevaluation periodinaseriesofBusinessPlans,whoseevolution For practical purposes that strategy statement was is described and commented upon in Appendix 2 superseded by a 2001 document, `WSP Renewal ­ and Section 4.2.3. This chapter summarizes what Strategy Update and Progress Report', prepared for WSP was aiming to do in the five years, while the the October 2001 Council meeting in The Hague, four annexes to this report (available on request) which re-stated the five `product lines' and added givedetailsofwhatwasbeingdoneinthefourregions. `sanitation and hygiene' to the agenda. It also emphasized, as its central theme, the achievement The Program's work at the start of this period was of institutional change and reform on a wide scale. planned within the framework of the 1999-2003 The Program Charter, also issued in 2001, used a Table 3.1: Summary ofWSP's Original Declared Strategy for 1999-2003 Label WSP's Statements, in its `Program Strategy: 1999-2003' `mission `to help poor people gain sustained access to improved WSS' statement' `...works with partners in the field to seek innovative solutions to the obstacles faced by poor communities, and strives to be a valued source of advice to achieve widespread adoption of thesesolutions' `mutually 1a generate knowledge (`learning') supporting 1b communicate knowledge objectives' 2 strengthen sector policies 3 improve investment effectiveness `products' 1 pilot and demonstration projects or 2 knowledge generation and management `product 3 sectoral networking lines' 4 policy support 5 support for large-scale investment projects (Each product has`expected outputs'defined in the 99-03 Strategy, Part 4) `learning ruralagenda agenda' urbanagenda small towns agenda `key 1 sharedvalues factors/ 2 diverseandhighlyskilledstaff building 3 qualityassuranceprocesses blocks' 4 partnering and networking 5 decentralized operational structure 6 governancearrangements 7 sustainable financial base Source: Water and Sanitation for the Poor: Innovation through Field Experience ­ Program Strategy: 1999-2003;WSP External Evaluation Report 19 Table 3.2: Three-year Targets in WSP's Logframe of 2001 Highertarget `Partners have launched national sector reform programs and established appropriate frameworks for investments in WSS.' Subordinate `At least 50 percent of focus countries have: targets adopted DRA-based investment strategies for rural and small townWSS; introducedimprovedinstitutionalarrangementsformanagementofWSSservices for poor communities in small towns, urban, and peri-urban areas; adoptedprocurementstrategiesandenhancedincentivesforprivatesectorprovision of goods and services; and integratedeffectivehygieneandsanitationpromotionandgender-sensitive approaches into large programs for community-based water and sanitation.' similarstatementofobjectivetothe1999-03strategy promotion, as well as piloted models for extending document. There was, from 2000 onwards, a sustainable WSS services to the poor. Thus in diagrammatic`StrategicResultsFramework',which FY 2001 the main features of the 1999-2003 restated the strategy in the form of a hierarchical strategy document were still discernible, but tree.5 Starting with the above five `product lines', it differentlyexpressed. named three`outputs' corresponding to policy and strategy support, investment support (improving Since 1998 the `mission statement' quoted in quality of the investments of other parties), and Table 3.1 has been replaced for some purposes by a knowledge management. This led to statements of new form of words, usually accompanied byWSP's higher objectives. Formulations about strategy namesothatitdoesnotneedtomentionwaterand during these years tended to interpret the concept sanitation.Thisreads:`Aninternationalpartnership of`helping poor people gain sustained access...' in for improving sector policies, practices, and themissionstatementintermsofhelpingtheWSP's capacities to serve poor people.' Statements about direct clients (usually government or service strategyareissuedfromtimetotime,mostrecently providers) to help their clients, the poor. This and in the Annual Review for FY 2003. other concepts concerning the Program's strategy Althoughthe1999-2003StrategyDocumentwasnot and ways of working were in general internalized formallyreplaced,thestrategyforthefiveyearswas by its staff, without much reference to the formal thus occasionally redefined in an ad hoc way, while statements quoted here. keepingtoarelativelyconstantandconsistentcore. The Business Plan for FY 2001 included a global The study of the detailed task sheets and tables logframe, putting its emphasis on effective within the successive Business Plans reveals an approachesbyWSSserviceproviders.Thatlogframe underlying pattern which can be seen to represent included time-related targets. To illustrate this, aconstantimpliedpolicyorstrategy.Bytheirnature, Table 3.2 quotes those at the three-year timescale nearlyalltheProgram'sactivitiesandprojectsserve (up to the end of FY 2003) within a 10-year theMillenniumDevelopmentGoals(MDGs)inone perspective.Theywerepresentedasobjective-level way or another. indicators under the heading`impact outcomes'. In the spirit of helping direct clients to help their Thesethree-yeartargetsaregenerallyattheoutcome clients (the poor), the Program's main strands of level, since their achievement on a wide scale workincludedhelpingothersectoractorstoimprove implies action by parties outside WSP. The policies,institutions,organizationalarrangements, accompanying 10-year targets involved large-scale and projects. These applied to rural, small town, institutional reforms and hygiene and sanitation andperi-urbanfields(theattemptinBP-FY 2001to 5One version of that framework was presented on page 6 of the mid-year review for FY 2000, we were shown another in April 2004. 20 External Evaluation Report add`sanitationandhygiene'tothosethreeagendas co-ordinating activities involving national and does not seem to have become a lasting feature, international bodies, as well as diagnostic studies, although there are some activities that attend to the transmission of ideas and methods from one hygienepromotion,oftenandrightlyasanecessary place and time to another, and capacity building complementtosanitationprojects).Theemphases for many sorts of sector actors. Details of the shiftedamongthese`agendas'fromtimetotimeand Program's aims and ways of working are in the from place, generally in response to needs and regionalannexesofthisreport(availableonrequest). opportunities. The context to which WSP was WSP's objectives are entirely consistent with the responding, in most of the countries concerned, Millennium Development Goals, and in particular included government policies of shifting towards theirTargets10and11whichaimforahalvingofthe decentralization of responsibilities for service proportion of people without sustainable access to provision to local bodies such as municipalities. safewaterandbasicsanitation.TheseMDGsarenot This made capacity building and communication much used byWSP as explicit planning objectives; particularly important. planningisrightlydoneintermsofmoresubtleand The help to other sector actors often involved the place-specific formulations of objectives. setting-up of country or regional sector networks, promoting communication between actors who 3.2 WSP's Focus Countries lackedit.Themeansbywhichsupportwasgivento Table 3.3 lists the focus countries in the first year of various bodies and processes also included many theevaluationperiodandthecurrentyear,showing Table 3.3:WSP's Focus Countries in FY 1999 and in FY 2004 Fiscal Year Africa East Asia-Pacific Latin America- South Asia Caribbean Benin Cambodia Bolivia Bangladesh BurkinaFaso China Ecuador India Côte D'Ivoire Indonesia Peru Nepal Eritrea Laos Pakistan 1998-99 Ethiopia Mongolia Sri Lanka (FY 1999) Ghana The Philippines Kenya Vietnam Malawi Mali Mozambique Niger Rwanda Benin Cambodia Bolivia Bangladesh BurkinaFaso Indonesia Ecuador India Ethiopia Lao PDR Honduras Pakistan Kenya The Philippines Nicaragua 2003-04 Mauritania Vietnam Peru (FY 2004) Mozambique Senegal Uganda Yemen Source: Information from WSP, May 2004 External Evaluation Report 21 that the geographical focus sharpened during the required to meet formally at least once a year, and fiveyearsfrom27countriesto22.WSPhasawritten informal consultations between some of its set of criteria for guidance on the selection of members take place between formal meetings. countries in which it should work, which was Itcananddoesappointsub-committees,including discussed with the Council in 2002. one for financial matters. The Charter specifies that the Program `is directed 3.3 Main Activities in the by its participating partners and administered by Evaluation Period and within the World Bank on behalf of its donors. Table 3.4 summarizes the main fields of work and The WSP manager and staff report through the the partners involved. Director, Energy and Water, to the Vice President, Inadditiontothesecountry-specificarrangements, Private Sector Development and InfrastructureVice central core funding is provided by DFID (UK), Presidency'.6 The Chair of the Council is defined as a Danida, Norway, Netherlands, SDC (Switzerland), Vice-President of the World Bank,7 although in and DGDC (Belgium). Some other funds are practice this role has always been delegated to a provided to WSP regions with only partial relevantDirectorintheBank,whooftenasksanother restrictions on their use, notably from SDC, Council member to take the Chair for particular Netherlands, AusAID, Luxembourg, Norway, Council sessions. The Chair is required to `provide SIDA (Sweden). leadership and resolve conflicts,represent theWSPC in interactions with other WSP stakeholders, and generallyseektofurtherthesenseofpartnershipthat 3.4 Institutional Arrangements has characterized the operations of theWSP'.8 and Governance The Program is an international partnership The institutional relationship with theWorld Bank reporting to a Council and administered by the is a special case, and important both for function World Bank. It works through four regional offices and for perceptions.The Program's website states, andasmallheadquarters,mostlylocatedinornear `Its special relationship with the World Bank ­ one World Bank offices. of the largest investors in water and sanitation services for developing countries ­ increases the Having started out in 1978 as a program financed Program's access to national policymakers and its mainly by UNDP and the World Bank, WSP is now ability to influence the sector.WSP staff contribute funded almost entirely by individual governments, to the World Bank by participating in World Bank referred to in this report as donors. Until 2000WSP projects, working closely with Bank staff, and was guided by a Program Advisory Committee leading thematic discussions and thinking within (PAC),whichdidnothaveformaltermsofreference the sector.'This establishes theWSP's identity as an until 1996. It included a considerable number of institution distinct from the Bank, and thus having representatives from governments and other a relationship with the Bank to the advantage of institutions, not all of them significant donors. In both, but not part of the Bank's structure and not 2000, a review of governance (described briefly in under the Bank's control. The relationship is Appendix 3)wascommissioned,andaPACworking complex and easily misunderstood because, as the group developed a new arrangement which was website also and correctly says, `the Program is enshrined in a Program Charter and adopted in administered by the World Bank and follows the March 2001. It provides for a WSP Council Bank's management and administrative processes.' comprisingcontributingdonors,onememberfrom That WSP website also goes on to say `[WSP] an affected country, one member from a strategic functions as an independent unit within the partner such as a leading NGO or global sector Department of Energy and Water in the organization,onesectorexpert,andrepresentatives Infrastructure Vice Presidency', which, subject to of UNDP and the World Bank. The Council is the interpretation of the vague term`independent', 6Program Charter, clause 7. 7Program Charter, clause 14. 8Program Charter, clause 15. 22 External Evaluation Report Table 3.4: Main Activities and Partners in Particular Countries, as on July 2003 Country Main Fields ofWork MainPartners AFRICA Benin Policy support in rural areas WB, DANIDA, GTZ, UNICEF, DGDC, and NGOs Burkina Faso Urban ­ large-scale UNDP,AFD,andDANIDA investments Ethiopia Rural water supply ­ recently WB,DGDC,BNWP,DFID,EU,NGOs,PPIAF, changed to sanitation UNDP, and UNICEF Kenya Urban/Reform GTZ, KfW, AFD, UNICEF,WHO, SIDA, and JICA Mauritania Urban/Small towns GTZ, KfW, AFD, UNICEF,WHO, SIDA, and JICA Mozambique Ruralwater SDC,UNICEF,andWaterAid, Senegal Large-scale investment projects AFD,WB,NGOs,andUNDP Uganda Small towns water ­ recently DFID,DANIDA,SIDA,KFW,AUSTRIA, changed to sanitation NWSC,KCC,andUWASNET Yemen Rural water The Netherlands Note: In addition to the country-specific partnerships listed here,WSP-AF works withWUP, UNCHS, PPIAF, Cities Alliance, Urban Management Program (UMP), Municipal Development Program, Africities, HTN, UNICEF,WaterAid,WSSCC,PSO,EcosanNetwork,SANDRES,SANDEC,ODI,PSI,PSEau,ITNs EASTASIA-PACIFIC Cambodia Rural/Policy reform Localpartners,WorldBank,ADB,UNDP,SIDA, WHO, UNICEF, and selected NGOs Indonesia Urban/Rural/Policy reform AusAID,World Bank, and UNICEF LaoPDR Urban/Rural/Policy reform Sida,ADB,UNDP,JICA,WHO,UNICEF, World Bank, and selected NGOs The Philippines Rural/Knowledge sharing AusAID,World Bank, ADB, and GTZ Vietnam RuralWater and Sanitation AusAID,DANIDA,FINNIDA,andUNICEF LATIN AMERICA & CARIBBEAN Bolivia (up to Peri-urbanWSS, some rural, SDC, WB operations, governments, water 2001) policy support (standards, etc.) utility companies, SIDA, GtZ, local and national government and private sector bodies, and NGOs Peru Urban, rural, small towns, CIDA, SIDA, SDC, ACDI, national government policy support and private sector bodies and NGOs,WB, USAID,PRONASAR,andPROPILAS Ecuador Investment support for rural SDC, local NGOs, local government, andWB (until 2002) and small-townWSS operations Central Policy advice (supporting SDC, GtZ, national government bodies, and America: legislation), support for NGOs Honduras and design of SIF projects Nicaragua SOUTHASIA Bangladesh WSS sector reform ADB,AusAID,DANIDA,DFID,SDC,SIDA,UNICEF, World Bank, and selected NGOs India Rural water and sanitation ADB,AusAID,CitiesAlliance,DANIDA,DFID,SDC, SIDA, UNICEF,World Bank, and selected NGOs Pakistan Institutional reform ADB, UNICEF,World Bank, and selected NGOs, DFID, and SDC Sources:Regionalevaluations,andinformationfromWSPheadquartersinMay2004 External Evaluation Report 23 can be seen as indicating that the Program is a part After recovering from a difficult period in FY02, of the Bank after all. Some small presentational when funding was at a low ebb, creating problems aspects give signals to the same effect: the visiting of continuity, particularly inWSP-LAC, funding has cards used by WSP staff bear the words become more stable in FY04. As of December 2003, `administered by the World Bank', in very small WSP has assured funding through FY07 of printbutaccompaniedbyaprominentWorldBank over US$ 48 million9, and additional probable logo;somepublicationsrefertotheProgramas`the contributions of a further US$ 14 million. These Water and Sanitation Program of the World Bank'. funds are from bilateral donors, typically in Consequences are discussed in the next chapter. three-yearagreements.Additionally,theWorldBank purchasesWSPstafftime(so-called`crosssupport'), WSP has a decentralized structure and style, with a whichamountstoUS$0.4-0.5millionperyear.This considerable degree of autonomy for the regional translates to approximately US$12.5 million per officesandtheirmanagers.Manyofthedonorsalso yearforFY04toFY07fromtheassuredcontributions, operate in a decentralized manner, with some or over US$ 15.5 million per year if the additional autonomy for their offices in various countries and contributionsmaterialize.Thecurrentexpenditure regions around the world. WSP thus relates to its rate is about US$ 14 million per year. donors at two levels ­ globally through the Council orbydirectcontact,andregionallythroughcontact A distinction is made between `tied' and `core' between leaders of the Program and the donors in funding. Tied funds are allocated by their donors to theregions.Thismulti-levelpartnershipstyleiswell specific regional, country, or project activities. Core anchored in the Charter's clause 8. One of the main fundingisnottiedinthisway,andisusedtosupport instruments in WSP's decentralized governance is global initiatives and for management and the use of Regional Advisory Committees (RACs). administrative costs. Additionally, the core monies ActiveparticularlyintheAfricaandEastAsiaPacific provide funds for regions during periods of fiscal regions,thesebringtogetherannuallyawidegroup shortfalls,andalsocomprisethe`contingencyfund', ofWSP'spartners,clients,andothersectoractorsto asourceofmoneyavailableforemergencysituations. discuss the Program's annual work plan, to hear Over the past few years, donor support has explanations of its priorities and approach, and to increasingly come from the donors' regional and provide a forum for stakeholders' needs. They help country budgets, and less from their headquarters. toavoidoverlappingactivities,andtopromotewide This has resulted in proportionately more `ownership'ofthepartnership'sworkineachregion. contributions being tied. Of the US$ 48 million, At present, there is no formal mechanism for them about three-quarters, or US$ 34 million, is tied. Of to report to the Council. the US$ 14 million core funding, about US$ 8 million is core contributions from donors, and 3.5 Funding and Finance about US$ 6 million comes from the 12 percent WSPexpendedsomeUS$52millioninthefouryears Global Program Management fee that is levied on FY 2000 to FY 2003 inclusive, an average of every trust fund. This level of core funding is just US$ 13 million per year but varying from US$ 12 to sufficient to maintain the HQ operations, support 15 million; details are summarized in Appendix 4. regional operations, and build up by FY03 a The relative expenditure in the four regions and contingency fund of some US$ 0.5 million. global work were: A commentary about the balance between rural, Africa,28percent urban, and small-town themes, also covering SouthAsia,22percent cross-cutting and other categories, is shown in EastAsiaPacific,20percent Section 4.2.2. LatinAmericaandCaribbean,12percent Globalthematicprogram,about10percent In recent years, the Program's maximum Globalmanagement,about9percent expenditure rate has been about US$ 15 million. 9This funding is comprised of US$7.8 million carried over from FY03, and annual total contributions of US$ 17.8 million (FY04), US$ 11.8 million (FY05), US$ 8.1 million (FY06), and US$ 3.1 million (FY07). 24 External Evaluation Report We consider that, if it were justified on grounds of consultant reviewed WSP's governance, focusing need and comparative advantage, an increase up particularly on the operations of the Program and to the order of 30 to 60 percent could be managed its Regional Advisory Committees. More recently, without the major restructuring of the institution several World Bank evaluations of aspects of and its ways of working. its work have covered some of WSP's activities and outputs. 3.6 Previous Evaluations WSP has been either the sole subject or one of the WSP has taken some important decisions based on subjectsofmorethan10evaluationsofvarioussorts the evaluations.The most important of these relate inthelast10years.Thesearelistedanddiscussedin to the Program's planning systems and the Appendix 3. An overall one was conduced for WB replacement of the PAC with the Council. However, and UNDP in 1996, well before our review period, ithasnotacceptedallcriticismsandhasonoccasion and in 1999 a series of evaluations of particular responded to evaluations with detailed rebuttal of regions were done for various donors. In 2000, a many of their points. External Evaluation Report 25 4 EVALUATION OF THE FIVEYEARS FY 1999 TO FY 2003 4.1 Introduction to this Chapter add policy reform to that list too, although that was Chapter 4 uses the framework, terminology, and already established as one of the objectives with its criteria set out in Chapter 2 and Appendix 1 to explicit product line. evaluatetheProgram'sworkdescribedinChapter 3. Discussionofstrategicandplanningissues,together Later statements indicate some improvement in withmonitoring,isinSection 4.2.Theevaluationfor clarity. The Strategic Results Framework of about thefive-yearperiodisthenpresentedinSections 4.3 2000wasdrawnupasahierarchicalends-and-means and 4.4, using the aspect titles explained in tree, with higher and lower levels of objectives, the Appendix 1, plus some additional headings and lattercorrespondingtooutcomesinourframework sections. Section 4.5 brings together some general andclearlydistinguishedfromoutputs.Theconcept conclusions of the evaluation in a discussion of of indirect work (`we help our clients to help their comparative advantage. The nomenclature and list clients,thepoor')hasbeenarticulatedfromtimeto of sub-sections fulfil the requirements of this time but not clearly built into strategy statements. evaluation's Terms of Reference, while adding some TheProgram's`missionstatement'(which,asabrief further aspects where we consider it useful. This slogan encapsulating what the organization aims chapter is based on the corresponding chapters of for and does, should not be expected to distinguish the four regional annexes, which provide some levels) has occasionally been supplanted or details.Mostsectionsinthisevaluationchapterhave replacedbyothers,butinourviewthishasnotmuch briefstatementsofourconclusions,inboldtypeand clarifiedtheProgram'srole. usually at the end of the section. Until recently, mostWSP documents did not make acleardistinctionbetweenoutputs,outcomes,and 4.2 Strategic and Planning Issues impacts.Wenotethatsinceabout2003,theProgram 4.2.1 Strategy and the has, for planning purposes, been using an Definition of the Program's Role outcome-oriented approach called Results Based ThewrittenstatementsaboutWSP'sobjectivesand Management (see Appendix 3 and Section 4.2.3 waysofworkinghavebeendescribedinSection 3.1. below).Theolderstrategicstatementsfrom1998and Inourview,theyshowacertaindegreeofconfusion, 2001, described in Section 3.1, are not consistent though it is debatable how serious this is. This is with this, and we feel there is a need for an updated particularlytrueofthe1999-2003strategydocument, and clearly articulated statement of objectives and whichhasnotbeenmuchusedanyway,buthasnot strategy, to underpin the improved planning been replaced by a single clear strategic statement. methods and to communicate whatWSP does, and Confusion is also found in subsequent statements, why, to new staff and to outsiders. It can be argued insomeoftheBusinessPlanformatsthathavebeen that the diversity of the situationsWSP faces in the used over the last six years, and in the ToR for this regions and countries would make too rigid a evaluation. Table 3.1, which quotes that strategy standard strategy statement counter-productive. document, illustrates this. Its three central lists, However, we feel that a well-structured statement called `mutually supporting objectives', `product could be formulated to avoid this problem, for lines', and `learning agenda', though there are instance by incorporating the fact that there may implicit ends-and-means chains between the first be several alternative means of achieving a single two, do not make such linkages clear and seem to end, each appropriate in a different situation. New havebeenusedbyWSPasunstructuredlistsofideas. slogans or `mission statements' should grow out of Theotherwisevaluable2001strategyupdateadded such a strategy statement, not substitute for it. `sanitation and hygiene' to the list of learning agenda, which had until then been a spatial Any updated strategy formulation would need to distinction.10Later there was a short-lived move to reflecttheemergingemphasisonwhatisnowavery 10WSP Renewal: Strategy Update and Progress Report; prepared for the October 2001 Council meeting at The Hague: Section 2.2. 26 External Evaluation Report pressing need in the sector, namely addressing of different`product lines', the design and targeting massive and widespread institutional failure and of publications, and to micro-decisions within the ineffectiveness in theWSS sector.While retaining a design of individual interventions. particular pro-poor character in its objectives, WSPshouldstrengthenthissortofwork,whichwill An important aspect of focus is the choice of improveWSS provision generally, not only for poor countries in which to work, and how many they people. Although this has been a major part of shouldbeinaworldsituationwherelackofadequate WSP's aim since at least 2001, a clear strategy WSS for poor people is widespread. In this matter statement with this emphasis might help to of geographical focus the Program is especially stimulate more funding. dependent on the policies and priorities of its donors. There have been suggestions that WSP We have discussed our perception of confusion in should become active in other countries, notable strategic statements with WSP people. The largeandimportantonessuchasNigeriaandChina. headquarters team does not agree with our However,theProgramcannotdothatwithoutsecure perception; it considers that an adequately clear and medium-term donor support and funding on vision of what the Program aims for and does has an adequate scale; to go into such a country with been discussed and internalized by all its staff, and just one or two people and a small budget would that this has guided them adequately through the notserveausefulpurpose.WSPhasclearcriteriafor last five or six years. At a pragmatic level, this is choosing countries, emphasizing the long process largelyborneoutbyourevaluationofthework,but of building up trust, which were discussed by the we also note that many of the Program's regional Council in 2002. The current way of doing things, staffhavewelcomedourevaluationframeworkina especiallythroughdialoguebetweenWSPanddonor way that seems to us to indicate a certain unfilled officers in the countries and regions themselves, need on their part for structure and clarity. Some seems to us the right way to proceed. WSP people at the center and the regions have told usthattheyhavedifficultyarticulatingtheirstrategy. By its definition,WSP has focused onWSS for poor There is also the question of the perceptions and people.The effectiveness of this focus is debatable, understanding of outside parties and of since the poor are often served by the same stakeholders, which cannot be helped by vague or institutionsandinfrastructureasricherpeople,and general improvements may serve the poor as well confused statements even if the insiders are orbetterthannarrow`pro-poor'interventions.One themselves clear. Strategic thinking in this sector is area where the Program's focus is sometimes notasimplematter,butitsinherentcomplexitycan questioned by stakeholders is the balance between be exaggerated by opaque or vague terminology. different sorts of urban WSS. In some countries, We conclude that there was at the start of the most of its urban interventions concentrate on review period an underlying lack of clarity about places where people live on officially recognized ends and means, especially in WSP's written plots of land, so that municipal authorities have a statements, and that subsequent formulations statutory duty towards them. In a situation where have not effectively replaced those statements. At people are spontaneously migrating from rural to a practical level, the consequences were not very urban areas in search of work, however, there will severe because the Program's implied objectives always be some people, such as newly arrived were usually understood by its staff. migrants, who live in informal or illegal `shanty towns'. Serving these people is often hampered by 4.2.2 Focus and Balance political or legal constraints. It is questionable Focus and balance are very difficult aspects for any whether,inthelongerrun,theurbanpoorarebetter smallorganizationworkingmainlybypartnerships served by focused attention to the immediate anddependentonmanyfundingsourceswiththeir problems of illegal settlements, or of poor people own varying priorities. WSP management takes moregenerally,orperhapsbyimprovingtheoverall great care with these matters. Focusing decisions effectivenessofurbanserviceproviderseventhough aremadeatseverallevels,fromthechoiceofwhich thiswillbenefitrichpeopletoo.Theanswerisbyno countries to work in, to the balance between rural, meansobvious,andneedswell-informeddiscussion smalltown,andurbanwork,therelativeimportance in each country, based on a clear understanding of External Evaluation Report 27 interrelatedobjectivesandcause-and-effectchains. services for the urban poor, enabling consumer Our view is that the Program is right to emphasize voice, and promoting condominial solutions. the significance of improvements to the effectiveness of urban service providers, in The rural and urban work were augmented with their entirety rather than just the bits that deal with the hiring of a Global Rural Thematic Coordinator the poor. Increasing urgency of urban work is, of and a Global Urban Thematic Coordinator (later course,noreasontoreduceeffortsonruralandsmall replaced by a Sanitation Coordinator). The urban town work. program has generally been slightly smaller than the rural program. At the region and country levels, the need to use resources efficiently means that Program activities As rural populations continue to leave the often tend to focus on one particular sector. The countryside, small towns have grown rapidly. This region annexes of this report give some place- has put a considerable strain on the water and specific comments about focus. WSP is very sanitation sector in such towns, where the conscious of the need to focus its limited resources population density and size of conurbation may be where they will best serve its objectives, and tries too high for rural approaches but too low for a to do so partly through its internal Business Plan standard urban utility approach. Towards the end process and partly through discussions with of the evaluation period WSP began dedicating donors, since in the end it is mainly the donors specific resources to looking for solutions to the who determine what will and will not be uniqueproblemsofsmalltowns,includinghosting implemented. Of course donors review their a small towns conference in Addis Ababa in 2003. policies from time to time, and may decide to shift ReporteddistributionofWSP'sexpenditurebetween the focus of their support.They need, however, for the three spatial themes rural, urban, and small the sake of their own objectives, to discuss such towns, plus other themes, is given in Appendix 4. shifts in good time withWSP, and generally to plan Year-to-year comparisons must be made with for four or five years in advance because that is the caution, since the figures depend on precisely how lifetime of the more usefulWSP interventions.The expenditures were classified each year. In our view withdrawal of support for continuing urban work thisclassificationofexpenditureisnotuseful,since in Bolivia in 2001, for example, appears to have mostofthesignificantaspectsoftheProgram'swork taken WSP by surprise. Many donors, in the cut across the spatial distinctions. (The MDGs do interests of their own focus, have restricted lists of not differentiate between rural, urban or small countries where their funds are to be used, and towns, and some countries, such as Pakistan, have within the framework of its own criteria the stopped using the spatial distinction.) In the past Program has done well in accommodating this and five years, the Program has rightly been allocating maintaining its own balance by working with moreresourcestothenon-spatialor`cross-cutting' several donors in each region. themes such as institutional reform, finance, poverty reduction strategy papers, sanitation, and Yet another aspect of focus and balance is that gender.These are discussed in Section 4.4.5. betweenrural,urban,andsmalltownwork.Despite urban migration, the vast majority of the world's In our view, the Program has succeeded in poor population continues to reside in rural areas. maintaining a good focus on its declared core Serving the rural poor has traditionally been the objective, namely helping its clients and partners focus of WSP's work, and remained the dominant to achieve sustainable access to WSS for poor portion of the workplan in the evaluation period. people. The focus has been sharpened over the evaluation period, both geographically and At the beginning of that period, WSP's urban work thematically. The continuity and hence focused on urban environmental sanitation (UES), effectiveness ofWSP's focus would be helped if its which primarily concerned municipal solid waste. donors could find ways to plan any change of However, as it became clear that the urban WSS priorities several years ahead. problems were wider, the Program expanded its workplan to cover WSS for the urban poor more 4.2.3 Planning, Monitoring, Internal Evaluation generally. This led to important work on financing As mentioned in Section 3.1 and described in 28 External Evaluation Report Appendix 2, WSP's work was defined over the but trying to ensure more consistent application of five-year evaluation period in a series of Business outcome-oriented planning and project design. Plans,whosenatureandformatchangedeveryyear. We welcome both these developments. Projects or activities seem often to have been planned for several successive years, but since The present planning procedure is somewhat subsequent Business Plans did not follow on from inflexible, especially regarding project budgets. their predecessors at the project level, monitoring Development is an inherently uncertain business, projects against those plans over the years was andnewprioritiesandconcernssometimesemerge seldom possible. This appears to have hampered unexpectedly. There has been a tendency in recent WSP's internal monitoring, which was reported in years, not just inWSP, to attempt to specify the way mid-year and end-of-year reviews.11 We have not money should be spent in increasing detail, even seen much evidence of useful monitoring against down to the number of days to be spent by each plans in the regions or at project level, though the person on each activity. This may be useful as a global reports summarize regional results, for management tool, but not as a criterion for instanceintermsofnumbersofprojectsorproducts monitoring,andnotwhenitinhibitsresponsiblere- delivered on time (in the earlier years the work was allocation of resources in response to changing defined mainly in terms of products). conditions. To some extent the budget rigidity is imposed on WSP by donors, who in turn may be Consistent multi-year work programming is subjecttotheirowngovernmentrules,soitisdifficult needed because most useful WSP interventions to avoid. But one of the comparative advantages of takeatleastfiveyearstobearfruitandreachastage the Program should be its ability to respond to where they can be considered finished and can be emerging situations more flexibly than larger and closed down, as WSP projects, without loss of morebureaucraticorganizations.Agoodexampleof valuablemomentum.Thatstagecanbeconsidered what can be done is the innovative agreement reached when other actors have taken over the between AusAid and WSP-SA, which provides a process thatWSP initiated, so that it is sustainable flexible fund that can be called upon to support any without WSP. activities which comply with certain agreed criteria. WSP management has been aware of the Flexibility in planning and budgeting is not always shortcomings of the Business Plan process and beneficial, of course, since it can provide formats, and over the evaluation period took steps opportunities for vagueness, undisciplined use of to improve it (details are in Appendix 2; the resources, and loss of accountability. All these can, improvements resulted partly from the piecemeal however, be countered by simple and pragmatic evaluationssince1999,asdescribedinAppendix 3). procedures for authorizing any reallocation Some pre-packaged and formalized procedures, ofresources. suchaslogframesorthe`results-basedmanagement Themid-yearreviewforFY 2004(outsideourreview framework', were brought in by outside specialists period) indicates that flexibility may be improving, and proved inappropriate when used in their and that advantage is now being taken of the much entirety,butoverafewyearsWSPstaffhavedistilled improved definition of projects in the Business out their useful features and rejected the rest in a Plans. That review's overall report includes details sensible manner. The latest version of the process, of projects that have been dropped or modified for embodiedinBP-FY 2004whichwaspreparedduring various specific reasons, which is a welcome our evaluation period but applies to work outside indication of some flexibility. it, is a considerable improvement on the previous few years, and is generally welcomed by WSP staff. Itisevidentfromourdiscussions,andborneoutby It would, however, benefit from some tightening of documents, that the willingness of different WSP definitions and objectives.We understand that the staff to undertake coherent planning and current preparation of BP-FY 2005 builds on that monitoring is very variable. At least in recent years, previousplan,notchangingitsformatsignificantly headquarters staff tend to try hard to achieve 11There were also Annual Reports, but these were for general information and not intended to evaluate performance against Business Plans. External Evaluation Report 29 systematic and coherent planning, while some hygiene, and development. The sector advisory regionalstaff,underpressuretokeepupmomentum supportworkisequallyrelevant,especiallyinAfrica. on their projects, see this as an unnecessary and Knowledgeandcapacitybuildingiswelltargetedat time-consumingtaskthatdistractsthemfrommore sector politicians and practitioners. Work in Latin usefulthingsandalsoinhibitstheirfreedomtowork America has been focused on countries with creatively. We consider that better application of significant numbers of poor people, and within clear but flexible multi-year planning, with a thosecountrieshasusuallybeenfocusedonservices sensibledegreeofrecording(inBusinessPlans)and for those people, with imaginative and useful lines subsequent monitoring, would enhance efficiency of work. and effectiveness enough to repay handsomely the timeittakestodo.Wealsoconsiderthatthepresent The Program has, on occasion, taken the lead in Business Plan format (as in BP-FY 2004) is a good promoting and developing new ideas, for example one, and when consistently used at project level withtheworkonstrengtheningtheWSSorientation should serve the purpose well and not waste time. of Poverty Reduction Strategy Papers in Africa. It is difficult to guarantee at the outset that such We conclude that planning and monitoring were initiatives will be effective in achieving useful weak in the earlier years of our evaluation period, outcomes and impacts, but the potential for the but that WSP has taken effective steps to improve futureisvaluable. themandtheyarenowmuchbetter.Thereremains a need to consolidate cost-effective ways of WSP's work on disseminating new knowledge and combining coherent, outcome-focused, multi- changingattitudesisalsooftenrelevanttoitshigher year planning with creativity and adequate objectives. There have been clear attitude changes flexibility, and to achieve consistent application insomeareas.Forinstance,Kenyanofficialsarenow ofsuchplanningandmonitoringproceduresover prepared to consider the extension of water and all regions and all projects. sanitation services into informal areas whereas the prevailing government view, even as late as 1997, was that such areas were completely illegal and 4.3 Evaluation against Our Framework should not be tolerated. 4.3.1 Relevance The successive subsections of this Section 4.3 WSP clearly has a comparative advantage for the evaluate the Program's work against the five-level workitdoes,becauseofitsglobalextentdespitethe hierarchy of Table 2.1, using the terminology limited number of focus countries. Another reason explained in Appendix 1. This sequence of is the experience of many of its staff and their subsections begins with relevance and ends with knowledge of where to find the expertise of others, the related topic of impact. aswellasitsskillindrawingoutandcommunicating lessons from one place and time to another. Also, Relevance is a measure of whether or not a set of because of its practical knowledge of participatory outputs and outcomes (if achieved) are likely to and collaborative methods and their integration contribute to an appropriate higher-level goal.The with appropriate and sustainable technology. extenttowhichWSP'sworkmatchesitscomparative In general, the Program has positioned itself to use advantage is also important. We evaluate the its comparative advantages in its regions Program's relevance in the five years against our and countries. understanding of its objectives as written in its various statements discussed in Chapter 3, and as The Program's main lines of work have thus impliedbyitsBusinessPlansandchosenpriorities. strengthened each other and have been relevant to the stated objectives.WSP's work is directly related Many examples in the annexes illustrate a good toTargets 10and11ofGoal7oftheMDGs,andalso degree of relevance in this sense. The rural water serves several other of those goals more or supply programs in India and Bangladesh, and less indirectly. formerly in Pakistan, Nepal, and Sri Lanka, are addressingtheissuesoflowWSScoverage.Further, There are a few activities of doubtful relevance, thefocusonruralsanitationaddressestheproblem for instance the work on solid waste management of low coverage and the implications for health, in Bangladesh, but the positive examples 30 External Evaluation Report predominate. We conclude that the large majority Sinceeffectivenessdependsonthereactionofother oftheProgram'sworkishighlyrelevanttoitsstated actors toWSP's work, the Program's relationship to and implied objectives. thoseactorsisimportanthere.WSPnotonlyrelates to other actors one by one, but also influences the 4.3.2 Effectiveness way they perceive and work with each other. In Effectiveness is success in crossing the important manycontexts,especiallyintheLACregion,people interface between outputs (under the control of regard WSP as an `honest broker' and a valuable WSP) and outcomes (outside that control).The test forger of links. In other contexts, the Program is ofeffectivenessiswhetheraWSPprojectoractivity, sometimes seen, justly or not, as a partisan player having achieved its outputs, has a lasting outcome in a conflict situation or a clash of attitudes to tendingtowardsimpact.Itisanimportantmeasure development (especially where the World Bank is ofhowwellaninterventionisdesignedandadapted perceivedaspartisanandtheProgramisseenasits to its context, and of how well WSP relates to the close ally). This requires sensitive handling by other actors whose changed attitudes or behavior regionalWSPstaff,anditmustnotbeassumedthat determines the outcomes. what works in one place will work everywhere. The interaction of different lines of work (`product Perceptions and attitudes of NGOs are also lines'inWSPterminology;seeTable 3.1,eventhough important. Institutional reform and strengthening thelistoflinesisnotideal)meansthattheprocesses is a politically sensitive process, and here WSP can leading to outcomes are not simple and linear.The be most effective when it establishes a close and chain from action to outcome does not lead once trustingrelationshipbothwithdonors/fundersand and neatly up the hierarchy, but loops back and withgovernmentofficialsandotherlocaldecision- across between the `product lines' such as makers. Good examples are work on rural WSS in policy support, knowledge-management, and Peru, and some urban work in India. Due to the capacitybuilding,eachenhancingtheeffectiveness political sensitivity of institutional reform, good of the others. This is an endorsement of WSP's opportunities forWSP to achieve lasting outcomes product lines, which are most effective when used tend to be quite rare. in parallel. WSP contributions to projects and programs Among projects close to the WSS-provision end of supported by the World Bank were often very the project range, the most effective ones were effectiveanduseful,because: probably those relating to outcomes to which Links with Bank-funded projects often provide governments and other local stakeholders were opportunities for WSP's limited funds to be used already committed. This confirms the importance strategically to leverage benefits from the much largerfundsavailablethroughaBankloan. of the upstream persuasion and attitude-changing Duetoitsreputationasan`honestbroker'andits workthatWSPincreasinglyundertakes,oftenunder linkswithlocalstakeholders,theProgramwasoften the title of policy support. A range ofWSP activities in a position to achieve results that would not have contribute to the development of ideas and been possible for the Bank working alone. attitudes, and the creation of `informed demand' WSP involvement in the design of World Bank- for appropriate outcomes among national and fundedprojectshasoftenmadethembetterfocused regional stakeholders. These include engagement onpoorpeoplethattheywouldotherwisehavebeen. with stakeholders through country-level WSP's work in this area is most effective when it coordination support, the dissemination of reform guides or challenges, rather than follows, the usual proposals through publications and workshops, orpriorwayofthinkingoftheotherpartiesinvolved technical support to reform processes, regional in project design. advocacy, financing mechanisms, and incentive structures. WSP also rightly emphasizes the The Program works in an environment importance for lasting outcomes of ensuring that characterized by uncertainty and resistance to different thematic and country products change.Insuchcircumstances,effectivenessislikely complement each other, for example by a country to be enhanced if procedures allow flexibility to strategy that links regional work on WSS in PRSPs revise projects in the light of the information and with country-specific work on the same subject. understanding that accrue as they unfold. External Evaluation Report 31 Effectiveness may be undermined if standard differentiated job codes on staff time sheets, it has procedures endeavor to tie down activities too notbeenpossibletorelateparticularoutputstothe tightly. WSP's mid-year and end-of-year reviews quantitative inputs that produced them. Checking provide opportunities to assess progress for ofwholeprojectsagainsttheirpreviouslyestimated individualprojects,andtoconsideranychangesthat inputs has usually been possible forWSP, and as far mightincreaseeffectiveness.Formostofthereview as we are aware no serious problems have period, however, these reviews focused, at least at been detected. projectlevel,onspecificproductdeliverablesrather than outcomes, and were not systematically used Time and the availability of detailed records over to improve effectiveness. Recent changes to the the five years do not permit us to check all projects internal reviews have begun to improve this. onebyoneforefficiency,sowebaseourassessment of efficiency on looking atWSP's work over the five In finding that WSP's work is generally effective in years and four regions, and studying a number of leading to outcomes, we note that this involves projects more closely. In the case studies that we matching approaches to conditions in the very analyzed, we detected no significant waste of WSP different regions. In EAP, the Program works largely resources,thoughinevitablysomeactivitiesaremore byinfluencingprojectsinruralandperi-urbanareas efficientthanothersare.Itcanbearguedthatability of participating countries. HereWSP advocates the to `stay the course' over the period required to use of community-driven development and self- achieve an outcome is more important than a provisioning approaches (including a high level of narrow view of efficiency. The sharpening of the facilitation and dialogue with communities in the Program's focus over the period of the evaluation development of, and presentation of choices to hashelpedtoensurethatscarceresourcesareused potential beneficiaries) for both water supply and as cost-effectively as possible and so should sanitation. In some places, especially the LAC contribute to the efficiency of the Program. regionwhichwasaffectedbyafundingcrisisin2001, effectiveness has sometimes been limited by a lack Use of bottom-up, demand-driven approaches, of continuity over several years, but was still instead of top-down, supply-driven approaches, achieved in many ways. These included beneficial maybesomewhatslowandlogisticallychallenging influence on the design of investment projects, in scaling-up, but the former are much more WSP'sattitude-changingwork,andtheintroduction sustainable and replicable when well done, which of sustainable methods in both management and raises their effectiveness and outweighs any loss of crude efficiency. In the LAC region particularly, technical fields. Further discussion is in the it was noticeable that WSP's collaborative way `effectiveness' sections of this report's four regional of working naturally tends to produce annexes (available on request). considerableleverage. We conclude that most of the Program's work was In the AF and LAC regions at least, the imaginative effective in achieving the desired outcomes, over useofmedium-termcontractsfundedbyindividual all product lines and regions. We do not detect international agencies, interns and short-term significant time trends within the five years. The consultants helps to ensure the flexibility without main limits on effectiveness were inflexibility of which efficiency would be impossible, given the budgetcontrolandlackofpredictablecontinuityof uncertainties about funding. The Program funding, both of which are often imposed on the endeavors to optimize the use of scarce resources Programbydonors. by identifying ways in which costs can be reduced 4.3.3 Efficiency without compromising effectiveness. For instance, Efficiency, or the cost-effectiveness of converting WSP staff often use cheap airfares, and emphasize inputs to outputs, is difficult to assess where many the need to ensure that procurement processes activitieshavetobecarriedoutformanyyearsbefore represent value for money. Efficiency in the use of a desired output is achieved. In addition, WSP staff time is usually good, and although those partnership activities tend to use resources from staff are expensive compared with some other several sources. Even for WSP's inputs alone, organizations in the sector, their caliber and ability because of the lack until recently of finely toproduceusefulworkfastcompensatesforthisso 32 External Evaluation Report that the Program, in our view, provides good value boundaries; RWS development approaches have for the donors' money. been evolved through an iterative learning and replication process spanning India, Nepal, Sri We conclude that WSP's efficiency, though Lanka,andPakistan.Sustainabilityismoredifficult inevitably variable, is generally high. The overall to assess, but good reports are being received from effect achieved by a small number of people is the Swajal Project in India. In the LAC region also, impressive. As always, efficiency could in principle deliberate replication between countries, with due beimprovedbyanarrowerfocus,butfocusisawider attention to varying contexts and adequate time, issue than that. Improved multi-year planning has shown considerable success, and some clear could also make slightly better use of resources. evidence of sustainability. That sustainability is 4.3.4 Sustainability and Replicability much helped by WSP-LAC's skill in promoting Sustainability is the capacity of an intervention to compatible and mutually reinforcing packages produce continuing impact over many years, and of technical and management-institutional replicability is its usefulness over a wider innovations.InEAP,thebottom-up,demand-driven geographical area than the site of the intervention. approach for community WSS service delivery appears to be genuinely sustainable and replicable Questions related to sustainability include: when well implemented. Influence on national Is the change achieved through the initiative policy,norms,andlawsisalsopositive,forinstance sufficiently robust to prevent it being gradually involving PRSPs in Africa and SIFs in LAC. WSP's eroded over time as those who are opposed to collaborationwithinternationalNGOs,suchasCARE change seek to return to the previous status quo? orWaterAid, also helps to propagate lessons widely. Do the stakeholders who are seeking to sustain and replicate change have the tools, in terms of We conclude that WSP is achieving both attitudes, knowledge, and systems, required for sustainability and replicability in nearly all of its thistask? work; this is commendable in a sector where sustainability is difficult to achieve. With regard to the first question, the strong WSP emphasis on supporting or `upstream' processes 4.3.5 Outcomes and Impact of that lead to sector reform means that, if successful, Program Interventions individual initiatives are likely to be effective in Since effectiveness, relevance, sustainability, and replicability all concern outcomes and impacts, supporting movement towards the higher goals. these latter topics have already been discussed to a The indications are that this emphasis has become considerable extent above. This section seeks to more pronounced over the period covered by the bring together a few conclusions about them. Our evaluation. The second question points to TermsofReferenceidentifythreequestionsrelating the importance of capacity building and to the impact of Program interventions: attitude-changing for achieving sustainability Whathashappenedasaresultofanintervention? and replicability. Whatrealdifferencehasthatinterventionmadeto Assessment of sustainability and replicability can its intended beneficiaries? only be definitively done several years after the Howmanypeoplehavebeenaffected? completionofanintervention,andwehavenotseen The first question is more concerned with outputs any deliberate evaluations or lesson-learning and outcomes, in our terminology, and can be reviewsofthisnature.Onthebasisofwhatwehave answered for most interventions. The second and read,seen,andheardintheregions,ourjudgement third relate more to impacts, which take time to be isthatmostWSPinterventionsareachievingagood take effect so that, in many cases, it is still too soon degree of sustainability and replicability in all the to provide a definitive answer to them. regions, or showing a strong likelihood of doing so in the future. There are clear indications that Program interventions have brought about changes, The annexes give some examples. One of the most including many at the outcome level. Our annexes interestingfeaturesoftheWSP-SAprogramhasbeen provide examples. Naturally, not all Program the replicability of approaches across national productshavebeenequallysuccessful. External Evaluation Report 33 The sanitation guidelines for Uganda provide an ultimatelymeaningthecapacitytoproduceimpacts example of a good output that was felt, both by whiletakingadvantageofitscomparativeadvantage government stakeholders and government staff, to and respecting that of others. As discussed in other have had less influence on field practice than had sections of this report, relevance needs to be been hoped. Another example of impact failing to assessed against a coherent vision of an reach what was hoped for is the provision of organization'sobjectivesandhowtheyrelatetoeach municipal capacity building materials for South other and to those of other organizations, and for Asia,thoughthecausesweremultiple.Themovein WSP that vision is currently unclear. the more recent Business Plans to view groups of products in terms of their contribution to overall In summary, the Program's work has evidently projectswillhelptoensurebetterimpactinthefuture. achieved relevant outcomes in diverse situations, with considerable impact already observed and An example of valuable outcome, with good morelikelytofollow.Someinterventionshavebeen prospects for impact, is the expanded rural water less successful than others on these criteria, but supply and sanitation program in India, with its that is to be expected in a difficult field and for an fruitful linkage with the Rajiv Gandhi Foundation. institutionthatisoftenleadingthewayratherthan The Government of India has embarked on a following proven recipes. national scaling-up program. While other parties were also involved, it was based on the policies and 4.4 Evaluation against Other Aspects approaches that WSP-SA had a major input in 4.4.1 Institutional and Governance Arrangements developing. WSP's current focus in Uganda is on TheProgram'sformalandgovernancearrangements capacity building for improved sanitation and this have been described above in Section 3.4. Some provides an example of an integrated process that stakeholders consider that it might be helpful to ensures that publications are linked to other field considerafewfundamentallydifferenthypothetical activities, thus increasing the probability that they models, each with advantages and disadvantages, will have a significant impact. toreplacethecurrentinstitutionalset-up.Examples In the EAP region, donors and participating might be: governments are largely adopting methods Independentfoundationstatus,notlinkedtothe promoted by WSP for community facilitation and WorldBankbutstillcontrolledbyacouncilofdonors and other stakeholders. This would requireWSP to user choice. These methods appear to be working compete on more equal terms with some other well in the field, in a variety of cultural and sector actors and might lead to more cooperative geographicsituations,toimproveprojectoutcomes perceptionsofWSPbyothersectoractors.Thismay and sustainability. WSP-EAP also reports positive makeiteasiertoapproachnewfundingsourcessuch changes in attitudes and practices among national as Bill Gates. governments and their participating institutions A private service provider to the sector, as`think following the use of MPA methods. When directly tank' or otherwise, according to the comparative involved in design of BankWSS projects,WSP-EAP advantages. hasbeenlargelysuccessfulinincorporatingproven methods. Extensive impact can also be observed in Having considered the potential merits of the LAC region, especially in conurbations in such arrangements, and the difficulties and Bolivia and Peru, where ongoing full-scale disadvantages that they would also entail, we do investments have been heavily and beneficially not consider that any fundamentally different influenced by WSP's earlier work on pilot projects, arrangement would be better than the current one diagnosticstudies,knowledgemanagement,policy whereby the Program is guided by a Council while support, updating of standards, and general being hosted and administered by theWorld Bank. attitude-changing. In view of the evident An organization of WSP's size needs to be sustainabilityoftheinterventions,greaternumbers logistically hosted somewhere, since it could not of people are likely to benefit in the future. cost-effectively set up a completely independent system for fiduciary control and accountability ThereisevidentlyaneedfortheProgramtoregularly that would command the confidence of donors. and consciously review the relevance of its work, The logistic, fiduciary, and administrative 34 External Evaluation Report arrangements of theWorld Bank, while somewhat Ourdiscussionswithstakeholdershaverevealedthat rigid and bureaucratic as is inevitable in a serious some people do indeed, for better or worse, see the organization of the Bank's size and turnover, are as WSPasanarmoragentoftheWorldBank.TheBank appropriate as any other set. Moreover, the itself sometimes encourages the same impression practical and functional advantages to WSP of by behaving as if it were in control of the Program, being located in or nearWorld Bank offices around rather than just administering it on behalf of the the world are considerable. The closeness usually Council. Some stakeholders also take the view that workstotheadvantageofbothparties.Weconsider WSPwouldbemoreeffectiveifitweremoredistinct that improvements to the present arrangement, from the Bank and less under the Bank's control. as discussed in this section, would be more useful Someholdthatoneofitsmostvaluablefunctionsis than any attempt at fundamental change. to challenge the Bank's ways of thinking and its approach to pro-poor service provision. What difficulties remain, under the present set-up, concern mainly the relationships between the Inpractice,theProgramisindeedlargelydominated Council, the Program Director and staff, and the by the World Bank, but this does not necessarily World Bank, and especially the way those indicate an inappropriate controlling urge on the relationships are (rightly or wrongly) perceived by Bank's part. Indeed, the persons on each side of the other sector actors. delicatedivide,namelytherelevantBankDirector12 (whochairstheCouncil)andtheProgramManager The 2001 Charter defines the governance with the (who is a former senior Bank official and is still on words `...directed by its participating partners and theBank'sSectorBoard)takesomecaretoseparate administered by and within the World Bank on their roles and distinguish between Bank and behalf of its donors', where the `participating Program interests and priorities. To the extent that partners' are presumably intended to be itdoessteerandevencontrolanddirecttheProgram, represented by the WSP Council. The meaning of the Bank does so partly because, in practice, this depends on what `directed' and `administered' no one is willing to do so adequately. (This also mean. This Charter sentence could be thought to reflects the Bank's high opinion of the Program's mean that WSP answers to its Council and is only usefulnessandcomparativeadvantage.)AllCouncil administered by the Bank, but the clause fails to membershavemanyotherrolesandactivities,and make that clear. Moreover, the Charter goes on to donotdevotemanydaysperyeartoWSP'sconcerns. saythattheWSPmanagerandstaff`reportto'senior We understand that some members are also World Bank officials. In many bureaucracies the constrained by the regulations and divisions of words `report to' are almost synonymous with `are responsibility within their own governments or controlled by'. If the intention was thatWSP would otherorganizations.SotheCouncilineffectchooses report to a Bank official on administrative matters, totakearelativelypassiveroleandtodelegatemuch while answering to the Council for other matters, of its authority and responsibility for WSP to the that has not been stated in the Charter. Taken Bank.Inmanyinstances,includingsignificantstaff together with the Charter's stipulation that the appointments, WSP or World Bank management Council is automatically chaired by a World Bank gives Council members opportunities to influence official, this can lead people to perceive WSP as decisions, and they opt not to do so. under a high degree of control by the Bank. While any alternative may be difficult to achieve, Thus, the Charter is not entirely clear about the we consider on balance that this degree of World World Bank's role in directing the thematic and Bankdominationmaynotbeaverysatisfactorystate technicalworkoftheProgram.Manypeopledonot of affairs. This is both because of its effect on the realizethat WSP'sBusinessPlansandotherstrategic functioningoftheProgram,andbecauseofitseffect documentsareapprovedandgivenauthoritybythe ontheperceptionsofotherparties.TheWorldBank Council. WSP sometimes says it is under the is a lending and funding organization, and is `oversight' of the Council, but this is not anchored naturallyaccustomedtoahighdegreeofcontrolover in the Charter and its meaning is not clear. what happens with interventions that it finances. 12Currently the Director, Energy and Water. External Evaluation Report 35 It is staffed by highly competent and motivated representative to speak more freely to the people, and is inevitably seen in some quarters as Council, or to its Chair between meetings, as the arrogant.ItisformallyassociatedwithotherBretton representative of one of the partners, and a very Woods institutions, and is perceived by some as a important one. Separating the Chair role from the partisan actor in wide global debates on such Bank should also reduce misunderstandings, subjects as private sector roles and globalization. improve the perceptions of outsiders, and enable The Program, on the other hand, is a partnership other stakeholders, whether represented on the andneedstohaveitsowndistinctpositionsonsuch Council or not, to approach the Chair more easily matters,especiallyprivatesectorrolesandpro-poor on matters concerning the World Bank. In policies where it innovates fruitfully in its very expressing this opinion about the chairmanship of particularsector.WeconsiderthattheProgrammight the Council, we are aware that any alternative beevenmoreeffectivethanitalreadyisifitsCouncil arrangementmightbedifficulttoachieve,giventhe took a more active part in directing its work. This current limited engagement of most Council would make WSP more clearly distinct from the members and the fact that all other Trust Fund World Bank, both functionally and in terms of the organizations hosted by the Bank have their perceptions of others, to the advantage of both. governing or steering bodies chaired by Bank officials. But we consider that an alternative The perceptions of other parties in the sector, and arrangement, such as an elected Chair, should at more widely, are significant. Negative perceptions, least be seriously considered. whether they are well founded in fact or not, can reduce people's willingness to co-operate in Within the decentralized internal governance of partnershipventures,orcanhamperthepersuasion WSP,therelationshipbetweentheheadquartersand and attitude-changing work of the Program. We the regions is both fruitful and difficult.The largely considerthatmoreattentionshouldbepaidtosmall autonomous regional managers (formally called matters of labelling, such as the use of the World Regional Team Leaders or RTLs) are senior and Bank logo onWSP products or visiting cards, or the experiencedpersonswiththeirowndistinctiveand use of standardWorld Bank e-mail addresses, with different styles and ways of working, and they face theaimofhighlightingtheProgram'sdistinctiveness. very different circumstances in their respective Theuse,evenbyothers,ofthemisleadingterm`the regions. WSP's headquarters staff, especially the Water and Sanitation Program of the World Bank' ProgramManager,havetotreadafinelinebetween needs to be avoided. trying to impose too much uniformity and central control on the one hand, or on the other hand AsubsidiarybutrelatedissueisthefactthattheWorld allowing healthy autonomy and diversity to merge Bank automatically chairs the Council, which we intoacounter-productivelackofaccountabilityand considerfundamentallyunhealthybecausetheBank consistency.Planningproceduresisoneareawhere has such a particular and special place among the headquarters has struggled, over the last five or six Program's partners and clients. We are not years of changing methods, to maintain the concerned here about who presides at Council necessary minimum of control over the regions. meetings, and the fact that the Chair is given to Diversity and creativity are sometimes misused as different members for different sessions appears excuses for avoiding responsible planning and satisfactory. Nor are we concerned with the style or disciplined use of resources in the service of clear actions of the current chairman. Our concern is goals.Weconsiderthatthecurrentmanagementhas ratherwiththeroleoftheChairbetweentheannual struck a good balance in this difficult area. If Council meetings when it is required to `provide anything, slightly more disciplined behavior by the leadership and resolve conflicts,represent theWSPC regions would be an improvement. in interactions with other WSP stakeholders...'. We see in the present arrangement the potential for The annual meetings of the Regional Advisory difficulties,especiallyfortheChairhimselforherself, Committees (RACs) represent a valuable element when `interactions with other WSP stakeholders' oftheProgram'sdecentralizedgovernancestructure, meansinteractionswiththeBank.Ideallywewould complementingthelessstructuredcontactbetween prefer to see the Chair elected by the Council for WSP regional staff and regional stakeholders, one suitableperiodsoftime.ThiswouldenabletheBank byoneorinsmallergroups,thatoccursthroughout 36 External Evaluation Report the year. Some mechanism for reporting the views experimentation and innovation, not at random of the RACs to the annual Council meetings might but relating to key sector issues. The Program's be an additional enhancement. responsiveness to opportunity is evidently appreciated by many of its sector partners, One practical consequence of the Program's status especially government people who work in is that its staff are formally employed by the World somewhat inflexible institutions. They tend to Bank, but do not have long-term contracts like turn to the Program with requests or ideas. normal World Bank employees. In fact, their The Program is usually able to respond, subject to contacts are typically for three years at a time thewillingnessofdonorswhichmaytakesometime becauseoftheshorthorizonofassuredfundingfrom toarrange. donors.This is discussed further in Section 4.4.7. TheProgramhas,insomeregionsandatsometimes We conclude that the fundamental institutional in the past, tried formulating Memoranda of and governance arrangements are satisfactory, Understanding (MoUs) signed with particular and for practical purposes the best available, but donorsorparticulardepartmentsoftheWorldBank. that the details could be improved and clarified. Althoughtheseshouldinprinciplehelptopromote InparticularthedegreetowhichWSPisinpractice and systematize responsiveness, they do not seem controlled by theWorld Bank should be discussed tohavebornemuchfruitbeyondayearortwo.(The by the Council and an agreed statement should recent ones with donors in South Asia and Africa replace the ambiguous provisions of the Charter. may be a positive exception.) WeconsiderthattheprovisionfortheCounciltobe automatically and always chaired by a World Bank TherehassometimesbeenatendencyforWSPstaff official could with advantage be changed. to follow current fashions of thinking in the development community without adequately 4.4.2 Responsiveness to Opportunity examiningtheirrelevanceandpracticabilityforthe The Program's responsiveness to opportunity has Program's particular work. This responsiveness to been good. This is partly from necessity, when fashion rather than opportunity can divert donors offer funds for specified sorts of work, but attention, distort priorities, and the design of WSP also talks to stakeholders and identifies projects, and also adversely affect other people's opportunitiesitself,sometimesthenbringingthem perception ofWSP. On the other hand,WSP is often totheattentionofdonors.Whenadonoroffersfunds innovative and sets fruitful trends which others tied to specific work, the most appropriate WSP follow. Examples are practical approaches to cost responsedependsonthecircumstances.Isthework recoveryandprivatesectorinvolvementinpro-poor proposed by the donor compatible with the WSS.OurviewisthattheProgramworksbestwhen Program's other activities? Will it take place in a it retains its confidence in its own judgement and country in which the Program already has a respondstorealproblemsandissues,onlyfollowing presence? The Program's response to SIDA's current trends when they match that judgement. proposalfor`ecologicalsanitation'inAfrica,namely Clear strategic thinking would help WSP people to negotiation leading to agreement on a project that distinguish appropriate ideas and ways, for the includedecologicalsanitationbutwasnotconfined Program's particular priorities and situations, from to it, provides a good model for the way in which inappropriate ones. such opportunities can be handled. Flexibility is needed to take advantage of Examples of good responsiveness are in the region responsiveness. Financial and budgetary rigidity annexes to this report. Contexts include is the main constraint on the Program's decentralization in Pakistan, Latin America, and responsiveness, and some possible improvements elsewhere; arsenic mitigation in Bangladesh;`Total are discussed below. Sanitation' in India; PRSPs in Africa. Sometimes, WSP identified fruitful fields of work that had not We conclude that the Program has shown a high been identified by other organizations. WSP's degree of responsiveness to opportunity and responsiveness to opportunity is often a listenswelltoitspartnersandclients.Moreflexible consequence of its ethos, in particular its budgetary arrangements might enable it to do recognition of the need to allow space for even better. External Evaluation Report 37 4.4.3 Finance and Resources are important to the working of the Program This evaluation does not include any financial becausetheyenableittoadjustbudgetstochanging auditing, but it appears to us that the finances are circumstances, within or between years, and on managed in an accountable manner. This is an occasion to bridge unexpected funding gaps advantage of having the Program administered by while minimizing the loss of staff, continuity, the World Bank. Transparency is not always and momentum. We consider that the Program's particularly good, however. Summaries for our effectiveness would be further enhanced if the evaluationperiodhavebeenhardforWSPtoprovide, proportion of untied or flexible funding could be andwehaveheardofacasewhereadonor'sauditors raised, provided that conscious measures were in hadproblemsobtainingfiguresforparticularregions place to avoid the increased financial flexibility or projects. This may be due to the rigid use of the being misused to allow vague or ill-considered World Bank's standard accounting procedures, resourceuse.Weareawarethatdonorsmayfacetheir which are not necessarily ideal for the Program's own constraints in reducing the extent of their purposes. It should not be difficult to provide control over use of their funds. necessary information in addition to the standard reporting formats, where there is a specific need, We have noted elsewhere that most of WSP's work and as far as we know any past problems have requiressustainedeffortoverperiodsoftheorderof already been solved. Efforts to control central fiveyearsormore,whichislongerthanmostdonors managementcostsovertheevaluationperiodhave areabletocommitto.TheProgramgenerallycopes succeeded in limiting them to about nine percent withthissituationbystartinganundertakingwitha of the turnover. nominal three- or four-year project, then following itwitharelatedprojectoranewphase,underanew The funding situation has been described in funding commitment though in effect a Section 3.5. Given the effectiveness ofWSP and the continuation of the same work. In the absence of largeglobalneedforsustainableandpro-poorWSS clearmulti-yearplanninguptoabout2003,thiswas (expressed in the ambitious MDGs), there was and done in a pragmatic and ad hoc manner, and with remains scope for WSP to expand its scale of the improved planning procedures now in place it operations, which was in 1999-2003 effectively shouldbecomeeasier. limited by donor funding and not by the Program's inherent capacity. We consider that the current We conclude that financial arrangements are management structure, with four regions, could satisfactory, that WSP could usefully increase the fruitfully handle an increase in turnover of up to scale of its operation by up to about 50 percent 50percent,iftheincreasedscaleofoperationswere without major changes, and that the proportion to be justified and the strategy were clear and of untied or core funding could with advantage convincing. This would, of course, depend on be increased. donor funding, and on being able to attract extra 4.4.4 Relationships between Actors staff of the necessary caliber and experience. An WSP is a small organization working through expansion much beyond that level might require multiplepartnerships,soitsrelationshipswithother somerestructuring. actors in the sector are crucial. They are generally good,andtheProgram'sroleishighlyappreciatedby The Program's responsiveness to opportunity, and many people in governments, the World Bank the continuity of its work at some times and places, centrally and regionally, international development havebeenlimitedbythefactthatalargeproportion organizations and NGOs, national and local NGOs, of its funding is tied by donors to particular and the private sector. Many sector actors take countries, types of work or projects. The trend for advantageofWSP,someevenapologizefordoingso, donors to make allocation decisions in regional but in our view the Program is there to be taken contexts increases this difficulty. WSP does have a advantageof.IfpeopletakeadvantageoftheProgram measure of untied or `core' funds, some given as a they are recognizing its comparative advantage. matter of policy by particular donors and some derivedfromthepercentageofallTrustFundmoney Wheretherearetensionstheyaresometimesdueto which is allocated to core funding. These untied insufficient clarity about objectives and how they funds,andpartially-tiedfundsallocatedtoregions, serve each other (Section 4.2.1), so that roles 38 External Evaluation Report become confused. Any institution working in sector participation, and if WSP is then seen as an development shares its higher goals with many uncritical follower of the Bank's line. The Program others, while lower ones are shared with fewer hasnotalwaystakenaconsideredandrationalview others, and some goals are almost entirely its own andhassometimesindeedbeentoowillingtoadopt niche. Poverty reduction, for instance, is one of the whatever attitude the Bank was displaying at any highest goals of WSP, sometimes articulated and one time. These problems do not, however, appear sometimes not. It is shared with many institutions to be widespread in any region. in the fields of food, health, wider infrastructure, economics,andfinance(suchasFAO,WHO,UNDP, NGOs:WSPworkscloselywithmanyNGOs,both ILO,IMF,GWP, innumerablebilaterals,andNGOs). international ones such as CARE, Catholic Relief ImprovedWSSisalowergoal,justoneofmanyways Services, Save the Children, andWaterAid, and also of reducing poverty, but it is still shared with many many local ones. We heard positive comments on institutions, including WSSCC and some NGOs, WSPfrommanyofthese.Theyoftenappreciatethe and private sector groups, some of whom by their Program'sworkaslink-maker­settingupnetworks, nature do not put central importance on poor establishingcommunicationbetweengovernment people. While looking up its objective hierarchy officials and NGOs, bringing NGOs' concerns to towards poverty reduction,WSP needs also to look bear in the design of the World Bank, and other across towards sister organizations that seek that internationally funded projects, initiating and same end by different means. By being clear about guidingcollaborationwiththeprivatesector.There which particular ends and means are its special remainsomesmallNGOs,especiallyoutsidethebig business and which ends are shared, with whom, cities,whicharenoteffectivelyreachedbyWSP(this and how, WSP could more clearly relate to and was emphasized to us in Kenya.) engage with the other actors, enhancing the Service providers, in both public and private complementarity and effectiveness of all of them. sectors:WSP'sworkhasbeenhighlyappreciatedby Remarks about particular categories of other the staff of some such organizations, who still sector actors: welcomecontactwithWSPpeopleandmakeuseof Governments:Thisisprobablythemostimportant the networks and publications even after formal WSP projects are finished. Aguas del Illimani, the category, since the crucial step from WSP outputs urban service provider in La Paz, is an example of a to outcomes and impacts often depends mainly on large private sector body that shows lasting the attitudes, policies, priorities, and actions of appreciation of WSP's role; it is part of the governmentdepartments.Relationshipsareusually international group SUEZ whose WSS operations good, often excellent, with government people claimtoservenearlyninemillionpoorpeoplein10 goingoutoftheirwaytotellushowmuchtheyvalue Africa, Asian, and Latin American countries, only the Program's work. There appear to be few or no one of which is aWSP focus country. tensions in the LAC, AF, and EAP regions. In South Asia also, many government officials expressed Other private sector bodies: WSP has good appreciation ofWSP's role and work, and evidently working relationships with some private sector had good working relationships with the Program; organizations outside theWSS sector for particular even allowing for some gap between rhetoric and purposes, for example in connection with the reality, this is a positive sign. This applied to both handwashinginitiativeinAfricaandLatinAmerica. state and federal levels in India, though there is Inthatconnection,peoplefromaradiostationand reportedly some tension between WSP and some a soap manufacturer are very appreciative ofWSP's state governments. This is in part an inevitable success in pushing the initiative ahead. reflection of the perennial Indian federal/state Consultants:WSPsometimeshirestheservicesof tensions, whichWSP cannot necessarily avoid. But local consultants, and their contributions to the thereareanecdotalindicationsthatWSP'sperceived sector are enhanced by the formal and informal associationwiththeWorldBankisoneofthecauses. networks of contacts set up by the Program. Tensions, and perhaps hostile attitudes toWSP, can arise if the Bank is seen as crudely prescriptive, for Donors:IntheregionsWSPgenerallyhasverygood instance on the merits of simple models of private relationships with the local representatives of the External Evaluation Report 39 donors who fund most of its work. These people notbeoverstatedandcanbeeasilyresolvedthrough appreciateWSP'sgeneralcontributiontothesector, betterdialogue. and also make use of the Program's capacity in furthering their own projects and programs. Knowledgepartners:Thiscategoryoverlapswith In South Asia, where many donors are active, most some others but refers to organizations which are ofthemarehighlyappreciativeoftheProgram;some also engaged in the generation, management, and of them told us they would like to be kept more dissemination of knowledge concerning WSS. closely informed and more effectively involved Examples are WEDC and IRC. Occasionally WSP in decisions. competes with these as a knowledge service provider,justastheysometimescompetewitheach The World Bank: WSP has often helped Bank other, but more usually they co-operate, share people in the preparation and design of loan knowledge, and use each other's services. The projects in the WSS field, and we have noted how knowledgepartnersplayvaluablerolesinthesector, well this can serve the objectives of both complementing that of WSP, in particular in organizations, and can giveWSP valuable leverage. building up the capacity of institutions for The Program's relationship with the Bank is sustainable WSS development. Relationships with sometimes affected by tensions surrounding its WSP are good. degree of working independence. Despite this, our observations during this evaluation, and what we Otherinternationalbodies:TheProgramCharter's have heard from others, lead us to conclude that in clause 8, in describing the partnership way of practicetherelationshipbetweentheBankandWSP working and naming various categories, makes is good, fruitful, and positive, and that such specificmentionofthese,namingtheWaterSupply problems as remain are largely ones of perception. andSanitationCollaborativeCouncil(WSSCC)and Thisisunfortunate,sincetheundoubtedandwidely the Global Water Partnership (GWP) as relevant recognizedadvantagesofthecollaborativebutarms- examples. There is specific provision in clause 14 length relationship between the Program and the for one such body to be represented on WSP's Bankareweakenedwhenitisnotcorrectlyperceived Council (as one of the types of `strategic partner by relevant sector actors. Other small institutions organization', though none is currently on the hosted by large and well-known ones suffer similar Council), and WSP is formally linked to both those problems, but that fact does not diminish them. organizations. It also has formal and working links to others, such as UNDP and WHO. Relationships Our region annexes contain specific comments betweenWSPandtheseorganizationsaregenerally about the four regions. In most regions and adequately good, though there are some slight countries, the WSP staff collaborates satisfactorily tensions and confusions over relative roles. withregionalorcountry-basedWorldBankstaff,and are appreciated. One slight indication of tension WSSCC has superficial similarities with WSP in its was reported in India, where the Bank's Country title and its focus onWSS, including pro-poorWSS, Director told us he was not fully aware of the WSP and it is supported by many of the same donors. work program although he understood the nature But the two are functionally distinct and different. andobjectivesof WSP.OtherBankstaffinNewDelhi We have met with the team who is engaged on a expressedaconcernthatthedifferencebetweenthe current evaluation of WSSCC, to discuss the Bank and WSP may not be understood by all positions of both bodies. Any overlap of roles government counterparts, and that there was a between them has reduced since about 2000 when danger of giving conflicting messages to the WSSCCmovedawayfromknowledgedevelopment governmentduringthepreparationofnewprojects. to concentrate more on advocacy and the Our discussions, however, found that this was not promotion of sanitation. WSSCC, though much the case and that the differences were well smaller than WSP and thinly spread, has a wide understood. This issue seems to arise from the fact global reach and a recognized advocacy activity that there was a discontinuity in the Bank's contact which is enhanced by its complete and clearly with some WSS stakeholders in India, while WSP perceivedindependenceofallfundingbodies.Itcan has maintained a higher degree of continuity in its influencepoliciesfromthisposition,whileWSPcan relationship to GoI.This misunderstanding should influencethemfromitsquitedifferentpositionasa 40 External Evaluation Report partnership involved not only in policy advice but through national and regional networks, many of also in institutional reform and the design of which WSP has set up and supports. It often sustainableinterventions.WSPcannotspeakforthe organizesworkshopsatwhichsectoractorsfromall wider constituency that WSSCC represents, theabovecategoriescanlearnfromeachother,and while WSSCC cannot match the technical and it was evident to us during this evaluation that this professional resources of WSP. However, since function ofWSP is fruitful and highly appreciated. persuasion and attitude-changing form important parts of what both organizations do, there is scope We conclude that WSP, as a partnership for collaboration and mutual assistance without organization, has many important relationships wasteful duplication. This has been happening, with other sector actors, and that most of them especially in Africa generally, and in connection are very satisfactory.There are some tensions,not with Africasan and SACoSan.We conclude that the necessarily of WSP's making, which could be two organizations do have distinct roles and are alleviated by good communication and by clear complementary to each other. More collaboration thinking and discussion about shared and couldthusbefruitfulintheserviceoftheobjectives separate objectives. of both organizations, although it must be 4.4.5 Cross-Cutting Issues recognized that the staff of both have limited time There are a number of aspects ofWSP's work which and many conflicting demands on their resources; do not fall neatly into sub-sector categories such as informal case-by-case collaboration may be more water supply or sanitation, nor into spatial ones cost-effectivethanformaltiesandroutinemeetings. such as rural, urban, and small towns. In fact they WSP could make more use of the link for tend to cut across all those, being relevant for dissemination and advocacy beyond its own focus instance in a rural water supply context or in an countries,andtoenhanceitsowndegreeofrealand urban sanitation one. This section discusses a few perceived independence from funding sources.We such issues and aspects, and relates them to concur with the following statement from the particularMillenniumDevelopmentGoals(MDGs). currentWSSCCevaluation:`Thereisaneedforsome harmonization on key policy issues while a) Gender,Poverty,and Livelihoods emphasizing the different roles and interests, WSP's efforts in relation to gender and poverty do ensuring that there is no apparent discord, no not usually form distinct projects alongside others, apparent confusion of roles, and no opportunity for but pervade all projects and interventions, linked others to exploit perceived differences between the to the general drive towards participatory two organizations.' approaches and community involvement. A major TheGlobalWaterPartnership(GWP),ofwhichWSP focus of WSP's work in the review period was, has always been an Associated Program, has however, the Participatory Learning and Action generally enjoyed good mutual co-operation with Initiative (PLA), which ran from 1998 to 2000. It WSP. This was quite intensive in GWP's early years brought together a global core group of individuals when it covered the whole water sector, including and agencies into a partnership which worked for WSS. More recently GWP has narrowed its focus, mainstreaming gender and poverty concerns in largely onto integrated water resource watersupplyandsanitation.13Withinitsoverallaim management, so role overlap is now less and there ofimprovingtheunderstandingofthelinksbetween is less need for detailed collaboration; the gender,participation,demandandsustainabilityof relationship remains good. WSP could perhaps community-managedwatersupply,andsanitation engagemoreintheGWPdebateonwaterresources services, the initiative had two objectives: management,toensurethatWSSreceivedadequate toaddtotheknowledgethatgreaterattentionto attention alongside other water uses. genderpaysoffinincreasedsustainability(poverty was soon added, recognizing that poverty is as With all these sector actorsWSP works both in one- important as gender among the elements of a institution-to-another meetings and contacts, and community's heterogeneity). 13Linking sustainability with Demand, Gender and Poverty ­ A study in community-managed water supply projects in 15 countries; WSP and IRC (Gross, van Wijk and Mukherjee); January 2001. External Evaluation Report 41 acapacitybuildingphase,tousetheknowledgeto directly serving MDGs 1 and 3. It is notable that help change the way programs and projects are manyoftheindividualsinvolvedinthePLAandMPA designed and implemented. workbetweenabout1998and2003arestillactivein WSP's regions, while others have moved on to The initiative began with a series of rapid relevantworkintheWorldBank.Thisfurtherextends assessments in projects known to have some andreinforcesgender-andpoverty-consciousways participatory and demand-responsive features, to of approaching WSS work. see if more gender-sensitive demand was linked to better outcomes. The core group reasoned that There have also been a few specific gender-related agency officials responsible for water supply and activitiesbyWSP,notablysomemulti-phaseprojects sanitation services would pay more attention to intheLACregionand,in2002,adiagnosticstudyof gender,iftheyhadevidencethatdoingsopaysoffin genderaspectsintheWSSsectorinPeruandBolivia better sustained services.The action research in 15 which guided the design of major projects. In countries provided empirical evidence that better early 2004, four macro-regional gender workshops sustained and used services were indeed were held in Peru, to disseminate WSP-LAC's significantly and positively associated with the proposed methodology. use of gender- and poverty-sensitive demand- responsiveapproachesinprojectimplementation, b) Hygiene and Public Health institutional practices, and policies. The connection between water and sanitation on the one hand and health on the other is so obvious This PLA initiative led to the Methodology for that people do not often bother to emphasize it. Participatory Assessment (MPA), also led by WSP Watersupplyandsanitationaremajorcomponents and IRC.14 By 2003, thanks to these WSP-IRC of public health efforts, and there is also a direct interventions,MPAwasawell-establishedandwell- linkage to such problems as HIV-AIDS. Caring for documented methodology that was being used in someonewithAIDS-relatedsymptomsisobviously Asia, Africa, and Latin America. It has also been more difficult in the absence of adequate water influential inWSP's East Asia Pacific work. supplyandsanitationfacilities.SincemostofWSP's work serves to promote sustainable WSS services Wehaveexaminedthesetworeportsandhaveseen for poor people, it directly serves health objectives, someresultsoftheapplicationofMPAintheregions. and, in particular, MDGs 4, 5, and 6. The documents are well-written and clear. They persuasively link the sustainability ofWSS services c) Corruption, Accountability, and Governance to equity, to accountability and to the inclusion Corruption is a problem in the provision of WSS of gender and poverty aspects by involving services, and many other sorts of services and communities systematically and effectively in the infrastructuretoo,inmanycountries.Becauseofits services' design, implementation, and operation. political, legal, and ethical implications it is, They go on to present a well-thought-through however, seldom mentioned, and then often using methodology to help sector actors to apply these polite and obscure wording such as `good lessons all over the world. Sustainability is not just governance'or`rent-seekingbehavior'.WSPseldom held up as a desirable objective ­ practical and makes any claims about contributing to the step-by-step ways of achieving it are presented. reductionofcorruption,butinourviewitcouldwith justice do so. The Program often promotes In our view, these initiatives have contributed very transparency and community participation in significantly to improvements in the effectiveness procurement processes, which must usually make and sustainability of WSS interventions in many corruptpractices,forinstancebriberyofmunicipal countries, and not only the interventions in which officials or service providers by contractors, more WSP was directly involved. In so doing, WSP has difficult to initiate and to hide. During one of our givenpracticalandusefulexpressiontogenderand field trips, we discussed this with local NGO people poverty concerns at the heart of its sector, thus working in rural water supply, and they agreed that 14Sustainability, Planning, and Monitoring in Community Water Supply and Sanitation: WSP/IRC (eds. Mukherjee and van Wijk); 2003. (This document followed an earlier one called the Metguide [Methodology for Participatory Assessment with Communities, Institutions, and Policy Makers] which had been developed during the PLA initiative in 1998.) 42 External Evaluation Report this is one of the positive by-products of the technical ones, which we consider appropriate methodologythattheyhadrefinedandwereusing. since the more difficult constraints on sustainable WSS provision for the poor are in those fields, There has also been one WSP activity specifically especially concerning the weakness of institutions. related to corruption within a broader context of WSP's most useful role in the field of technological governance.15 Thestudystartedbynotingthatinterest innovationisnowprobablythatofencouragingand inprivatesectorinvolvementwaslargelydrivenby`a enabling others, such as local NGOs or research growing consensus on the inability of public sector bodies, to undertake research and development. agencies to improve performance. This consensus WSP has sometimes played a leadership role in the emerges from a collective disappointment with thinking of the WSS sector, for instance about stagnant coverage levels, as well as increasing appropriate rather than stereotyped ways of accounts of inefficiency, ineffectiveness, and rent- involving the private sector, or ways of mediating seeking in public W&S agencies.' But it went on to subsidies through a variety of institutional models. observethat,inSouthAsiaatleast,privatesectoraction was not likely to take over the task of WSS provision When it comes to knowledge management and the entirely,sothepublicsectorneededsupport.WSSand distillation, filtering, and propagation of lessons itspartnersstudiedfourparticularSouthAsianpublic learned by or with others, WSP is very active and sectorinstitutionsthathad`exceededtheexpectations effective. Our annexes contain examples, such as placed on them by sector professionals'. These practical ways of scaling-up rural water supply and four success stories were analyzed in terms of sanitation in India and in Peru, arsenic mitigation responsiveness, accountability, and service to low- in Bangladesh, condominial approaches to peri- incomehouseholds.Inoneofthem,thestudynoteda urban WSS, and hygiene promotion as in the decreaseincorruption(involvingbothmeterreadings handwashing initiative. Often lessons are learned andprocurement)followingawidersetofgovernance by front-line workers such as NGOs or government reforms that paid special attention to accountability. officials who do not have the time or skills to digest Overall,thestudyprovidedvaluabledataandinsights andarticulatethelessonsandexpresstheminforms into governance in the public sector in general, that can be used by others. WSP is good at those including corruption as one aspect of it, and things,sosuchpeopleareoftengratefulforitshelp. commented on the limitations of private sector Another valuable activity is diagnostic work, solutions. This is of special value in contexts where especially when it contributes to the design of private sector approaches sometimes become the projectswithsignificantfunding.WSPiswellplaced fashionablethingtodo,andareappliedunthinkingly forthiswork,bothbyusingitsownexperiencedstaff or inappropriately. and by hiring and supervising consultants. Sharing of knowledge betweenWSP's regions takes Our finding is that WSP takes adequate account place through publications and occasional visits, ofgenderandpovertyissues,whichareinherently throughafewinter-regionaljointprojects,through significant in all WSS, and handles them well. theattendanceofkeystaffattheannualWaterWeek Othercross-cuttingaspectsofitsworkgivepositive inWashington,andthroughtheregularmeetingsof effects in a number of fields, reinforcing the regional team leaders. Program's direct WSS objectives and also serving wider ends and helping towards several of WSP takes great care with its publications, which it the MDGs. rightly regards as an important category of output. It now has communications specialists in 4.4.6 Knowledge, Communication, headquarters and regional offices. Some and Publications publicationsaregeneratedinternally,butWSPstaff Incontrasttoitsearlyyearsfrom1978,theProgram's also help other organizations with the preparation, knowledge generation work is now more likely to editing, polishing, targeting, production, and emphasize systems and processes, including dissemination of their written material; this is managerial or organizational aspects as well as a valuable activity. 15In Pursuit of Good Governance: Experiments from South Asia's Water and Sanitation Sector; WSP and Massachusetts Institute of Technology; April 2003. External Evaluation Report 43 The quality of the Program's published material that and ways of working. (In that specific case, the we have seen is usually high, with a distinctive style problemwasrecognizedbyWSP-AFandledtoamore and effective use of photographs. The technical integratedapproachtocapacitybuilding;detailsare content of publications is generally good and in Annex AF). More generally, WSP always needs to appropriate, though it is, of course, not possible to make deliberate and structured choices about what suiteverypossibleaudience.Inevitablyinsuchalarge to publish, ensuring relevance to its objectives, and anddiversesetofpublications,therearesomeweaker avoidingpublishingworkmainlytopublicizesuccess examples. An evaluation in 2002, described in or promote the Program's or an author's reputation, Appendix3,notedsomeshortcomings,andourreview which has happened in a few cases. of someWSP publications tends to confirm this.The annexes contain some region-specific examples and WSP could with advantage use a wider range of discussion. Closer supervision or editing of media, for instance virtual field trips on video, to publicationsbyexperiencedWSPstaffmighthelp,but bring field lessons to wide audiences without the the benefits of this must be weighed against the high need for extensive travel and simultaneous opportunity cost of their time. Some stakeholders in translation.TheProgramdoessometimesdistribute SouthAsiatoldusthatfewerpublicationswithagreater the publications of other organizations, but we depth of research and analysis would be preferable, considerthiscouldusefullybedonemore,andthey but we do not see a need for any major change on could distribute those ofWSP. priorities in response to that. We conclude that WSP's knowledge-related work Ouroverallfinding,however,basedonwhatwehave is important and fruitful as a complement to its seeninthetimeavailableandnotonanysystematic other activities, and is mostly well done. Quality collation and analysis of large numbers of and appropriateness of publications is usually publications, is that WSP's efforts since 2002 to good,thoughinevitablyvariableinadifficultfield improvethequalityofpublications,andmakethem where all staff are involved, including the less moreconsistent,haveborneconsiderablefruit.The experiencedones.Standardshaveimprovedinthe Programnowsystematicallydistinguishesdifferent course of the evaluation period. sortsofpublication,withdifferenttargetaudiences, 4.4.7 Staffing and Staff Skills anddifferentdegreesofsensitivitytopresentational WSPhasanexcellentcoreofstaffwithrelevantskills style. It has a clear system of checklists and and experience, but they are spread quite thinly. procedures for the distinct sorts, and if well used Staffing policy is regularly considered by WSP this should steadily improve the consistency of management,andtheregionshavedistinctpolicies qualityinsuchaspectsasclearobjectives,technical to match their diverse situations. Recent losses of content, readability, and visual presentation. We thematic specialists from the headquarters team consider it right that some classes of publication have left a gap.16 We understand that WSP intends should receive more of WSP's scarce time and to reduce these two headquarters posts to one, and attentionthanotherswhicharelesssensitiveorless that some specialists in regional offices are to be widelydistributed. given a global co-ordinating role. This raises the Continual attention to both content and difficult question of how WSP should distribute its presentation remain essential. Sometimes the thinking capacity between the center and the reduced effectiveness of a publication is not due to regions. Although a regional base has advantages any failing of its own but to a lack of coherent forinnovativethinkers,itlimitstheirrangeofinputs planning of the project or theme it is related to. The from others, and makes the propagation of their case of the Uganda Sanitation Guidelines illustrates ideas more dependent on publications, which may a generic problem with publications, that they be less useful (partly because less interactive) than sometimesfailtomakeasignificantimpact,perhaps workshopsandinformalcontacts.IftheProgramis because a publication on its own is insufficient to to reduce the strength or number of thematic bring about change in deeply entrenched attitudes thinkers in its headquarters it may need to counter 16In the course of FY04 two headquarters staff, senior WSS specialists functioning as global thematic co-ordinators, left WSP to take up permanent appointments with the World Bank. At the moment there are transitional arrangements for them to continue serving WSP part-time. 44 External Evaluation Report this by giving those in the regions an explicit remit, 4.4.8 Languages and Sub-Regions plus time, and funding, to travel to other regions or WSP works in several languages. The LAC region toglobalforatointeractwiththeirpeers.Thesizeof works within its region mainly in Spanish, and theHQstaffhasbeencuttoaverysmallcore(partly occasionally in Portuguese, but in English for most inresponsetocallsforreducedcentralexpenditure), dealings with the other regions and with and we consider it should not be cut further. The headquarters.ManyofthestaffarefluentinEnglish staff numbers in particular regions depend on the as well as Spanish, while some speak French also; choice of focus countries, for which there are languagedoesnotappeartobeasignificantproblem clear criteria, and on donors' funding decisions. orconstraint.SeveralWSPpublications,originating Language barriers do not appear to us to be a in LAC and in other regions, are available in both significant problem. Spanish and English versions. In South Asia, the mainlanguageisEnglish,butsomepublicationsare A balance is also needed between staff with well- translatedintosomeoftheregion'snumerousother grounded technical training and experience plus a working languages; WSP-SA appears to have good understanding (usually from experience, not languagedifferencesundercontrol.Thesituationis training) of people, motives, governance, and similar in East Asia Pacific. institutions,andontheotherhandstaffwithformal training in institutional aspects or social sciences. In the Africa region, the main office in Nairobi Theformerarenoteasytofindandrecruit;WSPhas operates mainly in English, while the Dakar office several but could use more. A third balance that operates mainly in French and deals with the needs to be struck in staffing policy is that between Francophone countries in West Africa and using young but very talented people and using Madagascar. The small office in Mozambique people with long experience in the sector, usually operates in Portuguese. Communication between gainedwithotherorganizationsinthepast.Usinga the offices is mainly in English. The regional mix of experienced and more junior staff is not a managementhasmadeconsiderableeffortsinrecent badthing,butrequiresthattheformerhaveenough yearstoovercomelanguagebarriersandinparticular timeandincentivetogiveguidanceandleadership; toimprovetheintegrationoftheFrancophonework. this has often been good, especially in Africa, but Meetings of the Regional Advisory Council are remainspatchy.OftentheProgram'simaginativeuse bilingual, in French and English, this having been ofconsultants,andsubcontractingtaskstothemor introduced during the evaluation period, and are NGOs, helps to alleviate problems. sometimes held in Dakar.The region has appointed aFrench-speakingcommunicationsspecialist,based SincetheProgram'sfundingisnotusuallyknownmore inDakar,inordertoincreasethenumberandquality thanthreeyearsahead,thestaffarenormallyemployed of French-language publications. Several French- on relatively short contracts, leaving them without speakingpeoplehavebeenappointedtotheNairobi security of employment and with an incentive to office, and substantial language training has been acceptmoresecurepostselsewhere,suchaswiththe provided for the staff. WSP leadership remains WorldBank.ThisreducestheProgram'scontinuityand conscious of the need for further efforts, not only in institutionalmemory,thoughitalsobenefitsthesector the Africa region, in provision for Francophone more widely in thatWSP serves as a training ground countries and partners, and is seeking help with the forotherorganizations.Therateofturnoverputsextra high cost of translation and interpretation; we supervision and training burdens on senior supportthiseffort. WSP managers. Arelatedissueisthatofsub-regionswithinsomeof We consider that the staff base in the evaluation thefourregions.ThemainexamplesareWestAfrica period has been strong, and remains so today. and Central America. As described in Annex AF But the Program's work poses big challenges, and (especially its Appendix AF-3), since merging the we see the need for continuing the internal review former two African regions into one the Program of future staffing needs, grounded in a very has made considerable efforts to strengthen the clear understanding of WSP's special objectives West Africa and Francophone elements, subject, of and its comparative advantage relative to course, to funding constraints. The use of a sub- other organizations. regional office in Dakar, with four professionals, External Evaluation Report 45 seems to us a sensible and satisfactory way to go own work, to ensure that lessons are learned and about this. The relationship between the Central used for the design of future projects. America sub-region and its regional office in Lima is different in that there is no language barrier, but 4.5WSP's Comparative Advantage again the arrangement seems to us to be sensible The above systematic evaluation of the Program's and cost-effective (some details are in Annex LAC). work in the fiscal years 1999 to 2003, backed by the 4.4.9 Response to Evaluations regional annexes to this report, lead to the and Self-Evaluation conclusionthatitisgenerallyrelevantandeffective As described in Section 3.6 above and in in achieving outcomes and impacts in its chosen Appendix 3, the Program's work was the subject of field.Itisalsogenerallyefficientinitsuseoflimited many piecemeal evaluations during the review resources, often highly so, and this is any way period. The Program and its donors responded to subject to the natural constraints of innovative those which dealt with governance by setting up partnership work. The Program's partnership style the Council in place of the PAC, which as described and its influence on the design of projects funded above has been a marked improvement. WSP also byothersgivesitgreatpotentialforleverage,which respondedtoevaluationsofitsownworkwithsome isoftenusedwell.Itsinstitutionalarrangementsare beneficialchangesthathavebeenincorporatedinto generally good, though require clarification and itsevolution.Inparticular,itinitiatedalongprocess perhapschangesonsomegovernanceaspects.WSP of revising its planning procedures, which as noted hasimproveditsplanningproceduresconsiderably above is now bearing fruit. towardstheendofthefive-yearperiod(whichisnow a year behind us), and continues to do so. We notice, in connection both with those evaluations and this one, that the Program's staff Since its early years concentrating largely on tendtorespondtoanycriticism,evenifmixedwith sustainabletechnology,WSPhasrightlymovedinto praise, in a markedly sensitive and defensive way. otherfields,coveringwaysofdoingthingsinawide Discussing this with them, we are told that it is sense. For example, the condominial approach to normal in World Bank circles, which in turn peri-urbanWSSbringstogethersometechnological probablyreflectsthehighstandardsassociatedwith innovations (such as local drains and pipes located that organization. It may also reflect a desire to and connected to main systems in a new way) with achievemeasurabletargets,andatendencytojudge new forms of private sector involvement and cost people's work in such terms, but we consider that recovery, and also with context-specific ways of too much emphasis on such targets tends to involving communities and service users. undermine the value of a`learning organization'. ThewaysinwhichothersectoractorsuseWSPindicate In a positive sense, this sensitivity reflects theWSP that they appreciate its comparative advantages, staff's dedication, professionalism, and pride in particularlyinthefollowingsortsofactivities: their work. But it does not seem to us entirely Attitude-changing advocacy, persuasion, and healthy, especially in an organization dedicated to capacity building activities generally, for instance partnership work and therefore to listening to others. In view of their caliber and achievements, the Program's skills in WSP people should not need to be defensive. The settingupandsupportingnetworks,especially Program works at the leading edge of a field which in areas where political, social or logistic factors is professionally and intellectually difficult make contact between sector actors difficult; and challenging, and no one should expect facilitating dialogue and exchange perfection. Inevitably in development work, some on important issues, while not dominating interventionsfail,orsucceedonlypartly. Itappears the discussion, for instance by conducting tousthatamoreself-questioningattitudeonWSP's workshops; part, together with a less defensive attitude to documenting and disseminating the lessons constructive criticism, would strengthen rather of experience; and than weaken its effectiveness. At the same time, transferring those lessons from one country WSP could take a systematically critical view of its to another. 46 External Evaluation Report Table 4.1:WSP's Comparative Advantage Aspect WSP Strengths Strengths of Other Actors, and Comments Attitude- Especially in recent years, WSP has been Some other actors are also active in changing, engaged largely in persuasion and attitude- advocacy, notably WSSCC. But WSP's advocacy, changing,withcentralandlocalgovernments workinthisareaismuchenhancedby persuasion, and other public sector people, and also with the combination with its other and the private sector and communities. characteristics noted in this table. capacity building Advice and WSP has developed considerable strength in Although some other organizations support on thisareainrecentyears,andisatrustedsource also do this, WSP puts particular `upstream of advice in many countries. emphasis on it and has a track record. issues' Innovative Useful innovation often needs good Otherorganizations,especiallyNGOs, thinking knowledge of local cultural, social, and maybewellplacedinsomecountries, organizational situations.WSP is well placed but there is probably none with a for this because of its field presence in many general advantage overWSP. countries and its continuous contact with other sector actors. Collecting, WSP has a long tradition in this field, Many others also do this, but WSP is distilling, and its improved and still improving one of the strongest and has an and communications strategy give it further especially wide range of contacts. disseminating strength. Its many contacts in the sector give information it a wide source base. Assisting in Largely because of its historical and None other in this sector has a similar the design of continuingrelationshipwiththeWorldBank, relationship with the World Bank, investment WSP is uniquely placed to influence project which is a major funding source, projects design, giving it important leverage. thoughsomemaybestrongwithother funders or in specific regions. Professional Withabout80staff,ahighpublicprofileinthe Few others can match this: and sector,andWorld-Bank-levelsalaries,WSPcan some NGOs have high technical technical recruit and retain a good number of highly competence in their fields but tend to capacity competent professionals, covering a wide have high staff turnover and/or range of technical and organizational limitations in recruiting due to knowledge and experience. lower salaries. The international organizations (WSSCC, GWP) do not matchthestaffnumbersandtechnical range ofWSP. Continuity In its focus regions and countries, WSP can GWPandWSSCCareusuallyrestricted of contact and does maintain continuous contact with intheiradvocacyandpersuasionwork keyactorssuchasseniorgovernmentpeople. to occasional visits or missions to External Evaluation Report 47 Aspect WSP Strengths Strengths of Other Actors, and Comments It can persuade and change attitudes particularcountries,whichmaynotfit patiently and continually, and can wait until well with the evolution of thinking in people are ready for the next message. target organizations. Global WSP has strong regional, sub-regional, and Probably no other organization in the knowledge country bases on three continents, covering sectorcanmatchthis,exceptperhaps base a good diversity of situations. It also has a big funding agencies. strong and improving system for extraction, storage, and dissemination of information. Independence WSP'sfunctionalindependencefromdonors Thisisakeyadvantageincomparison offunding and funders can help to engender trust and with funders, especially the World confidence in its judgement and advice Bank.Manyotherorganizationsshare (provided that WSP is perceived as this advantage, for example, WSSCC independent by the relevant actors). and NGOs. Relationship The Program's special relationship with the Nootherorganizationcanmatchthis. with funding World Bank, and to a lesser extent other But the special relationship with the agencies funders, gives it valuable leverage and World Bank needs to be continually influence, including indirect influence on watchedtoensurethatitdoesnotgive clients who are dealing direct with the Bank. rise to perceptions of Bank control. Trackrecord Twenty-five years in the sector, with an Most international organizations are evolving role. younger and do not have as strong a track record (for example, WSSCC, GWP).SomeNGOsdohavestrongtrack records(forexample,CARE,WaterAid). Ability to WSP has always included learning in its While this is a strength of WSP, many learn rhetoric. Most of its staff are willing and otherorganizationsalsolearn,andwe inclinedtolearnfromexperience,thoughthe do not consider this a particularly internal monitoring of current and past significant area of comparative projects is not systematic. advantageforWSP. Providing advice and support to clients on Innovative thinking, especially the Program's `upstream issues' such as policy and methods, abilitytobringtogetherorganizationalandtechnical especially related to institutional effectiveness. aspects, and testing innovative ideas in particular ThisflowsfromtheProgram'scollectiveexperience, culturalandgeographicsettings(orpersuadingand andmorespecificallytheexperienceofitskeystaff. helping others to test them). Collecting, distilling, and disseminating Maintainingthiscomparativeadvantagerequiresthat information, often derived from the work of others, WSP makes a conscious effort to base its ontechnologiesandonorganizationalmethodsand recommendations and advice on real experience and models, and more importantly on ways of not on the currently fashionable prescriptions that combiningthose.WSPdoesnotoftenhaveparticular happen to pervade the development community at comparative advantage on purely technological any one time. matters, partly because they are covered by many 48 External Evaluation Report others, but can usefully support those others with training members of the government department knowledgeandadvice.TheProgramiswidelytrusted dependedontheProgram'spriorknowledgeofhow andrespectedasaninformationsource,bybothlocal to make rural schemes sustainable (collected and and global sector actors. filtered from many sources), on its understanding ofthecountryandthedepartment,andonthetrust Guidingandassistingothersinthedesignoftheir that the government people already had in WSP's investment projects, where WSP is especially understanding of policy issues. effective when it maintains its distinct pro-poor insights,guidingratherthanfollowingthethinking We have identified in this chapter some issues where of the funders and implementers of projects. we consider there is room for improvement. These judgementsleadtoourrecommendationsinChapter6, Table 4.1 gives a brief discussion of WSP's butitshouldbesaidherethatthesituationisnotstatic. comparative advantages relative to other sector Inmanyaspectswehavenotedimprovementsduring actors which pursue the same or overlapping the evaluation period and since. objectives. It starts with the above five sorts of activitiesandthencontinuestomentionafewother ThereissomedangerthattheProgramascurrently aspects. This evaluation has concentrated on WSP staffed may be overstretched, especially in Africa and has not systematically examined the where so many countries have urgent WSS needs. comparative advantages of other organizations, South Asia may be in a similar position ­ although though some comments about a few of them are in India is nominally one country, its vast size, its Section 4.4.4above,socommentsinthistableabout diversity, and its federal structure make it more like other organizations are tentative. A systematic a continent for many practical purposes. review of how WSP fits in the spectrum of organizations would be part of the updating and In view of our positive conclusion about clarification of its strategy mentioned in comparative advantage, the question arises as to Section 4.2.1 and in our recommendations. howfarWSPcouldexpandthescaleofitsoperations without losing efficiency or effectiveness. Our An important component of WSP's comparative subjective judgement is that, given adequate advantage is that the Program's different types of funding and the corresponding staffing, the work and of expertise are not separate efforts but Program could handle 30 to 60 percent more work complementeachotherverypowerfully.Totakethe with its present institutional set-up of four regions, hypothetical but realistic rural water supply somesub-regionalhubssuchasCentralAmericaand example used in Chapter 2 to illustrate our WestAfrica,andaheadquartersstaffcomprisingup framework, WSP's success in persuading and to about 10 percent of the total. External Evaluation Report 49 5 FUTURE CONTEXT AND WORKING ENVIRONMENT 5.1 Trends in the Sector we would expect this trend to be maintained in We consider that the most relevant trends in the thenextfewyears,probablywithbetterandmore WSSsectorinthenextfewyears,continuingpresent successfuldecentralizationmodelsascountries trends,arelikelytobeincreasingurbanization,and gain experience and improve on simplistic continuing decentralization by governments models. Thiswillplaceheavydemandsonlocal (though perhaps more subtle and refined forms of government bodies such as municipalities, decentralization than in recent years). General regional or provincial governments (some of economic growth will probably continue in most which are newly constituted), who often have relevant countries. We see the main implications little experience to help them meet the forWSP to be: challenges. Traditional ways of providing WSS services, through state organizations and with a) Increasing pressure due to urbanization: little or no community participation or choice, While the importance of rural and small-town have been widely found to be ineffective and/or WSS will not diminish, the increase in urban unsustainable.WSPhasalreadyplayedaleading populations will make urban work even more role in documenting and understanding these important than it is already. One key feature of problemsandinfindingordevelopingsolutions urbanworkisthefactthatawiderangeofurban whicharecompatiblewithdecentralization,and services are mutually interdependent, so that which make appropriate and cost-effective use consideringWSS in isolation from other urban ofbothpublicandprivatesectorcapabilities.The servicesisnotfruitful.Anotherfeatureisthatan needforthissortofworkisexpectedtoincrease. urbanWSSserviceprovidertypicallyservespoor There is an equally pressing need to examine communities through the same physical and the advantages and disadvantages of new organizational systems as richer people. approaches and paradigms. Questions to be There is a need to recognize the `horizontal' asked include what forms of community dependency between different services in the participation work in what circumstances and same area, and `vertical' dependency of local which aspects of service provision should be services on higher order facilities and services, decentralized to what level. WSP will often especially for networked systems such as need to remain critical of the latest sewerage and piped water supply. Service `conventional wisdom' on issues of governance provision has to be organized into institutional and participation. domains one way or another, and any way may bring difficulties at the boundaries. Horizontal c) Economic growth and the need for linkagesmeanthatanintegratedorcoordinated bettersystems: approach to service provision will usually be Whethergenerallyorinurbanareasspecifically, better than a narrow sectoral one. Vertical economicgrowthcannotbeexpectedtooutstrip integrationmeansthateffortstoimproveservices populationgrowthbyanysignificantmargin,so for the urban poor cannot focus only on local there will continue over the next few decades to services to low-income communities but rather be a need for innovative and cost-effective on those services within their overall context. approaches to the provision of WSS services for This may mean that the most effective and poor people. But economic growth will fuel efficientserviceimprovementswillbenefitboth technological development that can be used for rich and poor. Ways may need to be found to pro-poor WSS. The main need will be not for support or subsidize service provision to `appropriate technologies', but rather for particulargroups. appropriate systems, encompassing b) Institutional strengthening under technologies, financial, and management decentralization: systems within an integrated whole that takes The last few years have been marked by policies account of social attitudes and structures. WSP of decentralization by many governments, and is well placed to work on this. 50 External Evaluation Report 5.2 Needs and Demands of WSP's large provincial governments in the last five Partners and Clients years,butthereisnowarealandperceivedneed WSP's strategic and direct partners (agencies and to extend support to regional and local donors), as well as its direct clients (governments government, which is fragmented and involves andserviceproviders)willfacecontinuingneedsin hundredsorthousandsofinstitutionsinsteadof thesector,andtheabovetrends.Inthatcontextthey a few in each country. WSP is not likely to be will continue to place demands on the Program. asked to serve these numerous institutions ThedonorsrightlyuseWSPtofurthertheirevolving directly,buttocontributetojointeffortsinorder development objectives, and those in turn to build up their capacity and knowledge and are influenced by the donors' dialogues with give them effective and sustainable models governments. With both donors and WSP and methods. increasingly using decentralized ways of working, the relevant discussion of demands often goes on d) Institutional weakness is still seen as a major in the focus countries, which is healthy in that the problem in most parts of the sector. This is not participantsareexposedtolocalconditionsandare caused by lack of capacity alone but also by usually in close touch with governments and other inherent weaknesses in current institutional country-level actors such as local NGOs.TheWorld structuresandsystems.Sosectorreformislikely Bankandotherfundingagencieswillalsoprobably to remain important among the clients' and continue to make demands on the Program, partners' demands onWSP. especially for help with project design, and e) There will probably be an increasing need to participation in the global discussion about ways demonstrate and make known the linkages and means in the sector. between WSS and other development aspects AfterstudyingtheProgram'sworkandlisteningtoa suchashealth,education,poverty.Thiswillneed wide range of stakeholders, we consider the clear thinking about objectives and how they following aspects particularly important: relate to each other in effective integrated development. a) Thereisevidentlyawidespreadviewinthesector that urbanWSS, especially sanitation, is of high f) More generally, and underlying the above, a and increasing importance, and WSP probably central need of the sector is the need to change needs to respond to this systematically rather attitudes, particularly attitudes to community thanpiecemealorlocally.Itraisesdifficultissues participation, to collaboration between public abouttargetingeffortstowardsservingthepoor, andprivatesectors,andtoappropriatematching since (as mentioned above) the same urban of technical and managerial innovation. serviceprovidersusuallyserverichandpoor,and The Millennium Development Goals are an a crucial determinant is often the effectiveness important expression of collective will, though the ofthoseinstitutionsandthesustainabilityoftheir ones relating toWSS (Targets 10 and 11) merely set methods and products. relative aims in terms of halving percentages of b) Many people in the sector still see a need populations without sustainable access to basic for increased emphasis on sanitation, while services.Thismeansthattheyrefertoaccessrather hygiene promotion and education are thanhigher-leveloutcomes,andrelyonchangesin also considered important as necessary statistics that are of questionable meaning anyway. complements to WSS provision if the desired The necessary linkages between different outcomes and impacts are to be achieved.Work development aspects and goals are not effectively in these fields is often difficult to finance, and dealt with by the MDGs themselves, so that they do the handwashing initiative, mobilizing private nottakeadequateaccountofintegratedpoliciesand sector funds, shows promise. methods.ButtheMDGsremainasignificantwayof c) The need for WSP's work in support of raising political awareness of needs.The world as a decentralizedorganizations willpersistinmost whole is evidently not going to meet the MDGs regions for decades to come. The Program has easilyby2015,ifatall,sothereisalargedemandfor been of considerable assistance to central and more of whatWSP has to offer. External Evaluation Report 51 6 RECOMMENDATIONS 6.1 Continuing with the b) The Council and WSP management should Program's Work continue to seek ways to increase the level of Our principal recommendation is that WSP funding. Many of the recommendations which shouldcontinuewithitsvaluablework,andshould follow, and especially those concerning strategy seekasignificantfundingincreaseforthenextfive and governance, could form the basis for to 10 years. improvedconfidenceintheProgramandafresh approach to potential new donors. By`significant increase', we mean something of the order of 30 to 60 percent, rather than 10 percent or c) WSP shouldbothplanandmonitoritsresource 100 percent. This is not intended as a precise use with a multi-year perspective, rather than measure, nor does it rely on any quantitative justwithinfiscalyearsasithastendedtodountil analysis; it merely reflects our subjective views of about2003.Thedate-specificoutcomeindicators the sector's needs, as discussed in Chapter 5, of the in the Business Plan for FY 2004 provide a good Program's comparative advantages described in basisforthis.Whatisneedednowistorelateeach Section 4.5, and of the scale of work that could be year'sBusinessPlantopreviousandfutureones, handled without the major restructuring of the by keeping consistent names and reference Program.The next five to 10 years will be crucial for numbersofprojectsandrepeatingtheirdefined the sector, not least in relation to the MDGs, whose objectives and timelines in successive Business time-scale runs to 2015. Plans, unless they are deliberately and explicitly changedforstatedreasons. To attract a funding increase of this order, WSP would, of course, have to persuade existing donors d) Donorsshould,ifpossible,seekwaysto commit to give more, and/or persuade new donors to start funds several years in advance, typically five giving. The remainder of this chapter presents our years for most purposes. The necessary moredetailedrecommendationsforactionbyWSP confidence should be provided by a clarified anditsCouncilwhichweconsiderwouldhelptodo strategy and enhanced multi-year planning, as this,aswellasimprovingtheProgram'seffectiveness recommended below. and efficiency. e) WSP should use its improved multi-year The recommendations are arranged in six groups, planning to make systematic and transparent generallyfollowingthesameorderasthediscussion arrangements for coping with the fact that andevaluationinChapter 4,andarenotinorderof donorsdonotusuallycommitfundsmorethan importance or urgency. Relevant section numbers three years in advance; this can include ofthisreport,wherediscussionofourreasoningcan planning longer projects but breaking them up befound,aregivenwherepossibleinsquarebrackets into phases for piecemeal funding. [...]. Most of the specific recommendations are f) The Program and the donors together should addressed to a named party such as the WSP seek ways of achieving a joint flexibility in management or the Council, identified in this resourceallocationbyincreasingtheproportion chapter by bold italic type, while for convenient of partially tied or untied or core funding, so referenceafewkeywordsineachrecommendation that new opportunities identified by any party are highlighted by means of bold upright type. can be quickly grasped. This should include the formalizationofintermediatefundingcategories 6.2 Finance and Resources that are not global core funding but are not [Sections 4.4.3, 4.2.3, 4.3] completely tied either. This can counter the a) WSP should continue to operate under World reduction of flexible funding that tends to Bank financial and accounting systems, accompany the shift to regional funding while seeking efficient ways to make relevant decisions by donors. Ways of maintaining information available and transparent to responsible resource use and adequate its partners. accountability should be agreed by the Council, 52 External Evaluation Report so as to give donors the necessary confidence in ii. The Council should debate the extent to the more flexible allocation. which WSP's thematic and technical work ought to be controlled by the World Bank, g) WSP should seek ways to improve flexibility and, if possible, replace the ambiguous of resource allocation in the course of each provisions of the Charter with a clear and project's life, since conditions change or agreed statement. project design may turn out not to be ideal. The structured multi-year planning of iii. The Council, WSP, and the World Bank projects which we advocate should include shouldalltakeconsciousstepstoensurethat provision for adjustments every year. Ways other parties understand the distinct nature should also be sought to allow mid-year of the Program and do not perceive it as a reviews of projects to adjust work programs subservient agent or arm of the World Bank; and budgets for the second half of each this should include but not be limited to financial year, with financial provision to attentiontosmallpresentationalsignalssuch make this possible (such as unallocated as the use of logos. reserves within core budgets or project budgets).Where practicable, donors and local iv. The Council should consider whether project partners should be involved in these or not any changes resulting from reviews, to take advantage of their insights, those recommendations require small to keep them informed, and to enhance the amendments to the precise terms of culture of multi-party ownership of projects theCharter. (which in turn can enhance sustainability). c) WSP central management should maintain its presentbalancebetweencontrollingtheregions 6.3 Institutional Arrangements and allowing them more autonomy, and and Governance the regions should respect that balance and [Section 4.4.1] comply with the necessary central discipline a) The Program should remain a partnership regarding planning, monitoring, and resource organization directed by its main stakeholders management. represented on its Council, while being hosted and administered by the World Bank. d) Awayshouldbesoughtforreportingtheviewsof the Regional Advisory Councils to the annual b) Within that structure, and as provided in Council meetings. clause 13 of the Charter, the Council should consider taking on a more active role than 6.5 Objectives and Strategy at present, through a delegated executive [Sections 4.2.1, 4.2.2] committee or otherwise, in the direction and While not seeing a need for any change in the controloftheProgram.Forexample,itmighttake Program's basic identity and objectives, we an active part in the choice of focus countries recommend that WSP should seek to clarify and and the selection of senior WSP staff such as a update its strategy by sharpening the definition of Program Manager or a Regional Team Leader. its objectives, working out carefully what aims are To this end: attheultimatelevel,suchaspovertyreduction,and i. The Council should consider changing its whatareintermediatelevels,asmeanstothehigher rules so that it elects its Chair from time to ends. (The Program's various current statements of time instead of being chaired automatically mission, objectives, and ways of working are not by an official of the World Bank. (This refers wrong, but they lack clarity and do not well reflect not to presiding at meetings but to the role of the present range of activities.) The distinction the Chair between the annual Council between outputs and outcomes could be useful for meetings when it is required to provide this process, though it must always be kept clear leadershipandresolveconflicts,representing who is the relevant implementing agency. This the WSPC in interactions with other updated strategy formulation should reflect the stakeholders.) emergingemphasisonaddressingthemassiveand External Evaluation Report 53 widespreadinstitutionalfailureandineffectiveness of staff skills, based on the recommended in the WSS sector, and the ways in which this can clarification of its objectives and role in the benefit poor people along with others. sector. Moving into a wider field than WSS may WerecommendthatWSPretainsafocusonWSSfor be inadvisable, if it would be too difficult to find poorpeopleinitsobjectives,buttakesabroadview and retain the necessary skills. Some skills can of the means to achieve them, not necessarily and should be developed by providing training concentrating on specifically `pro-poor' for existing staff, in fields such as change interventions. One of the benefits of a clear and management, risk management or outcome- updatedstrategywouldbetohelppersuadedonors oriented project planning. In particular, we to increase funding levels. recommend thatWSP should: try to recruit more people with technical Stepsinthisupdatingandclarifyingprocess,which backgrounds plus a good understanding should not need to be repeated more often than of people, motives, governance, and aboutonceinfiveyearsbutdonotneedtorefertoa institutions; fixed time period, could include: if not placing a core of thematic thinkers in 1. Reformulation of WSP's objectives in the form headquarters, give those in the regions an of a hierarchical tree of ends and means, set in explicitremit,plustimeandfunding,totravel context alongside wide development and to other regions or to global fora to interact human welfare concerns using poverty with their peers; reduction and health; consistently using the if using a significant proportion of relatively simplest possible terminology. inexperienced staff, make specific provision 2. Identification of higher and intermediate (with enough time and funding) for their objectives, activities, and fields of work which supervision, guidance, and monitoring by areWSP's particular niche; noting distinctions more experienced staff; and and overlaps relative to the niches of other acceptingthatmoststaffhavetobeemployed sector actors. on relatively short contracts, develop an explicit and transparent policy of providing 3. Using this to define and describe WSP's in-servicetrainingandexperiencetostaffwho comparative advantages relative to those of will then go on to carry WSP's insights and other actors such as networks and fora, ways of thinking into other sector advocacy specialists, research institutes, global organizations in their subsequent careers. and local NGOs, governments, donors, and the World Bank. b) Regarding WSP's position of not being able to give its staff long or open-ended employment 4. Applying the results of those steps to a review contracts(anychangewouldrequiremuchmore of the Program's thematic focus, aimed reliable medium-term funding), we hear with at maximizing its outcome-generating concern that recent or impending changes in effectiveness in the light of WSP's comparative World Bank procedures may make continuity advantage. of employment even more difficult for WSP to achieve. If this is the case, WSP should examine 5. Describing the resulting objectives and fields of the implications of any such changes for work clearly, without mixing ends and means. its staffing, and take steps to mitigate any 6. Finallyreviewingthewaytheclarifiedstrategyis negative consequences. expressed in brief form as a slogan or `mission statement'(whichinordertodescribeWSPbriefly 6.6 WSP's Ways of Working does need to mix ends and means in one OurrecommendationsforthewaysWSPgoesabout statement). itsworkfollowonfrom,andwouldcruciallydepend on,therecommendedclarificationofobjectivesand 6.5 Staffing and Staff Skills of relationships with other organizations. The [Section 4.4.7] recommendations in this section apply generally a) WSPshouldcontinueregularlytoreviewthemix to WSP management and staff. 54 External Evaluation Report a) The Program should strengthen its efforts to d) Inviewofthepoliticalsensitivityofinstitutional identify and spell out intended outcomes and reform,WSPshouldfurtherstrengthenitsefforts impactswhendesigningitsownprojects,rather tobuildupwhereverpossible closeandtrusting than concentrating on outputs (results of those relationships both with donors/funders and effortsarepatchysofar).Itshouldthusworkout with government officials and perhaps local improved and outcome-centered ways of politicians; this can enhance its distinct designing projects, not just describing projects advantagesasapartnerindependentoffunding already designed, making the distinction sources. [Section 4.3.2] between outputs under WSP's control and intended outcomes-impacts beyond that e) WSP should collate and develop its control.The intended outcomes should then be growing experience in serving decentralized includedinperiodicmonitoringandinasimple organizations as well as central or provincial standardizedprojectcompletionreport,usually governments, and seek ways to make it widely relyingonthesubjectivejudgementofWSPstaff availablewithinitsownoperationsandtoothers. and key local partners to asses how far the [Section 5.1] outcomes have been achieved. Internal f) WSP should consider using a wider range of guidelines and staff training may be useful, and media,forinstancevirtualfieldtripsonvideoto the improvements to project designing should bring field lessons to wide audiences, without primarilybeaninternalexercise,buildingonthe the need for extensive travel and simultaneous insights and experience of WSP staff, with little translation. [Section 4.4.6] or no reliance on outside experts.We do not see a need to change the format of the annual g) The Program does sometimes distribute the BusinessPlanssignificantly,onlytoimprovethe publications of other organizations, but we way they are prepared. The partners and clients consider this could be done more usefully, and of a project should preferably be involved in its theycoulddistributethoseofWSP.[Section 4.4.6] design. [Chapter 2, Section 4.2.3] h) WSP should seek more ways to exchange ideas b) WhenaWSPprojectisbeingdesigned,thiscareful and information between its regions, for formulation of the intended outcomes and instance by occasional visits by regional staff impacts should lead to the identification of the members to other regions. Examples of otheractors whoseattitudesandbehaviorneed innovativeworkinparticularregionsthatmight to change, and then to an analysis of their be applicable elsewhere are ways to reduce constraints,habits,motives,andpriorities.Then fixed costs, ways of working with WSSCC, the WSP project can be designed to work with upstream approaches, and private sector those factors, and can, if necessary, include involvement modes. elementstochangethem.Aclearframeworkand i) WSP should continue and strengthen its efforts methodology for project design, using this to overcome languagebarriers and to foster the analysis of the non-WSP actors, should make it work of sub-regions such asWest Africa; donors easier to achieve effectiveness and relevance should consider helping with the costs of more efficiently and consistently. Both this and translation and interpretation. the previous recommendation should not lead to staff spending more time on project design, j) WSP staff should have enough confidence in merely to using the time more efficiently. their own caliber and achievements to adopt a [Sections 4.2, 4.3] more self-questioning culture, together with a less defensive attitude to constructive criticism. c) When asked to help others to design an [Section 4.4.9] investmentproject,WSPshouldtakecarealways to offer and apply its own distinctive experience and judgement, avoiding being drawn into the 6.7 Relationships with Other waysofthinkingoftheotherpartiesunlessthose Global Organizations are demonstrably appropriate. [Section 4.4.4] [Section 4.3.2] WSP should reconsider how it works with other External Evaluation Report 55 organizations on the global WSS and related could do more to help each other in such matters fields, building on its review of objectives, as dissemination of publications and knowledge, comparative advantages, and strategy. Work with advocacy, persuasion, and the changing of NGOs is important, but we are more concerned attitudes. While care has to be taken not to about global bodies such as GWP and WSSCC overstretch the capacity and staff time of any whose fields abut or partially overlap that of WSP. organization, we consider that WSP could use In many cases, the various bodies in the sector existing institutional links more forcefully. 56 External Evaluation Report APPENDIX 1 Terminology for this Evaluation which in our hierarchy is the most important Figure1.1showshowtheRBMplanninglevelsrelate boundary.Effectivenessdescribesanintervention's tovariousaspectsofthisevaluation,inparticularto ability to bring about or stimulate the outcomes, the aspects identified in the `objectives of the giventheoutputs.Itisanimportantmeasureofhow evaluation' section of our ToR. well the intervention is designed and adapted to Thisdiagramshowsthefollowingevaluationaspects its context. (we use the neutral term `intervention' to mean a Sutainability and replicability: An inter- project, program or any other defined package of activities): vention's sustainability makes it last over many years, and its replicability enables its approach Efficiency: This describes an activity's or tobeappliedinmanyplaces,notjusttheplacewhere intervention's cost-effectiveness, as opposed to the intervention was undertaken; together these wastefulness,inusinginputs(resources)toproduce attributes enable the progression from outcomes to outputs; inputs include financial resources, the impacts.Thesetwocharacteristicsdonotnecessarily Program'shumanresources(itsmainasset),andthe come together; it is possible to replicate a project inputs of contributing partners. that is later found to be unsustainable, and a good Effectiveness: This refers to the interface between sustainable project can still be place-specific. For outputs(underthecontrolofWSPandparticipating valuable impact replicability is useful, while partners) and outcomes (outside that control), sustainability is essential. Figure 1.1: Planning Levels and Evaluation Aspects Hierarchy of Levels Evaluation Aspects Impacts of WSP work and links replicability sustainability Outcomes ofWSP work & (needs actors) relevance institutional of other opportunities of Outputs ofWSP activities ness to effective- strength resources Activities responsiveness efficiency Inputs External Evaluation Report 57 Relevance: This concerns the levels from outputs actors, through outputs that are designed to uptoimpacts;itisameasureofwhetherornotaset complement those of other actors, to the design of of outputs and outcomes (if achieved) are likely to interventions so as to maximize the chances of contribute to an appropriate higher level goal. achieving outcomes (effectiveness). Relevance is linked to sustainability and replicability; if a project is relevant, it is more likely In accordance with theTerms of Reference and our to be sustainable and replicable. view of relevant topics, the above six aspects are joinedinChapter 4byimpact,andbythefollowing Strength and nature of institutional links: aspects which are not specifically related to the These are crucial for several of the other hierarchyoflevels: aspects, especially efficiency, effectiveness, and Financeandresources sustainability/replicability. Focusandbalance Responsiveness to opportunities: This aspect, Planning, monitoring, and internal evaluation closely related to the previous one, concerns the Institutionalarrangementsandgovernance extent to whichWSP matches its work to the needs Relationshipswithothersectoractors andstrengthsofothersectoractors,allthewayfrom Gender inputandactivitylevelswhereWSPworkswithother Staffingandstaffskills 58 External Evaluation Report APPENDIX 2 Evolution of WSP's Planning and andkeyactivities',`outputs/results',and`indicators/ MonitoringMethods,1998-2003 milestones'. In successive years the table format WSP'sworkwasdefinedoverthefive-yearevaluation changed, for instance the one for fiscal year 2000 period in a series of Business Plans, whose nature (FY 2000,runningfromJuly1999toJune2000)hada and format changed every year. At the start, the column for `indicators of impact', while the next planning was being done by calendar years, and the year's plan featured a column headed `output 1998 document contained large numbers of `task indicators' that was explicitly linked to the three- planning sheets'. Some such sheets described year rolling perspective. But the individual `tasks' bundles of several activities extending over several frompreviousyears,eveniftheyhadbeenscheduled years. As well as listing activities, most of these 1998 torunonintoFY 2000orFY 2001,werenolongerto sheets explicitly set out the task's `overall objective', befoundundertheiroldnames(theyhadapparently `operational objective', `learning objective', `link never had reference numbers), so there was no to country activities', `comparative advantage', clearly discerniblecontinuity of planningfrom one `assumptions', `milestones', `outputs' (and fiscal year to the next.WSP staff gradually modified `communications/products'), `inputs' (in terms of thelogframeapproach,droppingmuchofwhathad staff weeks), `impact on sector at task completion', beenintroducedin1999butkeepingsomeaspects, and`keypartners'.Thisrepresentedasystematicand including the annual cycle. quite detailed planning system, though using a different terminology from ours (the `outputs', for IntheBusinessPlanforFY 2002,therewereregional instance,oftenimplyactionsbyotheractors,sothat and country impact statements, and also explicit some of them are what we would call outcomes). logframes for the regions. This time the three-year indicatorsweremorespecificthanforFY 2001,and This system was, however, not carried through to related to named countries. subsequent years. The planning shifted to a fiscal-year basis, July to June, with a transitional The more detailed Business Plan for FY 2003 arrangement for the first half of calendar 1999. In included regional strategy statements, but no that calendar year, partly as a result of a number of logframe.Theydidnotrelateclearlytotheprevious fragmented reviews of bits of WSP's work by or for year's impact statement and logframe, though for various donors, the Program hired a planning and practical purposes the main themes continued. At monitoring specialist to redesign the procedures. the global level it had descriptions of a`global rural He introduced the formal terminology of Logical strategy' and a `global urban strategy'. In this Business Plan the table was set out in terms of Frameworks(`logframes')forthewholeProgramand `projects' with individual reference numbers, but for regions and countries, but not for projects, theynowhadnoindividualstatementsofobjectives, together with a comprehensive annual process of outcomes or intended impacts, and their `product' preparing Business Plans, a mid-year review column usually comprised a list of activities. (normally in January), and an end-year evaluation. Planning was intended to be done on a 10-year Then, in FY 2003, partly as a result of a review and perspective, rolling so that in each year the period suggestion by the World Bank's OED, WSP hired extended 10 years ahead, and within this a more another specialist, charged with `modifying the detailed rolling three-year perspective. This approach from an impact assessment framework to planning scheme was adopted in varying degrees, a results-based management [RBM] one'. One and with varying levels of enthusiasm, by all parts outcome of this was the more conscious and of WSP over the next year or two. The task sheets deliberate packaging or clustering of work into were discontinued, and instead the Business Plans defined `projects', fewer in number and broader in included,alongsidesomelogframesforregionsand scope that the typical `tasks' of FY 1999. There was countries, tables in which tasks were represented also a shift towards defining and monitoring byrows,whilecolumnscoveredsuchmattersas(in outcomesratherthanoutputs(thesecondspecialist the plan for the remainder of 1998-99) `objectives used the same five planning level names as we do), External Evaluation Report 59 and towards coherent planning for more than one planning within the series of Business Plans. In the year at a time. BusinessPlanforFY 2004,however,these`flagships' were not mentioned as such, even though the ThelastfewmonthsofFY2003,whichendsourreview activities they had embodied continued. So the period, saw the preparation of the Business Plan for `flagships' seem to have been ephemeral, merely FY 2004 on this new basis, as an experiment to be fashionable flags briefly planted on lines of work developed further for FY 2005. In the Business Plan whichexistedlongbeforethelabelwasintroduced, for FY 2004 there are still country-specific strategy and often persisted after it faded. statements,andtableswithonerowperproject,but there are also individual`project sheets' that specify The fact that many of WSP's plans and documents eachproject's`desiredoutcome'(usuallyinthesense show only activities and outputs, leaving the in which we use the word, though sometimes outcome-level and impact-level objectives to be confused with outputs), and some explicit outcome guessedorinferred,islargelyareflectionofthelack indicators with target dates, and there are separate of clear statements of objectives, at what we call lists of activities and of products. There is again no outcome and impact levels, in the strategy clear continuity from previous years, the project documents and the successive annual Business numbering system being a new one, and most Plans, or at least the earlier ones.Those documents projectshavestartdatesinFY 2004itself,evenwhen did have logframes and `impact' or `strategy' they are obviously the continuation of processes statements in various forms, but they tended to be begun in previous years.The specification of inputs vague and general, describing problems and needs in the FY 2004 plan is related to a newly introduced and laying claim to comparative advantage, but set of project-specific job codes for staff to enter on seldom saying precisely what was to be done and theirtimesheets(calledIOsinWorldBankparlance), whatitshouldachieve.Rightattheendofourreview which will make monitoring of inputs easier than it period, the preparation of the Business Plan for has been in the past. FY 2004 began to remedy this, and WSP is already The detailed Business Plan for FY 2003 identified movingforward. `flagships' for each region and for two global strategies. These `flagships' were just some of the It is evident from this brief history that the planning themesonwhichWSPwasworkingatthetime.The andmonitoringsystemwasinastateofchangeover flagshiptableattheendofBP-FY 2003notablydeals most our review period. The process has, however, only with activities and outputs (`product improved considerably since about 2003, deliveries'), not objectives at outcome or impact and continues to do so. For the earlier years the levels,whicharedescribedinthetextoftheregional unsatisfactoryandchangingplanningformatshinder strategies.AnotherinterestingfeatureoftheFY-2003 any monitoring of multi-year processes (by WSP or flagship table is that it explicitly deals with us), and have been frustrating for theWSP staff.To a continuityfromtheyearbefore,byhavingacolumn large extent they have held to consistent long-term forFY 2002andsometimesshowingwhatactivities or medium term visions in particular sub-sectors, hadbeenheldoverbecauseofresourceconstraints. regions or countries, but they have often done this This is a rare and limited sign of explicit multi-year despite the planning system, not because of it. 60 External Evaluation Report APPENDIX 3 Previous Evaluations The following table lists some relevant evaluations that we have seen, and a brief discussion follows it. Title Author Date EVALUATIONS SPECIFICALLY OF WSP: An Evaluation of the UNDP-World Bank Water and `an independent Feb1996 Sanitation Program team' appointed by WB and UNDP The UNDP-World Bank Water and Sanitation Boman & Peck, for Feb1999 Program: The Andean Region; Results from the SIDA Monitoring Mission 1998 Three-YearReview1996-1999;AndeanRegion WSP May1999 External evaluation of the Swiss Agency for WSP and June1999 Development and Cooperation contribution to the CEPIS/PAHO UNDP-World Bank Water and Sanitation Program and to the Pan-American Center for Sanitary Engineering and Environmental Sciences (CEPIS/PAHO) Joint Assessment of the Regional Water and SIDA July1999 Sanitation Group for Eastern and Southern Africa (RWSG-ESA) DANIDA evaluation reports on West and Central DANIDA July1999 Africa and on South Asia (3 volumes) WSP comments on that DANIDA evaluation WSP undated WSP action plan in response to all those WSP Sep 1999 1999 evaluations Review ofWSP's governance Jon Lane Mar2000 WIDER REVIEWS WHICH ALSO COVER WSP BridgingTroubledWaters:AssessingtheWorldBank OED ofWB 2002 Water Resources Strategy (Publisheddocument) The World Bank's Approach to Global Programs: OED ofWB Aug 2002 (Grey cover An Independent Evaluation ­ Phase 1 Report Bank document) and Annexes ExternalConsultantEvaluationofKnowledgeBank: OED ofWB Nov2002 Water Supply Sector (ElizabethKleemeier and Keith Stallard) External Evaluation Report 61 Title Author Date Efficient,SustainableServiceforAll?AnOEDReview OED ofWB Sept 2003 (another of the World Bank's Assistance to Water Supply and version dated July and Sanitation Aug2003)(WorldBank document) Shared Knowledge: Innovations and Remaining OED ofWB 2003 Challenges (Catherine Gwin) (PublishedBank document) Water&SanitationSectorBoardAssessment...draft QAG of WB Dec2003 The 1996 Evaluation and defines the limits to its involvement in the ThisoverallreviewwasconducedfortheWorldBank sector. It recommended that WSP should develop and UNDP in 1996, well before our review period. a strategic plan for Africa, clarify its scope for There are some relevant points in it but many of involvement in different countries and its them are repeated in later evaluations ­ which is modes of operation, and precisely define the role maybe a lesson in itself.We note the following from of the Nairobi office and the Country Sector the1996executivesummary: Advisers. Other recommendations related to 1. No mission statement ­ that has been the development and implementation of a attended to. more effective management system and the development of marketing, communications, and 2. Needforreviewstoevaluateprogressagainst dissemination strategies. some work plans ­ some progress but still room for improvement. The DANIDA evaluation ofWSP's activities inWest 3. Need to develop means of measuring whether and Central Africa and South Asia reached some the Program is making a difference to the lives conclusions that were similar to those reached by of primary stakeholders . This relates to the SIDA team. It too noted the need for a strategic outcomesandimpactsandisdifficulttodo. framework,withinwhichtheProgramcouldbemore selective about its operations. Noting the strong 4. Need to move away from Bank rules and positive experiences in countries in which the procedures relating to personnel and need Program had a strong country presence, it argued to develop opportunities for career that support for sector policies, performed within development for staff members ­ probably individual countries, constituted a strong impossible to implement under the present comparativeadvantagefortheProgram. governance and funding arrangements. 5. Regionalgroupsshouldbestrengthenedand As in East and Southern Africa, existing planning perhaps made centers for specialized sector systems were found to be inadequate while expertise ­ this has been done to a monitoring and reporting systems were described considerable extent. asrudimentary.Theevaluationfoundthattherewas room for improvement in financial reporting Perhapsthemainlessontolearnnowfromthe1996 systems. It expressed concern that the close evaluation is that there are some things that can be relationship with theWorld Bank might restrict the changed and some that are very difficult to change. activities of the Program and force it to follow the Bank's agenda to the detriment of its own priorities The 1999 Regional Evaluations and concerns. Last, but not least, it found that the and Their Outcomes then Program Advisory Committee (PAC) had The SIDA evaluation of the East and Southern limited influence and that there was thus a need to Africa Program, carried out in 1999, noted the need developamoreeffectivechannelofcommunication for a strategy that clearly statesWSP's role in Africa between WSP and its donors. 62 External Evaluation Report WSP responded to these evaluations by: continued and that the need to explore new the Taking steps to improve project management consequences forWSP's finances remains. systems, as described in Appendix 2 of this report. ProducingaStrategyUpdateandProgressReport Lessons from OED (2001), which attempted to refocus the Program's and Other Evaluations agenda, clearly identify product lines, strengthen More recently, the Operations Evaluation governance and management processes, and Department of the World Bank (OED) has carried identify the actions to be taken to secure a out a number of evaluations with implications for sustainablefinancialbasefortheProgram. WSP. Each of these is now briefly reviewed. Thegovernancearrangementsweresummarizedin An OED review of the World Bank's water supply a singleWSP Charter, adopted in March 2001 at the andsanitation-relatedassistancewascompletedin first meeting of the WSP Council (see this report's September 2003. This review does not refer toWSP Section 3.4).This had replaced the PAC in a change directly but its findings have implications for the whichrepresentedadirectresponsetothefindings activities and future direction of the Program. The of the evaluations. review identified a need to modify the paradigm, predominant in the 1990s, which focused on The 2000 Review of WSP Governance developing systems for regulating private sector In March 2000, a consultant was commissioned by participation in the sector. Noting that there were the then Program Manager to produce a report on fewexamplesofinstitutionalizedregulationthrough WSP Governance. Much of this report was anindependentregulatorandthateffortstoregulate concerned with options for strengthening the PAC. through contracts could be undermined by It produced a number of possible governance macro-economicandpoliticalpressures,thereview models, thus facilitating the introduction of the noted the need to move from prescribing to WSP Council. implementing regulatory systems. WSP-AF is already providing inputs to efforts to implement This report makes a number of other important design and implement regulatory systems that are points which still appear to be relevant. These relevanttoconditionsinthecountriesintheregion. include the following: It may be that other regions could usefully increase 1. WSP does not exist as a separate legal entity. their efforts in this area. This increased focus on Rather, it is legally a part of the World Bank developingandregulatingappropriateformsofPSP and its finances are handled through the should form one element of a drive to develop World Bank system. WSP must follow World implementable sector strategies that can, in the Bankfinancialandadministrativeprocedures words of the OED, chart a route towards the MDGs. at all times. 2. Nowrittenguidelinesareavailableontheway InNovember2002,theOEDcompletedanExternal in which the line should be drawn between ConsultantEvaluationofwatersectorpublications. `the Bank's proper accountability and the ThereviewincludedanumberofWSPpublications. WSP's necessary degree of independence'. Publications were reviewed in terms of their 3. Somestakeholdersfeltaneedforgreaterefforts relevance, comprehensiveness, content, clarity, to be made to ensure that WSP is a true objectivity, and practicality. Depending on the partnership with beneficiaries, donors, and resultsofthisreview,theywereclassifiedintermsof others, feeling that they are respected as their content (relevance, comprehensiveness, and equals. (Is the fact that WSP routinely refers content) and their potential for `sharing and to government departments as clients rather application' (clarity, objectivity, and practicality). than partners a factor here?). Theresultsofthereviewforthe19WSPpublications This report also noted that donors were were as follows: decentralizing their budgets and staff to regional Publicationof`benchmark'quality 5 and/or country offices and suggested that this Satisfactory 5 might have repercussions on WSP's financing Weak content 2 arrangements. It seems that this trend has Weak sharing and application 3 External Evaluation Report 63 Unsatisfactory with respect is thus for a greater degree of central control over to both content and sharing/application 4 knowledge processes. At present, WSP has little direct involvement with the Bank's global The main specific weaknesses identified by the knowledge initiatives. However, given the fact that review were: knowledge generation and dissemination is an A lack of objectivity ­ in some cases a document important part of its remit, any move to centralize set out to prove rather than test a hypothesis and the control of these initiatives within the Bank marshalled its arguments accordingly. would have important consequences. Not least, it Some documents were too general and failed to wouldrepresentamovetolinkthevariousprograms provide practical advice on their findings morecloselytotheBankandwouldarguablyresult and conclusions. in a reduction in WSP's independence. Some documents failed to adequately address important issues, particularly politics and WSP's Response to Evaluations social factors. WSPhastakensomeimportantdecisionsonthebasis The methodology and quality of this OED review oftheevaluations.Themostimportantoftheserelate may themselves be open to criticism, and its to the Program's planning systems and the findings were challenged in detail by WSP, but our replacement of the PAC with the Council. However, rapidreviewofanumberofthedocumentssuggests ithasnotacceptedallcriticismsandhasonoccasion thattheOEDreviewwasgenerallyfairandthatthere responded to evaluations with detailed rebuttal of wereindeedproblemswithasignificantpercentage many of the points contained in them. The defence of publications. Given the fact that knowledge hasusuallybeenonthegroundseitherthatthepoints development and management is one of WSP's made in an evaluation were not justified or that the `productlines',thefindingthatalmost50percentof evaluators were dealing with the past and had not publications reviewed were unsatisfactory in some taken account of WSP's more recent efforts to deal way is worrying. Since the completion of that OED with problems. While WSP's responses do review,WSP has taken positive steps to improve its undoubtedly include valid points, they do point to quality control procedures for publications. This is theexistenceofaratherdefensiveculturewithinthe tobewelcomedand,togetherwiththeemployment organization. Our brief review of some of the of communications specialists in Africa and LAC, documents covered in the 2002 OED review of shouldhelptoaddresstheproblem.However,there documents,forinstance,ledustosimilarjudgements is no room for complacency. to those made by OED.This is an issue that needs to beaddressed.Itwillrequireactionboth: Oneissueforthefuture,relatingtotherelationship ByWSPitself,todevelopamoreself-questioning betweentheProgramandtheWorldBank,concerns attitude; and the latter's commitment to become a `global Bydonors,Bankmanagersandothertowhomit knowledge bank'. It aims to do this through three reports, to provide suitable incentives for the global initiatives, the Development Gateway, the development of this self-questioning attitude. Global Development Learning Network (GDLN) and the Global Development Network (GDN). An One aspect of the latter would be recognition that, OEDevaluationreport,producedin2003,notesthat given the complexity of issues relating to these initiatives have sparked considerable development in general and WSS in particular, the innovation, knowledge sharing programs, and results of interventions will not always be as activities but that they will only achieve their broad expected. Close adherence to a system that objectivesiftheyarebetterintegratedintotheBank's emphasizestargetsaboveeverythingelseisunlikely core business processes.The report concludes that to produce lasting results in such circumstances. there is a need for Bank management to exercise Rather, there is a need for a greater emphasis on more strategic direction and oversight over the evaluation of the strengths and weaknesses of past Bank'sknowledgeprocesses.Therecommendation initiatives, in order to learn from both. 64 External Evaluation Report APPENDIX 4 Finance sanitation, is obviously difficult, so the relative Detailed financial reporting is not a part of this amounts are subject to some uncertainty. Year-to- evaluation. The table below summarizes the yearcomparisons,inparticular,mustbetreatedwith Program's expenditure in the last four years of the caution because any change in the way things are evaluation period (FY 1999 cannot be included categorized could distort the relative amounts. For because amounts were broken down in different example,onegender-relatedruralsanitationproject ways at that time). Some details are in the region might be classed as rural expenditure while a very annexes to this report. similar one in another time or place might be classified as cross-cutting because of the gender The table gives a more detailed breakdown of most aspect. In particular, the apparent increase from of these expenditures for the same four fiscal years, FY 00toFY 01inrural,urban,andsmalltownwork, plus FY 04 for interest. The expenditure is broken withasimultaneousdecreaseincross-cuttingwork, downintothethreespatialcategories­rural,urban, may be at least partly due to this sort of effect (the and small towns, plus three other categories which accountingproceduresweredifferentinthoseyears inevitably cut across the spatial ones. Classifying from what they are now, and these statistics have expenditurethatpertainsbothtoaspatialcategory had to be reconstructed more recently using and to another one, such as gender work for rural product descriptions). WSP Expenditures, million US$ FY00 FY01 FY02 FY03 Total Percent HQ&G* 2.5 2.7 2.3 2.1 9.6 19% LAC 1.7 2.2 1.2 1.0 6.1 12% EAP 2.3 2.3 2.6 3.1 10.3 20% AF 2.9 4.4 3.4 3.9 14.6 28% SA 3.1 3.3 2.5 2.3 11.2 22% Total 12.5 14.9 12.0 12.4 51.8 100% Source: Information from WSP on April 27, 2004 *This category includes expenditures of theWSP global management and the global thematic program (approximately 50 percent each) External Evaluation Report 65 APPENDIX 5 People Met and Consulted Edward Jaombe, Manager of Luwero water supply system AFRICA REGION (on behalf of Management Contractor) WSP-AF (Kenya) Representatives of partner organizations Piers Cross, RegionalTeam Leader Dennis Mwanza, WUP Alain Morel, Urban Specialist SamuelWambua, NETWAS Andrew Makokha, SeniorWater and Rosemary Ropp, Maji na Ufanisi Sanitation Specialist Simon Kenny, (DFID), Telephone conversation Sapheth Mbuvi, Operations Analyst Andreas Knapp, Consultant EAST ASIA-PACIFIC REGION Meera Mehta, Senior Financial Specialist Thomas Fugelsnes, Economist WSP-EAP (Jakarta, unless otherwise noted) Jean Doyen, Consultant Richard Pollard, RegionalTeam Leader KeziahKihara,OfficeAdministrator Ariyanto Istandar, Water and Sanitation Specialist Christine Njung'e, Budget Analyst Ratna Josodipoera, Hygiene Education Specialist Alfred Lambertus, SeniorWater and Sanitation WSP (Dakar) Specialist Annie Manou Savina, Senior Community Nilanjana Mukherjee, Senior Community Development Specialist Development Specialist Janelle Plummer, Senior Institutional WSP (Uganda) Development Specialist Maimuna Nalubega, Country Sector Adviser Ann Thomas, Water Supply and Sanitation Anthony Waterkeyn, Consultant Consultant Yosa Yuliarsa, Regional Communication Specialist WSP Communications team Thomas Meadley, Country Team Leader Sarah deVilliers Leach, Communications (based in Vientiane, Laos) Specialist Jemima Sy, Country Program Officer Cecilia Martinsen, JPO Communications (based in Manila, Philippines) Specialist Jan Rosenboom, Country Team Leader (based in Phnom Penh, Cambodia) World Bank Makhtar Diop, Country Director, Kenya, Government officials in Indonesia Eritrea, and Somalia Dr Umar Fahmi, DG Communicable Wambui Gichuri, Economist AFTU1 Disease Control DjokaWartono, former Director,WSLIC 2 Government officials in Kenya Zainal Nampira, Project Manager, WSLIC 2 Engineer Lawrence Mwangi, ex Nairobi Basah Hernowo, Head ofWASPOLA City Council WorkingGroup David Stower, Deputy Director, Ministry Dr Ir. Suyono Dikun, Deputy Minister ofWaterResources ofInfrastructure Government officials in Uganda Representatives of partner organizations Paul Luyima, Assistant Commissioner of Health (Jakarta, Indonesia) Services and Head Environmental Health JimWoodcock,Water and Saniation Adviser, (MoH) USAID EngineerEphraimKisembo,ReformManager, Zabeta Moutafis, First Secretary, AusAID DWD HildaWinata,West Java Representative, UNICEF 66 External Evaluation Report Jaco Mebius, First Secretary,Water Resources Omar Macedo Ruiz, Centro de Estudios y Mgt., Royal Netherlands Embassy Solidaridad con América Latina Carlos Escalante Estrada, CENCA (Instituto de World Bank (Indonesia Country Office, Jakarta) DesarolloUrbano) Andrew Steer, Indonesia Country Director VirginiaBaffigo,CARE,Peru AniruddhaDasgupta,InfrastructureSector Juan Francisco Soto Hoyos, CARE PROPILAS, Coordinator Cajamarca,Peru Janet Hohnen, Health Sector Coordinator Walter Cabrera, CARE PROPILAS, Cajamarca, Peru (WSLIC2 Project) Mirtha Villanueva, CARE PROPILAS, Cajamarca, Peru LATIN AMERICA & CARIBBEAN REGION Alfonso Aguirre, CARE PROPILAS, Cajamarca, Peru WSP staff Percy Suarez, CARE PROPILAS, Cajamarca, Peru Luis Tam, Regional Team Leader Juan Salazar, CARE PROPILAS, Cajamarca, Peru RafaelVera, Peru Country Coordinator Eduardo Bustamante, President of JASS (users Oscar Castillo, Institutional and Community association for water supply scheme), Development Specialist Monterey, Cajamarca, Peru Rocío Flórez, Handwashing Coordinator José Barragán,Vice Ministry of Basic Services, Martin Ochoa, Coordinator for Central America Bolivia Jorge Luis McGregor, Small Towns Management Antonio Terrazas,Vice Ministry of Basic Services, Models Specialist Bolivia Maria Luz Perez, Communications Consultant EnriqueTorrico,Vice Ministry of Basic Services, Norma Chávez, Language Team Assistant Bolivia Cecilia Ubillus, Administrative Assistant Juan Carlos Sumaeta, Fundación SUMAJ HUASI & Luciana Mendoza, Communications Assistant CAUDAL, Bolivia Zoraida Mantilla, Regional Team Assistant Francisco Guachalla, Catholic Relief Services, Bolivia Stakeholders Betty Soto, Bolivia Jean Bernard Parenteau, CIDA/ACDI, Peru EnriqueTorrico,Bolivia María Elena Bodero de Eran, CIDA/ACDI, Peru Roberto Bianchi, Aguas del Illimani, Bolivia Beatrice Meyer, SDC/COSUDE, Peru Tania Jaldin Trigo, Aguas del Illimani, Bolivia Gilbert Bieler, SDC/COSUDE, Peru Mario Flores Chávez, Aguas del Illimani, Bolivia Guillermo León Suematsu, National Director of Albero ChávezVargas, Aguas del Illimani, Bolivia Sanitation, Ministry of Housing Construction Humberto Puerto, RASCA (Central America and Sanitation, RegionalWatsan Network), Honduras alsoVice-President of SEDAPAL, Peru Hugo Cobo, FIS/KfW, Honduras Javier Hernández Campanella, PRONASAR, Peru Givanny Ayestas, FIS Watsan Pilot Project, Roxana León, PRONASAR, Peru Honduras Andrew Michell, Alicorp, Peru ThomasWalder, SDC, Honduras DalmiVillar, Radio Programas del Perú Yadira Recinos, SDC, Honduras Juan Manuel Calvi del Risco, Radio Programas Edith Rivera, Save the Children, Honduras delPerú Nora Reyez, MINSA, Peru SOUTH ASIA REGION Carmen Calle, MINSA, Peru Jaime Salcedo, SEDAPAL, Peru Javier Acosta, SEDAPAL, Peru WSP-SA RolandoVásquez, SEDAPAL, Peru (New Delhi, India, unless otherwise noted) Luis Valencia, CEPIS Junaid Ahmad, RegionalTeam Leader Juan CarlosValencia, Plan International Marianna Kullappa, State Coordinator Carlos Bendezú, Fondo de las Americas (AndhraPradesh) Roger Agûero Pittman, Servicios Vandana Mehra, Regional Communications EducativosRurales Specialist External Evaluation Report 67 David Savage, Sr. Municipal Development Patel, Sheela, Director, SPARC Specialist Patra, Lalit Mohan, Project Officer, UNICEF Mark Ellery,Water and Sanitation Specialist Rajamani, M, Ministry of Urban Development, Shekhar Shah, Lead Economist Jt Secretary Salman Zaheer, Lead Utilities Specialist Ray, Samarjit, Chairman, Andhra PradeshWater Conservation WSP-SA (Dhaka, Bangladesh) Sanan, Deepak, Government of Himachal Shafful Azam Ahmed,Water and Sanitation Pradesh, Secretary Rural Development & Specialist and Acting Team Leader, PanchayatiRaj Bangladesh CountryTeam Shah, Ajay, Ministry of Finance, DEA Khawaja Minnatullah, Sr.Water and Sanitation Srivastava,Vivek, ExWSP India Country Team Specialist Subramanium, Ramesh, AusAID Senior Tanveer Ahsan, Urban Specialist ProjectOfficer Kazi Adil Ahmed Shafi, Consultant (RWSS) Tripathi, PK, CEO, Delhi Jal Board Md. Abul Fayez Khan, Program Assistant Thirupathiah,Dr, APPARD,Hyderabad Van Norden, Henk, UNICEF Chief, WES WSP-SA (Islamabad, Pakistan) RajaRehanArshad,TeamLeader,Pakistan Bangladesh Country Team Abdullah, Md., Deputy Chief, Local Government Division, Ministry of Local Government Others Ahmed, Prof M Feroze, Department of Civil India Engineering, BUET Abhyankar, GV,World BankWater and Sanitation (Bangladesh University of Engineering & Specialist Technology) Andersson, Owe, SIDA Head Development Ahmed, Shehlina, PLAN Health Adviser Cooperation Claydon, Timothy, Country Representative, Banerjee, S, Ex. Joint Secretary, Ministry Urban WaterAID De Groot, Carel, Danida Coordinator Development Dyer, Rodney, DFID Engineering Adviser Barrett, Alison,World Bank Regional Adviser Edwards, Paul, Chief,Water and Environmental Cities Alliance Sanitation Section, UNICEF Benmassaoud, Rachid,World Bank Hossain, Khandker Zakir, Program Director, Operations Adviser WaterAID Briscoe, John,World BankWRM Adviser Hossain,Yakub, Deputy Director,VERC Chakravaty, Sunita, SIDA Program Manager (Village Education Resource Center) Chary, ProfV Srinivas, Senior Faculty, Howard, Guy, DFID Engineering Adviser Administrative Staff College of India (Arsenic Mitigation) Chaudary, Nazmaul,World Bank Economist DEC Kemper, Karin, World Bank, SASE Curtis, Ian, DFID Senior Adviser Khan, Rifat Shahpar, Senior Program Officer, Doolan, Sean, DFID Environmental Adviser AusAID Durgaprasad, CEO, Zila Parishad Chittoor (AP) Motaleb, Abdul, Danida Program Officer Khatua, BC, Government Maharashtra Principal Qasem, A H M Abul, Ministry LGRD and Secretary, W&S Cooperatives Kumar, Ajith,WSP State Coordinator, Maharashtra Racki, Jeffrey,World Bank, SASE Kutty, PVValsala, Director, Rajiv Gandhi National Rashid, SMA, Executive Director, DrinkingWater Mission NGO Forum for DWSS Misra, Smita, World Bank NDO Economist Swann, Peregrine, Senior Infrastructure and Mohandas, Palat, Secretary, Ministry of Livelihoods Adviser, DFID Rural Development Waldvogel, Markus, Counsellor (Development), Murty, JVR,WSP State Coordinator, Maharashtra SDC, Bangladesh Nagireddy, Secretary, Panchayati Raj and Rural Wallich, Christine,World Bank Development, Ministry of Andhra Pradesh Country Director 68 External Evaluation Report Washington Roche, Robert, WSS Specialist, Africa, World Bank Saghir, Jamal, Director, Energy andWater Sara, Jennifer, WSS Specialist, LAC, World Bank Department, World Bank Van den Berg, Caroline, SeniorWSS Specialist, INF, Stottmann,Walter, Program Manager,WSP WorldBank Iyer, Parameswaran, Senior WSS Specialist, WSP Van Ginneken, Meike, Water and Sanitation Kolsky, Pete, Senior WSS Specialist, WSP Specialist, World Bank Roa, Donna, Senior Communications Officer, WSP Others Paynter, Nat, Consultant, WSP Armon Hartmann, Retired, formerWater Adviser, Gerrard,Chris, Senior Evaluation Officer, OED, SDC WorldBank Willem Ankersmit, Retired, formerWater Adviser, Haxaire,Marie Claude, Resource Manager, INF, DGIS WorldBank MartinWalshe,Water Adviser,EU Jagannathan Vijay, Sector Manager - Middle East/ Dennis Mwanza, former Managing Director,WUP North Africa,World Bank Brian Baxendale, Divisional Engineering Adviser, Janssens,Jan,ProgramManager,Bank-Netherlands West Africa, DFID WaterPartnership Koenig, Peter, SeniorWater Resource Management Darren Saywell, former Program Manager,WSSCC Specialist, LAC, World Bank Gourishanker Ghosh, Executive Director,WSSCC Locussol, Alain, WSS Specialist, South Asia, World Barbara Evans, Consultant, former Global Urban Bank Thematic Leader,WSP McKecknie, Alister, QAG, World Bank BruceGross,Retired,formerProgramManager, Mejia, Abel, Sector Manager, Latin America/ WSP Caribbean,World Bank Stephen Bass, Head of Profession, Environment, Ravat, Anwar, Chief Administration Officer, INF, DFID WorldBank and two others External Evaluation Report 69 APPENDIX 6 Documents Seen and Consulted [see also Appendix 3 concerning previous evaluations, which are not repeated here] Title Author or Issuing Organization Date WEBSITE www.wsp.org: a well-designed and informative website, regularly updated, with news and many contacts WSP GLOBAL ANNUAL PLANS, MID-YEAR REVIEWS AND REPORTS RWSG-ESA, Jan-Dec 1998 (reputedly the 1999 WSP (presumed) undated Business Plan) WSPWorkplan,Jan-Jun1999 WSP Jan1999 FY 2000 Business Plan WSP undated (untitled, loose papers in file) FY 2001 Business Plan WSP undated (untitled, loose papers in file) FY 2002 Business Plan WSP undated (untitled, loose papers in file) WSP FY 2003 Business Plan WSP undated WSP Mid-year review (apparently FY 2000) WSP undated 2001Mid-yearreview WSP undated WSPFY2002Mid-termreview WSP undated WSPFiscalYear2003 WSP undated Mid-termreview 98-99WSPReport WSP undated 99-00Report WSP undated 00-01Report WSP undated 01-02Report WSP undated WSP Annual Report WSP undated (July2002-June2003) (draft) 70 External Evaluation Report Title Author or Issuing Organization Date PUBLICATIONS LearningWhatWorks: A 20-Year RetrospectiveView MaggieBlack 1998 on International Water and Sanitation Cooperation Taking Sustainable Water Supply Services to Scale: BNWP/WSP (Jennifer Davis, December A Discussion Paper ParameswaranIyer) 2002 Condominial Water and Sewerage Services: Costs WSP-AND/Viceministerio de undated of Implementation of the Model (El Alto-Bolivia, Servicios Básicos, Bolivia/ (ca2002) Pilot Project) ­ Economic and Financial Evaluation ASCID (Vivien Foster) Sistemas condominiales de alcantarillado sanitario PAS-AND/Viceministeriode undated ­ Cambios en la disponibilidad de infraestructura Servicios Básicos, Bolivia/ (ca2002) sanitario y en hábitos de hygiene a partir de las ASCID (Nelba Canelli) implantación del proyecto piloto: Un enfoque cuantitativo; Evaluación de Impacto (Proyecto Piloto El Alto ­ Bolivia) Sistemas condominiales de alcantarillado sanitario PAS-AND/Viceministeriode December ­ Guía de procedeimientos (Proyecto Piloto El Alto Servicios Básicos, Bolivia/ 2001 ­ Bolivia) ASCID (Mery Quitón, Adele Martínez) Informe delTaller destinado a la`Presentación de Vice Ministerio de Servicios 2001 los Resultados de la Implantación del Sistema Basicos (Bolivia); PAS-AND; Condominial en La Paz y El Alto' ASDI BetterWater and Sanitation for the Urban Poor: WaterUtilityPartnership/WSP July2003 GoodPracticefromsub-SaharanAfrica Linking sustainability with Demand, Gender, and WSP and IRC (Gross, van Wijk January2001 Poverty ­ A study in community-managed water andMukherjee) supply projects in 15 countries Sustainability, Planning, and Monitoring in WSP and IRC (eds Mukherjee 2003 Community Water Supply and Sanitation and vanWijk) Meeting the Financing Challenge for Water Supply MeeraMehta May2003 and Sanitation WSP/WorldBank In Pursuit of Good Governance: Experiments from WSP and Massachusetts April2003 South Asia'sWater and Sanitation Sector Institute of Technology (Davis, Ghosh, Martin, Samad, Tankha, Zia and Prunier) External Evaluation Report 71 Title Author or Issuing Organization Date Water Supply and Sanitation for Small Towns and WSP/WorldBank/Ministryof 2002 Multi-Village Schemes (Proceedings of an Water Resources, Ethiopia, international conference in Addis Ababa, June BNWP/World Bank Insititute 2002),Vol 1, conference summary Willingness-to-Pay for Arsenic-Free, Safe Drinking WSP/WorldBank/Bangladesh August2003 Water in Bangladesh (Fighting Arsenic: Listening Rural Advancement Committee to Rural Communities) (Junaid Ahmad, BN Golder, Smita Misra, M Jakaria) Estudios de Base para la Implementación de Vice Ministerio de Construcción May2003 Proyectos de Agua y Saneamiento en el Area Rural y Saneamiento, with help from Dirección Nacional de Saneamiento, UGP-PRONASAR, COSUEDE, AGUASAN- COSUEDE, PAS-LAC Abastecimiento de Agua por Gravedad para ProAnde (Nicolas Marinof), November PoblacionesRuralesDispersas with editorial support from PAS- 2001 AND, also supported by UNICEF, COSUDE, CHRISTIAN AID, VOLENS La Contratacion por la Communidad ­ PAS-AND (Oscar Castillo) November Empoderamiento y Saneamiento Rural en 2001 elPerú WaterTariffs and Studies in South Asia (a series of PPIAF/WSP (ed Clarissa December papers): 1. Understanding the Basics; Brocklehurst) 2002 2. A Scorecard for India; 3. Tariff Structures in Six South Asian Cities Achieving Sustained Sanitation for the WSP-EAP(NilanjanaMukherjee) April2001 Poor ­ Policy and Strategy Lessons from Participatory Assessments in Cambodia, Indonesia, Vietnam Desinfectión Solar del Agua ­ Guia de Aplicacion SANDEC (translation of June2003 EAWAG/SANDEC publicaqtion (ISBN3-906484-24-6))withhelp from PAS-LAC, COSUDE, UNICEF El Crédito para Agua y Saneamiento Rural: una IEP (CarolinaTriveli), for PAS- February Propuesta AND 2003 Memoria deTaller sobre Sistemas de Información PAS-AND,Vice Ministerio de November Sectorial para el Saneamiento Básico Rural Infraestructura 2000 72 External Evaluation Report Title Author or Issuing Organization Date PROPILAS: Nota de Campo 1: CARE-PROPILAS, with editorial September la Selección de Contratistas para Ejecutar help from PAS-AND, and 2000 Proyectos de Saneamiento Rural support from COSUDE, CARE PROPILAS: Nota de Campo 2: CARE-PROPILAS, with editorial August2001 la Selección de Municipios y Comunidades help from PAS-AND, and 2002 support from COSUDE, CARE PROPILAS: Nota de Campo 3: CARE-PROPILAS, with editorial May2002 La Participación del Sector Privado en el help from PAS-AND, and Saneamiento Rural support from COSUDE, CARE PROPILAS: Nota de Campo 4: CARE-PROPILAS, with editorial July2002 Dos Modelos en la Provisión de Servicios de Agua y help from PAS-AND, and Saneamiento Sostenibles support from COSUDE, CARE Lecciones Aprendidas del Proyecto Piloto de Agua COSUDE, with help from PAS- Nov2002 y Saneamiento PROPILAS en Cajamarca, Perú LAC (1st and 2nd editions) Aug2003 Peru: Saneamiento Básico Rural ­ Analisis World Bank (confidential draft, April1999 Sectorial y Estrategia report19209) Los servicios de agua y saneamiento en el Perú: un ACDI and PAS-LAC Aug2001and diagnostico y estadisticas 2ndedition May2003 Project Appraisal Document on a Proposed Loan TheWorld Bank, August2002 in the amount of US$50 million to the Government Report No. 24173 of Peru for a National RuralWater Supply and Sanitation Project Servicios de agua en zonas periurbanas de Lima PAS/UniónEuropea March2002 Metropolitana ­ la experiencia del Proyecto (Giovanni Bonfiglio) Alimentación de Agua para Pueblos Jovenes (APPJ) La Sostenibilidad de los Sistemas Autonomos de SER for SEDAPAL none (2003) Abastecimiento de Agua del Projecto APPJ in Lima, (SER seems to be part a incios del 2003 of CASMA) Estudio de las Practicas de Hygiene en Indivduos y SER none FamiliasdePoblacionesPeri-urbanasdeLima Metropolitana Planificacion del Saneamiento Ambiental Urbano: WSP-SA with acknowledgement August Lecciones de Bharatpur, Rajasthan, India to DFID 2002 Health and Hygiene Education inWater and CARE andWSP April2001 Sanitation Projects (also available in Spanish) External Evaluation Report 73 Title Author or Issuing Organization Date Metodologias Participativas en Educación ProAnde and UNICEF, May2003, Saniataria ­ un Adaptión de PHAST para published byWSP-LAC second ComunidadesRuralesAndinasdelPerú edition EvaluaciónProgramaAguasan­Informe COSUDE Aug-Nov 2002 HealthinyourHands­Public-PrivatePartnerships WSP July2003 for Health: A review of Best Practices (Ann Thomas,Valerie Curtis) in the Health Sector La Salud en tus Manos ­ Alianza Público/Privada the alliance, Lima, Peru ?2003 para Promover el lavado de manos... Plan de Negocios La Salud en tus Manos...Executive Report La Salud en tus Manos...la alianza global entre los WSP-LAC, mentioning none, sectores publico y privado para promover el lavado COSUDE, MINSA, USAID probably de manos con jabon 2003 La Salud en tus Manos...estudio de Apparently by PRISMA, none, comportamientos en lavado de manos con for the alliance probably jabonen zones urbano perificas y rurales del Péru 2002or2003 Red Nacional de Agua y Saneamiento de Centro RRAS-CA, with support from none America (RRAS-CA) WSC (Martin Ochoa) Construyendo la equidad de género: (Jesica Nino de Guzman, 2nd ed., metodologia e instrumentos para su inclusión en Mecedes Zevallos C) WSP of the May2003 proyectos de agua y saneamiento WB (sic), with acknowledgment to theWB Gender Division Norma Boliviana NB 688, Installaciones Sanitarias Dirección Nacional de 2001 ­ Alcantarillado Sanitario, Pluvial yTratamiento de Saneamiento Básico, DIGESBA, AguasResiduales Bolivia Building Partnerships to Serve the Urban Poor: WUP/WSP Summary Proceedings ­ FinalWorkshop of WUP Project No. 5, Abidjan, Côte D'Ivoire, November 19-21, 2001 List of Water and Sanitation Program-Africa WSP-AF February (WSP-AF)Publications,1998-2003 2004 Meeting the Challenge ofWater and Sanitation WSP-ESA 1999 Services for the Poor in East and Southern Africa: AThree-yearRegionalBusinessPlanFY2000-2002 74 External Evaluation Report Title Author or Issuing Organization Date New Designs forWater and Sanitation WSP/PPIAF May2002 Transactions: Making Private Sector Participation WorkforthePoor Private Sector Participation inWater Supply and BNWP,World Bank Sanitation Services in Sub-Saharan Africa: Decision-Maker'sWorkshop:Summary Proceedings and Outline for a Roadmap: Dakar, Senegal,February13-15,2002 Report of the Regional Program Advisory WSP Committee (RPAC) Meeting, March 1-2, 1999, Nairobi,Kenya Report of the Regional Advisory Committee (RAC) WSP Meeting, November 6-8, 2002, Dakar, Senegal Small-scale Independent Providers ofWater and FaridMohamed November Sanitation to the Urban Poor: 1999 A case of Nairobi, Kenya UrbanWater and Sanitation Reform Strategy Government of Uganda, July2003 Ministry ofWater, Lands, and Environment Water and Sanitation in Poverty Reduction Strategies, Workshop proceedings, Regional Workshop,Nairobi,June 17-19,2002 Water Supply and Sanitation in Africa: How to Unpublished paper measure progress toward the Millennium Development Goals? Water Supply and Sanitation in Poverty Reduction WSP Pre-print October Strategy Papers in Sub-Saharan Africa: 2003 Developing a Benchmarking Review and ExploringaWayForward WSP-ESAMid-YearReviewof WSP February WorkProgramFY2000 2000 WSP Strategic Program Planning and Management WSP-ESA Undated System ­ PPMS NETWORK NEWSLETTERS Agua ­ Boletín del Comité Sectorial de Agua y Comité Sectorial de Agua y Saneamiento, Saneamiento Lima,Perú External Evaluation Report 75 Title Author or Issuing Organization Date AguaYaku ­ Comite Sectorial de Agua y Comite Sectorial de Agua y Saneamiento, Saneamiento Ecuador Caudal ­ Revista Sectorial de Agua y Saneamiento caudal@sumaj.org Cuenta Gotas -- boletin de la RRAS-CA RRAS-CA WSP NEWSLETTER ACCESS ­ an electronic newsletter available by WSP continuing signing on or accessible through theWSP website series WSP FIELD NOTES Urban Sewerage and Sanitation: Lessons learnt Joint publication with DILG June2003 from case studies in the Philippines Philippines and AUSAID The Best-laid Plans ­ Revisiting Community-based WSP-SA March2002 RuralWater Supply Schemes in Uttar Pradesh Lower Costs with Higher Benefits ­Water and WSP/Vice-Ministry of Basic undated Sewerage Schemes for Low Income Households ­ Services/SIDA (Vivien Foster) (ca 2002) Lessons from the El-Alto-Bolivia Pilot Project Fighting Arsenic: Listening to Rural Communities WSP/WorldBank/Bangladesh December ­ Findings from a Study onWillingness to Pay for Rural Advancement Committee 2002 Arsenic-Free, Safe DrinkingWater in Bangladesh (Junaid Ahmad, B N Golder, Smita Misra, M Jakaria) Promocion de la salud y la hygiene a través del Field note byWASP-LAC, 2nd ed., sistema escolar en el Péru mentioning COSUDE and Kallpa April2003 Nota de Campo ­ la micromedición en el area PAS-LAC, mentioning CARE, June2003 rural: una experiencia exitosa en El Salvador USAID (Ricardo Mairena) Nota de Campo ­ escuela y casa saludable: una PAS-LAC, mentioning UNICEF, September experiencia exitosa en Honduras SANAA, ASDI, CRS 2003 Nota de Campo ­ soluciones innovadura para el PAS-LAC, mentioning FHIS, September suministrodeaguaencomunidadesrurales KfW, SANIPLAN, INBAS, SODIS 2003 dispersasdeHonduras NotadeCampo­Lasnormastécnicasylaampliación PAS-LAC ­ decoberturadeacueductoyalcantarilladohacislos sectores pobres ­ El proceso de modificación de la Norma Boliviana NB 688 Nota de Campo ­ la Asociacion de Usarios en la PAS(severaloffices),with October Gestion de Servicios de Agua en Localidades contributions from ACDI, SUM 2001 RuralesMultiples Canada,andtheusersassociation 76 External Evaluation Report APPENDIX 7 Terms of Reference for this Evaluation government level (to modify policies and remove This appendix reproduces the Terms of Reference structural constraints, as well as to create national that were issued by WSP in 2003. At the time of incentivesforinnovation);atlocalgovernmentlevel contract agreement in December 2003 the timing (to maximize innovation and learning across was changed to accommodate a start in January municipalities and to build up capacity); with local instead of in December, and during the course of and national civil society and the private sector the evaluation various other changes were agreed (to increase understanding of sustainable service betweenWSP and the evaluation team. delivery and to develop new marketable skills); and with local communities (to participate These Terms of Reference describe activities required effectively in the reform debate and to become for the full evaluation of the operations of theWater responsible consumers). and Sanitation Program (WSP) during the period of fiscal years 1999-2003. The evaluation is being WSP's field-level interventions thus complement carried out by WSP management on behalf of the andcombinewiththeworkofothergroupsworking WSP Council. moreattheleveloffinancing(suchasinternational development banks, UNICEF, bilateral programs), Background internationaladvocacy,monitoring,andevaluation TheWaterandSanitationProgramisaninternational (such as WSSCC, WHO, and UNICEF through the partnership which aims to provide increased access JMP), international networking and capacity to improved water and sanitation services for poor building (IRC, WEDC, Sanitation Connection) and people living in rural and urban areas across the pure research (MIT, LSHTM). world. Financing is provided by a consortium of The Water and Sanitation Program aims to donors while the Program is managed by theWorld maintain the following combination of relevant Bank, which also retains fiduciary responsibility for characteristicswhichtogethergiveitacomparative thepartnership.TheProgramhasfourdecentralized advantage to deliver impact in multiple countries regionaloperations(Africa,SouthAsia,EastAsiaand down to the local level: Pacific,andLatinAmerica)andapproximately60staff based in field offices. Management is provided by a 1. GlobalPresence.ThenetworkofWSPoperations teamcomprisingregionalmanagersbasedinthefield and skills deployed around the world enables it andasmallfinancial,technical,andmanagerialunit toquicklydrawinrelevantglobalexperiencewith based in Washington, DC. The Program's governing low transactions costs and create links between Council, consisting of core donors and partners, countries grappling with the same problems. overseesthedirection,strategies,andworkprogram Additionally, the network provides an excellent of WSP. Governance issues are outlined in the vehicle to maximize impact of lessons learned ProgramCharter. within regions. ComparativeAdvantages 2. Regional Presence. The ability to flexibly WSP is one of a range of agencies operating deploy staff (from WSP and through partner internationally and regionally in the water supply organizations) facilitates regional learning and and sanitation sector.WSP aims to deliver a unique the sharing of capacity and experience set of services which complement both the work of between countries. its clients and of other international organizations. WSP specifically seeks to have impact on the 3. Positioning Within Countries. The Program's capacity of local and national governments to long-termengagementwithpolicydevelopment effectivelysupportthedeliveryofsustainablewater and capacity building provides an opportunity andsanitationservices.ToachievethisimpactWSP forpartner-projectstoinfluenceradicalchanges recognizes that some interventions must be made atanationalscaleandtotransfertheirinfluence consistently over long periods of time at national beyond project or national boundaries. External Evaluation Report 77 ProgramStrategy,1999-2003 plans are produced by each region, by each global The current Program Strategy Document (Water thematic group (Rural and Urban) and by WSP and Sanitation for the Poor: Innovating from Field management. There are thus at least seven work Experience,1999-2003)providestheoveralldirection planning areas in each annualWSP Business Plan. for WSP operations. The strategy had four areas of Theseannualworkplansarereviewedinthecontext substantive focus; Rural Water Supply and of overall program resources and finalized by the Sanitation, Services for the Urban Poor, Supply WSP program management team (PMT). While Chains, and Small Towns. To define WSP activities, some changes in nomenclature have occurred in four `product lines' were identified: Investment this period, annual work plans broadly contain Support, Pilots, Networks, and Knowledge referenceto: Management(whichincludedStudies,FieldNotes, FlagshipActivities:Significantareasofwork,with and Learning Events). a clearly defined objective of measurable scale and significance; The 1999 strategy document was developed in Outputs: The impacts expected of each flagship response to a multi-donor evaluation of WSP activity; and operations carried out in the same year. The Products:Discretedeliverableswhichareexpected evaluation called for tighter management controls to contribute to each of the outputs. and greater accountability for delivery of products and assessment of impact. In response, a number Work Planning,2004 of management innovations were introduced, including more rigorous business planning and In FY03,WSP undertook a comprehensive study of monitoring. The formation of the WSP Council its business planning and monitoring structure, (replacingtheearlier,lessformalProgramAdvisory specifically examining how impact could be Committee)wasalsoadirectoutcomeofthisearlier measuredmoreaccurately.Thefindingsshowedthat evaluation. Since 2000, formalized consultation while WSP was planning well, a few changes could with Regional Advisory Committees (RACs), greatlyimproveitsabilitytoevaluatetheeffectofits formalized business planning (documented in activities.The changes recommended included: Annual Work Plans, or AWPs), regular progress Project Sheets: Generating page-long summaries reporting,andthedevelopmentofdetailedstrategy of the initial goals of every project. papersfortheCouncil'sconsiderationandapproval, OutcomeIndicators: Identifyingthetoolstohelp have all led to greater clarity inWSP operations. measure the impact of projects at a level beyond simple outputs; and The current strategy and activities of the WSP can Flagships:Limitingthenumberofflagshipactivities. thereforebestbeunderstoodbyreferenceto: the strategy document and subsequent updates Underthenewplanningstructure,theBusinessPlan presented to the Council; now contains an annex with all the project sheets theproceedingsofRegionalAdvisoryCommittee for the Program. Included in the sheets are the (RAC) meetings; outcome indicators, along with the desired AWPs,six-monthlyProgressReports; outcomes and background information of the the Annual Report; projects. The indicators, as developed, should be backgrounddocumentspreparedforWSPCouncil referenced in developing the consultants' strategy meetings; and for2004-2008. minutesofWSPCouncilmeetings. EvaluationofImpact,1999-2003 Work Planning The current strategy paper was intended to guide Nomenclature,1999-2003 WSP operations up to 2003. Recognizing that the Theevaluationwillfocuson,andbestructuredwith sectoriscontinuallyevolving,aperiodicreviewand referenceto,theannualworkplanspreparedbyWSP evaluationwaspartofthestrategyidentifiedin1999. countries, regions, and thematic groups. Work Given the existence of a Council body which brings planning has been refined over this period, with together key funding partners and technical progressive improvements being made to the advisers, it is proposed that such a periodic review system of planning and monitoring. Annual work nowbecarriedoutundertheauspicesoftheCouncil, 78 External Evaluation Report withfullbuy-infromindividualfinancingpartners, in other annual work plans (that is, in other to minimize costs and maximize the resources regionsorthematicareas)? available to support a detailed evaluation and planning exercise. The evaluation will cover the C. Efficiency(Ameasureoftheoutputs­qualitative period1999-2003. and quantitative ­ in relation to the inputs) a) To what extent were the impact and benefits arising from activities commensurate with Objectives the level of effort and resources expended? Overall objective b) What were the most efficient areas of The overall objective of the current exercise is to operation for WSP activities (by country, evaluate the impact of WSP activities over the past region or thematic area of work)? fiveyears(1999-2003). c) Were activities cost-efficient? d) Were objectives achieved on time? Objectives of the Evaluation, 1999-2003 e) Was the project implemented in the most The evaluation will assess WSP operations in the efficient way compared to alternatives? period 1999-2003. This assessment will be made in terms of relevance, impact, effectiveness, D. Impact (The positive and negative changes efficiency, responsiveness, and the sustainable producedbyadevelopmentintervention,directly benefits of the Program's activities, and overall or indirectly, intended or unintended) institutional arrangements. Each of these aspects a) WhathashappenedasaresultoftheProgram of WSP operations will be assessed for the period or project? under consideration as follows; b) What real difference has the activity made to thebeneficiaries? A. Relevance (The extent to which the activity is c) How many people have been affected? suited to the priorities and policies of the target group, recipient, and donor) E. Institutional Arrangements (An assessment of a) To what extent are the objectives of the the appropriateness and effectiveness of the Programstillvalid? partnerships and other institutional b) Did WSP have a comparative advantage in relationships in whichWSP engages) the sector, and if so, was it correctly defined a) How appropriate, effective, and efficient andarticulated? were the institutional and management c) Were the stated objectives ofWSP operations arrangementsofWSPduringthisperiod? commensurate with this comparative b) What were the most effective relationships advantage, clearly defined in strategy (and specifically partnerships) under which documentsandmeasurable? WSPoperated? d) Are the activities and outputs of the Program c) Whatweretheadvantagesanddisadvantages of WSP's institutional relationship with the consistent with the intended impacts WorldBank? andeffects? d) How appropriate and effective were WSP's e) Are the activities and outputs of the Program governance, organizational structure, and consistent with the overall goal and the staffing profile in realizing a relevant, attainment of its objectives? effective, and efficient business plan? B. Effectiveness(Ameasureoftheextenttowhichan F. Sustainability(Ameasureofwhetherthebenefits aid activity attains its objectives) of an activity are likely to continue after donor a) To what extent were the objectives achieved/ funding has been withdrawn) are likely to be achieved? (This aspect to be a) To what extent did the benefits of a project assessed across all annual work plans.) continue after the project ended? b) Which activities were most effective in b) What were the major factors that influenced contributing to the achievement/non- the achievement/non-achievement of achievement of stated objectives, what were sustainability of the project? For example, the characteristics of these activities and to what were the lessons learned? what extent could they have been replicated External Evaluation Report 79 G. Responsiveness to Opportunity This Consultancy a) How responsive is the Program to Purpose opportunities and demands that arise from WSP management requires detailed inputs to this clients,regionalorinternationalsectorbodies? process from a team of independent experts in b) Is the management, planning, and financial the sector. The consultancy team will carry out structureflexibleenoughtoallowtheProgram the evaluation exercise, whose objectives are to respond to demands as they arise? described above, and will oversee a consultative c) Do the benefits of such responsiveness process which will enable WSP Management to outweigh the inevitable delay or disruption make recommendations to the WSP Council. to the planned program of work? d) How highly valued is such responsiveness by Outputs theProgram's clients? The consultancy would result in the H. Finances development of: a) Financial Sustainability: Has the Program a) Criteria and evaluation indicators for effectively raised funds in this period? Has it assessing the relevance, effectiveness, pursuedtherighttypeoffunding(forexample, efficiency, and responsiveness of WSP global vs. regional, large, untied vs. small, operations; tied)? What can be done to improve the b) Specific criteria or an approach for assessing sustainability/efficiencyoffundraising? theimpactofWSPactivitiesrelatingtosector b) Financial Management: Has the Program reform and strengthening;17 managed its finances effectively and c) An assessment of performance against the efficientlyinthisperiod?Dothetaskmanagers evaluationindicatorsintheperiod1999-2003; manage their budgets appropriately? What d) An assessment of one project from each can be done to improve the situation? region that has been completed in FY04.The c) Impact of Financial Management and assessment will be compared against the Funding Uncertainties upon Program termsandindicatorsdevelopedintheproject Effectiveness: How has the funding sheets as found in the FY04 business plan; availability affected the Program's e) Anassessmentoftheeffectiveness,efficiency, performance (for example, have delays in and responsiveness of the WSP institutional available funds hurt performance? Are arrangements in force in this period; and regionalorglobalfundsmoreeffective)? f) Recommendations for future strategic focus forWSP. I. Recommendations for the Future a) What is the new reality of the sector and how Deliverables does it impact on WSP operations? The consultancy will have the following b) WhatarethenewneedsanddemandsofWSP deliverables: clients and partners? c) What are the most effective interventions (products, thematic areas, modes of InceptionReport operation, etc.) for WSP with its given track The inception report will contain: recordandcomparativeadvantages? a) List of key documents and resource people for d) What are the potential financing the evaluation development exercise; opportunitiesforWSPservicesinthecoming b) Work program for the evaluation development period; and exercise; e) Are WSP's mission, goals, and strategies c) Draft of detailed program for regional visits and still appropriate? consultation meetings; 17This is a focus area in the 1999-2003 strategy and annual workplans, but the impact of discrete WSP activities is difficult to measure because: (i) change may take place in a long timeframe; (ii) changes are unlikely to occur without input from a range of partners, making the impact of discrete WSP activities hard to assess; (iii) to be really effective, WSP has to create a sense of real ownership on the part of governments, rendering it even more difficult to assess the impact of WSP specifically. 80 External Evaluation Report d) First cut of criteria and indicators for assessing b) Synthesized assessment of the key sector issues therelevance,effectiveness,andresponsiveness relevanttoWSPstrategyandassessmentoflikely ofWSP operations; and levels and sources of funding in the coming e) List of specific questions and concerns relating five-yearperiod; to the evaluation to which WSP management c) Global recommendations for key activities, and/or Council will respond. outputs, and product types which should form the basis of WSP work programming in the Draft Global Evaluation Report coming five-year period; and The draft global evaluation will form the basis for d) Global recommendations for institutional moredetailedconsultationattheregionalleveland arrangements for both the coming five-year will contain: period, and, if relevant, in the longer term. a) Draftassessmentofgeneralperformanceagainst agreedindicatorsintheperiod1999-2003; Activities b) Draftassessmentofthegeneraleffectivenessand Preliminary Global Evaluation efficiency of the institutional arrangements in a) Assembleandreviewcurrentstrategydocument, force in this period; annual Business Plans, six-monthly progress c) Preliminary proposal for important activities reports, proceedings of regional retreats and (two-threeperregionandone-twoglobal)which RACs, documentation prepared for the WSP willformthebasisofcasestudiestobecompleted Council, minutes of Council meetings, minutes during the rest of the assignment; of PMT meetings, and any other relevant d) Preliminary assessment of the key global sector managementdocumentationwhichisavailable. issues relevant to WSP strategy, as well as likely All available documentation will be made levels and sources of funding in the coming available byWSP management; five-year period; and b) Make a preliminary assessment of the internal e) Detailed work program for finalization of consistency between the strategy, annual work evaluation through regional meetings, and plans, and activities/products delivered and preparation of regional strategies. develop draft indicators for evaluation of the relevance, effectiveness, efficiency, and responsiveness of WSP operations, the Four Draft Regional Evaluations institutional structure, and the impact of Based on four regional consultations and further institutional reform activities; investigations at the regional/country level, the c)Hold preliminary meetings/phone conver- consultancy will produce for each region: sations with WSP management, key staff a) A finalized assessment of performance against (past and present if appropriate), and WSP agreedindicatorsintheperiod1999-2003; Council members; b) A finalized assessment of the effectiveness and d) Develop a list of key contacts (individuals and efficiency of the institutional arrangements in organizations) who will participate in the force in this period; and evaluation exercise; and c) A finalized assessment of the key sector issues e) Prepare and present to WSP management an relevant to WSP strategy as well as levels and inception report (see above). sourcesoffundinginthecomingfive-yearperiod. Regional Consultations GlobalEvaluation a) Prepare a draft workplan, in consultation with The consultancy will finalize the four regional WSP regional management staff, for the reports, incorporating feedback received, and completion of a regional consultation in each of also prepare a brief global overview document thefourregions; which provides: b) Prepare a draft global evaluation (see above) a) Synthesized overall evaluation of performance which will form the basis of regional and institutional arrangements in the period consultations. This should benefit from 1999-2003,withreferencetoregionaldocuments detailed input from WSP management, and forfurtherdetail; will not be released to partners and clients External Evaluation Report 81 without the prior approval of relevant regional report for submission to WSP Donor Sub- and global WSP staff; Committee for comment ­ March 22, 2004; c) Hold detailed meetings (by phone if iii)Incorporate Sub-Committee's comments into appropriate) with regional WSP management draftreportfordonormeeting­May1,2004;and to review the draft global evaluation iv) Produce final version of the global evaluation ­ and plan for regional consultation meetings. June15,2004. Agree on a final workplan in each region including timetable, and a list of key Staffing and Resources invitees and people/organizations who Staffing will be consulted at the regional and The firm will provide a review team consisting of national levels; threetofourprofessionalsexhibitingarangeofskills. d) Incorporate feedback into draft global Specific skills of the team will constitute: evaluation; e) In each region ­ prepare a program and Team Leader/ Institutional Specialist background material for a regional The team leader will be a specialist with at least 15 consultation meeting which will be organized years of experience in the area of infrastructure and by WSP regional staff. This meeting will be service delivery, with a focus on water supply and attended by at least one member of the sanitation, of which at least five years should be consultant's team, at least one member ofWSP experienceofworkingfor,orcloselypartneringwith, managementstafffromoutsidetheregion,and international organizations working in developing WSP staff from the region; country contexts. f) Participate in regional consultation meetings in each region. Contact and consult on an Specialist expertise should relate to institutional individual basis with key members of the PAC, transformationandreforminsocialservicedelivery RACs, country advisory groups,WSP staff, and areas at the national level in developing or WSP Council to assess quality and impact of developedcountries. interventionsatlocal,regional,andgloballevel The team leader/institutional specialist will: (scope likely to include consultations with at Leadtheteam,providingdirectionandoversight least five key actors in each region); to the evaluation exercise; g) Analyze within each region the activities and Design and oversee the Program for evaluation, impacts of the Program over the past including taking responsibility for planning five years, taking into account relevance, regional travel, coordinating meetings with key effectiveness, impact, efficiency, responsive- informants, and ensuring there is adequate liaison ness, institutional arrangements, and and coordination with regionalWSP offices; sustainability; LiaisedirectlywiththeWSPProgramManagerand h) Analyze the current sector situation and key keep him informed of the activities of the team; relevant issues in each region which should Take the lead in developing the indicators for shape WSP future operations (issues would include, for example, the MDGs, current assessingWSPperformance,particularlyinthearea thinkingonsectorreform,theroleoftheprivate of institutional reform; sector, the call for greater action on hygiene Take the lead in evaluating the institutional promotion and sanitation, etc); and arrangements within which WSP operates, i) Preparefourdraftregionalevaluations. including its institutional positioning, govern- ance, management, partnerships, financing, Completion and staffing; a) Assemble and review feedback on regional Provide oversight and continuity for the four evaluation papers and write draft interim regional evaluation exercises; and report for submission to WSP for comment ­ Take lead responsibility for ensuring all reports March 1, 2004; are produced in time, and be available for review b) Review feedback on draft interim report. and discussion with WSP Council members in the Incorporate Program's comments into interim later stages of the consultancy. 82 External Evaluation Report UrbanPoverty/SanitationSpecialist Costsoffourregionalconsultationmeetings; The urban poverty/sanitation18 specialist will have Costs associated with the participation of WSP at least 10 years of developing-country expertise in Council members, RAC members, and other the area of improving service delivery to poor partners in all consultations; communities and households in urban areas. At Rapidandtimelyinputtodraftreportsproduced least five years of this experience should be directly by the team; and linkedtothedeliveryofwatersupplyandsanitation Costsofproductionoffinalevaluationdocument services.Thisexpertisemayhavebeengainedinthe for formal distribution to WSP partners private sector, NGO or public sector. and clients. The urban poverty/sanitation specialist will: Cost Estimate Participate as a full member of the team in the The cost estimate should include the costs of evaluation development; LeadtheevaluationoftheurbanportfolioofWSP consultant time, travel and associated expenses, communications, production of all reports (eight work, including evaluation of regional/national copies of each), and other expenses. products and activities, the development of appropriate indicators of success, and the Payment will be made on a lumpsum basis on the development of a future strategy; and production of the following reports: Take the lead in two of the regional evaluation Mobilization 30% exercises. Interim Report 40% Final Report 30% Rural Water Supply and Sanitation Specialist The rural specialist will have at least 10 years of developing-country expertise in the area of Management improving water supply and sanitation service This consultancy will be managed by Walter delivery to poor rural communities. At least five Stottmann, Program Manager, WSP. All written yearsofthisexperienceshouldbedirectlylinkedto submissions will be made to Mr Stottmann. At the thedeliveryofwatersupplyandsanitationservices request of the Council, the consultancy team may withinawidersocialdevelopmentframework.This be requested to make presentations directly to the expertisemayhavebeengainedintheprivatesector, WSP Council either as a whole, or through the NGO or public sector. Chairman,JamalSaghir,Director,EnergyandWater, World Bank. Theruralwatersupplyandsanitationspecialistwill: Participate as a full member of the team in both Proposed Methodology and Timeframe the evaluation development; Lead the evaluation of the rural portfolio ofWSP For this evaluation,WSP suggests that it will take a work, including evaluation of regional/national team of four approximately 40 person-weeks products and activities, the development of altogether to complete. Specific requirements for appropriate indicators of success; and each of the team members are defined above. Take the lead in two of the regional evaluation WSP proposes the following timeline for the exercises. evaluation exercise: 1. Early December. The four-member evaluation WSP Contributions team comes to Washington, DC to meet with WSP will provide significant inputs to the process WSP management. The purpose of this visit is for of the evaluation exercise. These contributions the evaluation team to become familiar with include: WSP and its processes.This will also allow the team Stafftime:Uptotwostaffweeksforeachregional time to examine WSP's files and documents. team. Up to four weeks for Program Manager and WSP staff members (five in total) will plan on the management team in Washington, DC. designating two work-days each to assist the team. 18Expertise in sanitation would be an advantage in all three members of the team, but at least one member must have specific expertise in this area. External Evaluation Report 83 Additional administrative support will be available 5. End-March. Team incorporates changes and as necessary. distributesinterimreporttodonorsub-committee, (One-two weeks) allowing three weeks for comments. 2.EarlyJanuary.Theteamevaluatestheregions.One (Three days) teammembertravelstoeachoftheregionstomeet 6. Mid-April. Sub-committee responds with withtheWSPregionaloffices,clientrepresentatives, comments. Evaluation team incorporates and other officials. comments, producing the draft report in time for (Two weeks) the Council meeting in early May. 3. End-January to February. The evaluation team (Three days) writes report, producing it at the end of February. 7. End-May. Report is presented to Council, and (Fourweeks) comments are noted for the final version. 4. Early March. Team leader returns toWashington (--) to discuss the draft interim report with WSP and 8.Mid-June.Council'scommentsareincorporated, incorporate suggested changes. and final version produced. (Two-three weeks in DC) (Three days) 84 External Evaluation Report The ITAD team comprised: Michael Snell Robert Boydell KevinTayler Charles Chandler WaterandSanitationProgram 1818HStreet,N.W.Washington,D.C.20433,USA Phone: (1-202) 473-9785; Fax: (1-202) 522-3313; E-mail: info@wsp.org Africa(WSP-AF) World Bank, P.O. Box 30577, Nairobi, Kenya Phone: (254-20) 322-6000; Fax: (254-20) 322-6386; E-mail: wspafrica@worldbank.org LatinAmericaandtheCaribbean(WSP-LAC) Water and Sanitation Program, Banco Mundial, Mision Residente del Perú Álvarez Calderón 185, San Isidro, Lima 27, Perú Phone: (51-1) 615-0685; Fax : (51-1) 615-0689; E-mail: wspandean@worldbank.org EastAsiaandthePacific(WSP-EAP) World Bank, P.O. Box 1324/JKT, Jakarta 12940, Indonesia Phone: (62-21) 5299-3003; Fax: (62-21) 5299-3004; E-mail: wspeap@worldbank.org SouthAsia(WSP-SA) World Bank, 55 Lodi Estate, New Delhi 110 003, India Phone: (91-11) 2469-0488, 2469-0489; Fax: (91-11) 2462-8250, 2461-9393; E-mail: wspsa@worldbank.org Website:www.wsp.org