WATER GLOBAL PRACTICE A WSP TOPIC BRIEF Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework Problem Statement: Value of a provide decision makers with sufficient and compa- Set of Universal Metrics rable ­ evidence on numbers of systems, types of sys- Globally, the proportion of people living without tems, and performance of systems. Given this global improved drinking water was halved between 1990 challenge, it might be valuable to have a standardized and 2010; however, inequities remain between and set of indicators that could be adopted and adapted within countries. For example, eight out of 10 peo- by countries, thus facilitating improved national and ple who are still without access to improved drinking global reporting and analysis. water sources live in rural areas. Countries are now aiming for the Sustainable Background to the Study Development Goal (SDG) 6, which calls for universal Countries have developed their own monitoring and equitable access by 2030. This represents a far more ­ indicators—many of which have commonalities but are ambitious challenge at a time when many rural water not necessarily exactly the same, and may not be sim- systems in developing countries are not function- ilarly comprehensive. Decision makers in all countries, ing, or are performing below expected levels. Recent however, would likely benefit from ensuring that their data suggest that although 78 percent of water point monitoring framework produces a standard set of indica- schemes are functional at one time, almost 15 percent tors against which to compare their rural water systems. of water points fail after one year and 25  ­percent of By having one standardized global set of indicators, coun- water points are non-functional by their fourth year tries could begin to assess sustainability across aspects (Banks et al. 2016). that are common to all situations, and in the long term Although there is general understanding among to adapt their own monitoring system toward alignment ­ professionals about these low levels of performance, for producing this set. The adoption of such global there are few country monitoring systems that indicator set could also facilitate and contribute—along 1 with better country monitoring systems—to the global set of indicators and definitions exists, which hampers reporting on safely managed drinking water for the Joint comparability and benchmarking. This reflects the fact Monitoring Programme of UNICEF and the World Health that i) there are different levels of technical complexity Organization, among others. The recent SDG baseline that set rural water services apart from those in urban report illustrated that only 19 countries were able to settings (for example, a broader mix of technologies report both urban and rural data for safely managed from rainwater harvesting, simple wells with hand drinking water services, underscoring the need for  a pumps, to more complex piped systems with water better monitoring of service levels. treatment) and ii) urban indicators assume that there is an established utility-like provider while in rural areas This briefing note provides a summary of a recent there are many types of service provider, from infor- World Bank study to set out such a global set of indi- mal water committees to local governments and even cators. It presents the methodology and the proposed private providers. metrics which were derived from the empirical find- ings. It closes with a discussion on next steps and ways forward to disseminate the findings and engage with Study Methodology key global and national sector monitoring initiatives. The study examined rural water service delivery metrics based on host country definitions of what is Efforts to Date considered “rural.” As a result, a range of different sce- The most notable experiences for gathering and shar- narios arise, including concentrated rural communities ing common information on rural water supply are the (often referred to as “rural growth centers”), rural (typ- Water Point Data Exchange (WPDx) and the Sistema ical village centers), and dispersed (scattered, low den- de Información de Agua y Saneamiento Rural (SIASAR, sity) rural populations. Rural Water Supply and Sanitation Information The study was conducted in three main phases. In the System). The WPDx1 is a platform that collects water first phase, in an iterative manner, a conceptual frame- point data from different sources and processes them work was developed, based on a literature review, into a common format to allow data comparability. identifying three broad dimensions of rural water SIASAR2 is a country monitoring system, adopted by indicators: 11 countries so far, which is able to capture both piped water system information and point-sources (instead • service levels (the characteristics of water that users of water point functionality by WPDx) and produce receive) sustainability indexes. • functionality (the physical condition and functioning of a supply system) In the urban sector, a common, global set of indi- cators has been developed under the International • sustainability factors, considered in two parts, Benchmarking Network for Water and Sanitation namely: (IBNET) system to measure the quality of service and 3 i) the performance of the service provider in its role performance of service providers. These indicators of operation, maintenance, and administration include those related to the level of service provided (for example, compliance with water quality standards ii) the extent of external back-up support to the ser- vice providers. and continuity of supply), and secondly to the technical and financial performance of the provider (for exam- In the second phase, using this conceptual framework, ple, in terms of non-revenue water and financial balance a range of indicators sets from countries and develop- sheets). In the rural sector, however, no such common ment partners were analyzed, including 20 national 2 Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework monitoring systems and 20 monitoring frameworks government capacities are likely to be limited. In from donors (including from the World Bank, UNICEF, such a context, formally recognized and legally and USAID). The study also took into account previous established service providers may only just be studies conducted on rural water functionality and emerging, while service authorities may not have sustainability, and particular attention was paid to the clear mandates. An example of this metric is whether SIASAR model from Latin America. there is a service provider in place or not under the “Presence of a legally established service provider” In the third phase of the study, the results of the empir- indicator. ical data were used to identify a shortlist of indicators and associated metrics to be included in the main out- • The basic metrics category embeds the minimum set put, which is a proposed global framework. In addition (as above), but includes indicators with more com- to developing the global indicator set, the study drafted plex (mostly non-binary) metrics. Indicators in this definitions of these indicators and recommended data category are applicable in contexts where service collection protocols and approaches. provision can be assumed to be broadly established, although not necessarily in all types of communi- ties or across all geographic regions. Compliance Proposal for Global Indicator Framework with legal requirements to be established as service Based on the findings of the empirical study of provider is an example of a basic metric under the 40 example frameworks, a global framework of indi- “Presence of a legally established service provider” cators for rural water service provision was developed. indicator. To ensure that the framework is as flexible as possible and can be applied to different types of technologies, • The advanced metrics category embeds both the ­ minimum and basic metrics or implicitly assumes country contexts, and different capacities and types they are being met, depending on the context. In the of service providers, three levels of metrics have been setting of an established professionalized service developed for each indicator: minimum, basic, and delivery (for example, through utility provision), advanced, as set out below (see table 1): minimum indicators such as presence of a recog- • The minimum metrics category 4 should be applica- nized service provider are not assessed because it ble and used in contexts of less advanced institu- is assumed that all systems are managed by a pro- tional and policy frameworks or weak governance, fessional operator. Such indicators are more likely where service providers, service authorities, and to be  applied in contexts where access levels are TABLE 1. Proposed “Three by Four” Indicator Matrix Dimensions Minimum Basic Advanced Service levels Access and continuity of supply Accessibility, availability, quality Affordability, reliability, user satisfaction Functionality Water system physical condition (hand pumps and piped systems) Sustainability: service provider Presence and limited Developed assessment of service Performance optimization performance performance assessment of provider performance metrics service provider Sustainability: service authority Presence and limited Developed assessment of service Performance optimization or technical assistance provider performance assessment of authority performance metrics performance service authority Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework 3 high  and the challenge is now orientated toward categories of indicator metrics, recognizing that coun- optimizing the performance of service provision tries and their monitoring systems are at different (for example, reducing operating costs or improving ­ levels of development. quality of service). Dissemination and testing of this proposed framework In total, 24 indicators were selected as being key to is the next step. This process is not without challenges, monitoring rural water supply services. For each indica- as every country presents a unique context; often with tor, at least one associated metric was identified and a fragmentation of monitoring efforts in rural areas. metrics were categorized into “minimum,” “basic,” or The alignment of country monitoring systems with “advanced.” Indeed, some indicators or metrics were the proposed core set of global indicators may thus be found to belong across all categories (as table 1 indicates). a gradual process. However, the global monitoring of For example, functionality metrics for water points, safely managed drinking water for the SDG achieve- which measure the physical condition of a water facility ment could be one of the incentives for countries to at a specific point in time, are identical across the three move toward the adoption of a core set of indicators. categories. The final selection of relevant indicators for The pathway to adopting and adapting a global core set rural water supply monitoring is presented in appendix of indicators to measure the performance and sustain- A. Service levels indicators are presented in table A.1; func- ability of rural water requires a set of short and medium tionality indicators are shown in table A.2; ­ service pro- term actions: vider performance indicators are shown in table A.3; and, finally, sustainability indicators for the service authority • Short-term recommendations include further validation by a set of rural water practitioners. or technical assistance provider are shown in table A.4. Dissemination efforts with regional and global part- This initial proposal for a global framework includes a ners such as Rural Water Supply Network (RWSN), generic data collection protocol that sets out a descrip- Joint Monitoring Programme (JMP), Sanitation and tion of each indicator; an explanation of the metrics Water for All (SWA), and African Ministers Council used to measure these indicators (categorized as mini- on Water (AMCOW) will be critical to galvanize sup- mum, basic, and advanced); the data to collect in order port for the adoption of the global rural water indi- to inform the metrics; the unit of analysis for each indi- cator framework. Finally, ensuring that existing cator (that is, what is being measured); and the poten- platforms, such as WPDx and SIASAR are compat- tial sources of data (that is, whether it was derived ible with the indicator framework, are also recom- from household, service provider, or service author- mended as short-term actions. ity). Further work is being undertaken to field test this protocol and the need for data aggregation. • In the medium term, there should be engagement with key platforms such as SWA and JMP and regional platforms such as AMCOW and FOCARD- Next Steps and Way Forward APS (Regional Forum for Central America and The global indicator framework developed through Dominican Republic for Water and Sanitation) this study is based on a consultative process, working for the adoption of this framework. Longer-term with national partners and some of the key global play- actions recommended include the creation of a ers engaged in sector monitoring. It has been delib- platform linked to the IBNET initiative to sup- global ­ erately kept to a limited and manageable number of port governments in collecting data, establishing a indicators. Critically, it includes a “menu” of options in database, sharing data, tracking trend lines and pro- terms of the proposed minimum, basic, and advanced viding access to common definitions and protocols. 4 Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework Appendix A: Proposed Global Indicator Framework TABLE A.1. Service Levels Indicators and Metrics Metrics Indicator Minimum Basic Advanced Type of source (1) The type of source a person Same as minimum Same as minimum uses, as per JMP definitions. Accessibility (2) n/a Travel time of a round trip to fetch Same as basic water in minutes, or whether the source is located on premise. Availability (3) Proportion of time that a service Binary: Household responding Same as basic is provided to households, positively to having water taking into account planned available when needed. interruption (continuity). Quality (4) n/a Frequency and percentage of Same as basic water quality test that falls within national standards for water quality, further subdivided into bacteriological (E. coli) and specific physiochemical parameters (arsenic and fluoride). Reliability (5) n/a n/a Proportion of time that a service is provided to a particular household, taking into account unplanned interruption. Affordability (6) n/a n/a The amount spent on water in relation to a household’s total consumption. User satisfaction (7) n/a n/a Overall satisfaction with the service, satisfaction over quantity, satisfaction over quality. Note: n/a = not applicable. TABLE A.2. Functionality Indicators and Metrics Metrics Indicator Minimum Basic Advanced Functionality (at level of Multi-category: functioning, partial, or non- Same as minimum Same as minimum individual hand pump) (8) functioning; based on the results of the discharge or leakage test. Physical condition of the water Water infrastructure condition index, based on Same as minimum Number of breakdowns/ supply infrastructure (system physical condition of main components of the leakages or leaks per performance) (9) water system (intake, reservoir, and so on). kilometer of pipe. Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework 5 TABLE A.3. Sustainability Indicators and Metrics Metrics Indicator Minimum Basic Advanced Governance Presence of a legally Binary: Whether there is a Compliance with legal Same as basic established service service provider in place or not. requirements to be established as provider (10) service provider. Staffing (11) Presence of at least one skilled Multi-category: whether the Staff ratio expressed as number of staff member needed to carry service provider has organizational full-time equivalent (FTE) per unit out the tasks associated with charts, job descriptions for (number of connections or cubic their position. all positions including regular meters sold) which indicates the size staff, volunteers, and board of the service provider. members, and whether these posts are filled. Performance in operation and maintenance Maintenance (12) Binary: Whether any type of Percentage of breakdowns over Ordinal score for asset management maintenance has been carried last 12 months repaired within the planning. out in the last 12 months. established (national) norm for response time. Chlorination (13) Binary: Whether the service Same as minimum Residual chlorine concentration in provider is carrying out milligrams per liter (or parts per chlorination or not (for piped million). schemes only). Coverage (14) Percentage of the population Same as minimum Same as minimum served by a service provider in its service area. Non-revenue water (15) n/a n/a Difference between water supplied and water sold (non-revenue water). Financial management Tariff structure (16) Type of tariff structure, Same as minimum Whether the tariff is based on an including not levying a tariff. adequate tariff calculation. Financial management (17) Binary: Whether the service Multi-category: Whether the Presence of financial reports provider has a general ledger service provider has, and keeps, including all required elements for or cash-book. updated monthly or annual informed decision making (billing financial reports. receipts, operating expenditure, volume of water produced, volume of water sold). Tariff collection efficiency Percentage of users with The ratio between the amount the Same as basic (18) outstanding debts. income from water bills and the total amount that was billed over the last financial year. table continues next page 6 Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework TABLE A.3. Continued Metrics Indicator Minimum Basic Advanced Financial sustainability (19) Ratio between operational Ratio between current assets and Ratio between all current and non- income and expenditure current liabilities of the service current assets and all current and during the last financial provider (liquidity ratio). non-current liabilities (solvency year (working ratio). ratio). n/a n/a Short- and long-term debt payments (ratio between average monthly income and average monthly expenditure) (debt-service ratio). Environmental and water resources management Source, catchment, Binary: Whether the service Binary: Whether the service provider Ordinal scale on the number and and water resources provider has undertaken any type has a source water protection plan types of source, catchment or water management (20) of source, catchment, or water or wellhead protection plan in resources management plans and resources management activity place and implements the plan on a activities undertaken by the service in the last 12 months. regular basis. provider. Customer relations Complaints handling Binary: Whether the service Same as minimum Binary: Whether a customer care mechanism (21) provider holds regularly mechanism exists. scheduled, publicly announced Percentage of complaints or meetings, or other mechanism to requests that is handled within an provide feedback to users over a established period. given period. Note: n/a = not applicable. TABLE A.4. Sustainability Indicators and Metrics: Service Authority or Technical Assistance Provider Performance Metrics Indicator Minimum Basic Advanced Service authority presence and functions Service authority Binary: Presence of a Percentage of sanctioned positions Percentage of allocated financing capacity (22) service authority, as per the for rural water in the service available for functioning in the service legislative and administrative authority structure that are filled. authority or technical assistance role in requirements of the country. relation to what was calculated as being required over a 12-month planning period. Service authority Binary: Whether the service Binary: Whether the service Percentage of communities, systems, support functions (23) authority provided any type authority has in place and has or providers that have met, out of the of support function to rural applied a proactive schedule of universe of communities, systems, or water scheme operators in support visits to rural water scheme providers in the service area (during the the last 12 months. operators in the last 12 months. last 12 months). table continues next page Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework 7 TABLE A.4. Continued Metrics Indicator Minimum Basic Advanced Performance in monitoring role Presence of an Binary: Whether an Binary: Whether the information Multi-category or ordinal scale: information system (24) information system is in system has been updated in the last Information system contains updated place at the level of the 12 months. data on service levels, functionality, and service authority (or any service provider performance. designated third party). 4. Typically, only the most basic levels of service are provided. Notes Indicators belonging to this category usually have binary metrics 1. http://www.waterpointdata.org. (for  example, the presence or absence of a service provider). This “minimum set” comprises only 18 of the 24 indicators. 2. Currently SIASAR (http://www.siasar.org) is being applied in Honduras, Nicaragua, Panama, the Dominican Republic, Costa Rica, Oaxaca (Mexico), Peru, Bolivia, Colombia, Ceará (Brazil), and Paraguay. 3. IBNET (http://www.ib-net.org) is an online database for water and Reference sanitation utilities’ performance data. It supports and promotes good Banks et al. 2016. “What’s Working, Where, and for How Long: A 2016 benchmarking practice among water and sanitation services by pro- Water Point Update.” PowerPoint presentation, 7th RWSN Forum, viding guidance on indicators and definitions, facilitating the estab- November 29, Abidjan, Côte d’Ivoire. Available at https://rwsnforum7​ lishment of national or regional benchmarking schemes, and .files.wordpress.com/2016/12/full_paper_0150_submitter_0239_banks​ undertaking peer group performance comparisons. _­brian.pdf. © 2017 International Bank for Reconstruction and Development / The World Bank. Some rights reserved. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. This work is subject to a CC BY 3.0 IGO license (https://creativecommons.org/licenses​ by/3.0/igo). The World Bank does not necessarily own each component of the content. It is your responsibility to /­ determine whether permission is needed for reuse and to obtain permission from the copyright owner. If you have questions, email pubrights@worldbank.org. 8  SKU W17059