WATER GLOBAL PRACTICE Testing, Piloting, and Validation of the Rural Water Indicator Global Framework MARCH 2020 in the African Context Brian Banks, Ethel Mendez-Castillo, Miguel Vargas-Ramirez, Sabrina Zimmermann, and Libbet Loughnan About the Water Global Practice Launched in 2014, the World Bank Group’s Water Global Practice brings together financing, knowledge, and implementation in one platform. By combining the Bank’s global knowledge with country investments, this model generates more firepower for transformational solutions to help countries grow sustainably. Please visit us at www.worldbank.org/water or follow us on Twitter at @WorldBankWater. About GWSP This publication received the support of the Global Water Security & Sanitation Partnership (GWSP). GWSP is a multidonor trust fund administered by the World Bank’s Water Global Practice and supported by Australia’s Department of Foreign Affairs and Trade, the Bill & Melinda Gates Foundation, the Netherlands’ Ministry of Foreign Affairs, the Rockefeller Foundation, the Swedish International Development Cooperation Agency, Switzerland’s State Secretariat for Economic Affairs, the Swiss Agency for Development and Cooperation, the U.K. Department for International Development, and the U.S. Agency for International Development. Please visit us at www.worldbank.org/gwsp or follow us on Twitter #gwsp. Testing, Piloting, and Validation of the Rural Water Indicator Global Framework in the African Context Brian Banks, Ethel Mendez-Castillo, Miguel Vargas-Ramirez, Sabrina Zimmermann, and Libbet Loughnan © 2020 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. ­ argas-Ramirez, Sabrina Zimmermann, Please cite the work as follows: Banks, Brian, Ethel Mendez-Castillo, Miguel V and Libbet Loughnan. 2020. “Testing, Piloting, and Validation of the Rural Water Indicator Global Framework in the African Context.” World Bank, Washington, DC. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights​@ worldbank.org. Cover photos (left to right): Miguel Vargas-Ramirez/World Bank; Dominique Mercier; Curt Carnemark/World Bank. Cover design: Jean Franz, Franz & Company, Inc. Contents Executive Summary vii Abbreviationsix Chapter 1  Background 1 Notes 2 Chapter 2  Existing Data-Sharing Efforts 3 Notes 5 Chapter 3  Summary of Development of the Rural Water Metrics Global Framework 7 Phase 1: Conceptual Framework 7 Phase 2: Empirical Study 7 Phase 3: Proposing Global Indicators 8 Chapter 4  Overview of the Validation Study 11 Chapter 5  Project Methodology 13 Preproject Planning 13 Development of the Survey Tool 13 Field Implementation 15 Data Analysis 21 Notes 28 Chapter 6  Project Findings 29 Feasibility Assessment 29 Usefulness Analysis 40 Limitations 43 Note 44 Chapter 7  Recommendations 45 Individual Indicators 45 General Implementation Approach 51 Next Steps 55 Notes 57 Appendix A  Summary Findings 59 Appendix B  Originally Proposed Rural Water Metrics Global Framework 63 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context iii Appendix C  Local Stakeholders Engaged and Capacity Assessment 67 Monitoring Systems and Documentation Reviewed as Appendix D  Part of the Study of 40 Frameworks 69 Dimensions, Categories, Indicators, and Metrics Identified as Appendix E  Part of the Study of 40 Frameworks 73 References79 Figures 1.1. Growth of Water Point Data Collected: New Data Records Collected by Year: 2007–2017 2 3.1. Proposed Domains for the Rural Water Global Framework 8 5.1. Data Collection in Burkina Faso 18 5.2. Data Collection in Kenya 19 5.3. Data Collection in Sierra Leone 20 5.4. Possible Categorization of Indicators 27 6.1. JMP Drinking Water Ladder 41 7.1. Potential Collaboration for Global Monitoring Frameworks 57 Tables 5.1. Roles and Permissions for Stakeholders in Akvo Flow 17 5.2. Differences among Countries in Implementing Pilot of Rural Water Indicators Framework 18 5.3. Feasibility Score Derived from Immediate Feedback Question 22 5.4. Example of a Feasibility Score for “Don’t Know” Answers 23 5.5. Example of Enumerator’s Assessment of One Survey Question 23 5.6. Example of Thresholds for Difficult Questions 24 5.7. Example of Assessment of More Difficult Questions 25 5.8. Pathway for Questions Considered Less Difficult 25 5.9. Questions to Determine Usefulness 27 6.1. Surveys Collected Per Target 29 6.2. Survey Questions Marked Difficult Based on Immediate Feedback 30 6.3. Survey Questions Marked Difficult Due to “Unable to Answer” 30 6.4. Survey Questions Marked Difficult Due to Duration 31 6.5. Survey Questions Marked Difficult Based on Difficulty Feasibility Score 32 6.6. Survey Questions Marked Difficult Due to Challenges 32 6.7. All Survey Questions Identified as Difficult 33 6.8. Approaches to Defining Functionality 38 7.1. Differences between Core Harmonized and Reference Indicators 46 iv Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 7.2. Recommended Indicators as Core Harmonized Indicators and Reference Indicators 46 7.3. Indicators Ready for Use as Core Harmonized Indicators 48 7.4. Rural Water Monitoring Sub-Sector 52 B.1. Initially Proposed Indicators 63 C.1. Service Authority 67 D.1. National Monitoring Systems Documentation Reviewed 69 D.2. World Bank Projects Reviewed and Related Documentation 70 D.3. Indicator Frameworks Used by Other Development Partners and Related Documentation 71 E.1. Dimensions, Categories, Indicators, and Metrics of Existing Frameworks 73 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context v Executive Summary This document presents the findings of the pilot conducted in Burkina Faso, Kenya, and Sierra Leone to determine the feasibility and utility of the indicators proposed by the World Bank in the Rural Water Metrics Global Framework. Through standard indicators, the proposed framework aims to facilitate improvements in national and global reporting and analysis, which would improve rural water services around the world. This document provides background on the framework; shows how it relates to other efforts to harmonize rural water data; and outlines development of the framework. It also describes the pilot—its methodology, findings, and limitations—and offers recommendations regarding the indicators themselves; suggests an implementation approach; and proposes a pathway for collection of the data using integration into national monitoring framework. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context vii Abbreviations DGESS General Directorate for Sectoral Studies and Statistics DGIS Directorate-General for International Cooperation DHS Demographic and Health Surveys DISE Integrated Monitoring and Evaluation System (Dispositif Intégré de Suivi Evaluation) DREA Regional Directorate for Water and Sanitation FTE full time equivalent GDP gross domestic product GIS geospatial information systems GLAAS United Nations-Water Global Analysis and Assessment of Sanitation and Drinking-Water GMHTT Global Monitoring Harmonization Task Team IBNET International Benchmarking Network for Water and Sanitation Utilities JMP Joint Monitoring Programme for Water Supply, Sanitation and Hygiene MDG Millennium Development Goal MEA Ministry of Water and Sanitation MICS Multiple Indicator Cluster Surveys MIS management information system MoWR Ministry of Water Resources MWE Ministry of Water and Environment NGOs nongovernmental organizations NMIP National Mapping Information Project PAMSIMAS Water Supply and Sanitation for Low-Income Communities III Project PIMS Post Implementation Monitoring Surveys RASARP Annual Report for Water and Waste Services in Portugal (Relatorio Annual dos Servicos de Aquas e Residuos em Portugal) RCAP Rural Community Assistance Program RWSS rural water supply and sanitation SDG Sustainable Development Goal SIASAR Rural Water and Sanitation Information System (Sistema de Información de Agua y Saneamiento Rural) SIBS Water and Sanitation Information System Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context ix SIP Performance Indicator System (Systeme d’Indicateur des Performances) SIS sector information system SIT sustainability index tool SNIS National Sanitation Information System (Sistema Nacional de Informações sobre Saneamento) SUI Rural Unified System for Public Rural Services (Sistema Único de Informacíon de Servicios Públicos Rural) SWA Sanitation and Water for All TA technical assistance UAV unmanned aerial vehicle UNICEF United Nations Children’s Fund USAID United States Agency for International Development WASH water supply, sanitation, and hygiene WGS 84 World Geodetic System, 1984 revision WHO World Health Organization WPDx Water Point Data Exchange WPMS Water Point Mapping System x Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Chapter 1 Background Worldwide, more than 600 million people in rural areas lack basic access to drinking water.1 Despite significant investments in rural water services during the past several decades, the number of people in this situation has decreased by less than 50 percent during the period of Millennium Development Goals (MDG).2 For every urban dweller who cannot access a basic level of drinking water, more than three people in rural communities face the same challenge.3 Improving rural water services has been challenging for many reasons, including relatively high cost per person (Naughton 2013); remote and difficult-to-reach locations; decentralized services requiring a spread-out support mechanism (Naughton 2013); and relatively high failure rates (Banks and Furey 2016). Compounding these challenges has been the lack of robust data about rural water services. Although other development sectors, such as health (Nigeria Federal Ministry of Health 2014)4 and edu- cation (UNESCO Office in Nairobi 2015),5 have comprehensive data sources that cover rural areas, har- monized monitoring data for rural water services (especially water points) are much more limited. Water points refer to a unique point of access to water, such as boreholes, wells or handpumps; they are in contrast to water systems where water is distributed to several access points across a pipe network. The dearth of comprehensive data relates directly to one of the most intractable challenges in rural water services: sustainability. As many as 25 percent of rural water services fail in the first four years (Banks and Furey 2016), limiting progress toward universal water services. Anecdotal evidence abounds, but the lack of quality data covering long periods of time poses a major challenge to identify- ing drivers of lasting rural water services. Understanding the conditions necessary for longevity ser- vices that last is paramount for the rural water sector, but acquiring it requires an unprecedented quantity of quality data. Data collection of rural water services has grown dramatically over the past decade; however, it is massively fragmented (Banks and Furey 2016). Different water institutions often collect different data, using different tools to achieve different objectives. When—and if—they share data, they may use for- mats that are difficult for others to access, or they may have proprietary data structures. In addition, each data source is often relatively small. The average nongovernment water point inventory contains fewer than 5,000 records.6 When these small data sources cannot be combined because of differences in structure, the value of collection drops considerably. People who have tried to address issues in rural water services find that the lack of harmonized data makes it “very difficult to determine how large scale the issue is, or what steps need to be taken” (Internal document from Sanitation and Water for All (SWA). The challenges in harmonizing rural water data come from the nature of the sector. As noted by Smits, Mansour, and Lockwood (2017), “there are different levels of complexities that set rural water supplies apart from urban supplies, including a broader mix of technologies in use … [and] different types of service providers.” The current work on this document builds directly on this previous effort and aligns with the recom- mended next steps of “testing its feasibility and applicability.” The pilot assesses the feasibility and Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 1 FIGURE 1.1. Growth of Water Point Data Collected: New Data Records Collected by Year: 2007–2017 120,000 100,000 Water point records 80,000 60,000 40,000 20,000 0 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Year Source: https://www.waterpointdata.org/, 2018 usefulness of the 24 indicators suggested in the initial research by testing them in diverse contexts in three African countries (Burkina Faso, Kenya, and Sierra Leone). The data collected do not represent rural water services in these countries, but the diverse contexts do show how these indicators might work across a range of contexts. Notes 1. Counting total rural population with limited service, surface water, or unimproved water access in 2015 data are from Household Data (database), WHO/UNICEF (World Health Organization and the United Nations Children’s Fund) Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (JMP), Geneva, https://washdata.org/data#!/table?geo0=region&geo1=sdg. 2. Counting total rural population with limited service, surface water, or unimproved water access in 2000 and in 2015 data are from Household Data (database), WHO/UNICEF JMP, Geneva, https://washdata.org/data#!/table?geo0=region&geo1=sdg. 3. Counting total urban and rural population with limited service, surface water, or unimproved water access in 2015 data are from Household Data (database), WHO/UNICEF JMP, Geneva, https://washdata.org/data#!/table?geo0=region&geo1=sdg. 4. For more information about the Health Management Information System unit under Strategic Information Department of Ministry of Health, Swaziland, see its website at http://www.hmisswaziland.com/page/about.php. 5. For more information about the Education Management Information Systems function and unit under Department of Basic Education of the Republic of South Africa, see its website at https://www.education.gov.za/Programmes/EMIS.aspx. 6. For more information about the Water Point Data Exchange (WPDx), see the WPDx website at https://www.waterpointdata.org/. 2 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Chapter 2 Existing Data-Sharing Efforts Awareness of the importance of harmonizing water data has sparked several efforts to overcome the many challenges. The earliest success occurred in the urban sector, where the International Benchmarking Network for Water and Sanitation Utilities (IBNET) provided “access to comparative information that will help to promote best practice among water supply and sanitation providers world- wide and eventually will provide consumers with access to high quality and affordable water supply and sanitation services.”1 IBNET has grown dramatically since its launch in 1994 with 12 utilities to more than 2,000 in its peak in 2015. To fuel this growth, IBNET has provided four essential components that reduce barriers to harmonizing data: •• A common set of data definitions •• A minimum set of core indicators •• Software to allow for easy data collection and calculation of the indicators •• Resources to analyze data and present results Although IBNET has harmonized key water-related data with these resources, it remains focused on utilities, leaving massive gaps in the rural water landscape. The first effort that successfully coordinated data in the rural sector was the Sistema de Información de Agua y Saneamiento Rural (Rural Water and Sanitation Information System; SIASAR), which addressed the challenges of harmonizing government data in the rural water sector with a focus on Latin America. Innovations have included integration of areas of reporting (communities, service pro- viders, technical assistance providers, and water supply systems). SIASAR “defines methodologies to aggregate the information in thematic indices” (Rural Water and Sanitation Information System 2016). These indices provide a user-friendly overview, using letter scores of A through D, to summarize find- ings. In addition to these innovations, SIASAR provides core data-sharing infrastructures, such as stan- dardized questionnaires (both paper and digital [Rural Water and Sanitation Information System app]2) and a dictionary of definitions. This information system has achieved scale in Latin America with par- ticipation by Bolivia, Colombia, Costa Rica, the Dominican Republic, Honduras, Nicaragua, Oaxaca (Mexico), Panama, Paraguay, and Peru. Another multicountry water data-sharing initiative is the Water Point Data Exchange (WPDx). It provides a data-exchange standard for sharing information about communal water points—including point sources and small water distribution systems—and a data repository that complies with the standard. This initiative was based on a comprehensive review of what 70 actors throughout the sector already collected and on feedback from global stakeholders. By involving these concerned parties, the developers ensured easy adoption of the standard. This approach has allowed the inte- gration of data collected outside of dedicated WPDx efforts, ensuring broad application throughout the sector. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 3 Although both help harmonize rural water data, WPDx and SIASAR differ in several ways. First, WPDx focuses on water supply points; it does not include separate data collection about communities, service providers, and technical assistance. Some information regarding the basics of these domains is col- lected, but the process is not as thorough as that of SIASAR. Second, WPDx has a less intensive approach to data sharing, requiring governments to provide only data, some basic metadata, and permission to share data, whereas substantial government engagement and ownership are SIASAR cornerstones. WPDx does encourage data sharing by all stakeholders, including nongovernmental organizations (NGOs), universities, and the private sector. This broad approach has harmonized more than half a mil- lion data records from dozens of countries, establishing WPDx as the world’s largest harmonized data set on rural water services. Both WPDx and SIASAR provide unique value to the rural water monitoring sector, but gaps still exist. WPDx has achieved considerable scale, but it collects relatively simple data and does not cap- ture all aspects of water services. SIASAR has succeeded in collecting robust data and getting notable buy-in from host governments, but deep engagement requires significant investment in time and resources. One final effort to harmonize global data about water—in this case, water governance—is the United Nations-Water Global Analysis and Assessment of Sanitation and Drinking-Water (GLAAS). Implemented by the World Health Organization (WHO), GLAAS seeks “to provide policy- and decision-makers at all levels with a reliable, easily accessible, comprehensive, and global analysis ­ of the investments and enabling environment to make informed decisions for sanitation, drink- ing-water and hygiene.”3 GLAAS also contributes to the United Nations-Water Integrated Monitoring Initiative for Sustainable Development Goal (SDG) 6, with a mandate for tracking progress toward SDG targets 6.a and 6.b. Going beyond collecting data only from participating countries, GLAAS collects harmonizes data from external support agencies, reaching 75 countries and 25 external support and ­ agencies, according to the 2017 report (WHO/UN-Water 2017). The WHO provides a range of support to encourage harmonization of collected data, including survey forms, a survey implementation plan, and, notably, a country feedback form, all of which promote continuous improvements. GLAAS con- ducts its survey every two years. Collectively, these tools provide robust data about many aspects of rural water services related to those covered in the proposed rural metrics. However, gaps remain, and globally, many rural water stakeholders are collecting data of a disparate nature outside of the four discussed frameworks. In addi- tion, except for SIASAR, each data framework primarily captures a single analytical level—for example, data on a service authority, a water point, or a household’s service level. Although SIASAR goes beyond a single level of analysis, its growth has been limited, partly because of its rigor and detail. Because these disparate data sources lack any approach for interoperability, it is difficult to achieve a holistic perspective that looks at the many levels at which a water system operates. To address these gaps, the World Bank launched an initiative to “have a standardized set of indica- tors that could be adopted and adapted by countries, thus facilitating improved national and global reporting and analysis” (World Bank 2017a) Further, “a global set of indicators will help focus on achieving sustainability more clearly on the sector agenda, identify future investment needs, improve 4 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context sector management, enable the comparison of progress across countries and regions, and permit a standard to extract information from different monitoring systems” (World Bank 2017b).Commissioned by the World Bank and led by Aguaconsult in 2017, this effort generated a proposal for a global indica- tor framework, building on a robust conceptual framework and an empirical study of frameworks from countries and development partners. This study advances the World Bank’s effort to develop a global indicator framework by piloting the proposed indicators and analyzing the feasibility and ­ utility of the proposed indicators. Notes 1. For more information about IBNET, see its website at https://www.ib-net.org/about-us/. 2. http://siasar.org/en/download-app. 3. For more information about GLAAS, visit its website at https://www.who.int/water_sanitation_health/monitoring/investments/glaas/en/. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 5 Chapter 3 Summary of Development of the Rural Water Metrics Global Framework The initial development consisted of three main phases. In the first phase, a conceptual framework was developed to use as a foundation for the new proposed indicators. The second phase used this concep- tual framework to analyze nearly 40 indicator frameworks used by national governments and develop- ment partners. The third and final phase built on the reviewed indicator frameworks to propose indicators, definitions, metrics, and data collection protocols. Phase 1: Conceptual Framework Within the conceptual framework, several key terms apply to this study and the original effort. The authors (Smits, Mansour and Lockwood, 2017) of that document use the following definitions: •• Domain (or dimension or criterion) refers to the broad category of what is monitored. The study uses domain of indicators to refer to a broad category of indicators. For example, a domain of indicators could refer to service levels. •• An indicator group refers to a subcategory of a domain, which consists of several indicators. There may be several levels of subgroups, particularly in the domain of sustainability factors. •• Indicator refers to an indication of the state or level of the domain or indicator group. In general, sev- eral indicators are needed to capture a domain or indicator group. •• Metric refers to the specific expression and unit of measure of the indicator. This may be a quantifiable unit, such as meters, or a binary expression (whether something is there) or an ordinal scale. •• Parameter refers to the measurable factor(s) that make up the indicator. For example, to measure cov- erage with improved supplies, two parameters are required: first, the number of households with access to improved supplies and second, the total number of households in a certain area. The conceptual framework then explored three different and related domains, shown in figure 3.1. The authors of the conceptual framework provided additional insights into the relationships of the three domains, noting that “a water supply service is sustainable if it functions and provides agreed levels of service (or improves on them) throughout the life-cycle of the infrastructure, reflecting the fact ­ that a sufficient number of enabling conditions, or sustainability factors, are in place. Sustainability, functionality and service levels are therefore three interrelated, but not interchangeable concepts.” (Smits, Mansour and Lockwood 2017) Phase 2: Empirical Study In this phase, the authors of the conceptual framework identified potentially relevant indicator frame- works before carrying out their review. In total, “40 indicator frameworks were reviewed, of which 17 are national monitoring systems, nine World Bank project monitoring systems, three studies of Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 7 FIGURE 3.1. Proposed Domains for the Rural Water Global Framework • Underlying factors related to likelihood that service will continue to perform adequately Sustainability factors • Partly measurable, partly predictive • Multiple dimensions • Multiple institutional levels • Characteristics of the flow of water Service levels • In theory, measurable at the household level, though in practice also at the facility level • Whether a certain facility is working Functionality • Directly measured at the facility level Source: Smits, Mansour and Lockwood 2017. World Bank projects, and seven indicator frameworks of other development partners. In addition, some 11 meta-reviews were studied, informing the review on how domains and sub-groups of indicators are conceived, and/or to zoom into a specific domain.” (Smits, Mansour and Lockwood 2017) An analysis was then completed to compare these different frameworks to identify indicators that provide insights into each domain, and the metrics that can be used for each. For each indicator framework, the review provided descriptions; information about structure; specific indicators included in the domains; and the method for processing and weighing indicators toward aggregate indicators or scores. The authors then analyzed commonalities and differences to identify common domains and groups of indicators. Phase 3: Proposing Global Indicators Building on the indicator frameworks reviewed in Phase 2, the authors proposed a list of globally appli- cable indicators. Guiding this effort was the principle “that it must be universally applicable, i.e. it can be used across different contexts and capacities to deliver and monitor rural water services, both between and within countries.” (Smits, Mansour and Lockwood 2017) Achieving this required identifi- cation of indicators that could span the diverse contexts of rural water services, including capacities, different technologies, and approaches to providing service. To ensure universal applicability, the con- ceptual authors proposed a tiered system of metrics, including minimum, basic, and advanced. They wanted each increasing metric level to include (explicitly or implicitly) the previous metric so that each level ensures applicability to more established contexts or where service providers have higher levels of capacity. As noted by the original authors, “the advanced set of indicators, which include the minimum and basic set, or for some data assume that these are already met, and thus exclude those indicators.” 8 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context (Smits, Mansour and Lockwood 2017) As an example of different levels of the metrics, the minimum metric for the service authority capacity was proposed as “presence of a service authority, as per the legislative and administrative requirements of the country,” whereas the advanced metric looked at “percent of allocated funding available for functioning in the service authority/ technical assistance role in relation to what was calculated as being required over a 12-month planning period (or other).” Therefore, the advanced metric assumes that the minimum metric is already met. Overall, this approach yielded 24 indicators. Each included at least one metric that was categorized as minimum, basic, or advanced, and some included multiple metrics across these categories. A total of 18 advanced, 15 basic, and 18 minimum metrics were proposed. Based on these, a series of next steps became apparent: integrating the proposed metrics into the World Bank’s own work; disseminating the indicators among rural water practitioners through easy-to- absorb documents so that they easily could apply to ongoing initiatives; and harmonizing the proposed metrics with other efforts, such as the Water Point Data Exchange (WPDx) and the Sistema de Información de Agua y Saneamiento Rural (Rural Water and Sanitation Information System; SIASAR). Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 9 Chapter 4 Overview of the Validation Study The validation study builds on the first effort and aligns with the recommended next steps of “testing its feasibility and applicability” (Smits, Mansour and Lockwood 2017). The goal was to pilot the 24 indica- tors from the initial research and assess the feasibility and usefulness of each indicator. To achieve this, the 24 proposed indicators were tested in diverse contexts in three African countries (Burkina Faso, Kenya, and Sierra Leone). Although the data collected was not designed to be representative of rural water services in these countries, the researchers intentionally selected the diverse data collection con- texts to help illustrate how these indicators might work across a range of contexts. For this reason, data presented in this report should not be used to understand any characteristics of water services or stakeholder capacity. The results are not representative in any way, and they should not be extrapolated to suggest any findings about services or capacities on a district, national, or regional scale. Further, this was a pilot and some questions were found to be invalid because of a lack of under- standing or ability to answer. Thus, the data collected should not be used to draw conclusions about the services covered in the survey or the stakeholders engaged. The only appropriate use for the data is to evaluate the feasibility and usefulness of the indicators. No conclusions should be made with respect to services, capacity, trends, or any other aspect of water service provision. This research project had four distinct phases: pre-project planning, development of the survey tool, field implementation, and data analysis. Each phase provided opportunities to learn about the feasibil- ity and applicability of the indicators in the field. Country and location selection took place during the preplanning phase, as did development of an analysis framework for utility and feasibility. During the development of the survey tool, the researchers collaborated with the Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (JMP) and with World Bank staff to transform the guidance in the proposed metrics into a robust survey tool suitable for the field. During the field implementation phase, enumerators in Burkina Faso, Kenya, and Sierra Leone attended training, a necessary step to support the collection of more than 600 records across the three countries. Finally, during data analysis, researchers assessed qualitative and quantitative input from the previous two phases to determine the feasibility and usefulness of the proposed metrics. Based on this analysis, the authors are proposing several key recommendations for future implementation at scale. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 11 Chapter 5 Project Methodology Preproject Planning The researchers decided to conduct the study in Africa because of the continent’s notable rural water challenges and its difficulties with data collection. More than one-third of the world’s rural population without basic water access lives in Sub-Saharan Africa.1 In the face of such massive challenges, monitor- ing and evaluation are critical parts of the solution. The challenges regarding rural water in Africa extend to data as well. Although successful examples of country-led Water Supply, Sanitation and Hygiene (WASH) monitoring efforts exist in some African countries, no efforts to harmonize indicators in Africa have achieved the same level of engagement as the Sistema de Información de Agua y Saneamiento Rural (Rural Water and Sanitation Information System; SIASAR) has achieved in Latin America. The researchers placed a priority on diverse contexts to achieve global metrics. Striving to push the metric limits, they selected vastly different pilot contexts, giving consideration to socioeconomic devel- opment, regional characteristics, and existing operations in each country. Given these criteria, they selected three—Burkina Faso, Kenya, and Sierra Leone—to represent different contexts in terms of gross domestic product (GDP) per capita (499 to 1,507 dollars) (World Bank, 2017)2; regional characteristics (both anglophone and francophone countries); geographic diversity (from eastern to western Africa); percentage of rural water access (74 percent to 83 percent with at least basic access) (WHO/UNICEF JMP3); and implementation approaches (ranging from hiring private enumerators to leveraging enumer- ators implementing existing mapping efforts by the government). At the same time, all three countries already had been involved in large-scale data collection and monitoring of rural water services, demon- strating a demand for data and an ability to do basic monitoring. Within the countries, the researchers selected locations based on existing efforts that could be expanded; populations that could meet the target for data collection; and relationships with government authorities. Development of the Survey Tool The global metrics provided a foundation for piloting this new framework. However, transforming the concepts into a survey and guidelines for implementation required considerable effort. The developers followed the guidance in the proposed global metrics as closely as possible, recognizing the expertise required to arrive at the proposed metrics. Therefore, the pilot survey tool had only necessary changes, all of which were documented in the survey development process. The research team first worked to ensure that indicators aligned with new guidance from the Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (JMP) wherever possible and updated them accordingly (WHO/UNICEF JMP3). This would ensure the original project goal to “draw universal conclusions … in support of SDG monitoring efforts.” Where possible, questions from JMP became part of the survey. Following that step, the research team worked each indicator into one or more survey questions. The team made several changes to align with best practices for survey development. Specifically, they Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 13 avoided asking multiple questions within a single question, ensured that no math needed to be done in the field, integrated skip logic to make sure that all questions were relevant to ask based on previous answers, and reviewed the order of questions to ensure clarity. The initial research plan for this phase of the validation had proposed including only the advanced metric, but an in-depth review of the indicators showed that this would limit its scope. As noted by the original authors, “such indicators can only be applied in contexts where the sector has put in place the basic elements of sustainability, has most likely achieved first time access and is now orientated towards optimizing the performance of service provision (e.g. reducing operating costs).” (Smits, Mansour and Lockwood 2017) According to this guidance, advanced indicators should be used only in specific con- texts. Testing these indicators alone, in all contexts, would contradict the original guidance on applying the indicators. The research team wanted to test a fuller range of the proposed indicators, so members developed survey questions using the lowest and highest level of metrics proposed. For example, in the case of an “availability” indicator—which has only proposed minimum and basic metrics—they included both. In cases where the conceptual authors proposed minimum, basic, and advanced metrics, only the minimum and advanced metrics became part of survey questions. In the final survey, 79 questions cap- tured the parameters required for 39 metrics, which included 16 at the minimum level, five at the basic level, 16 at the advanced level, and two at all three levels. The research team omitted seven basic ques- tions, including the seven with distinct indicators at all levels. All questions, metrics, and indicators appear in appendix A. For each survey question, the research team noted the unit of analysis (that is, household, water sys- tem, and so on) and provided potential lists of responses. Wherever possible, researchers pulled responses from definitions or other information in the initial list of proposed metrics. Where necessary, definitions appeared as help text in the digitized surveys, and they became part of the training. In some cases, as noted later in this document, the research team developed additional definitions according to feedback from field teams. Lastly, the survey developers included questions designed to capture immediate feedback and supply data necessary to conduct the feasibility analysis. They asked respondents to assess previous answers, typically in terms of the confidence of their answer. To do this, explicit questions were added to the survey asking respondents to confirm how confident they were in the previous answer on a scale from 1–5. Feedback from World Bank staff and the research team led to the development of four revised surveys that targeted specific units of analysis—households, service authorities, service providers, and water point observations—which enabled researchers to question the most appropriate respondents and facil- itate analysis of different perspectives. For example, researchers could analyse data to compare how households and providers of drinking water services felt about the provided service and note any signif- icant discrepancies in perspective between the service provider and recipient. The additional data also could add context to information collected by other surveys and provide context to water service-­ related findings from other survey instruments. The researchers also developed a list of supplementary queries hypothesized to add value to the data without requiring significant effort. After the researchers reviewed them with the World Bank team, a final set of supplementary questions became part of the four questionnaires. These questions did not 14 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context include the 79 fundamental questions evaluated later in this document, but the researchers did use the supplementary queries to help inform recommendations. The final versions of the four surveys incorporated the last edits and rounds of feedback, contained questions to assess all 24 of the originally proposed rural indicators, and included the feedback queries as well as the supplementary ones. In addition to the four field surveys, the researchers created one additional questionnaire to collect feedback from enumerators on their perceptions about the feasibility of the other surveys. Finally, all five questionnaires (including the enumerator feedback questionnaire) were translated into French by a translation firm and reviewed by a native French-speaking water expert in Burkina Faso. Field Implementation Approaches to field implementation varied among the three countries to fit the local contexts. However, in all three countries, researchers used standard general approaches. Sampling Researchers used a convenience sampling methodology for the study, and general principles guided inclusion and exclusion per survey type for all countries as well as country-specific methodologies and criteria. The expected sample sizes per survey were the following: •• Water point observation surveys: 60 to 75 observation surveys per country •• Household surveys: 60 to 75 surveys per country •• Surveys of water service providers: six to 60 surveys per country •• Surveys of water service authorities: one survey per country •• Enumerator surveys: one survey per enumerator The following survey-related definitions applied to all three countries: Water point observation survey: A water point is a “1. Point source from which water is abstracted, such as a borehole, well or spring, and 2. Water supply/distribution points, such as a hand-pump installed on a borehole or a standpipe in a small piped network” (Magara 2018). Household survey: Respondents were members of a household likely to use a given water point. They agreed to participate in the survey by answering affirmatively to the informed consent question of the survey. The respondents were adults (18 years or older). Where appropriate, the enumerator asked for the person who collected water in the household. If that person was unavailable, the enumerator asked for any other adult in the household. In Burkina Faso, the enumerator asked for the head of the house- hold first because of social norms. Water service provider survey: Respondents were representatives of the water service provider for a given water point. They could include a community water committee, a corporate utility, an association, or any other mandated or ad hoc service provider that operated, maintained, and/or provided other support to the water point. Enumerators identified the highest ranking or most senior member of the water service provider to respond to the survey. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 15 Water service authority survey: Respondents were members of a water service authority, which has authority over water service providers. In many countries, this mandate has been devolved to local gov- ernments; in others, central or regional agencies are service authorities. Akvo project coordinators and supervisors provided introductions to respondents to ensure enumerators spoke with the highest-rank- ing officials within water service authorities. Enumerator survey: Each enumerator filled out an enumerator survey after completing data collection. Enumerators working on this pilot had prior experience working with Akvo in the selected communi- ties. In Burkina Faso and Sierra Leone, these enumerators had engaged with service providers through previous data collection efforts and knew how to find the providers. In new intervention communities, such as the ones in Kenya, enumerators worked with local ministry representatives and community members to identify service providers. Replacement and Substitution Enumerators identified water points, households, water service providers, and water authorities that met the established definitions and criteria as well as the country-specific criteria. If an enumerator initially selected a water point, household, service provider, or water authority and then realized that it did not meet the definitions or criteria, the enumerator ended the survey and did not submit it in the mobile data collection tool. This avoided the data point from syncing to the online database. Mobile Data Collection Project personnel used Akvo’s Flow app to collect all survey data. Akvo staff entered surveys into the Flow system and assigned them to mobile devices on which enumerators completed surveys. Once surveys were submitted, the app stored data on the devices and synced all data points when the device was connected to the Internet, allowing for data collection in remote environments with no connectivity. Akvo Flow has built-in features that contribute to higher data quality and aid in verification to con- firm identification of data points. These included the ability to document photos and geolocation and to generate automatic metadata, such as submission time stamps, survey duration times, and unique identifiers for all data points and submissions. Survey digitization also considered question settings, such as double entry for open-text questions to avoid typing mistakes, setting valid ranges for number questions and disabling manual entry of geodata. The app impedes submission of a form when man- datory fields have not been completed or when, according to the question settings, there is a mistake in the data entered. The app notifies the enumerators what questions they need to review. All ques- tions in the surveys were mandatory; thus, enumerators were trained to use codes 9999 for “don’t know” or 0000 for “does not apply” to ensure they could submit responses even when the questions were not answered. Once data were synced to the online database, users with the right levels of permission were able to see and manage submissions on the online workspace. Akvo Flow’s monitoring feature allowed users to register a data point, which was assigned a unique identifier. That point could then be monitored with separate survey forms linked through that unique identifier. This feature linked household surveys with water point surveys and, in some cases, with the water service provider. 16 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Data Management The Akvo Flow software allows for different roles and permissions on the online workspace. The app managed data submitted by the enumerators. Table 5.1 gives user permissions for managing data in Akvo Flow. Data generated through the pilot are co-owned by the World Bank and the governments of Burkina Faso, Kenya, and Sierra Leone, which have access to all data files generated by Akvo Flow. All relevant data can be accessed on the Water Point Data Exchange (WPDx). Ethics and Informed Consent All participants in this pilot, including respondents and enumerators, were informed of the potential benefits and risks involved in participating. The surveys of enumerators, households, water authorities, and water service providers contained a question regarding informed consent to document respon- dents’ willingness to take part in the survey after considering all potential benefits and risks. Country-Specific Plans In order to adapt to various local contexts, detailed approaches to implementation varied across the three countries. Table 5.2 summarizes the main differences in implementation. A brief description of the process followed in each country appears here. Burkina Faso Implementation occurred in close collaboration with the General Directorate for Sectoral Studies and Statistics (known as DGESS) and the Regional Directorate for Water and Sanitation (DREA) of the Ministry of Water and Sanitation (MEA). The pilot leveraged Akvo’s previous work with the ministry when the inte- grated system for monitoring water services, known as DISE was set up. To this end, the methodology to implement the pilot aligned with the methodology used by DISE (Integrated Monitoring and Evaluation TABLE 5.1. Roles and Permissions for Stakeholders in Akvo Flow Stakeholders Roles and permissions in Akvo Flow Enumerators • Submit data via app • Edit data in app prior to submission (once data are submitted, Akvo Flow does not allow editing via the app) Field supervisors • View and edit data in online workspace • Download reports Field coordinators • View and edit data in online workspace • Download reports Lead researchers • View data in online workspace • Edit data to clean data and remove errors • Download reports World Bank staff • View data in online workspace • Download reports Local authorities • May view data Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 17 TABLE 5.2. Differences among Countries in Implementing Pilot of Rural Water Indicators Framework Areas of difference Burkina Faso Kenya Sierra Leone Water quality testing E. Coli, arsenic, and residual E. Coli, fluoride, pH, and No testing chlorine turbidity (funded by ongoing Watershed water quality mapping programa Starting point survey for collection/ Households (following Water point Water point linking surveys the government’s DISE methodology) Number of enumerators One Four Two Independent collection or joint effort Independent Joint effort with Watershed- Independent with other data collection effort sponsored water quality mapping Primary survey language French English English Note: DISE = Integrated Monitoring and Evaluation System (Dispositif Intégré de Suivi Evaluation). a. For more information on the Watershed program, visit its website at https://watershed.nl/. FIGURE 5.1. Data Collection in Burkina Faso 1. The enumerator 2. The households 3. The enumerator 4. The enumerator followed the DISE directed the enumerator identified one collected methodology to to the water point, service provider data from the service randomly select where the enumerator per community authority overseeing the households in six performed the water to collect the six communities communities. quality sampling and service provider at municipal level. water point survey. survey. Community A Key Water point Service provider Household Service authority Community B Note: DISE = Integrated Monitoring and Evaluation System (Dispositif Intégré de Suivi Evaluation). 18 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context System --Dispositif Intégré de Suivi Evaluation), which takes households as the entry points for linking them and water point surveys. A schematic of the process appears in figure 5.1 using Community A and Community B as examples and to illustrate that the service authority is overseeing several communities. The one enumerator in Burkina Faso collected data in the Hauts Bassins region; the commune of Karangasso-Vigué, the main town that administers 25 villages; and in six of these 25 villages: Dan, Deguele, Karangasso-Vigué, Soumousso, Poya, and Wara. The enumerator completed the job in 20 days. Kenya The pilot in Kenya took place in conjunction with water quality mapping planned by Laikipia County within the scope of the Watershed program.4 Funded by the Directorate-General for International Cooperation (DGIS) and implemented by Akvo, IRC, Simavi, and Wetlands International, the Watershed program aims for improved governance in the WASH sector that is responsive to the interests of margin- alized groups. Laikipia County officials agreed to collecting the rural metrics indicators simultaneous to the Watershed program’s water quality mapping. The indicator surveys provided the basis for both the pilot and the water quality mapping because the surveys already contained most of the variables foreseen in the scope of the water quality mapping. In addition to E. Coli and fluoride, the Watershed program made collecting pH and turbidity data possible through the surveys. A schematic of the data collection appears in figure 5.2 using Community A, FIGURE 5.2. Data Collection in Kenya 1. The enumerators mapped 2. At every fifth water point, the 3. The enumerators all water points in the county enumerators identified a household collected data and tested them for E. Coli, that collects water from that water from the service authority fluoride, pH, and turbidity. point and the water service provider at county level. (Co-funded by the Watershed program) that maintains the water point. The enumerators then collected the household and water service provider surveys from those sources. Community A Key Water point Service provider Household Service authority Community B Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 19 a community with several water points, and Community B, a community with only one water point, as examples and to illustrate that the service authority is overseeing several communities. Data collection took place in the three subcounties of Laikipia: Laikipia East, Laikipia North, and Laikipia West. Four enumerators worked on alternating schedules and altogether completed the work in 32 days. Sierra Leone Researchers conducted the pilot in close collaboration with the Ministry of Water Resources (MWR) and chose enumerators based on their experience in prior activities to map water points with Akvo Flow. A schematic of the data collection process in Sierra Leone appears in figure 5.3 using Community A, a community with several water points, and Community B, a community with only one water point, as examples and to illustrate that the service authority is overseeing several communities. The two enumerators spent 17 days collecting field data, after which enumerators and authorities completed the surveys. Enumerators collected data in the Port Loko district, the most populous dis- trict in the north and the second most populous district in Sierra Leone. Chiefdoms included in the sample were Bureh Kasseh Makonteh, Kaffu Bullom, Koya, Lokomasama, Maforki, Marampa, Masimera, and TMS. FIGURE 5.3. Data Collection in Sierra Leone 1. The MWR and research team 2. For every water point mapped, the 3. The enumerators randomly selected 75 water points enumerators identified (a) a household collected the in the intervention area from that collects water from that water point service authority survey the National WASH inventory, and (b) the water service provider that at the county level. from which the enumerators maintains it. The enumerators then collected the water point survey. collected data from those sources. For the water points where the enumerators could not identify a water service provider, no water service provider survey was completed. Community A Key Water point Service provider Household Service authority Community B Note: WASH = water supply, sanitation, and hygiene. 20 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Data Analysis Although the data collected represent a valuable contribution of this research, the primary goal was to determine the feasibility and usefulness of the proposed metrics. Achieving this goal required further analysis of the data, both quantitative and qualitative. Throughout the research process, five primary sources of data enabled systematic reflection and documentation of insights. That served as the main sources of evidence for feasibility of the survey questions and, to a lesser degree, usefulness of the data. The five primary sources of data included: •• Feedback on survey design: All changes to the survey merited documentation throughout develop- ment. These changes, and their rationale, provided key qualitative insights into the feasibility of the metrics as originally presented. •• Posttraining feedback report: Insights from the training session and field pretest were captured in country-specific posttraining reports that contained questions raised about the metrics during train- ing, comments and observations on the questions, and any adjustments to contextualize the metrics survey. The reasons for suggesting changes to the survey were documented. •• Immediate feedback questions: The surveys included feedback queries to document respondents’ level of confidence in answering the previous question(s) or to solicit objective evidence about their responses. For example, if the service provider indicated keeping a revenue ledger, the feedback question asked whether the provider could furnish the ledger while the enumerator was conducting the survey. •• Postdata collection feedback report: In this reflection process, supervisors commented on the pro- cess. They addressed challenges and successes during data collection; provided feedback on the sur- veys about feasibility, comprehension, and relevance; and supplied additional feedback from third parties involved. •• Final enumerator survey: Enumerators provided feedback on every question in the surveys, specifi- cally the time respondents spent answering each question relative to other questions, the perceived difficulty level of each query, and reasons why respondents considered a question difficult. Enumerators also summarized problems finding participants, challenges in the field, and suggestions for future implementation. In the three countries, these five primary data sources provided rich structured evidence to under- stand the feasibility and challenges of implementing the survey, considering issues such as reliability, comprehension, duration, and others. These data sources provided critical insights to determine the feasibility of collecting data, but deter- mining usefulness was a greater challenge. The primary difficulty: Many actors may wish to use this data in different ways, and each of them might find a specific piece of data more or less useful. In the case of WPDx, users employ data in hundreds of ways every year. The next section explores the approach to overcoming this challenge in more detail. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 21 Determining Feasibility Quantitative Assessment of Feasibility Assessment of the survey’s quantitative feasibility occurred at the question, or parameter, level. Through this approach, the survey team recognized that if any parameter needed to calculate whether an indica- tor was harder than others to collect, it would be difficult to determine the overall indicator. Similarly, if all the parameters needed to calculate an indicator are less difficult to collect, calculating the indicator itself would be easier. Three primary data sources provided the foundation for quantitative analysis: •• Responses to immediate feedback questions asked of enumerators and respondents during the surveys •• Data collected from respondents via survey questions •• Data collected from enumerators about their experiences in implementation To achieve consistency across these data sources, the researchers processed the findings to determine a numerical feasibility score. Immediate Feedback Questions Analysts averaged and normalized scores for the three immediate feedback questions. Survey developers had framed most of these questions as a Likert scale, ranging from 5 (extremely confident) to 1 (not confi- dent). Some questions, however, were binary (that is, 1 for yes or 2 for no), or they had fewer than five options. All questions were ordinal: Higher scores indicated greater feasibility for the survey question(s) under scrutiny, and lower scores represented greater challenges. In these cases, normalization occurred by dividing the average score by the number of potential options and multiplying by 5 to arrive at a score out of five. The resulting number was the feasibility score for the immediate feedback questions. Analysts determined a feasibility score for every query with immediate feedback questions, as shown in table 5.3. Possibilities for limitations do exist. Because all fields were required, some respondents may have answered that they didn’t know the answer to a question, but enumerators still asked them about their confidence in answering. In some cases, they may have responded that they were very confident that they didn’t know the answer. This would have indicated a high level of confidence, but in reality, they lacked the confidence to answer the survey question. However, because the methodology of the report captured both people unable to answer and those who had a low level of confidence in answer- ing, the research team does not expect this situation to affect the findings. TABLE 5.3. Feasibility Score Derived from Immediate Feedback Question Respondent How many minutes does it take to collect water? How confident are you on that answer? Respondent 1 30 minutes 5 – Extremely Confident Respondent 2 30 minutes 4 – Very Confident Respondent 3 45 minutes 3 – Confident Respondent 4 40 minutes 2 – Somewhat Confident Average Confidence Score: 3.5 22 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Data Collected from Respondents Analysts studied data collected from respondents to determine the percentage of valid “don’t know” responses for each indicator, as shown in table 5.4. If an enumerator did not ask a question (for example, if it was a follow-up query to a survey question the respondent didn’t answer initially), the analysts did not include this data in their computations. Thus, the analysts only considered the percent of respon- dents who didn’t know an answer to a question they could reasonably know the answer to. Enumerator Survey Analysts assessed data regarding (a) the time respondents spent on each question relative to other ques- tions, (b) the difficulty level of each question as perceived by respondents, and (c) the challenges each question presented. The pilot collected responses about question duration via a Likert scale, from much shorter to much longer. The average score determined duration. The analysts used a similar process to determine average level of difficulty for each question. Finally, the study team assessed data regarding challenges associated with each question: •• Did not understand the question •• Question was not relevant to respondent •• Could not recall the answer •• Did not have any way to know the answer •• Felt the question would be better directed to someone else •• Did not feel comfortable answering the question Enumerators could select as many challenges (0–6) as they felt applied to each question. Analysts calculated the number of challenges for each question, including all enumerators and all possible chal- lenges in the summations. An example is shown in table 5.5. TABLE 5.4. Example of a Feasibility Score for “Don’t Know” Answers Respondent How many minutes does it take to collect water? Respondent 1 30 minutes Respondent 2 45 minutes Respondent 3 Don’t Know Respondent 4 * Average “Don’t Know” Score: 33% * Intentionally left blank because respondent 4 did not answer the question and the blank line was not accounted for in the denominator. TABLE 5.5. Example of Enumerator’s Assessment of One Survey Question Post-Survey Enumerator Feedback for “How many minutes does it take to collect water?” How long did this question How difficult was this question Which challenges did you face on this Enumerator take? for respondents to answer? question? Enumerator 1 1 – Much longer than others 1 – Much more difficult than others Did not understand; Could not recall (2) Enumerator 2 3 – About the same 4 – Slightly easier than others Uncomfortable answering (1) Enumerator 3 4 – Slightly shorter than others 4 – Slightly easier than others Did not understand; Uncomfortable answering (2) Duration Feasibility Score: 2.7 Difficulty Feasibility Score: 3 Challenge Feasibility Score: 1.6 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 23 The analysis presented in tables 5.3–5.5 provided five feasibility scores for every question. These scores included: •• Immediate feedback feasibility score (for questions that asked immediate feedback questions) •• Unable to answer feasibility score •• Duration feasibility score •• Difficulty feasibility score •• Challenges feasibility score Setting a Threshold for More Difficult Questions Each feasibility score presented a spectrum, so no binary distinction was possible between “more diffi- cult” and “less difficult” questions. Thus, analysts identified thresholds to identify difficult questions. In addition, each feasibility score had a different range, so the thresholds needed to account for the distribution of scores. In all cases, the threshold was set at two-thirds of the total range. For example, if a feasibility score ranged from 1 to 4, with 4 being the most difficult, the threshold was set at 3. All ques- tions with scores in the most difficult one-third of the range (that is, above 3) were tagged as “more dif- ficult.” This process occurred for the five feasibility scores. An example is given in table 5.6. If any of the five feasibility scores exceeded the threshold, the question itself was identified as “more difficult.” See table 5.7 for an example. Linking Difficult Questions to Difficult Metrics and Indicators Because the assessment occurred at the question level, the analysts aggregated the findings to identify more difficult metrics (that is, if any question needed to collect the required parameters was identified as more difficult) and more difficult indicators (that is, if all metrics within the indicator were more dif- ficult). For an indicator to be identified as less difficult, a less-difficult pathway was required from the question to the metric and then to the indicator, as illustrated with question A1 in table 5.8. TABLE 5.6. Example of Thresholds for Difficult Questions Question Average Challenge Feasibility Score (Higher = More Difficult) How many minutes does it take to collect water? 1 How satisfied are you with the quality of water? 1.5 How much money do you spend on water monthly? 4 Min: 1 Max: 4 Feasibility Threshold @ 2/3 of Range: 3 Question Difficult Based on “Challenge” Feasibility Threshold of 3? How many minutes does it take to collect water? No How satisfied are you with the quality of water? No How much money do you spend on water monthly? Yes 24 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE 5.7. Example of Assessment of More Difficult Questions Difficult Based Difficult Based Difficult Difficult Difficult on “Immediate on “Unable to Based on Based on Based on Difficult Question Feedback” Answer” “Duration” “Difficulty” “Challenges” Overall? How many minutes does it Yes Yes take to collect water? How satisfied are you with the No quality of water? How much money do you Yes Yes Yes Yes spend on water monthly? TABLE 5.8. Pathway for Questions Considered Less Difficult Question Metric Indicator Question Difficult? Difficult? Difficult? 1.1 What is the main source of drinking water for members of your household? Not Difficult Not Not Difficult Difficult 1.2 What is the main source of water used by members of your household for other purposes? Not Difficult 2.1 Where is that water source located? Not Difficult Difficult Difficult 2.2 How long does it take to go there, get water, and come back? Difficult Qualitative Feasibility Assessment Several sources provided qualitative assessment, including the enumerator survey, posttraining feed- back, and postsurvey commentary. These sources typically reflected upon survey implementation over- all, rather than on specific questions. Content analysis identified key themes. Determining Usefulness Data collection provides information for a productive purpose, such as improving water services through maintenance or new development, but the potential benefits do not end there. Specifically, these indi- cators should “provide core management information to all participants in the delivery of RWSS ser- vices from the community-based organization at one extreme to national government as the other.” The researchers who conducted the pilot recognize the extensive range of potential uses for the data collected in this pilot. Global indicators could help solve challenges. First, they could improve the way that national moni- toring takes place. An illustrative set of metrics could be used as a reference, with metrics adapted and/ or removed to fit local context. Second, the metrics could standardize data across countries. Harmonizing data could improve decision making by actors operating on a global scale, such as donors, governments, multilateral agencies, and nongovernmental organizations (NGOs). This also would allow simplified benchmarking at a local level. Additionally, this approach could reduce costs for monitoring and evalu- ation because developing a single set of guidance principles could reduce the effort and funding each country needs to invest to monitor key rural water services. Lastly, such data could open the door to harmonized analysis so powerful tools could be developed once and applied across all countries with harmonized data. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 25 Focusing on the 24 rural indicators, this pilot provides a robust starting point for developing or improv- ing national or subnational indicators. As such, the primary objective is to evaluate the proposed rural metrics to decide upon the most feasible and useful ones for standardization, enabling users to harmo- nize data across borders. Thus, the evaluation of usefulness occurred with a lens toward helping multi- national stakeholders make more informed decisions and provide more effective support. This benefit, however, may be perceived to be less direct. The advantages of standardizing the metrics apply first to international organizations, but these metrics also must provide immediate value to local and national stakeholders to encourage adoption of these metrics. As such, the research team identified indicators as useful if they could support decision making at both global and national levels. If the indicator could offer value only to the international community, it likely would not be adopted at a local level. If the indicator could provide value only at a local level, an effort to standardize and harmonize that metric across borders would have limited value. The major challenge in analyzing the usefulness of the metrics is that, at both local and global levels, many stakeholders could potentially use data for a nearly unlimited number of uses. In order to system- atically assess which metrics are useful at both a local and global level, a two-step evaluation process has been used. The research team identified three arenas of data usage: accountability, learning, and steering. Team members developed illustrative questions, with one question for each usage area and decision level. These six questions formed the foundation for determining the value of the data. Based on the author’s experiences supporting data usage across the rural water sector and on input from the World Bank, the set of questions provided the first framework for evaluating the usefulness of each indicator. Recognizing the importance of the Sustainable Development Goals (SDGs) in the water sector, one question looked specifically at the SDGs. This ensured that any indicator supporting SDG measurement was identified as useful. Potential approaches for answering each of the six questions were then developed. Each indica- tor used in answering one of the three local questions to indicate value was identified as locally useful. Similarly, each indicator used in answering one of the three global questions to indicate value was iden- tified as globally useful. See table 5.9 below. Researchers recognized that they could limit themselves and overlook useful material by pre identifying questions, so they interviewed key stakeholders at the local and global levels to solicit input on the most useful metrics. They conducted 10 semi structured interviews with representatives of service authorities in two countries, international NGOs, and multilateral stakeholders. Each interviewee identified the top three indicators in each domain (that is, functionality, service level, and sustainability) that they were most likely to use on a regular basis. Indicators recommended by the service authorities were identified as locally useful. Indicators suggested by regional and global stakeholders were identified as globally useful. In the final analysis, an indicator was designated as useful if it was identified as both locally useful and globally useful. Developing Recommendations Ultimately, the research team identified each indicator as more difficult or less difficult and more useful or less useful based on the feasibility and usefulness analysis. Figure 5.4 captures the possible categoriza- tion of each indicator. 26 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE 5.9. Questions to Determine Usefulness Purpose of Illustrative activities using Illustrative Local questions Illustrative global Global questions to data usage data local data user to indicate value data user indicate value Accountability • Demonstrating Civil society What aspects of International NGO To what extent is progress against stated organization service delivery the government commitments is the service successfully delivering • Holding others provider failing on SDG 6.1 targets? accountable for their to deliver in each roles and responsibilities district? • Engaging others through advocacy Steering • Making evidence-based District What type of International In which country management decisions government investment can donor should investment • Mobilizing new external have the greatest in infrastructure be resources impact for each prioritized, and in community? which is technical • Improving impact and assistance relatively efficiency of budget more needed? allocations Learning • Identifying approaches University How does University What type of cost that deliver more this country recovery approach is sustainable outcomes compare to other correlated with the • Creating new models for countries in the highest level of spot service delivery water services functionality in the that people region? • Developing strategies receive? that increase inclusion and equity Note: NGO = nongovernmental organization; SDG = Sustainable Development Goal. FIGURE 5.4. Possible Categorization of Indicators More usefull Core harmonized indicator Core harmonized indicator for review More difficult Less difficult Retain only as reference Retain only as reference Less usefull Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 27 According to the recommendation matrix, three possible scenarios exist: •• If the indicator was not determined to be useful at both global and local levels, the research team recommends it only as a reference indicator to focus on those metrics that are more useful when stan- dardized and harmonized. •• If the indicator was found to be more useful and less difficult, the research team recommends it as a core harmonized indicator. It should be maintained as it was implemented in the pilot survey. •• Finally, if the indicator was useful but identified to be more difficult, the research team recommends it as a core harmonized indicator, although it merits review to explore the possibility of a less difficult method to capture the metric in a standardized way. Notes 1. Data are from Household Data (database), WHO/UNICEF (World Health Organization and the United Nations Children’s Fund) Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (JMP), Geneva, https://washdata.org/data/household#/table?geo0=region&geo1=sdg. 2. Data are from World Bank Open Data (database), World Bank, Washington, DC, https://data.worldbank.org/indicator/NY.GDP.PCAP.CD​ ?locations=ZG&year_high_desc=false. 3. Data are from Household Data (database), WHO/UNICEF JMP, Geneva, https://washdata.org/data/household#/table?geo0=region&geo1=sdg. 4. In 2016, Akvo worked with AfDB, DFID, Statistics Sierra Leone, and UNICEF to support the MoWR in mapping the water points in the whole country (https://wateratlas.akvotest.org/). This exercise aimed to become a baseline for SDG indicators. One hundred and eighty enumer- ators from various NGOs and ministry staff were trained by the ministry and Statistics Sierra Leone. Since 2016, local efforts have endeav- ored to update data with some NGOs, but nothing has been organized at the national level. When this report was written, Akvo was collaborating with MoWR to collect water quality data in 1,100 communities throughout the country, as well as to work on other ongoing monitoring activities with UNICEF Sierra Leone. Together with IRC, Akvo has written a five-year plan that outlines the pathway to sustain- able country-led monitoring of WASH in Sierra Leone. 28 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Chapter 6 Project Findings The enumerator survey proved to be the most challenging across all countries, mostly because of its length, which disincentivized enumerators from completing it. Table 6.1 summarizes the number of data points collected per survey type in each of the three countries. Feasibility Assessment Quantitative Analysis Findings Researchers used the quantitative methodology described in chapter 5 to evaluate the feasibility of each query in the survey, which consisted of 79 questions across 24 indicators. The findings reported in this document relate to individual parameters, not to indicators as a whole. These findings cap- ture only survey questions aimed at gathering information on required parameters from the original indicators and do not capture information about any supplementary questions or feedback questions. Immediate Feedback Feasibility Score Across all four questionnaires (that is, household, service authority, service provider, and water point), the normalized immediate feedback scores ranged from 2.74 to 3.67. Higher scores represented greater levels of feasibility, and lower scores were relatively more difficult. In many cases, this referred to the confidence with which respondents answered questions, but it also captured enumerator responses to how long questions (or sets of questions) took to answer. For example, one question capturing respondent feedback asked about monthly expenditure on all goods. A query that captured enumerator feedback looked at the time required to collect all of the photos or all components of a pipe scheme. Given the range, the research team determined that a value of two-thirds resulted in a threshold of 3.05. All scores below that number were marked as more difficult, which was the case for 10 questions (see table 6.2). TABLE 6.1. Surveys Collected Per Target Country Number of surveys collected/target Data collection dates (2018) Household Water point Service provider Service authority Enumerator Burkina Faso 75/75 75/75 48/6 1/1 2/2 September 26–October 12 Kenya 50/60 154/60 44/60 2/1 4/5 August 30–October 22 Sierra Leone 90/75 90/75 12/12 1/1 1/4 August 28–October 17 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 29 TABLE 6.2. Survey Questions Marked Difficult Based on Immediate Feedback 2.2 How long does it take to go there, get water, and come back? 5.1 During the last year, how many hours has water stopped flowing without warning due to technical breakdowns, corrective maintenance works, or non-regular rationing? 9.1.1 What is the observable condition of the water intake? 9.1.2 What is the observable condition of the water reservoir? 9.1.3 What is the observable condition of the tap? 14.2 How many households in your service area use this water infrastructure on a regular basis? 19.1.1 How much money, in local currency, was spent on operations and maintenance for the water service in the last twelve months? 19.1.2 How much money, in local currency, was received as revenue from tariffs in the last year? 19.2.1 What is the annual net income in local currency, considering all revenue and subtracting all costs and depreciation? 19.2.2 What are the total liabilities in local currency, including short term and long term, that will need to be paid in the coming year? TABLE 6.3. Survey Questions Marked Difficult Due to “Unable to Answer” 6.2 How much money, in local currency, did your household spend on basic goods and services (i.e. food, water, medical, etc.) in the last month? 14.1 How many households are in your service area? 15.1 How much water (in liters) was produced from this water source in the past month? 19.1.1 How much money, in local currency, was spent on operations and maintenance for the water service in the last twelve months? 19.2.2 What are the total liabilities in local currency, including short term and long term, that will need to be paid in the coming year? 23.2.1a How many communities are under the service authority area? 23.2.2a How many communities received support from the service authority in the last year? Unable to Answer Feasibility Score Across the four questionnaires, the maximum percent of unable to answer responses was 61 percent (service provider question 19.2.2: What are the total liabilities, in local currency, including short term and long term, that will need to be paid in the coming year?). The minimum was 0 percent because many questions received a valid answer from all respondents. Setting the threshold for difficult at two-thirds resulted in a threshold of 40 percent. Seven questions exceeded this threshold and were marked as dif- ficult (see table 6.3). Duration Feasibility Score The enumerator survey yielded duration scores for each question; scores ranged from 1.14 to 5.00, with 5 representing a question that took much longer than the average question to answer. However, there was only one average score of 5, which was significantly higher than other scores in this category. This score related to arsenic testing, which was completed by a single enumerator in only one country. Because this wasn’t a representative score, this value was removed from the range, producing a range of 1.14 to 4.29 and a two-thirds threshold of 3.24. A total of 24 questions were ultimately marked as difficult as a result of the enumerator feedback on the duration, the largest number across all data sources (see table 6.4). 30 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE 6.4. Survey Questions Marked Difficult Due to Duration 2.2 How long does it take to go there, get water, and come back? 3.1.1 On average how many hours a week is water available? 4.1 What is the level of E. Coli in the water? 5.1 During the last year, how many hours has water stopped flowing without warning due to technical breakdowns, corrective maintenance works, or non-regular rationing? 6.1 How much money, in local currency, did your household spend on water from your primary water source in the last month? 6.2 How much money, in local currency, did your household spend on basic goods and services (i.e. food, water, medical, etc.) in the last month? 9.2.1 How many times during the last year did a breakage or new leak occur in the conveyance or distribution network of the water supply system? 9.2.2 How many kilometers of pipe are in the conveyance and distribution network of the water supply system? 10.2.1 Does the service provider comply with legal requirements? 12.2.1 Does the service provider have an inventory of all infrastructure assets? 12.2.3 Has this inventory been updated in the past six months? 12.2.4 Is this inventory being used for planning maintenance and replacement works? 14.1 How many households are in your service area? 14.2 How many households in your service area use this water infrastructure on a regular basis? 15.1 How much water (in liters) was produced from this water source in the past month? 15.2 Of all of the water produced from this water source in the past month (in liters) how much was paid for? 18.1.2 How many customers does this system have? 18.1.3 How many customers have outstanding bills? 18.2.1 What is the total value, in local currency, of water billed every month? 18.2.2 What is the total revenue, in local currency, collected every month? 19.1.1 How much money, in local currency, was spent on operations and maintenance for the water service in the last twelve months? 19.1.2 How much money, in local currency, was received as revenue from tariffs in the last year? 19.2.1 What is the annual net income in local currency, considering all revenue and subtracting all costs and depreciation? 22.2.1 How much funding in local currency, including salaries, was projected to be required by the service authority over the last twelve month budget period? Difficulty Feasibility Score The range of enumerator scores capturing the perceived difficulty of each question ranged from 1 to 3, with higher numbers representing greater levels of difficulty. At two-thirds of the range, the threshold was set at 2.33. Only three questions were marked as difficult based on this threshold (see table 6.5). Challenges Feasibility Score Enumerators selected from six possible challenges that might have increased the difficulty of each ­ question; they could indicate 0 to 6 challenges per query. The research team added the number of chal- lenges indicated by each enumerator to give a grand total per question across the seven enumerators Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 31 TABLE 6.5. Survey Questions Marked Difficult Based on Difficulty Feasibility Score 6.2 How much money, in local currency, did your household spend on basic goods and services (that is, food, water, medical, etc.) in the last month? 13.2.1 What is the residual chlorine concentration in mg/l? 15.1 How much water (in liters) was produced from this water source in the past month? Note: mg/l = milligrams per liter. TABLE 6.6. Survey Questions Marked Difficult Due to Challenges 5.1 During the last year, how many hours has water stopped flowing without warning due to technical breakdowns, corrective maintenance works, or non-regular rationing? 6.2 How much money, in local currency, did your household spend on basic goods and services (i.e. food, water, medical, etc.) in the last month? 9.2.1 How many times during the last year did a breakage or new leak occur in the conveyance or distribution network of the water supply system? 10.2.1 Does the service provider comply with legal requirements? 12.2.1 Does the service provider have an inventory of all infrastructure assets? 14.1 How many households are in your service area? 14.2 How many households in your service area use this water infrastructure on a regular basis? 15.1 How much water (in liters) was produced from this water source in the past month? 18.1.2 How many customers does this system have? 18.1.3 How many customers have outstanding bills? 18.2.1 What is the total value, in local currency, of water billed every month? 18.2.2 What is the total revenue, in local currency, collected every month? 19.1.1 How much money, in local currency, was spent on operations and maintenance for the water service in the last twelve months? 19.2.1 What is the annual net income in local currency, considering all revenue and subtracting all costs and depreciation? in three countries. The total number of challenges for each question ranged from 0 to 16. A threshold of two-thirds resulted in a limit of 10.67 challenges without being labeled as more difficult. A total of 14 questions exceeded the threshold and were marked as more difficult (see table 6.6). Quantitative Summary Out of 79 questions, 31 were identified as more difficult, representing 39.2 percent of the total survey. Twenty-two of the 31 queries identified as more difficult were questions for which respondents needed to provide a number as an answer (24, if including water quality testing answers). Additional confidence in the methodology was assured by the majority (19 out of 31) of more difficult questions being identi- fied as difficult by more than one data source (that is, percentage unable to answer and challenges iden- tified). The list of difficult questions appears in table 6.7. 32 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE 6.7. All Survey Questions Identified as Difficult 3.1.1 On average, how many hours a week is water available? 4.1 What is the level of E. Coli in the water? 5.1 During the last year, how many hours has water stopped flowing without warning due to technical breakdowns, corrective maintenance works, or non-regular rationing? 6.1 How much money, in local currency, did your household spend on water from your primary water source in the last month? 6.2 How much money, in local currency, did your household spend on basic goods and services (i.e. food, water, medical, etc.) in the last month? 9.1.1 What is the observable condition of the water intake? 9.1.2 What is the observable condition of the water reservoir? 9.1.3 What is the observable condition of the tap? 9.2.1 How many times during the last year did a breakage or new leak occur in the conveyance or distribution network of the water supply system? 9.2.2 How many kilometers of pipe are in the conveyance and distribution network of the water supply system? 10.2.1 Does the service provider comply with legal requirements? 12.2.1 Does the service provider have an inventory of all infrastructure assets? 12.2.3 Has this inventory been updated in the past six months? 12.2.4 Is this inventory being used for planning maintenance and replacement works? 13.2.1 What is the residual chlorine concentration in mg/l? 14.1 How many households are in your service area? 14.2 How many households in your service area use this water infrastructure on a regular basis? 15.1 How much water (in liters) was produced from this water source in the past month? 15.2 Of all of the water produced from this water source in the past month (in liters) how much was paid for? 18.1.2 How many customers does this system have? 18.1.3 How many customers have outstanding bills? 18.2.1 What is the total value, in local currency, of water billed every month? 18.2.2 What is the total revenue, in local currency, collected every month? 19.1.1 How much money, in local currency, was spent on operations and maintenance for the water service in the last twelve months? 19.1.2 How much money, in local currency, was received as revenue from tariffs in the last year? 19.2.1 What is the annual net income in local currency, considering all revenue and subtracting all costs and depreciation? 19.2.2 What are the total liabilities in local currency, including short term and long term, that will need to be paid in the coming year? 2.2 How long does it take to go there, get water, and come back? 22.2.1 How much funding in local currency, including salaries, was projected to be required by the service authority over the last twelve-month budget period? 23.2.1a How many communities are under the service authority area? 23.2.2a How many communities received support from the service authority in the last year? Note: mg/l = milligrams per liter. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 33 Qualitative Analysis of Feasibility Findings Researchers reviewed qualitative feedback from the enumerator surveys, posttraining reports, and postsurvey reports to arrive at overall findings. The team also identified challenges faced in each coun- try during the pilot and pinpointed key themes across the three countries. Burkina Faso The late decision to integrate water quality testing for E. Coli, arsenic, and residual chlorine into the pilot delayed activities by a month and a half to allow time for procuring, shipping, and getting supplies out of customs (a lengthy and costly process because free chlorine reagent is labeled as a dangerous good). However, the delay did accommodate translation and digitization of the French surveys, a process that required several rounds of review to ensure consistency with the English versions. Rains made it hard for the enumerator to access certain areas, but the protocol, which called for house- holds to be matched with water points and service providers, also caused unexpected delays because people weren’t always available on the same days in each village. For example, if the water service pro- vider for a specific household was not available on the day the household and water point surveys were completed, the enumerator waited until they could be reached. In some cases, that was not possible. Despite these challenges, data collection was completed over a period of three weeks, just one more week than initially planned. Kenya Enumerators in Kenya faced several challenges. Laikipia East and Laikipia West subcounties have a good road infrastructure, which made those areas accessible. However, the road infrastructure is poor in Laikipia North, the subcounty farthest away from county headquarters in Nanyuki. Rains and secu- rity conditions were not conducive to completing data collection as planned in that region; thus, the team had to adjust the protocol, which initially called for enumerators to visit a household every fifth water point visited for Watershed work. To complete data collection faster, the team proposed that enu- merators visit households more frequently and focus on the most accessible geographic areas, a change that the World Bank approved. Despite these efforts, though, enumerators did not achieve the target sample size for households and water service providers primarily because of limitations of time and resources. Sierra Leone Changes within the Ministry of Water Resources at the beginning of the pilot caused delays, but once researchers secured approval, the team proceeded with the activities. Enumerators completed an initial round of surveys and, upon review of the water point survey, noticed that the sampling did not include enough diversity in the type of source. Therefore, they com- pleted a second round to capture a more diverse sample that was also more in line with the 2016 water point mapping. This caused the water point sample size to exceed the target of 75. Because the protocol called for a one-to-one match of water points to households, the number of households surveyed also reflected this increase. 34 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Data collection occurred around the provincial roads, which often have concentrated population cen- ters. To diversify population samples, enumerators also tried to visit remote households; however, this wasn’t always possible because of road conditions during the rainy season. Rains also resulted in adjust- ments to the initial proposed timeframe. Finally, as a result of changes within the ministry, Akvo was unable to get a signed authorization letter to collect data. However, the ministry authorized it, as evidenced by the fact that the Ministry of Water Resources (MoWR) methods and evaluation coordinator, who is in charge of coordinating the ministry’s field exercises, participated and supervised the data collection. Collaborating with the Watershed program provided both advantages and disadvantages. The advantages included: •• Cross-sharing of data •• More efficient training •• Reduction in costs •• Added value for both investments—the World Bank benefited from being able to draw upon the water quality information collected by the Watershed efforts, and the Watershed venture benefited from having access to the data on households as well as service providers and authorities •• Greater number of water points tabbed for data collection On the other hand, the need to coordinate efforts among more actors made data collection in Kenya more complex than in the other two countries. For example, the researchers had difficulty coordinating use of vehicles, which the county office was supposed to provide. Competing priorities within the coun- try office created the need to hire private vehicles to proceed with data collection, an unforeseen cost and delay. The three countries experienced delays in implementation, but in general, teams largely abided by the country protocols in Burkina Faso and Sierra Leone, where all but the enumerators survey targets were met or exceeded. In Kenya, continuous rains and related feasibility challenges led to modifications of the data collection protocol. Despite those efforts, target samples in Kenya were not achieved. Key Themes Overall, field supervisors and enumerators found that respondents generally were willing to answer survey questions. Mobile data collection worked well throughout the pilot, and enumerators who col- lected water quality data indicated it was a manageable process. Review of the final implementation reports and insights gathered from final conversations with field staff generated. Leveraging Existing Relationships Good relationships among principals were critical to the pilot’s success. Akvo was a key player in previ- ous Water Supply, Sanitation and Hygiene (WASH) monitoring exercises in Burkina Faso and Sierra Leone, and it had worked closely with county officials in Laikipia, Kenya, before the idea germinated to pilot the rural metrics framework. These relationships helped researchers get permissions and Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 35 authorizations; improved coordination and communication; helped the team enlist dedicated enumer- ators; led to shorter training times and sharing of resources (that is, premises and equipment for train- ing); and provided access to previous inventories consulted for sampling. By having good relationships already in place, researchers had fewer stumbling blocks and, consequently, fewer mistakes overall. Enumerator Capacity and Experience Akvo’s understanding of the context, its prior experience, and its working relationships enabled research- ers to select data collectors from a pool of knowledgeable and trustworthy enumerators familiar with the WASH sector, the technology to be used, and key actors in the intervention area. Their knowledge and prior experience with WASH surveys were among the most important factors that contributed positively to data collection. The field supervisor from Burkina Faso noted, “We were working with a very experi- enced team of data collectors and supervisors. In practice, this meant that feedback on the questionnaires and logistical challenges were addressed quickly to keep quality data collection ongoing. Also, these data collectors are ministry engineers and know a lot about the situation in the country.” Respondent Familiarity As mentioned earlier, prior data collection in the pilot areas gave the enumerators advantages, including familiarity with the area, water points, households, and key actors. This was particularly true in Burkina Faso and Sierra Leone. Similarly, community members were familiar with surveys about drinking water, having encountered them in the past, which made it easier for them to understand questions in the household and water point surveys. However, respondents in several households expressed frustration with not receiving feedback on the results of or decisions made after these surveys. Seasonality and Logistics All three countries faced challenges with seasonal rains. In Kenya, this was the primary reason for not completing the target sample size. All three field supervisors indicated that the research team should consider data collection when planning because heavy rains can compromise the safety of enumerators, create logistical problems, raise competing priorities among local actors, and increase costs. Some of these concerns often led enumerators to adjust data collection plans. In Sierra Leone and Kenya, this resulted in avoiding some of the most remote areas, which could have skewed efforts to collect repre- sentative data. The field supervisor in Sierra Leone noted, “The period of data collection was in an intense monsoon period. Data collectors have struggled with very bad roads, which led to motorcycle breakdowns, minor accidents, and overall delays in collection. At the same time, daily rain on the data collectors while being outside sometimes led to hard moments in keeping up the spirit. As much as possible, (financial) sup- port was given to the data collectors to keep them going. In the future, we should avoid the monsoon period or provide finance for cars as this is not a given fact for the ministry in rural Sierra Leone.” Supporting Documentation and Guidelines Field coordinators in the three countries received the surveys, illustrations of water supply and sanita- tion facilities, and definitions included in the original metrics guidance or added during the survey 36 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context design phase that is, for the observable elements of the piped scheme, included definition about the potential answers: good, poor, acceptable, and deficient). The field teams then participated in sessions to clarify information while they digitized the surveys and prepared for training. The field teams raised questions about definitions and about the interpretation of questions that had not been anticipated. For instance, they wanted to know the meaning of inventories, operations, and legal requirements and ques- tioned whether enumerators needed to be able to explain terms such as inflation and depreciation. Answers to questions asked in Kenya—first to start digitization and training—became part of the guid- ance provided to the other countries. In addition, necessary definitions were added as tool tips in Akvo Flow, making them available to enumerators. All doubts, changes, and questions about definitions appear in the posttraining reports. Before data collection began, digitized surveys were reviewed at least three times per country. Nevertheless, queries still needed answered about definitions, skip logic, and question order or phrasing. The lack of supporting documentation for the surveys and questions proved troublesome. For instance, in Burkina Faso, they lacked supporting documentation—particularly about definitions and skip logic—. Furthermore ongoing changes to the English versions used in Kenya and Sierra Leone were not reflected in the French version because the changes occurred, while the surveys were being trans- lated to French. This caused several misunderstandings in Burkina Faso using the French version of the surveys. An important oversight: the definition of functionality. It initially did not appear in the surveys used in Kenya and Sierra Leone. In these countries, the definition was heavily debated during training ses- sions. The field supervisor in Kenya noted, “During the training, this question (about functionality] had a lot of debate. We encountered a tap water from piped system as a source of water point. However, one had to use a pliers in order to open it. Therefore, the question as to (whether it was functional] or not was discussed. The result of its functionality was mostly due to mismanagement. Therefore, restricting unnecessary use … . At the end it was agreed that it was partly functional.” Moving forward, indicators could consider the definitions of functionality in “Developing Rural Water Metrics for Sustainability: An Assessment Of Existing Indicators Of Sustainability” by Smits, Mansour, and Lockwood (2017), and orig- inally described by Wilson et al. (2016). This paper identified six types of metrics for defining function- ality. The functionality metric used for this report—and recommended for core harmonized indicators—is a hybrid between multiple categories and flow rate in relation to design yield. Specifically, the metric proposes using the time required to fill a standard-sized category to determine the category. This approach as shown in table 6.8 could be helpful in further refining the proposed rural metrics. The project coordinator became aware of the definition oversight prior to training in Burkina Faso, so additional guidance was provided on the Jerry Can methodology described in the surveys that mea- sured the time required to fill a 20 liters jerry can. This training proved effective. The field supervisor in Burkina Faso said, “The definition of the functionality of water points according to the rural metric data collection has been well understood by the enumerator. He made sure that the indications given correspond to the realities on the ground before informing the question on the functionality of the water point.” Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 37 TABLE 6.8. Approaches to Defining Functionality Type of metric used Description Binary, not defined By default, working or not working. Binary, defined Working at the time of visit, in use, not in use. Multiple categories Functional, not functional, needs repairs, semi-functional, minimally functional, functioning through difficulties, broken, missing parts, seasonal. Flow rate in relation to Whether the borehole produced a flow rate (for example, in liters per minute or the time to fill a design yield standard size bucket) that is at least equal to the design yield at the time of visit. Tiered definition Several different levels of assessment and indicators are used to assess functionality. As a minimum, functionality is assessed using a binary approach of working or not working but can be examined in greater detail using several levels of assessment. Sustainability assessment A broader assessment, which includes several factors indicating the reliability of the water supply. Functionality is one of several factors considered to assess sustainability. Source: Smits, Mansour, and Lockwood 2017. Relevance to Local Actors and Survey Customization Field supervisors in all three countries as well as enumerators in Sierra Leone and Burkina Faso indi- cated that the effort to standardize the metrics proved problematic because it did not offer enough space to contextualize. This input was particularly salient because the field supervisors and the enumerators with whom they worked were experts in the sector, including government officials and other experi- enced enumerators. They valued having a set of indicators that countries could choose from based on their needs but found the rigidity of a standard problematic. For example, questions within service provider and service authority surveys did not match the realities of those countries.1 An enumerator in Sierra Leone noted, “Hardly any rural water is paid for in Sierra Leone, which means questions about consumption and pricing are mostly not applicable. Also, questions about inflation are maybe relevant for urban water supply but can’t be asked to a local service provider … [to contextualize] we would tailor the questionnaire in a way that we would not ask questions that are not applicable to the situation of the country.” Troublesome Questions Enumerators in Kenya and Sierra Leone indicated that household respondents were generally not com- fortable answering questions about income, expenses, or finances in general. The data collection super- visor in Sierra Leone noted: “Questions related to expenses, income, or finances are often received with suspicion, which is in the culture of Sierra Leone. Households do not want everybody in the community to know about their financial situation.” Similarly, in Kenya, enumerators reported having trouble documenting financial information from water committees. The field supervisor noted, “It was impossible to take a picture of the financial doc- uments, mostly due to consent. An enumerator had to first seek consent from either the chairman, trea- surer, or secretary. And in most cases, none of these persons were present at time of collection, and if they were reachable via phone, they did not allow photos of the financial books to be taken.” 38 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context In Sierra Leone, enumerators indicated having most trouble with questions in the service provider and service authority surveys, especially with those that required recalling a number, such as ques- tion 15.1: How much water (in liters) was produced from this water source in the past month? The field supervisor indicated that those types of questions are “hard to answer, as hardly anyone keeps track of this.” These findings closely align with the quantitative findings, which indicate that 22 of the 31 questions identified as difficult above were ones where respondents had to answer a number. Similar feedback from Sierra Leone highlighted questions where the enumerator constantly had to revise sentence structure and modify delivery for respondents to understand questions. The enumera- tor noted, “the complexity of the questions in relation to the reality of the environment sometimes forced the enumerator to ask several questions in order to obtain the desired answer—for example, the number of hours of water shut down in the year, the question on inflation, water billing, complaint han- dling times, the volume of water withdrawn, the maintenance of the structure, the financial aspects for the service provider, or even questions on the chemical parameters of the water put to the service pro- vider and the service authority.” Finally, in Kenya, enumerators indicated that they had trouble with open-ended questions; they felt they had to revise the question because of the lack of guidance (that is, predefined options) that would help the respondent understand the question and answer accurately. In the water point survey, for example, some questions needed precise answers. Examples: Question S9.1: What entity was responsi- ble for installation of this water point? and Question S10.1: What inventories have captured this water point in the past? Water Quality Testing Water quality testing was performed in Burkina Faso and Kenya. Although procurement and ship- ping of water quality kits caused delays in Burkina Faso, enumerators in both countries found it simple to conduct the tests. The county government of Laikipia in Kenya also expressed interest in including additional parameters, such as electrical conductivity, phosphates, and total dissolved solvents. The E. Coli tests required a 24-hour window to get results. Therefore, to ensure data collectors could submit water point data in a timely manner, a separate survey linked to the water point was set up spe- cifically for E. Coli tests. This allowed the enumerator to finalize data collection at the water point, send the data (time-stamped at that moment), start the E. Coli test, and add the E. coli result via a different form once results were ready the next day. This process seemed to have worked well. Mobile Data Collection Field supervisors in all three countries reported advantages in using mobile data collection. However, they noted the importance of complying with best practices, such as ensuring a full charge in device batteries and providing mobile data to enumerators to ensure that they could sync data daily, even in low-connectivity environments. Dashboards set up by Akvo to monitor progress helped steer field activ- ities where needed. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 39 Usefulness Analysis The feasibility analysis occurred at the question level because the indicator consisted of several ques- tions and the difficulty depended on those component questions, but the usefulness analysis occurred at the indicator level. Key Use Case Assessment Findings Examining the three local use cases as well as the three global use cases and figuring out how those questions would be answered enabled the research team to analyze the usefulness. Indicators Required to Answer Local Questions “What aspects of service delivery is the service provider failing to deliver in each district?” Answering this question relies on one key piece of metadata (that is, district), as well an overall understanding of the degree to which each aspect of services is being delivered. Thus, indicators that touch on elements of service delivery are useful in answering this question, which would involve averaging the scores for each service level by district and identifying those districts with a score below a minimum acceptable threshold. “What type of investment can have the greatest impact for each community?” Although there are many ways to approach this question, this analysis will view the question as “what investment can best move communities up the water access ladder.” Answering that question simply requires identifying where on the water access ladder communities currently stand. The water access ladder developed by the Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (JMP) provides a useful framework. See figure 6.1. With this approach, the first step would be determining whether people have access to at least basic water. Answering that would require understanding the type of source for a given community (Indicator 1). This would identify people who are still at the unimproved level. If they are at least using an improved source, confirming if communities have access to basic water services would require understanding whether that source is within 30 minutes round-trip collection time as required by the JMP definitions for drinking water service levels. This could be answered with the accessibility question. Finally, to differentiate between basic and safely managed, data on the water quality and availability are required. Thus, this set of indicators could be used to determine, at an individual community level, which investment would most improve water services. “How does this country compare to other countries in the water services that people receive?” One approach to answering this question would be to compile a composite score for the average service level in each country. At the start, this would require critical metadata on the country of origin for each data record. Each country could then receive a score based on an index developed from the service level indicators, like those used in Question 1. The scores could then be normalized and weighted or simply averaged to create the index. Compiling such an index for each country would allow the question to be answered. Indicators Required to Answer Global Questions “To what extent is the government successfully delivering on SDG 6.1 targets?” Answering this question would first require collecting from each country its national Sustainable Development Goal (SDG) targets. 40 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context FIGURE 6.1. JMP Drinking Water Ladder Safely managed Drinking water from an improved water source which is located on premises, available when needed and free from faecal and priority chemical contamination Basic Drinking water from an improved source, provided collection time is not more than 30 minutes for a roundtrip including queuing Limited Drinking water from an improved source for which collection time exceeds 30 minutes for a roundtrip including queuing Unimproved Drinking water from an unprotected dug well or unprotected spring Surface water Drinking water directly from a river, dam, lake, pond, stream, canal or irrigation canal Source: https://washdata.org/monitoring/drinking-water. Assuming those targets called for certain levels of basic and safely managed water sources, answering the question would require data to determine who has basic service and who has safely managed ser- vices. Because basic services must meet just two criteria—an improved source and a round-trip collec- tion time not exceeding 30 minutes—only two indicators are required, with one addressing the source (type of source) and one looking at collection time (accessibility). Identifying the percentage of respondents with a safely managed source is slightly more complex because the water service must meet the basic criteria in addition to being from priority contaminants, accessible on premises, and available when needed. Determining whether the water is on premises can be done by looking at the same indicator that was used to determine the collection time. However, ensuring that the water is free from priority contaminants would require the addition of the water quality indicator. Determining availability will also require the availability indicator. Other aspects of service levels, such as afford- ability, are referenced in SDG 6.1 but are not a part of the JMP service ladder and have not been included here. “In which country should investment in infrastructure be prioritized, and in which is technical assistance relatively more needed?” To answer this question, the analysis must consider the investment that will result in sustainable water services compared with the investment that will be lost because of water Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 41 systems that don’t work. At the simplest level, this can be done by capturing the spot functionality rate of water points in the country. A relatively higher level of functionality would suggest that a greater portion of investments are sustained, whereas a lower level of functionality would suggest that much of the investment in infrastructure would result in nonfunctioning services and could perhaps be better used in supporting the enabling environment to improve the functionality rate in the future. Calculating this functionality rate can be done by looking at the two functionality indicators. Although an invest- ment in technical assistance could be further targeted by looking at additional indicators, the question itself could be answered by looking at the two functionality indicators and selecting an acceptable level of lost investment to use as a threshold. “What type of cost recovery is correlated with the highest level of spot functionality in the region?” Answering this question requires segmenting data by geographic region, likely based on the country name. In addition, the analysis requires an overall average of spot functionality, considering both hand- pumps and piped schemes, as well as the tariff structure. This information then allows for analysis of the relationship between the tariff structure and the functionality rate. Overall Findings on Usefulness In addition to identifying indicators needed to answer these questions, the research team conducted 10 interviews at local and global levels to get potential user feedback on indicators they were most likely to use and how they might use them. Interview subjects included academics, donors, government offi- cials, and regional stakeholders. Based on this input and the key use case assessments, the research team identified 13 total indicators considered useful at local and global levels. Challenges to the Use of the Metrics The overall lack of standardized metadata is the primary challenge of using the metrics. With an overar- ching goal of harmonizing data across diverse contexts and across borders, the lack of metadata is a notable absence. Standardizing the indicators themselves is necessary but not sufficient. Without meta- data, no way exists to combine data while retaining useful information related to the data source. As noted by the Global Monitoring Harmonization Task Team (GMHTT) of the Sanitation and Water for All (SWA) partnership, “It is important to collect metadata about the water point that can be used to reference the waterpoint point in time and space.” Beyond locating data in time and space, this meta- data should provide information related to the origin of the data, allowing for quality assessment and follow-up where needed. Although this metadata could be considered to be ancillary and not critical at this stage, much of the hard work invested in this standard will be frustrated if using harmonized data in practice is not possi- ble. For this reason, the metadata that should be collected with the metrics should be clearly articulated from the outset. A recommended set of metadata, pulling from the draft GMHTT standard and broadly aligned with the metadata structure used by the Water Point Data Exchange (WPDx), has been included in ”Recommended Metadata” below. Furthermore, clear definitions and instructions on how to calculate and interpret the indicators are necessary to ease implementation and make it more consistent. A glossary of terms and specific indica- tor reference sheets should be included in a future field manual. 42 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Limitations This pilot was completed in a systematic and rigorous manner, but significant limitations could nega- tively affect a global rollout of the metrics as proposed. The first area of limitations involves methodology, including the subjective decision to select a thresh- old for feasibility at two-thirds of the total range for each data source and to limit the number of inter- views to 24 to determine usefulness. In addition, this survey tested only one version of each question and only one approach to each metric. It is possible that the selected approach to reach a given indicator was more difficult than other possible approaches. For example, it would have been possible to arrive at the same metric asking slightly different questions than those included in the questionnaire. The last limitation in methodology involves the selection of respondents. Moving forward, more guidance could ensure the optimal respondent selection to maximize the number of answers a respondent could pro- vide, though this may prove to be logistically challenging. The next major limitation is in regard to sample selection for field validation. Although the research team made every effort to ensure diversity in implementation, the resources available for this effort limited the scope of work to three countries, all in Sub-Saharan Africa. Significant diversity across Africa and around the world could not be captured in these countries, but such diversity could add complexity to implementation of the metrics. Additionally, within the countries, data were collected only in a select area, which was not representative of the entire country. Additional challenges could emerge when attempting to carry out the metrics in areas that are more difficult to access or to provide water services through different types of systems than were covered in the pilot. Additionally, this validation exercise relied heavily on the researchers’ networks, often using experi- enced enumerators who had provided similar support in the past. The work occurred in contexts where the researchers had strong relationships with government officials. Implementation at scale would require a much larger cadre of enumerators, requiring more resources and more training. Finally, where relationships with governments are weaker, implementation could face additional delays or lack of cooperation. Much of the complexity that was experienced in this evaluation would grow significantly, along with costs, though it is difficult to predict the exact effects given the limited scope of the pilot. This includes the need for additional translations to other languages, the identification of different and complex rela- tionships between services and service providers, and cultural factors that may limit the ability to carry out surveys. Importantly, this effort could not capture the efficiency of implementing the metrics as a single effort at scale. For example in Kenya, 50 household surveys, 154 water point surveys, and 44 service provider surveys were collected. This was possible to do as a single effort, using one budget and one set of enu- merators, because the total numbers were relatively low. At scale, one could imagine many more house- hold surveys, water point surveys, and service provider surveys in a single country. The average water point survey took about six and a half minutes, but that does not include the time to get to the location, engage stakeholders, and complete other preparations. At scale, there may be massive inefficiencies in having the same enumerators try to work at all levels, and it may be more efficient to split data collec- tion into different efforts based on level of analysis. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 43 Furthermore, the pilot was, by design, intended to share resources with existing efforts, such as the water quality mapping exercise sponsored by the Watershed program in Kenya and ongoing activities Akvo had underway in Burkina Faso and Sierra Leone. Thus, determining the real cost of the pilot is limited by cost sharing among the various programs, which also limits the researchers’ ability to esti- mate the costs to take this exercise to scale. However, the pilot reveals the following as potential cost considerations for future exercises: enumerator, supervisor, and coordinator fees; accommodation and daily subsistence allowances; insurance (that is, health and incidental insurance for data collectors); water quality tests hardware, taxes, and shipping; fuel and transportation; training materials and venue, if necessary; flights; mobile data or Internet plans to sync data on devices; mobile data collection software; translation fees; lead researchers fees; and other miscellaneous expenses (that is, bank fees, printing, gloves, batteries, raincoats, and so on). Although the growing complexity suggests that costs might increase, some economies of scale might be gained, which would decrease implementation costs. Another potential effect on costs could be linkages with other monitoring efforts, although potential effects are difficult to predict based on the limited scale of the pilot. Lastly, data cleaning could become a significant barrier. Often an afterthought, data cleaning can require significant resources. Even among three surveys, harmonizing data with such small tweaks as spelling, language, or the format for a don’t know answer—all of which were implemented during digitization—can require a fairly substantial amount of effort. Robust standards with minimal flexibility ­ can be used, but it is highly likely that some level of data cleaning will still be needed, and it is difficult to determine how this might affect the overall data harmonization effort. Overall, the limited reach of this effort exposed the researchers to only a subset of the challenges and complexities that may come with implementation on a global scale. Note 1. As a result of the purpose of the pilot to validate the metrics and surveys, field teams received instructions to avoid changes to survey questions unless items needed to be adapted to improve the understanding. However, all questions were mandatory, and they could not be deleted from the survey. Changes to the surveys were documents in the post training reports. 44 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Chapter 7 Recommendations Individual Indicators Core Harmonized Indicators Versus Reference Attributes This research has identified that all indicators are not equally valuable to harmonize, and all are not equally difficult to collect. Based on this, the researchers recommend that the proposed indicators be divided into two classifications: core harmonized indicators and reference attributes. See table 7.1 for a side-by-side look at the classifications. This approach allows the intensive work required to harmonize data across different actors to focus on those indicators that are most useful at different levels. At the same time, by removing the pressure to standardize the reference attributes, it allows countries to mod- ify them to be more relevant in the local context or potentially exclude them if they are not useful. Both types of indicators can be used in parallel, with some countries choosing to build on the core harmo- nized indicators by using adapted reference attributes (or even their own indicators). This approach—to identify a core of harmonized indicators that all data collection efforts can easily incorporate—balances two key priorities. First, countries need to collect locally relevant data. This can vary dramatically across countries because standards and goals for water access; local technical aspects, such as the presence of arsenic in a country; and even policy objectives may differ. It is critical that any proposal neither limit nor discourage governments from collecting data they need to improve their local water services. However, as the original research showed clearly, despite this diversity, there are certain pieces of information that are broadly collected. This evaluation has shown that within this larger set of com- monly collected data, a subset of data also has proven useful at a global level. This presents an opportu- nity to achieve a second priority: comparability. Identifying a set of core harmonized indicators enables disparate data collected at a local level to be aggregated at a global level. This provides a foundation for unprecedented analytics, evidence-driven policy, research, targeted investments, and trend analysis. Thirteen indicators have shown themselves to be relevant at local and global levels, making them ideal candidates for harmonization. These indica- tors were not harmonized—different actors collected them in slightly different ways—because of a lack of coordination. No substantive reasons existed for the minor differences. Of course, it is critical that these globally harmonized core indicators not add an undue burden to existing monitoring efforts. This research has found that six of the 13 indicators were not difficult to collect. This ensures that the addi- tional burden of collecting these core indicators would be minimal. For the remaining seven indicators, recommendations have been provided on how data could be collected for these indicators in a way that reduces the monitoring burden. This ensures that in cases where the core harmonized indicators are not yet being collected, the addition of these indicators would be a relatively light effort. By harmonizing these 13 indicators across all global monitoring efforts, significant progress is possible at a global level. In this way, governments are still able to have flexibility across the things they collect, both in terms of what they collect and how most of those indicators are measured and defined. At the same time, Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 45 standardizing core indicators that are applicable in diverse contexts with no modification allows for unprecedented harmonization and comparability. The minor trade-off of standardizing these indicators to achieve comparability is ultimately a positive trade, simplifying monitoring for countries and provid- ing immense value to the international community. By enabling comparability, these data can allow for the identification of global trends and benchmarking across countries, two types of analysis that are difficult without harmonized data across borders. Table 7.1 shown the differences in use and collection methods between core harmonized indicators and reference indicators. Analysis of the indicator feasibility and usefulness via the recommendation matrix (see table 5.4) leads to recommendations of the following indicators for each classification. The matrix weighs the usefulness attribute more heavily. If an indicator was determined to be globally and locally useful, it was recommended as a core harmonized indicator. Usefulness was the only attribute used to separate core harmonized indicators from reference attributes. Feasibility, however, did not factor into the deter- mination of which indicators should be core harmonized indicators. Instead, the feasibility assessment determined which indicators already deemed core harmonized indicators should be implemented as piloted and which needed additional review. The results of the recommended indicators by type is sum- marized in table 7.2. TABLE 7.1. Differences between Core Harmonized and Reference Indicators Core harmonized indicator Reference indicator Should be collected in all cases Can be collected when useful Should be implemented based on standardized guidance Can be tailored to fit local context Should follow standardized guidance provided by the World Bank Can develop custom country guidance Stored at a national level and harmonized in a global repository Captured and stored at a national level Used to inform local and global decisions Used to inform local decisions TABLE 7.2. Recommended Indicators as Core Harmonized Indicators and Reference Indicators Core harmonized indicator Reference indicator Type of source Presence of a legally established service provider Accessibility Staffing Availability Chlorination Quality Nonrevenue water Reliability Tariff structure Affordability Financial management User satisfaction Tariff collection efficiency Handpump functionality Source, catchment, and water resource management Piped system functionality Complaints handling mechanism Maintenance Service authority capacity Coverage Presence of an information system Financial sustainability Service authority support functions 46 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Reference Attributes The originally proposed metrics, when applied as recommended in regard to different levels of metrics, can provide useful information to national and local governments working to improve water services. The researchers who implemented the pilot found no evidence that any proposed metric should not be considered as part of a national or local monitoring system. Based on the rigorous work of the original researchers, the pilot team found that all the proposed metrics could be useful and feasible when tai- lored to the local situation, and the team recommends them as reference attributes. These provide a valuable starting point when countries are looking to ensure that their monitoring and evaluation sys- tems are aligned with best practices. Within the reference attributes, the levels proposed for each metric (that is, minimum, basic, and advanced) provide valuable guidance on identifying the types of metrics that may be most useful in each context. Eleven attributes are recommended to keep only as reference attributes because they were not iden- tified to be both globally and locally useful with regard to the usefulness analysis (including predeter- mined questions and expert interviews). There were no structured data collected on why certain indicators were not prioritized as useful. Given the approach of semistructured interviews, respon- dents shared their priorities rather than feedback on every indicator. That said, insights from the interview helped to shed some light on the lack of prioritization. Regarding the cost recovery indica- tors (for example, nonrevenue water, tariff structure, financial management, tariff collection effi- ciency, and financial sustainability), several respondents said they felt that financial sustainability effectively captured the results of the other indicators. This efficiency perspective was common among respondents. It was interesting to note some consistency in the prioritization of indicators, or lack thereof. Fully one-quarter of the proposed indicators were not prioritized as useful by a single respondent at any level, suggesting monitoring this information is not critical to improving water services. These included: •• Staffing •• Nonrevenue water •• Financial management •• Tariff collection efficiency •• Source, catchment, and water resource management •• Complaints handling mechanism Core Harmonized Indicators When it comes to globally standardized metrics, this research has refined the initial recommendations further to identify core harmonized indicators. With a goal of standardized global adoption, the metrics focus must not be difficult to collect. This subset of core harmonized indicators and metrics provides a global core of data that can be robustly standardized and thus easily shared across borders to provide value at a global level. Within the subset of core harmonized indicators, the different levels proposed (that is, minimum, basic, advanced) become less relevant. If the metrics are to be truly standardized and global, with data Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 47 TABLE 7.3. Indicators Ready for Use as Core Harmonized Indicators Indicator Metric Type of source The main type of source used by the household for drinking water Availability Number or percentage of household responding positively to having water available when needed User satisfaction Overall satisfaction with the service, satisfaction over quantity, satisfaction over quality Handpump functionality Multicategory classification: functional, partially functional, not functional, abandoned Maintenance Whether any type of maintenance has been carried out in the past twelve months Service authority support Binary: whether service authority provided any type of support function to rural water scheme functions operators in the past twelve months coming from the broadest array of countries possible, these metrics must be feasible for countries at all stages of progress. Thus, the core harmonized indicators should not be categorized by these levels and should be used across all contexts. A total of 13 indicators were found to be useful as core harmonized indicators. Of these, six have at least one metric that was not difficult to collect. The indicators and the corresponding metric listed in table 7.3 are recommended for implementation as they were implemented in the pilot. These indicators were found to be more useful and less difficult overall and less difficult within each indicator. No changes are required. The remaining seven indicators have been identified as useful to include in the core harmonized indi- cators, though they were also identified as more difficult, and are recommended for further review. Some general guidance can be applied across many of these. For example, asking respondents to recall a specific number was typically found to be more difficult than selecting a categorical range of values. Further, analysis of specific numerical values can be more challenging than analysis of a categorical response to a question. Accessibility This indicator was identified as difficult because of the question regarding the time it takes to collect water. These answers were found to have a low level of confidence by respondents and take a rela- tively long time to collect by enumerators. This could be simplified by asking a binary question if the water source is not on premises. One possibility: Does it typically take you more than 30 minutes to collect water, round-trip? This may be easier for respondents to answer than estimating a specific amount of time, as indicated by the findings that recall of specific numbers can be difficult for respondents. Quality This indicator was deemed difficult because of the time required to complete the E. coli test. This could be a result of confusion over the methodology. Although it did take longer than other questions to answer fully (approximately 24 hours), the field component of it (sample collection at point of use) did not necessarily take longer. This perspective is bolstered by feedback from enumerators who noted the use of Akvo Caddisfly for water quality testing was easy and efficient. Additionally, limited feedback was collected on this indicator because not all countries tested water for E. coli. 48 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Given the weak evidence that this is too difficult and the importance of water quality in the Sustainable Development Goals (SDGs), the additional effort required to collect this indicator may well be worth- while. In some instances, as was the case in Kenya during the pilot, the government supporting the survey effort has already given a clear demand to collect this type of data, further reducing any potential barrier. Overall, the researchers recommend keeping this indicator as it was piloted, despite it being flagged as more difficult in the assessments. Reliability This indicator was found more difficult based on immediate feedback, duration, and challenges. This may be because of the exacting nature of the question used, which looked at the number of hours that water stopped flowing. Given the challenges faced in asking respondents to recall a specific number, another approach could be to ask whether water is usually available when it is supposed to be. Affordability The affordability indicator had two questions to capture the required parameter. The first, regarding how much households spend on water in the past month, was flagged as difficult only because of the duration. In contrast, the other question, regarding total amount of monthly expenditure, was identi- fied as difficult based on four feasibility scores (a high rate of unable to answer as well as all three enu- merator feasibility scores). Thus, one option could be to focus on the amount that people spend on water because this was a seemingly easier question for respondents to answer, with fewer feasibility scores exceeding the threshold. The overall expenditure data could be benchmarked against national level statics, such as gross domestic product (GDP) per capita or other widely available measures. Piped System Functionality This indicator was identified as difficult based on the immediate feedback question, which was answered by the enumerator in this case. This question could potentially be simplified by focusing on the distri- bution network rather than requiring a visit to all the other major components (that is, reservoir intake). This could also harmonize the type of information received between the handpump functionality and the piped system functionality because the handpump indicator looked only at whether water was available and not the condition of the entire handpump. Coverage Service providers had a low level of confidence and struggled to answer the questions related to cover- age. Enumerators also found these questions relatively more time-consuming, and they faced multiple challenges when trying to get answers. Direct discussion with service providers might help to identify a more feasible metric to capture the coverage indicator. Financial Sustainability Each of the four questions related to financial sustainability were found to be difficult based on at least two different data sources. A simpler approach may be to ask whether the service provider had any financial reserves at the end of the past year. Although slightly different, this may be easier to answer because it doesn’t require specific numeric recall yet it still captures whether the service provider could Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 49 cover costs in the past year. Because this indicator is looking at the financial sustainability of an entire service provider, rather than a specific water point, the specific management model will not change the use or interpretation of this indicator. If it looked instead at whether revenue from a given water point covered the costs for the same water point, interpretation might differ by management model. For example, a single water point for a large utility may be operated at a loss but still be financially viable as a result of cross-subsidy. Because this indicator looks at the level of service provider rather than individ- ual water point, that issue is avoided. Recommended Metadata Based on proposed use cases, several pieces of standardized metadata are required to enable meaning- ful analysis. These should be included explicitly in all future publications of the proposed metrics. Specifically, the following data must be incorporated with all standardized metrics: Date of survey: This allows data users to determine the timeliness of data and thus its relevance to a specific use case. Additionally, this allows multiple data points over time to be organized appropriately, providing critical longitudinal data. Given the diversity of date formats (that is, MM/DD/YYYY versus DD/MM/YYYY) used around the world, using the ISO 8601 format (YYYY-MM-DD) would ensure consis- tency and avoid confusion between similar looking dates (i.e., 7/6/2020 and 6/7/2020). Data source: Some use cases may require official government data, whereas others can make use of any available data. Thus, it is important to clarify the source of the data for this reason and to allow users to follow up with the original data collectors in case any questions arise. Water point location: The specific latitude and longitude of each water point enables the use of analysis through geospatial information systems (GIS). This type of analysis allows water points to be catego- rized by any boundary (that is, political boundary, watershed, and so on). This removes the need to collect the name of the district and allows for locating the water point within a given district even if political boundaries change. Furthermore, it allows for assessing the proximity of water points to other locations, such as populations, roads, health facilities, or physical features (steep hills and so on). Because many different data point exist, World Geodetic System 1984 (WGS 84)1 is recommended to avoid any potential errors in transformation. This is an extremely common datum, and it is the one most often used by mobile data collection tools. Additionally, decimal format with at least four decimals is recommended to ensure sufficient precision (approximately 11 meters at the equator). Country: The name of the country isn’t needed when data are collected locally, but it is valuable to include when the data are aggregated across borders, as is the specific objective of this effort. Having the country name provides a broad opportunity to validate the coordinates, and it also simplifies analysis and filtering of data. Water service infrastructure identification: Where a physical unique identifier is available on a piece of water infrastructure, it should be included. This allows for tracking water points over time and ensures that all data collected refer to the same water piece of infrastructure. Infrastructure photo: Many similar pieces of water infrastructure are located in close proximity. For example, a community may have several handpumps within a small area. Photos are valuable to help enumerators identify infrastructure for future assessment. 50 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context General Implementation Approach Permissions The pilot was, by design, supported on the relationships and ongoing work Akvo was doing in the three countries to ease implementation. Despite this advantage, getting written permission letters from gov- ernment officials was a long and arduous process, particularly in Sierra Leone. Future efforts should start in advance to gather all approvals, permissions, and required insurance. Documentation Although the research team provided definitions that members anticipated to be necessary, digitizing the surveys and training the enumerators revealed concepts that were not understood in the field and not anticipated by the research team. These included legal requirements and operations. These ques- tions are documented in the training reports. Overall, feedback from all countries indicated the need to have clearer documented guidelines for collecting the metrics. Experience from other sectors, such as health, indicate that for people to be able to collect and use valid information, documentation should include, at a minimum, a comprehensive glossary of terms; greater detail on parameters and metrics; guidance on transformation (how to mix or summarize the parameter to arrive at a metric); indicator analysis and interpretation; suggestions for alternative data sources; and validation processes. An explicit rationale for each indicator, documented through indica- tor refence sheets, could help countries determine the best way to contextualize according to that rationale and, therefore, not risk losing alignment with the rationale if survey questions are modified. ­ Differences in languages should be taken into account in the process of generating the documentation. Data Presentation Based on the extremely diverse use cases identified in the usefulness analysis, it would be beneficial to present the data resulting from this effort in its raw form, rather than as an index or other aggregation. This allows different users to aggregate data in the ways that are most useful. If the data are believed to be too complex for users to interpret, the data could be aggregated in a recommended way, as long as the raw data were to also be provided and equally accessible. Links with Other Global Data Efforts The rural metrics initiative was developed to address important gaps in the harmonization of water-­ related data, but it is not the only related effort. Other data collection and harmonization activities, including those mentioned in previous chapters, provide a unique value to the water sector. Moving forward, these metrics should be evaluated to determine indicators already being captured through other large-scale efforts. Table 7.4 provides an overview of the rural water monitoring subsector, categorized by the unit of analysis and the scale of the effort. ­ This assessment shows that the rural metrics fit within a broader landscape, where there are opportu- nities for both leverage and leadership. The most efficient approach in moving forward would be to formally collaborate with other data-related efforts, where appropriate, to ensure interoperability while reducing duplication. Existing data efforts that already have achieved scale and that touch on similar areas may not have a need for the rural metrics to focus on those areas. Instead, perhaps, the Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 51 TABLE 7.4. Rural Water Monitoring Sub-Sector Unit of analysis Rural water monitoring efforts Service authority Rural metrics SIASAR GLAAS Service provider Rural metrics SIASAR WPDx Household Rural metrics SIASAR WPDx MICS/DHS/JMP Water point Rural metrics SIASAR WPDx Scale of initiative 1–10 countries 10–50 countries 50–100 countries 100–193 countries (piloting) (scaling) (worldwide) (at scale) Note: Darker coloring suggests more complete coverage at that level. Light means active, dark means that this is a primary focus. DHS = Demographic and Household Surveys; GLAAS = United Nations-Water Global Analysis and Assessment of Sanitation and Drinking-Water; JMP = Joint Monitoring Programme for Water Supply, Sanitation and Hygiene; MICS = Multiple Indicator Cluster Surveys; SIASAR = Rural Water and Sanitation Information System (Sistema de Información de Agua y Saneamiento Rural); WPDx = Water Point Data Exchange. recommendations of rural metrics could be integrated into similar existing data collection efforts. In other cases, where gaps exist, there may be opportunities for rural metrics to play a larger role in global monitoring. To advance this collaborative approach, a small convening is recommended to bring together the leadership of all major standards that touch on rural water data to explore a distributed approach. Under this framework, each individual data effort would have the global mandate for one or more units of analysis, and the stewards of all efforts would work together toward interoperability. For example, the World Bank team that leads rural metrics and the Sistema de Información de Agua y Saneamiento Rural (Rural Water and Sanitation Information System; SIASAR) could be identified as the global lead for collecting data about service provider capacity. Other groups would be identified as leading data efforts on water point information, governance process information, and so on. This would allow each stakeholder to become an expert in one area and one type of survey. At the same time, ensuring interoperability with the data would allow each organization (and all water sector stakeholders) to evaluate data across all units of analysis. For example, data about a water point col- lected under one data effort could be analyzed with data about the service provider from another data effort run by a different organization. This would enable specialization and focus while also reducing duplication. Linking Surveys Given the scope of the rural metrics across different levels of analysis, it is important to plan from the outset for effective links among the different surveys. This type of interoperability allows for identify- ing which water points are primary sources for which households, as well as which service providers and service authorities are connected. These connections allow for richer analysis in addition to trian- gulation of key pieces of information across the surveys. However, in practice, this connectivity is difficult because of several factors, including different methodologies resulting in different surveys completed first, relying on availability of certain actors, as well as a lack of consistent naming of water points. In this pilot, Akvo Flow’s monitoring feature allowed for an easy way to link the household, service provider, and water point surveys; however, enumerators experienced trouble staying on schedule 52 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context when people from households or service providers were unavailable. This kept them from being able to complete all three surveys at once. These types of situations caused delays that also affected costs because enumerators had to either delay their travel to the next site or return to a previous community, thereby increasing the amount of working time and complicating logistics. Efforts to collect data at scale while linking surveys need to consider additional time and resource requirements to accommodate these kinds of delays. Furthermore, the use of physical unique identifiers that increase the consistency of water point names can improve linking. Another option would be to increase the use of geospatial analysis to link water points and household surveys. In addition to linking between the surveys in one implementation, it is valuable to consider approaches to link future data collection efforts to the existing data collection. Tools such as Akvo Flow and SIASAR provide simple monitoring features that allow for updating data on the same entity over time. Other services, such as the Water Point Data Exchange (WPDx), use algo- rithms to attempt to associate data points across time. Although the evidence from the feasibility study is limited, the initial findings still indicate the wide range of different relationships among the four questionnaires, especially between service providers and water points. Institutional arrangements vary widely, and though the service authority in Kenya reported more than 500 service providers under its jurisdiction, only three were reported in Burkina Faso. Based on this, a significant variation can be expected in the relationship between the number of water point surveys linked to each service provider survey. Emerging Technologies Around the world, innovative technologies can improve monitoring at various levels, including the abil- ity to track missed calls, blockchain immutable ledgers, physical remote sensors, remote sensing using aerial imagery, and unique identifiers. “A blockchain is a database that is shared across a network of computers. Once a record has been added to the chain it is very difficult to change. To ensure all the copies of the database are the same, the network makes constant checks.” (Murray 2018) One area likely to dramatically shape the monitoring landscape is remote sensing of water point use and functionality. Organizations such as SweetSense, Oxford University, charity: water, and WellDone are all experimenting with remote sensors at various scales. These sensors can help provide real-time insights on domains such as functionality and service level. As their use grows, it will be important to consider how this technology impacts the implementation of the rural metrics. An additional innovation is the use of physical unique identifiers (Davis 2012). Similar to national ID numbers or license plates, physical unique identifiers can ensure that anyone collecting data on a spe- cific water point can relate that data to the same water point. Without this, an installing nongovernmen- tal organization (NGO) may refer to the water point by its own internal database ID, a government official carrying out a water point inventory may name the water point for the name of the community, and a university doing research may identify it by the head of a nearby household. Having multiple names for a single water point severely limits the possibility of sharing data about the specific water point. Ensuring that a unique ID is physically available and human readable (i.e. including letters/numbers, and not just a barcode) can ensure that any stakeholder monitoring that water point can communicate the specific one they are referring to in all data sources. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 53 Another technology that can help reduce the cost of monitoring is tracking missed calls. Because missed calls from a cell phone do not connect, the user is not charged anything for making the missed call. However, these missed calls can still be used to signal important information. This approach is widely used outside of the development sphere for everything from social communication (Heavens 2007) to voting for your favorite contestant on a reality show (Anand 2016). This same technology is being used to enable water committees to report whether their water points are functioning. Communities can receive a text from their water service provider asking whether their water point is working. If it is, they can place a missed call to one number. If not, community representatives can make a missed call to a different number. In both cases, the community can share feedback without any cost. In some cases, these missed calls can even be made without any credit. By analyzing the number that placed the call—and which line was called—software systems can determine the func- tionality of a given water point. This approach has already been used in Angola, Cambodia, and Democratic Republic of Congo.2 As satellite imagery resolution improves and unmanned aerial vehicles (UAVs or drones) prolifer- ate, an emerging opportunity is to use aerial imagery to monitor water points. This can be done by analyzing images to look at proxies for functionality, such as soil moisture, overgrowth at the water point, or even the presence of people using the water point. Machine learning can help to automate pattern recognition. Similar approaches are already using satellite imagery to monitor the number of cars in parking lots3 and identify wildlife from space (Steer 2018). Although satellites can cover more territory in a single photo, drones provide other advantages, including potentially lower costs (Thomas et al. 2018), more customization of imagery collection, and higher-resolution images. This can help address some of the challenges posed by conventional satellite imagery, including low image resolution and midday flyover times that might hide shadows needed to identify people and miss common water collection times. These aerial imagery approaches are still in their infancy and require further research. A final area for exploration may be using the blockchain to track monitoring data in an immutable ledger. SweetSense, in partnership with IBM, has done pioneering work on blockchain for water data (SweetSense 2019). Beyond monitoring, potential applications include enabling and tracking the sale of water credits. Institutional Infrastructure As the other harmonized water standards have shown, a standard alone does not lead to a globally har- monized data source. Instead, a rigorous and well-resourced institutional infrastructure is required to bring a standard to life. This includes setting up governance structures that can modify the standard as new evidence emerges and the sector grows, managing outreach and engagement with stakeholders to increase uptake, and developing a website that includes structures for uploading and downloading data. This institutional support has been a critical element of success across SIASAR; the United Nations- Water Global Analysis and Assessment of Sanitation and Drinking-Water (GLAAS); the WPDx; and the International Benchmarking Network for Water and Sanitation Utilities (IBNET), and it must be consid- ered in rolling out the rural metrics as a global standard. 54 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Data Quality Metrics The proposed indicators have not included metrics that track the quality of the data collected. Potential metrics could capture information related to coverage, validation by the relevant government authority, completeness of each survey record, or time since data were collected. These metrics could help deter- mine the usefulness of the metrics themselves for specific use cases. Next Steps The pilot clearly showed growing momentum for harmonized metrics for rural water services. Initial feedback from the three countries indicate interest to continue exploring adoption of the metrics, provided that surveys can be better adapted to local contexts. During the pilot In Burkina Faso, the World Bank held a seminar to share the rural metrics and work on the pilot more widely among local stakeholders. The session was well-received; it heightened desire among local authorities to con- tinue exploring further use of the metrics. The pilot helped advance the landscape for globally harmonized rural water metrics, but these metrics are not yet fully ready for implementation ­ at scale. First, the World Bank should engage experts to develop metrics for the indicators recommended as core harmonized indicators. To save time, the experts should base these metrics directly on ones already deployed with a proven track record of feasibility. The experts should develop them in a sim- ilar fashion to how the initial metrics were developed but with an additional focus on ease of implementation. Once a final set of proposed harmonized indicators have been developed, including the metadata, two primary approaches are possible. Independent Approach The first approach would be for the World Bank to roll these metrics out directly. This would require first developing appropriate infrastructure. An independent approach should include the required guidance, data collection tools, governance processes, and data repository. Once this is done, the World Bank could advocate use of these metrics; advocacy measures could include requiring use of these metrics in all World Bank-funded rural water programs. Coordinated Approach An alternate approach—and the one recommended by the pilot researchers—would be to use momen- tum from the pilot to develop a coordinated approach to global rural water data. Based on the review, each level of analysis of the four questionnaires (that is, household, service authority, service provider, and water point) receives the most comprehensive coverage at the greatest scale by a different effort. Implementing rural metrics could reduce duplication by identi- fying a lead effort for each level of analysis and focusing on ensuring coordination and interoperability. At the water point level, WPDx already has harmonized data on more than 500,000 water points (including small water systems and point sources) around the world, reaching more than 50 countries. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 55 At the household level, the Joint Monitoring Program for Water Supply, Sanitation and Hygiene (JMP) (along with Multiple Indicator Cluster Surveys [MICS] and Demographic and Health Surveys [DHS]) has collected millions of representative surveys around the world. At the service provider level, SIASAR has shown unparalleled expertise in collecting robust data. At the service authority level, GLAAS has a strong track record of collecting rich data on the enabling environment. Although each of these ventures may not include the exact proposed core harmonized indicators, each touches on similar indicators and might be able to add the proposed indicators to its existing data collection frameworks without much difficulty. This would include integration into survey templates, guidance, and databases. As a first step, the World Bank could convene a meeting among these actors to share findings of this research, propose a harmonized approach, and offer to serve as custodian of this integrated effort. In addition to integrating the recommended core harmonized indicators, key players also would need to commit to interoperability. They would have to agree on ways to connect data. This interoperability could be achieved by focusing on geographic location or other key information. For each given water point in WPDx, information about the relevant households, service authority, and service provider could be accessed. Connected databases could then be developed that allowed users on a local, national, regional, or global level to see all of the data collected about rural water services in a harmonized platform. This approach has an advantage because each of the ventures has already begun developing the guidance, data collection mechanisms, and other infrastructure required to bring each type of data collection to scale. Each of these efforts has already taken on and solved many of the challenges a new monitoring effort would be likely to face when growing to scale. Additionally, the broad collec- tive reach of these efforts provides a strong starting point for harmonizing the global data landscape. The immediate next step would be to complete an analysis of the proposed core harmonized indi- cators versus the existing monitoring efforts at that scale to see where the proposed metrics are not yet included. Where they are already included, this research can enhance them, and where they are not, this research provides a basis for inclusion. Building on that, the World Bank could then con- vene a meeting among the key stakeholders—including Global Water Challenge; JMP; the United Nations Children’s Fund (UNICEF); the United States Agency for International Development (USAID); and the World Health Organization (WHO)—to align on the process. Moving forward, the World Bank could eventually offer to serve as a secretary for these efforts, providing support on the integration of metrics and interoperability across the efforts. This approach is modeled in figure 7.1. This distributed approach would accelerate a harmonized global rural water data landscape by build- ing on existing momentum, infrastructure, and even data. The end result would rapidly achieve the initial goals of the rural metrics effort of “a global set of indicators will help focus on achieving sustain- ability more clearly on the sector agenda, identify future investment needs, improve sector manage- ment, enable the comparison of progress across countries and regions, and permit a standard to extract information from different monitoring systems.” (World Bank 2017b). 56 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context FIGURE 7.1. Potential Collaboration for Global Monitoring Frameworks Service authority GLAAS Service provider SIASAR Household JMP/ MICS/ DHS Water point WPDx Note: DHS = Demographic and Health Surveys; GLAAS = United Nations-Water Global Analysis and Assessment of Sanitation and Drinking- Water; JMP = Joint Monitoring Programme for Water Supply, Sanitation and Hygiene; MICS = Multiple Indicator Cluster Surveys; SIASAR = Rural Water and Sanitation Information System (Sistema de Información de Agua y Saneamiento Rural); WPDx = Water Point Data Exchange. Notes 1. World Geodetic System, 1984 revision (WGS 84) available at https://www.nga.mil/ProductsServices/GeodesyandGeophysics/Pages​ /WorldGeodeticSystem.aspx. 2. For more information on SeeSaw, see its website at https://greenseesaw.com/page/. 3. For more information on Orbital Insight, see its website at https://orbitalinsight.com/products/consumer/. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 57 Appendix A Summary Findings Agreeing on a global core set of indicators that can be integrated into existing monitoring efforts or har- nessed as a foundation for new monitoring efforts will improve the efficiency of the sector and still enable collection of locally relevant information. Over the past eight decades, the World Bank has played a critical role in supporting the development of rural water services and institutions, investing over $5.5 billion on projects in the past five years. Despite this massive investment, sustainability of these services has remained a challenge. Globally, approximately 25% of water points fail within the first four years.1 Worldwide, projects supported by the World Bank and development partners have generated signifi- cant amounts of data about rural water services. This information can play a role in understanding and developing solutions to current sustainability challenges. Further, evidence-based analysis can acceler- ate water access by enabling insights at all levels—from global policy levels down to local operations. Currently, each World Bank water project and study develops their own unique monitoring frame- works, with only a few simple Core Sector Indicators in common. As a result, rural water data are diffi- cult to harmonize, and it has not been possible to bring together the data collected across the rural water projects by different agencies to learn about sustainability challenges at scale. Fragmented data limits measuring impact by countries, development partners, and other stakeholders, each developing and implementing their own monitoring frameworks. The WASH Poverty Diagnostics2 implemented in 18 countries also revealed the close correlation between extreme poverty and lack of rural water services. These findings highlighted that reducing extreme poverty will require larger and more sustainable investments in rural areas. SIASAR,3 the Rural Water Monitoring System in Latin America, has been suc- cessful in showing how harmonized data can help improve rural services in middle-income countries. The need to enhance the Development Agencies’ approach to rural water data is a growing strategic imperative. The return on this investment will be significant. Research can be accelerated when researchers have easy access to clean data, allowing for unprecedented learning throughout the water sector. Project review can also be accelerated when practitioners are able to access key information on the context of proposed pro- grams using standardized information. Finally, project implementation can be more efficient, as detailed data can be used in planning processes to prioritize investments where they can have the greatest impact. Although the data required to accelerate rural water access and harness the data revolution are being collected, and increasingly so, it remains too fragmented to use at scale. Slight differences in formats and limited global data sharing architecture mean that data are difficult to share, harmonize, and use. Accelerating progress toward universal water access and achieving the SDG will require a more sophis- ticated approach to rural data harmonization. Challenges In the urban context, the World Bank has demonstrated clear leadership on improving the global evidence-base through the work of IB-Net.4 However, as is often the case, the rural sector has been ­ Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 59 left behind. Investing in data harmonization is particularly urgent as the unique context of rural water services exacerbates data challenges. While urban utilities may serve millions of people under one ser- vice provider and one set of data, rural service provision tends to be much more decentralized. Rather than harmonizing data from a limited number of urban service providers in a given country, under- standing rural services may require harmonizing data from hundreds or even thousands of rural water services. This fragmentation of data leads to three specific challenges: Planning, Implementation, and Monitoring: The lack of harmonized data increases the amount of resources required (both time and money) to plan, implement, and monitor projects. Because finding and accessing relevant data is difficult, new data are constantly collected for project design and moni- toring, even though data may already exist. In addition to the data collection itself, resources are repeat- edly spent developing new monitoring frameworks from scratch. Research and Innovation: Beyond implementation, the lack of a central repository severely hampers efforts to research, learn, and innovate. The lack of standardized data means that researchers are forced to either use smaller, less representative data sets, or carry out new data collection activities—both of which slow the progress of innovation and research in the sector. Standardized analysis of data is also impossible, requiring instead that analytical tools must be constantly developed to match different monitoring frameworks. This limits the use of data. Understanding Trends: The fact that rural water data are not currently harmonized across country bor- ders poses a further set of unique challenges. To start with, the lack of detailed global data makes it difficult to understand global trends that transcend specific countries. Related to this challenge, the lack of robust multi national data eliminates the possibility of benchmarking to identify “bright spots” and areas where tailored support is required. The inefficiencies noted above ultimately increase the costs of reaching universal access. Scarce resources are often spent addressing the inefficiencies of fragmented data, rather than improving water services. In a resource-scarce environment, a lack of global metrics increases costs and reduces the use of costly data that have already been collected. The lack of global metrics is holding back prog- ress on the SDGs. Opportunity: Global Standard, Local Flexibility The World Bank is proposing the global water sector implement global metrics that effectively balance national objectives and the global imperative to improve rural water data. This approach recognizes that every country has unique needs and capabilities related to rural water information. Among this diver- sity of needs, however, some information requirements are common across nearly all countries. In these cases, where many countries are already collecting similar data, the differences in data collection frame- works are often the result of a lack of coordination rather than a meaningful need for diverse approaches. While every country and stakeholder should collect data, they need to match the local context; there is a clear opportunity to make minor enhancements to monitoring frameworks to ensure that key data can be shared and harmonized globally. Creating a global core set of indicators that can be integrated into existing monitoring efforts or harnessed as a foundation for new monitoring efforts will improve the efficiency of the sector and still enable collection of locally-relevant information. As illustrated below, 60 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context countries with existing monitoring frameworks can simply integrate the Global Core Metrics for Rural Water Supply. In other cases, where no monitoring framework exists yet, these metrics can serve as a starting point for monitoring. Core Country 1 Country 2 Country 3 Country 4 Metrics To make integration of these indicators as easy as possible, the global indicators should be based on existing monitoring frameworks that are already in use. Following a World Bank review of 40 different existing rural water indicator frameworks from national and development partner project systems, a set of 24 existing indicators was identified as a potential set of Global Core Metrics for Rural Water Supply.5 A subsequent review that piloted the proposed metrics across three countries in Africa with some of the greatest data collection challenges (Burkina Faso, Kenya, and Sierra Leone) evaluated the proposed indicators in terms of feasibility and usefulness and further refined the list to 13 core indicators and 11 reference attributes. The proposed indicators reflect what is already being collected in many cases—either through com- mon elements of national monitoring frameworks or existing global standards. Where the information is not already being collected, evaluations have shown that these indicators and metadata can easily be integrated to ensure alignment and still enable stakeholders to monitor the information that is locally relevant. Implementation Approach A set of 13 indicators form a core set of metrics across four levels of analysis: water point, household, service provider, and service authority. Additional metadata at each level enable the identification of this data in time and space. Moving forward, any Rural Water project that collects data at any of these four levels will be encouraged to integrate the relevant core indicators using the standardized approach. Where existing indicators among the proposed core indicators already exist in widely adopted monitor- ing frameworks, such as the JMP tools, MICS, DHS, GLAAS, SIASAR, or WPDx, the indicators will be used in the same format. This will reduce duplication and enable interoperability. This “collect-as-you-go” approach will ensure that future data collection, regardless of institution or government leading the col- lection, can be collected in a harmonized format and can contribute to global rural water learning.6 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 61 Next Steps As a first step, the Global Core Metrics for Rural Water Supply will provide a full set of guidelines to enable Rural Water Practitioners and clients to begin collecting data in a standardized way in its projects. Moving forward, the Global Core Metrics for Rural Water Supply could provide a platform for sharing data from different sources across diverse stakeholders. This platform would be a flexible interchange, allowing data to come from different sources and able to provide data directly to other platforms. Information from different sources could be harmonized into a single data repository, using the Global Core Metrics for Rural Water Supply as a framework. These metrics will first be piloted throughout the World Bank’s own portfolio in partnership with external stakeholders, including governments and development partners. As the global dataset grows, the Global Core Metrics for Rural Water Supply initiative could provide tailored support to enable the use of the harmonized data in planning, implementation, and monitoring efforts. Annex A: Summary of All Indicators List of All Global Core Metrics for Rural Water Supply, by Survey: Household HH1. Percent of households using an improved drinking water source HH2. Percent of households that have water accessible within 30 minutes (total collection time) HH3. Percent of households reporting sufficient water available when needed HH4. Percent of household income dedicated towards water HH5. Percent of households with reliable water service HH6. Percent of households satisfied with overall water service supply Water Point WP1. Percent of water points at risk of E. Coli infection WP2. Percent of functional hand pumps in geographic area WP3. Percent of functional taps/ points of collection in geographic area Service Provider SP1. Percent of service providers that have carried out preventive or corrective maintenance in the last 12 months SP2. Percent of all households in service areas using water services SP3. Percent of service providers reporting availability of funds at the time of monitoring Service Authority SA1. Percent of service authorities providing support to rural water scheme operations in the last 12 months List of All Reference Attributes, by Survey: Service Provider RHH1. Presence of a legally established service provider RSP1. Staffing RSP2. Chlorination RSP3. Non-revenue water RSP4. Tariff structure RSP5. Financial management RSP6. Tariff collection efficiency RSP7. Source, catchment and water resource management RSP8. Complaints handling mechanism Service Authority RSA1. Service authority capacity RSA2. Presence of an information system Notes 1. https://www.rural-water-supply.net/en/resources/details/787. 2. http://www.worldbank.org/en/topic/water/publication/wash-poverty-diagnostic. 3. https://www.siasar.org/ 4. https://www.ib-net.org/about-us/ 5. World Bank. 2017. “Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework.” World Bank, Washington, DC. 6. In line with the Principles for Digital Development, the Addis Accord, and the World Bank’s work on the Data Revolution, all data collected in compliance with the Global Core Metrics for Rural Water Supply will be shared openly (based on open data best practices and World Bank guidance [https://spappscsec.worldbank.org/sites/ppf3/Pages​ /previewpage.aspx?DocID=18ec8892-2cd1-458f-8797-50a83313dcac]). 62 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Appendix B Originally Proposed Rural Water Metrics Global Framework All information comes from “Rural Water Metrics for Sustainability: A Global Framework,” written in 2017 by Aguaconsult. It has been slightly adapted for formatting purposes. No content has been modified. TABLE B.1. Initially Proposed Indicators # Indicator and Level Metric SERVICE LEVEL 1 Indicator: Type of source Minimum metric The main type of source used by the household for drinking water Basic metric The main type of source used by the household for drinking water Advanced metric The main type of source used by the household for drinking water 2 Indicator: Accessibility Minimum metric Basic metric Travel time of a roundtrip to fetch water in minutes Advanced metric 3 Indicator: Availability Minimum metric Proportion of time that a service is provided to households, taking into account planned interruption (continuity) Basic metric Number or % of households responding positively to having water available when needed Advanced metric 4 Indicator: Quality Minimum metric Basic metric Frequency and percentage of water quality test falling within national standards for water quality - further sub-divided into bacteriological (e-coli) and specific physiochemical parameters (arsenic and fluoride) Advanced metric 5 Indicator: Reliability Minimum metric Basic metric Advanced metric Products of the frequency and average duration of unplanned interruptions in the supply during the last year 6 Indicator: Affordability Minimum metric Basic metric Advanced metric The amount spent on water in relation to a household’s total expenditure on consumption table continues next page Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 63 TABLE B.1. continued # Indicator and Level Metric 7 Indicator: User satisfaction Minimum metric Basic metric Advanced metric Overall satisfaction with the service, satisfaction over quantity, satisfaction over quality FUNCTIONALITY 8 Indicator: Handpump functionality Minimum metric Multi-category classification: functional, partially functional, not functional, abandoned Basic metric Multi-category classification: functional, partially functional, not functional, abandoned Advanced metric Multi-category classification: functional, partially functional, not functional, abandoned 9 Indicator: Piped system functionality Minimum metric Water infrastructure condition index based on functioning and physical condition of main components of the water system (intake, reservoir, etc.) Basic metric Advanced metric Number of breakdowns/leakages per kilometer of pipe SUSTAINABILITY: SERVICE PROVIDERS 10 Indicator: Presence of a legally established service provider Minimum metric Whether a service provider is in place or not Basic metric Compliance with legal requirements to be established as service provider Advanced metric 11 Indicator: Staffing Minimum metric Presence of at least one skilled staff needed to carry out the tasks associated with their position Basic metric Multi-category: whether the service provider has organizational charts, job descriptions for all positions including regular staff, volunteers and board members and if these are filled Advanced metric Staff ratio expressed as number of FTE per unit (no. of connections or cubic meters sold) that indicates the size of the service provider 12 Indicator: Maintenance Minimum metric Whether any type of maintenance has been carried out in the last 12 months Basic metric Percentage of breakdowns over last 12 months repaired within the established (national) norm for response time Advanced metric Ordinal score for asset management planning 13 Indicator: Chlorination Minimum metric Whether the service provider is carrying out chlorination or not (for piped schemes only) Basic metric Advanced metric Residual chlorine concentration in mg/l 14 Indicator: Coverage Minimum metric Percentage of the population served by a service provider in its service area Basic metric Advanced metric table continues next page 64 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE B.1. continued # Indicator and Level Metric 15 Indicator: Non-revenue water Minimum metric Basic metric Advanced metric Difference between the volume of water produced and the volume of water which was sold and paid for 16 Indicator: Tariff structure Minimum metric Type of tariff structure including the absence of any tariff levying Basic metric Advanced metric Whether the tariff is based on an adequate tariff calculation 17 Indicator: Financial management Minimum metric Binary: whether the service provider has general ledger and/or cash-book Basic metric Whether the service provider keeps updated, monthly/annual financial reports Advanced metric Presence of financial reports including all required elements for informed decision making (billing receipts, operating expenditure, volume of water produced, volume of water sold) 18 Indicator: Tariff collection efficiency Minimum metric % of users with outstanding debts over a period of time Basic metric Ratio between the income from water bills and the total amount that was billed over a period of time Advanced metric 19 Indicator: Financial sustainability Minimum metric Operating cost coverage ratio: ratio between operational income and expenditure during the last financial year Basic metric Liquidity ratio: ratio between current assets and current liabilities of the service provider Advanced metric Solvency ratio: ratio between all current and non-current assets and all current and non- current liabilities 20 Indicator: Source, catchment and water resource management Minimum metric Whether service provider has undertaken any type of source, catchment or water resources management activity in the last 12 months Basic metric Binary: Whether the service provider has a source water protection plan or wellhead protection plan in place Advanced metric Ordinal scale on the number and types of source, catchment of water resources management plans and activities undertaken by the service provider 21 Indicator: Complaints handling mechanism Minimum metric Whether the service provider holds regularly scheduled, publicly announced meetings or other mechanism to provide feedback to users in a given time period Basic metric Advanced metric % of complaints or requests that is handled within established time period table continues next page Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 65 TABLE B.1. continued # Indicator and Level Metric SUSTAINABILITY: SERVICE AUTHORITIES/TA PROVIDER 22 Indicator: Service authority capacity Minimum metric Binary: presence of a service authority, as per the legislative and administrative requirements of the country Basic metric % of sanctioned positions for rural water in the Service Authority structure that is filled Advanced metric % of allocated funding available for functioning in the service authority/ technical assistance role in relation to what was calculated as being required over a 12-month planning period (or other) 23 Indicator: Service Authority support functions Minimum metric Binary: whether service authority provided any type of support function to rural water scheme operators in the last 12 months Basic metric Binary: whether service authority has in place and applies a pro-active schedule of support visits to rural service provider operators in the last 12 months Advanced metric % of communities/systems/ providers met out of the universe of communities/systems/ providers in the service area [during the last 12 months] 24 Indicator: Presence of an information system Minimum metric Binary: whether an information system is in place at the level of the Service Authority (or any designated third party) Basic metric Binary: whether the information system has been updated in the last 12 months Advanced metric Multi-category or ordinal scale: information system contains updated data on service levels, functionality and service provider performance Note: FTE = full time equivalent; TA = technical assistance. 66 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Appendix C Local Stakeholders Engaged and Capacity Assessment Table C.1 summarizes the service authorities engaged in the three-country evaluation and provides key data on each. TABLE C .1. Service Authority Support Provided in Presence of MIS Country Service Authority Name Past Yeara (Updated at Least Annually) Burkina Faso Service Communal de L’eau et de L’Assainissement 2, 3, 4, 7 Unknown Kenya Laikipia County Government 1, 2, 3, 4, 5, 6 No Kenya Nanyuki Water and Sewerage Company 2 Yes Sierra Leone Ministry of Water Resources-Water Directorate 1, 2, 3, 4, 5 Yes Note: MIS = management information system. a. Description of Support Services 1. Performance monitoring 2. Technical advice and information 3. Administrative support (e.g., help with tariff setting) 4. Organizational support (e.g., to achieve legal status) 5. Conflict resolution 6. Identifying capital maintenance needs (including advice on financing) 7. Training and refresher courses Service Providers The three-country evaluation engaged 104 service providers. Note that these summary findings are not representative of the countries selected, and this information is designed only to summarize data collected. Total surveys by country: •• Burkina Faso (48) •• Kenya (44) •• Sierra Leone (12) What type of service provider is this? •• Community management (48) •• Direct government operation (1) •• Institutional management (that is, health care facility, school and so on) (1) •• Private operator or delegated management (4) •• Other (1) Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 67 Has the service provider undertaken any type of water resources management activities aimed at ensuring sustainability of the water resources used by this water source in the past 12 months? •• Yes (74) •• No (28) •• Don’t know (1) Does the service provider comply with legal requirements? •• Don’t know what the legal requirements are (19) •• Know the legal requirements but don’t know if the service provider complies with all (6) •• No legal requirements exist (13) •• No, the service provider does not meet any of the legal requirements (4) •• The water service provider complies with some of the legal requirements (43) •• Yes, the service provider complies with all legal requirements (18) Does the service provider have available a ledger that includes both revenues and expenses? •• Yes (25) •• No (50) •• Don’t know (28) Is there someone responsible for operations and/or maintenance? •• Yes (81) •• No (23) On average, how many hours a week is water available? •• 0–40 hours per week (15) •• 41–80 hours per week (7) •• 81–120 hours per week (36) •• 120–168 hours per week (40) 68 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context Appendix D Monitoring Systems and Documentation Reviewed as Part of the Study of 40 Frameworks All information in this appendix comes from appendix 3 of “Rural Water Metrics for Sustainability: A Global Framework,” written in 2017 by Aguaconsult. It has been slightly adapted for formatting pur- poses. No content has been modified. TABLE D.1. National Monitoring Systems Documentation Reviewed Region Country Monitoring System Documentation reviewed or sources of data 1. Africa A. Ethiopia OneWASH National Monitoring & National WASH Coordination Office (2017). DraftOneWASH Evaluation system National Program. Key Performance and Supplementary Indicators With Corresponding Query Formats. B. Ghana A new Sector Information System CWSA (2014). Framework for Assessing and Monitoring (SIS), incorporating the District Rural and Small Town Water Supply Services in Ghana Monitoring and Evaluation System C. Liberia Waterpoint Atlas http://wash-liberia.org/data-maps/ D. Mozambique SINAS Sistema de Informação MACARIO Luis, B.-N. E. (2015). Towards a Comprehensive Nacional de Agua e Saneamento Water Sector Information System in Mozambique. Ministry of PRONASAR – National Rural of Public Works, Housing and Water Resources. National Water Supply and Sanitation Water Directorate (DNA), Mozambique. The World Program Bank. (2016). Implementation Completion and Results Report, Water Services and Institutional Support Project, Mozambique. The World Bank, Mozambique. E. Sierra Leone Water Point Mapping http://washdata-sl.org/ F. Tanzania Water Point Mapping System Rural Water Supply Division (RWSD), Ministry of Water. (WPMS) (2015). USER MANUAL - Processes & procedures for updating rural water point data in Tanzania. Ministry of Water, The United Republic of Tanzania. G. Uganda Golden indicators established MWE (2016). Water and Environment Sector Performance by the Ministry of Water and Report MWE. Data Collection for Point Water Sources Environment (MWE) 2. Asia A. India Swajal Sector Information System http://swajalsis.uk.gov.in/ (Uttarakand) (Swajal SIS) B. Nepal National Mapping Information Ministry of Urban Development. Department of Water Project (NMIP) Supply and Sewerage (2014). Nationwide Coverage and Functionality Status of Water Supply and Sanitation in Nepal C. Philippines Listahang Tubig National Water Resources Board, Philippines. (2015). Listahang Tubig - A national Water Survey. National Water Resources Board, Philippines. D. Timor Leste Water and Sanitation Information WaterAid, ITAD and IRC (2016). How can ICT initiatives be System (SIBS) designed to improve rural water supply? table continues on next page Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 69 TABLE D.1. continued Region Country Monitoring System Documentation reviewed or sources of data 3. Europe A. France Observatoire des services d’eau et Eaufrance. (2014). Rapport national des donnees SISPEA. d’assainissement Observatoire des services publics d’eau et d’assainissement, France B. Portugal Relatório Anual dos Serviços de ALEGRE Helena, M. R. (2013). Guia de avaliação da Águas e Resíduos em Portugal qualidade dos serviços de aguas e resíduos prestados (RASARP) aos utilizadores - 2,. A geração do sistema de avaliação. Laboratorio Nacional De Engenhara Civil, Portugal. 4. Latin A. Brazil SNIS – Sistema Nacional de Ministero das Cidades, Servicos de Agua e Esgotos. (2015). America and Informações sobre Saneamento Sistema Nacional de Informacões sobre Saneamento – SNIS. Caribbean Brazil. B. Colombia SUI Rural – Sistema Único de República de Colombia. (2014). Informe Sectorial Pequeños Información de Servicios Públicos Prestadores. Superintendencia de Servicios Públicos Rural Domicilarios, Republica de Colombia. C. Haiti SIP – Système d’Indicateur des DINEPA (2016). Rapport SIP Juillet 2016 Performances D. Multiple SIASAR – Sistema de Información Engineering Sciences and Global Development. (2017). countries de Agua y Saneamiento Rural SIASAR and its contribution in monitoring the Sustainable Development Goal on water and sanitation. ESc&GD 5. North A. USA Rural Community Assistance RCAP (2014). Water System Technical, Managerial and America Program (RCAP) Financial Capacity Assessment TABLE D.2. World Bank Projects Reviewed and Related Documentation Country Project Name Related Documentation Ghana Sustainable Rural Water & The World Bank. (2010). Project Appraisal Document (54672-GH), Sustainable Sanitation Services Water and Sanitation Project, Ghana. The World Bank, Ghana. Haiti Sustainable Rural Water DINEPA. (2014). Programme Eau Potable Et Assainissement en Milieu Rural Supply and Sanitation Project (EPAR). DINEPA, Haiti The World Bank. (2015). Project Appraisal Document (PAD1060) Sustainable rural and small towns water and sanitation project, Haiti. The World Bank, Haiti. India (Kerala) Kerala Rural Water Supply The World Bank. (2011). Project Appraisal Document (64658-IN), Second Kerala and Sanitation Project II rural water supply and sanitation project, India. The World Bank, India. Andres, L., S. Deb, M. Gambrill, E. Giannone, G. Joseph, P. Kannath, M. Kumar, P.K. Kurian, R. Many, and A. Muwonge (2016). Sustainability of Demand Responsive Approaches to Rural Water Supply: The Case of Kerala. Research Working Papers Series No 8025. The World Bank. India Maharashtra Rural Water The World Bank. (2014). Project Appraisal Document (76172-IN), Maharashtra (Maharashtra) Supply and Sanitation Project rural water supply and sanitation program, India. The World Bank, India. India (Punjab) Punjab Rural Water Sector The World Bank. (2015). Project Appraisal Document (PAD1174), Punjab rural Improvement Project water and sanitation sector improvement project. The World Bank, India. table continues on next page 70 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE D.2. continued Country Project Name Related Documentation Indonesia Water Supply and Sanitation International Bank for Reconstruction and Development. (2016). DPSP in Rural for Low-Income Communities Water Service in Indonesia - Synthesis Report. World Bank, Indonesia. The World III (PAMSIMAS) Bank. (2015). Project Appraisal (PAD1532), Second additional financing for the third water supply and sanitation for low income communities/ community based water supply project (PAMSIMAS). The World Bank, Indonesia. Nepal Rural Water Supply & The World Bank. (2014). Project Appraisal Document (81863-NP), Rural Water Sanitation Improvement supply and sanitation improvement project, Nepal. The World Bank, Nepal. Vietnam Results-based Scaling Up Socialist Republic of Vietnam. (2017). Program Operational Manual, Results Rural Sanitation and Water based scaling up rural sanitation and water supply program (2016-2021). Socialist Supply Program Republic of Vietnam. The World Bank. (2015). Program Appraisal Document (100485-VN), Results based scaling up rural sanitation and water supply program, Vietnam. The World Bank, Vietnam. Vietnam Rural Water Supply & Socialist Republic of Vietnam. (2016). Operational Manual – Results based rural Sanitation (Program for water supply and sanitation under national target program 2013-2018. Socialist Results) Republic of Vietnam. Ethiopia, WASH Poverty diagnostics Oxford Policy Management. (2016). WASH Poverty diagnostics Nigeria - Indonesia and studies in each of the three Assessment of informal private water service providers in Bauchi City, Nigeria. Mozambique countries World Bank, Nigeria. The World Bank. (2015). Indonesia WASH poverty diagnostic: Improving service levels and impact on the poor. The World Bank. India Sustainability Evaluation Ernst & Young. (2015). Sustainability Evaluation Exercise (SEE) of Uttarakhand (Uttarakhand) Exercise of the Uttarakhand Rural Water Supply & Sanitation Project (URWSSP). Dept of drinking water and Rural Water Supply and sanitation, Govt of Uttarakhand The Energy and Resources Institute (TERI). Sanitation Project (2015). Impact Analysis of Uttarakhand Rural Water Supply & Sanitation Project (URWSSP). State Water and Sanitation Mission, Government of Uttarakhand, India. The World Bank. (2016). Implementation Completion and Results Report (ICR00003689), Uttarakhand rural water supply and sanitation project, India. The World Bank, India. TABLE D.3. Indicator Frameworks Used by Other Development Partners and Related Documentation Organization Name of Framework Scope Documentation Reviewed Dutch WASH Alliance Sustainability Monitoring Framework 6 countries Joint Monitoring Indicator framework for Sustainability Global WHO/UNICEF (2017). Safely Managed Drinking Program Development Goal target 6.1 Water UNICEF Sustainability check Global UNICEF Sustainability Indicators and Factors USAID Sustainability Index Tool (SIT) 9 countries USAID Sustainability Index Tool (SIT) Water For People Service level and sustainability checklist 8 countries Water for People (2017). Monitoring Framework WaterAid UK Post Implementation Monitoring Surveys 14 countries WaterAid. Water Sustainability Indicators (PIMS) Note: UNICEF = United Nations Children’s Fund; USAID = United States Agency for International Development; WASH = water supply, sanitation, and hygiene. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 71 Appendix E Dimensions, Categories, Indicators, and Metrics Identified as Part of the Study of 40 Frameworks All information in this appendix comes from Annex 2 of “Rural Water Metrics for Sustainability: A Global Framework,” written in 2017 by Aguaconsult. It has been slightly adapted for formatting pur- poses. No content has been modified. Note that this summary reflects the wide range of indicators captured in the comprehensive review, and not all indicators were put forth for the initially proposed rural metrics. Of these, 24 indicators were identified for inclusion in the initially proposed rural metrics. The determination of the initial 24 indicators is described in more depth in chapter 3 of ­ the report. TABLE E .1. Dimensions, Categories, Indicators, and Metrics of Existing Frameworks Group of Sub-Group of Domain Indicators Indicators Indicator Metric Functionality Functionality [at Binary: yes/no level of individual Multi categories functioning, partial, non-functioning, as a handpump] result of the yield Functionality Ratio between number of functioning water points and [at higher total number of water points surveyed geographical Fraction of abandoned handpumps and non-abandoned levels of scale] handpumps multiplied by total number of days without service per year Service Levels Type of source Category as per JMP classification Coverage or % of households in a geographic area using an improved access type of source Population served by a service provider / population in the service area Accessibility Travel time of a roundtrip to fetch water in minutes One-way distance between homestead and water point Crowding, defined as the number of users sharing a water point Availability % of households responding positively to having water available when needed Number of hours of service per day % of households expressing being satisfied with the quantity received Continuity Proportion of time that a service is provided, taking into account planned interruption table continues on next page Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 73 TABLE E .1. continued Group of Sub-Group of Domain Indicators Indicators Indicator Metric Reliability Frequency of unplanned interruptions – expressed in different units of time, kilometers of pipeline or number of users Duration of unplanned interruptions Product of frequency of unplanned interruptions and average duration of these interruptions Proportion of time that a service is provided, taking into account unplanned interruption Seasonality Binary: water available throughout year or not Binary: whether a water point has dried up for at least a month during the last year Flow in the water source in relation to reference demand Proportion of time a service is not provided, due to insufficient water resources availability Quantity Net volume of water consumed per person or per household Percentage of households expressing being satisfied with the quantity received Quality Binary: compliance with frequency of water quality testing % of water quality test or the last water quality test that falls within national standards – further sub-divided for different bacteriological and physiochemical parameters % of tests for residual chlorine that fall within standards Acceptability % of users that rates water quality as acceptable % of users that is satisfied with water quality Affordability Ratio between total expenditure on water supply and the total expenditure or income of the household Binary: presence and/or application of subsidy mechanisms Pressure Sustainability Water Water catchment Ordinal scale to describe catchment condition factors system status Ordinal scale on relation between the discharge at moment performance of monitoring and at the moment of commissioning Water safety plan Intake Ordinal scale to describe its physical condition and infrastructure functionality status Treatment and Presence of treatment and disinfection infrastructure disinfection Physical condition including functionality of treatment and infrastructure disinfection infrastructure Percentage of treatment capacity that is being adequately used in relation to its original dimension table continues on next page 74 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE E .1. continued Group of Sub-Group of Domain Indicators Indicators Indicator Metric Conveyance Ordinal scale to describe physical condition of conveyance and distribution and distribution infrastructure infrastructure Percentage of conveyance and distribution pipes over a certain age that have been rehabilitated Number of breakdowns per kilometer of conveyance pipes Storage Ordinal scale to describe physical condition of storage infrastructure infrastructure and frequency of cleaning Number of days of system autonomy, i.e. total volume of storage capacity divided by the daily volume needed to provide all users with a reference supply Metering % of users with (functioning) water meters Binary: presence of macro metering at production site Users User satisfaction % of water users that is satisfied with the overall water services, and with specific parameters of water quantity and quality Number of complaints per 1000 users User participation % of users attending meetings of the service providers % of users contributing [in kind] to the upkeep of water points Service Governance Presence of Binary: is there a service provider or not providers of the service service provider provider Legal status Binary or multi-category: compliance with legal of the service requirements to be established as service provider provider Binary: whether the Board of the service provider has been democratically elected Capacity of the Binary: whether the water committee has received any [board of] the training service provider Frequency of Number of meetings during the last time period meetings Gender balance in Number or percentage of women in the Board of the the Board of the service provider service provider Performance in Staffing Staff ratio or labor productivity expressed as number of operation and FTE per unit that indicates the size of the service provider maintenance Binary: presence of a skilled technical employee Time it takes to source an external repair service [by an area mechanic or similar] Access to tools, Binary: whether the service provider is equipped with tools spare parts and for maintenance or inputs (like chlorine) inputs Time it takes to source spare parts table continues on next page Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 75 TABLE E .1. continued Group of Sub-Group of Domain Indicators Indicators Indicator Metric Carrying out Binary: whether preventive and/or corrective maintenance of preventive is being carried out and corrective Binary or ordinal score for asset management planning maintenance Frequency of carrying out preventive maintenance Time it takes for corrective maintenance to be carried out Chlorination Binary: whether the service provider is carrying out chlorination or not Residual chlorine concentration in mg/l Physical water Difference between total volume of water produced and losses total volume metered through household meters Energy use Pumping hours efficiency Energy use per unit of water produced in kWh/m3 Financial Tariffs structure Binary: whether a tariff is levied or not management Multi-category: type of tariff structure including not levying a tariff Binary: whether the tariff is based on an adequate tariff calculation Height of the tariff in amount of money per unit of water Bookkeeping Binary: whether the service provider has a bank account practices and up-to date general ledger and cash-book Financial Binary: whether the service provider shares data on its accountability financial performance Subsidy Binary: whether a subsidy or solidarity mechanism is in mechanisms place Amount of money put into the subsidy fund per m3 billed Profitability Ratio between operational income and expenditure during the last financial year Solvency ratio Ratio between all current and non-current assets and all current and non-current liabilities Debt service ratio Short and long term debt payments / average monthly income – average monthly expenditure) Savings The amount of money saved by the service provider Liquidity Ratio between current assets and current liabilities of the service provider Collection The ratio between the amount the income from water bills efficiency and the total amount that was billed Percentage of users with outstanding debts Non-revenue Difference between water supplied and water sold water table continues on next page 76 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context TABLE E .1. continued Group of Sub-Group of Domain Indicators Indicators Indicator Metric Environmental Undertaking Binary: whether the service providers undertakes any type and water of source, of source, catchment of water resources management resources catchment and activities management water resources Multi-category based on ordinal scale on the number and management types of source, catchment of water resources management measures by the activities undertaken by the service provider service providers Percentage of water resources captured that meets license standards Percentage of treatment waste that is disposed of adequately Customer care Presence of Binary: whether a customer care mechanism exists complaints handling mechanism Attendance to Percentage of complaints or requests that is handled within complaints established time periods Service Governance Presence of a Presence of a service authority, as per the requirements of authorities of the service service authority the country authority Resources of Staffing Percentage of sanctioned positions that is filled the service Ratio between number of staff and number of authority communities Financial Binary: whether the service authority has calculated the resources for budget required for its functioning in the service authority/ the functioning technical assistance roles of the service Binary: whether the budget allocated is sufficient in authority / relation to what was calculated technical assistance provider Financial Binary: whether the service authority has calculated the resources budget required for major repairs for capital Binary: whether the budget for major repairs is sufficient maintenance Logistical Multi-category: presence and condition of IT equipment resources Multi-category: presence and quality of internet connectivity Multi-category: presence and condition of transport equipment Multi-category: presence and condition of water quality testing equipment table continues on next page Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 77 TABLE E .1. continued Group of Sub-Group of Domain Indicators Indicators Indicator Metric Performance Presence of Binary: whether an information system is in place in fulfilling an information roles system Effectiveness Binary: whether the authority provides a certain support function Number of support functions provided Service Service authority Binary: whether the benchmark for frequency of support authority coverage visits is met coverage Percentage of communities/systems/providers met out of the universe of communities/systems/providers in the service area Number of supports provided to more than 50% of communities / Total number of type of support carried out Availability of Binary: whether spare parts are available spare parts and services Note: FTE = full time equivalent; JMP = Joint Monitoring Programme for Water Supply, Sanitation and Hygiene; kWh/m3 = kilowatt hour per cubic meter; mg/l = milligrams per liter. 78 Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context References Anand, Shefali. 2016. “5 Things You Can Get in India with a Missed Call.” The Wall Street Journal, May 13. https://blogs.wsj.com​ /briefly/2016/05/13/5-things-five-things-you-can-get-in-india-by-using-a-missed-call/. Banks, Brian, and Sean Furey. 2016. “What’s Working, Where, and for How Long: A 2016 Water Point Update.” Poster presented at the 7th RWSN (Rural Water Supply Network) Forum, Abidjan, Côte d’Ivoire, November 29. https://www.researchgate.net/publication​/312027512​ _What’s_Working_Where_and_for_How_Long_A_2016_Water_Point_Update. Davis, Susan. 2012. “Why Physical Unique Identifiers on Water Points Will Improve Sustainable Services.” WASHfunders blog. https:// washfunders.org/why-physical-unique-identifiers-on-water-points-will-improve-sustainable-services/. Heavens, Andrew. 2007. “Phone Credit Low? Africans Go for ‘Beeping.’” Reuters, Sept. 25. https://www.reuters.com/article/us-africa-beeping​ /phone-credit-low-africans-go-for-beeping-idUSHEA92325720070926. Magara, Peter. 2018. “Giving Data a Soul: A Case Study on Use of Water Point Data in Uganda.” IRC Blogs, August 3. https://www.ircwash.org​ /blog/giving-data-soul-case-study-use-water-point-data-uganda. Murray, Maryanne. 2018. “Blockchain Explained.” Reuters Graphics June 15, 2018. http://graphics.reuters.com/TECHNOLOGY-BLOCKCHAIN​ /010070P11GN/index.html. Naughton, Meleesa. 2013. “3 Innovative Ways to Manage Rural Water Supply.” The Water Blog, June 25. https://blogs.worldbank.org/water​ /3-innovative-ways-manage-rural-water-supply. Nigeria Federal Ministry of Health. 2014. “Nigeria Health Information System Policy.” Federal Ministry of Health, Abuja, Nigeria. http://ehealth​ 4everyone.com/wp-content/uploads/2015/09/Nig-Health-Info.pdf. Rural Water and Sanitation Information System app. http://siasar.org/en/download-app. Rural Water and Sanitation Information System. 2016. “Conceptual Model SIASAR 2.0 Version.” https://siasar.org/sites/default/files/documents​ /siasar_2.0_conceptual_model.pdf. Smits, Stef, Goufrane Mansour, and Harold Lockwood. 2017. “Developing Rural Water Metrics for Sustainability: An Assessment of Existing Indicators of Sustainability.” World Bank, Washington, DC. Steer, George. 2018. “Scientists Are Using Satellite Technology to Track Whales from Space.” Time, Nov. 1. http://time.com/5441967/whales​ -camera-space/. SweetSense. Feb. 8, 2019. “State of California Tackles Drought with IoT & Blockchain.“ http://www.sweetsensors.com/applications​ /groundwater/. Thomas, Evan, Luis Alberto Andrés, Christian Borja-Vega, and Germán Sturzenegger. 2018. Innovations in WASH Impact Measures: Water and Sanitation Measurement Technologies and Practices to Inform the Sustainable Development Goals. Directions in Development—Infrastructure. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/29099. UNESCO (United Nations Education, Scientific and Cultural Organization). 2015. “Enhancing Capacity for a Stronger and More Efficient EMIS in Kenya.” UNESCO Regional Office for Eastern Africa, Nairobi. http://www.unesco.org/new/en/nairobi/about-this-office/single​ -view/news​/enhancing_capacity_for_a_stronger_and_more_efficient_emis_in/. World Bank. 2017a. “Toward a Universal Measure of What Works on Rural Water Supply: Rural Water Metrics Global Framework.” World Bank, Washington, DC. http://documents.worldbank.org/curated/en/607191503560633102/pdf/119073-23-8-2017-14-21-21-W.pdf. ———. 2017b. “World Rural WS Sustainability Metrics.” Report No: AUS22416. World Bank, Washington, DC. WHO/UN (World Health Organization/United Nations). 2017. “Financing Universal Water, Sanitation and Hygiene under the Sustainable Development Goals.” UN-Water Global Analysis and Assessment of Sanitation and Drinking-Water (GLAAS) report. World Health Organization, Geneva; United Nations, New York. Testing, Piloting, and Validation of theRural Water Indicator Global Framework in the African Context 79 SKU W19060