WATER AND SANITATION PROGRAM: LEARNING NOTE Key findings • Collecting feedback from citizens on service levels provides a useful reality check to service providers’ SLB CONNECT FOR URBAN reports on their own performance. For example, citizens in Indian cities surveyed by SLB Connect WATER SUPPLY AND reported experiencing poorer water quality than service SANITATION providers said they delivered. • Various modes of collecting Using ICTs for citizen feedback surveys to mainstream demand-side feedback can be used in a complementary manner monitoring depending on the local context and objective of the feedback December 2016 exercise. Household surveys conducted using mobile aided personal interview methods tend Introduction citizen engagement for improving to be more resource intensive SLB Connect is an initiative of service delivery, including through but also more representative than the World Bank’s Water and the development of the National telephone surveys and SMS polls. Sanitation Program, developed in e-Governance Plan. As ICTs continue partnership with India’s Ministry to develop, there will be increasing • Granular data showing of Urban Development (MoUD), to opportunities to make processes performance at the level of for gathering citizen feedback more zones and wards, are helpful in complement the Ministry’s Service attracting the attention of local Level Benchmarks (SLB) program. intelligent, inclusive and efficient decision makers – as is making Under the SLB program, providers – and thereby leverage its use in those data publicly available in of water supply, wastewater, solid decision making for improving service easy-to-understand maps, graphs waste management and storm-water outcomes. and tables. These also facilitate drainage services report data on a integration in local planning standardized set of 28 performance Problem statement processes. indicators. Grants to municipalities Even where water and sanitation have been linked to reporting on • ICT platforms offer not just the services in India exist, they are advantages of speed and scale, these indicators, and, over time, often of poor quality. Historically, but can also help overcome local SLB data have become an integral the government has focused constraints in resources and component of India’s urban program primarily on creating infrastructure, expertise, for example, with a formulations. and not enough on ensuring that default questionnaire template the infrastructure was designed which is customizable using SLB Connect complements SLB to meet the needs of citizens and a question bank facility, and data by gathering feedback from the used effectively to deliver quality experts remotely monitoring citizens who use those services. The service outcomes. The SLB program data collection at multiple demand-side data are intended to was a step forward in shifting the locations simultaneously. focus from infrastructure creation to These functionalities of SLB improve tracking of service outcomes, Connect were leveraged by the provide a reality check for supply- outcome monitoring. However, the Government of India for conduct side data reported by providers, data it collected came only from the of city sanitation ratings in 73 identify problems with service service providers without adequate cities across the country. outcomes at local (ward/zone) level, quality checks, and usually did not identify inequities by user groups reflect the views of the citizens who (for example, households in slum actually use the services. This called settlements), and ultimately improve into question the reliability of the SLB service providers’ accountability to data, and their usefulness as a tool citizens. for engaging citizens and increasing service providers’ accountability. The SLB Connect program leverages the use of Information and Action Communication Technologies (ICTs) SLB Connect was developed to for collecting feedback, which fits with systematically track the service the Government of India’s broader experience of citizens through interest in using ICTs to strengthen feedback surveys using a variety 2 SLB Connect Figure 1: Performance aspects and citizen service experience captured by SLB Connect Profile Water Supply Sanitation Feedback Age Access to water Access to toilets Satisfaction Gender Continuity Toilet usage Willingness for repeat Dwelling type Adequacy Access to sewerage survey Address Water quality network Contact number Income profile Complaint redressal Alternate disposal Suggestion to service Ease of bill payment system provider of ICTs. Mobile to web systems were deployed the An online survey management module (see Figure 2 most, wherein local agents were employed to interview for a sample screenshot) enables managers to track a cross-section of citizens in their homes, to get their the progress of surveys in real time, and customize the feedback on various aspects of the water and sanitation questions as required. Quality control mechanisms are services they receive (see Figure 1 for a summary of data inbuilt, including predefined validation checks, exceptions collected via the questionnaires). The agents recorded the flagging, performance reports on individual enumerators citizens’ responses using an Android mobile application, and supervisors, and field visits for validation. an approach known as MAPI (Mobile Aided Personal Interview). The phone’s Global Positioning System (GPS) The data are analyzed and presented real time, on an records the location where the data are collected, and the online dashboard using graphs, tables and maps, with time. The phones used were standard Android handsets traffic light color-coding for easier understanding (see costing around US$130. Figure 3 for an example). Demand side metrics are used to Figure 2: Sample screenshot of survey records collected www.wsp.org SLB Connect 3 Figure 3: Example of an online dashboard www.wsp.org 4 SLB Connect Figure 4: Implementation of SLB Connect Implementation till now…. Pilot phase (2012-14) - Pilot phase (2012-14) 3 cities in 3 states MAPI surveys of 7,500 households (city level and community level surveys) Partnered with other agencies for 2 cities Scaled up demonstration (2014-15) 5 cities in 3 states (population 3.5 million) MAPI surveys of 28,500 households Repeat Surveys using additional feedback Integrated with Swachh Bharat Mission city sanitation ratings and modes World Bank project 1 city covered in Pilot phase Telephone survey (2014) – 3,000+ households SMS polls (2015) assess service levels and enable comparisons with SLB of these cities was done in consultation with the MoUD data as well as across wards/zones in the city. Data can and concerned state governments. In each city, a scoping be broken down by user group (for example, slum/non- study was first conducted to understand its institutional slums) or geographic unit (for example, zones/wards), arrangements and the status of service delivery, and the and comparisons can be made between locations or over questionnaire was customized accordingly – with input time in the same location. The data are made available to also solicited from local functionaries. SLB Connect has decision makers and the general public: initial meetings now surveyed around 35,000 households in eight cities are held to discuss findings with city functionaries, then across six states. an outreach meeting is held to share results with other stakeholders, and key findings are summarized in flyers Findings (print and electronic) in the local language. Finally, the The SLB Connect surveys provided concrete, survey findings are made available through the publicly relevant data which served as a useful reality check accessible online dashboard. In addition, SMS messages to the SLB reports of service providers on their own are sent to all survey respondents and councilors, performance. On questions of infrastructure provision, providing a link to the flyers and the website. it was found that survey data generally validated the reports of service providers. On questions related to SLB Connect began as a pilot in 2012 in Pimpri service quality, however, there were often significant Chinchwad, where around 5,200 responses were gaps between the reports of service providers and collected. Informed by the feedback, the city set up the experiences of service users. For example, in a new complaint mechanism and undertook outreach Raebareli, Jabalpur and Varanasi, between 42 percent to increase awareness of it. In 2014 and 2015, at the and 73 percent of survey respondents said that their city’s request, other survey methods – computer-aided water supply had been dirty at least once in the telephone interviews and an SMS poll – were tested to preceding three months – whereas service providers follow up and see if citizens’ experiences had improved in these cities reported 96-98 percent compliance with on specific service aspects. water quality standards. Granular data analysis also helped reveal inequities between slum and non-slum After further pilots in Mehsana and Delhi, improvements areas, as also across wards, with peripheral areas of were made to the system’s functionality and household cities generally seeing poorer service levels. surveys were scaled up in 2014-15 for implementation in five further cities: Raebareli, Varanasi, Ajmer, Jhunjhunu The survey results also drew attention to the inadequacy and Jabalpur (see Figure 4 and Table 1). The selection of existing formal complaint mechanisms, revealing that www.wsp.org SLB Connect 5 Table 1: City and survey sample details Jabalpur (MP) Varanasi (UP) Raebareli (UP) Ajmer (Rajasthan) Jhunjhunu (Rajasthan) City profile Total population 1,069,292 1,597,051 191,056 542,580 118,473 Population in slum areas 45% 19% 23% 20% 4% Agency responsible for Municipal Capital works: Jal Municipal Water: State agency (Public Health water and sanitation Corporation Nigam (state agency) Corporation Engineering Department) Operations and Sanitation: Municipal Corporation maintenance: Jal Kal (Municipal Corporation) Sample details No. of wards 70 90 31 55 45 No. of respondents (HHs) 6,693 9,330 3,134 5,500 3,823 Respondents from slum 48% 22% 26% 18% 4% areas Note: HHs: households; MP: Madhya Pradesh; UP: Uttar Pradesh. citizens’ use of telephone, SMS or online channels was Experience in Pimpri Chinchwad with the follow-up negligible. In most cities, only around 5 percent of citizens surveys using telephone and SMS found that, while had lodged a complaint, mostly in person, and only a third cheaper and quicker than MAPI surveys, these methods of complaints had been resolved. had some drawbacks: household status (for example, whether in a slum or not) could not be validated; fewer The authorities in all cities accepted the survey findings, female respondents were reached; and respondents were though they subsequently displayed varying levels of reticent about answering questions on toilet use. interest in acting on the concerns the surveys revealed. Some cities publicized the survey results on their Drawing from the experience gained under SLB municipality websites; some integrated them in proposals Connect, MoUD adapted the SLB Connect platform and for funding under national urban programs such as the survey approach for undertaking City Sanitation Ratings Atal Mission for Rejuvenation and Urban Transformation (called ‘Swachh Survekshan’) across 73 cities as part of (AMRUT), Smart Cities and Swachh Bharat Mission. the Swachh Bharat Mission. Field data were collected within just two weeks in January 2016 and 80,000 citizen The responsiveness of city functionaries was found to responses were collected for the rating assessments, depend on several factors including their general comfort using Interactive Voice Response System (IVRS). The levels with participatory processes and familiarity with SLB Connect platform and methodology is also being ICT; degree of decentralization of service delivery function used for a baseline survey of 150,000 households for the to local body; the local survey partner’s capacity and World Bank-supported Karnataka Urban Water Supply working relationship with city authorities; the degree of Modernization Project, and subsequent annual feedback cohesion among local political leaders; and the extent to surveys to monitor service levels. which there were already planning processes underway for service improvements, into which survey findings Key learnings could feed in. As far as possible, demand-side metrics should be aligned with supply-side indicators. Aligning the The SLB Connect ICT platform enabled surveys to be questions asked in citizen surveys with data reported conducted at scale in a short time frame and with a by service providers (see Table 2 for examples) makes limited number of technical experts. It also proved able it easier to generate interest among stakeholders who to cope with diverse environments, reaching a significant are already familiar with service provider metrics, to percentage of respondents who are female, below the track outcomes, strengthen monitoring,and hold service poverty line or living in slum areas. providers to account. Creating a common vocabulary of www.wsp.org 6 SLB Connect Table 2: Comparison of selected SLB metrics and SLB Connect metrics derived from feedback data SLB (reported by service provider SLB Connect (results of household surveys) % HHs with individual/shared household piped connection for % HHs reporting individual/shared household piped water supply connections as primary source of water supply Duration of water supply Median value of responses on duration of supply Quantity of water supplied per capita per day % of HHs reporting adequate supply to meet the needs of the family % of water samples meeting specified standards % of HHs reporting no incidence of dirty water supply in last 3 months % of complaints resolved in one day % of HHs that lodged complaints, reporting resolution in 1 day Revenues collected as a percentage of revenues billed % of HHs reporting regular receipt of bills, and that find the location, timing of bill payment to be convenient % of household connections with functional meters % of HHs reporting functional meters % of HHs with access to individual, shared or public toilet % of HHs reporting access to an individual, shared or public toilet % of properties with connection to sewer network % of HHs reporting connection to sewer network Note: HHs: households. service metrics helps consumers and their representatives of seeking citizen feedback – MAPI surveys, SMS polls, to dialog more effectively with service providers. telephone surveys, formal complaint mechanisms, and crowd sourced ratings – and they have different Designing surveys to be replicated at scale could strengths and weaknesses, as illustrated in Figure 5. SLB help to address local capacity constraints. Most local Connect primarily used MAPI surveys, which are relatively bodies lack the capacity to undertake citizen surveys. resource intensive but have the advantage of reaching a SLB Connect suggests how to make this easier: representative sample of citizens – not only those who provides a default template which is customizable happen to have access to particular technologies, or with different questions from a question bank; enables who are highly motivated to respond. They are therefore surveys conducted in multiple locations to be remotely especially suited to capturing detailed feedback at the monitored online in real time, making best use of a start of a project in a context of poor service delivery. limited number of experts. It provides a platform which Cheaper methods, such as SMS polls, should be is hosted by an agency with the requisite credibility viewed as complements, rather than as interchangeable and technical expertise, could serve as a monitoring substitutes. unit in the government, a regulatory agency, a research institution or civil society organization. How citizen feedback is expected to improve services should be clarified in advance. Citizen feedback ICTs can improve impact by providing credible, can potentially be leveraged in several different ways transparent, immediately actionable information. to improve service delivery, such as informing service Making all the data accessible – including rejected providers about gaps; mobilizing public opinion to records, enumerator details, time stamps, geo- generate pressure on service providers; or feeding into coordinates and photo images – made the integrity of plans to improve policies or infrastructure. Clarifying in SLB Connect’s data collection process more transparent advance how feedback is expected to be leveraged could to stakeholders, increasing its credibility. In contrast to help to decide the extent and nature of civic mobilization traditional survey methods – which often take months to and stakeholder engagement required to accompany the report, by which time the findings are dated – the system survey. also enabled results to be made available to decision- makers immediately. It may be difficult to generate popular demand for service improvements. The survey results revealed low Differing ways of obtaining feedback have different expectations on service delivery: even though service advantages and drawbacks. There are various ways levels were reported as relatively poor, nonetheless 80- www.wsp.org SLB Connect 7 Figure 5: Comparing feedback channels SLB Connect Mobile to SMS or Telephone Crowd Sourced Complaint Web HH IVRS Poll Survey Ratings/Feedback Monitoring Survey Representative (respondent mix) Only customers who Respondent mix uncertain: gender, income, literacy bias use formal channels (for SMS polls) Quality of feedback (reliability, details) Fewer questions possible: interpretation errors likely Citizen motivation (Response rate) Use of informal “Why bother?” Difficult to sustain channels when provider response is poor Intra-city analysis Possible if Possible if geo-tagged; unlikely for SMS polling & online feedback geo-tagged Scale (No. of citizens reached) Resource requirement Note: HHs: households. 90 percent of respondents said they were partially or incentives by integrating these processes in government fully satisfied. This suggests there may be limits to the programs and formulations; and developing orientation extent to which civic mobilization is possible for service modules for citizen groups and political representatives on improvement. The survey also showed that customers with how to interpret the resulting data and use it for advocacy. complaints often address them to their local councilors, who could potentially play a greater role in putting What else do we need to know? pressure on service providers. A suitable pressure point What potential does the SLB Connect platform in this regard could be the metric measuring perceptions have to be adapted for other purposes? The benefits of whether services have improved or deteriorated offered – in terms of speed, scale, accuracy, data compared to the previous year. integrity and transparency of process – may be more widely applicable. For example, the MoUD has already Advocacy and capacity building are needed for adopted the SLB Connect platform to undertake City standardized performance indicator frameworks to be Sanitation Ratings (called ‘Swachh Survekshan’) adopted and mainstreamed. Development agencies can as part of the Swachh Bharat Mission. The initial play a critical role in this, in partnership with the national implementation done in January 2016 covering 73 government. Interest and buy-in of local functionaries cities is now being further scaled up to cover 500 cities can be increased by involving them in the preparation of in the next round of ratings planned for 2017. questionnaires; employing locals, such as students, rather than outsiders to gather the data; and providing granular What lessons can be drawn from discrepancies results that are more relevant for local functionaries. between demand- and supply-side data? Some trends were common to all cities, such as citizens reporting The national government can help states and cities better access to toilets than did service providers. In to adopt citizen feedback processes in several ways. contrast, trends were mixed for water supply. In the two These include providing guidance, capacity building cities surveyed in Rajasthan, for example, where water and advisory support; hosting or making available ICT- supply is the responsibility of the state rather than cities, based solutions for feedback processes; strengthening water supply access was reported by service providers to www.wsp.org 10 How Sustainable are Sanitation Outcomes in Clean Village Prize Winners? be lower than according to citizens surveyed by SLB Connect – the How can the World Bank best support clients in adopting citizen About the project opposite of results in other cities. It engagement interventions? The recent World Bank directive Service Level Benchmarking, Citizen would be useful to explore further Voice and Performance Improvement what incentives might exist for overly mandating inclusion of beneficiary Strategies in Urban Water Supply conservative reporting, and also feedback processes can give further and Sanitation (UWSS) in India is a whether these are influenced by impetus – but these processes WSP project focused on improving institutional arrangements. should not be limited to self-reporting accountability for service outcomes options such as helplines, SMS in the UWSS sector, by providing How can insights from SLB or online feedback. They should support for strengthening supply and also include mechanisms to collect demand side monitoring processes Connect be integrated with feedback proactively, and identify under national programs in India, those from similar experiences and integrating use of performance elsewhere? It would be useful to factors that may inhibit citizens’ data into decision making by public consolidate learnings from SLB use of formal grievance redressal providers, with specific focus on Connect with those from other mechanisms. services to the poor. World Bank experiences in citizen By Vandana Bhatnagar and engagement using ICTs, such as Andrew Wright Maji Voice (Kenya), Vozelectrica (Dominican Republic) and Citizen For more information, see report SLB Feedback Monitoring Program Connect: Mainstreaming citizen feedback (Pakistan). There is potentially scope on service delivery using ICTs: Findings and lessons from ICT-based feedback surveys on to reduce the costs of future such water supply and sanitation services in Indian initiatives by creating a platform to cities by Vandana Bhatnagar, Nidhi Batra and share knowledge. Kanak Tiwari The Water and Sanitation Program is a multi-donor partnership, part of the World Bank Group’s Water Global Practice, supporting poor people in obtaining affordable, safe, and sustainable access to water and sanitation services. WSP’s donors include Australia, Austria, Denmark, Finland, France, the Bill & Melinda Gates Foundation, Luxembourg, Netherlands, Norway, Sweden, Switzerland, United Kingdom, United States, and the World Bank. The findings, interpretations, and conclusions expressed herein are entirely those of the author and should not be attributed to the World Bank or its affiliated organizations, or to members of the Board of Executive Directors of the World Bank or the governments they represent. © 2016 International Bank for Reconstruction and Development/The World Bank www.wsp.org