88785 Making Mobile Feedback Programs Work LESSONS FROM DESIGNING AN ICT TOOL WITH LOCAL COMMUNITIES Panthea Lee Merrick Schaefer PERSPECTIVES ON SOCIAL DEVELOPMENT Briefing Notes From Nigeria May 2014 PERSPECTIVES ON SOCIAL DEVELOPMENT Briefing Notes From Nigeria Does providing a free text messaging (SMS) reporting tool, which facilitates communication between farmers and the state-level Project Implementation Unit (PIU) overseeing the Fadama III project, improve the quality of service of the program? This note highlights initial outcomes from a pre-pilot exercise conducted in 2013 with two program objectives: First, to explore whether working with local Community-Based Organizations (CBOs) to train farmers to use and promote the tool was an effective means to get and maintain user adoption; and second, to examine whether the ICT-generated information and system helped the PIU better manage the project. Two main conclusions can be drawn from this pre-pilot. Firstly, it was found that CBO engagement led to significantly larger uptake and use of the system. And secondly, the State PIU only minimally used the reports that were being sent by farmers – despite institutional excitement about the system and a strong mandate from management to use the reports. The findings presented here are based on the deployment of MyVoice, a prototype SMS tool, in six communities in the Federal Capitol Territory (FCT) of Nigeria. The tool was rolled out over a two-week period through training sessions in the community. The tool was then left operational for an additional four weeks for the communities and government to utilize it. The data informing this note was from all six weeks. Despite the fact that the prototype had significant technical and usability limitations (which were well-understood by the implementing team), its deployment allowed the team to test and learn about the social adoption and use of the tool, as well as to refine the program, training and final software for a more formal pilot and launch. This responsive and adaptive approach helped the team build a more locally relevant and useful system. BACKGROUND: THE FADAMA PROJECT AND ICT-ENHANCED SOCIAL ACCOUNTABILITY The Fadama project, a World Bank supported program, aims to reduce rural poverty and increase food security. The primary beneficiaries are small holder farmers organized in Fadama Farmer User Groups (FUGs), and the project facilitates their -1- May 2014 access to financial and technical resources through matching grant arrangements. The MyVoice initiative utilizes an ICT-based system to enhance the reporting mechanism between FUGs and the overseeing state-level PIU to increase transparency and accountability by making available real time information about the Fadama III program. If successful, this initial model can be expanded to include intermediary levels of Local Government Associations (LGAs) and Fadama Community Associations (FCAs), and it could be scaled from the initial pilot in FCT and Nasarawa State and to the full country. A Fadama farmer sending an SMS to the ICT system. MyVoice has two ICT components: 1) a mobile phone-based tool using SMS that allows FUGs to provide structured and unstructured feedback on funding and services provided by the Fadama project; and 2) a web-based dashboard displaying actionable information and tools for the state- and federal-level PIUs, as well as ways to track follow-up on specific issues. Prior to launching the two-state pilot, a prototype system was deployed to six communities in FCT to help refine the program and tools. The pre-pilot and prototype showed a fair amount of positive and expected behavior. There was sustained use of the system during the pre-pilot period, and roughly 30 percent of the messages came from untrained users. Data from system use shows that, on days where there were specific outreach activities, spikes occurred in usage. There was also steady use of the system (i.e., 1-2 messages a day) on most days, regardless of outreach activities. All six communities targeted for deployment used the system. 40 percent of the farmers who were trained never used the system, though this exceeded expectations since there was minimal control over who was trained. This is due to fact that farmers themselves chose whether to attend the training, and the times of the trainings were not designed around the farmers’ schedules. Overall, the pre-pilot demonstrated -2- PERSPECTIVES ON SOCIAL DEVELOPMENT Briefing Notes From Nigeria steady use of the system and general interest in reporting. The initial pre-pilot provided insights into many aspects of the MyVoice system, but there were two areas of the program design focused on: • Lessons learned from training and promoting MyVoice in communities with and without local CBO support; and • Lessons learned from PIU adoption of the web-based dashboards that provide actionable information and follow-up tools. Box 1: Telecom Arrangements In order to deploy a tool that allows beneficiaries to use SMS to communicate with the government through web based software, a number of arrangements need to be negotiated with the local Mobile Network Operators (MNOs) in country. Then specific services need to be procured from the MNOs, which include a reverse billing shortcode enabled to work across multiple networks and communicate with the MNO’s Short Messaging Center (SMC) through either of the internet protocols known as HTTP of SMPP. Further explanation of what is needed is provided below. • Shortcode: This is a 3-5 digit special telephone number. It is easy to publicize, remember and type and the single number works across all MNOs which normal long numbers do not. For promotion and adoption of mobile services shortcodes are critical due to their simplicity. Shortcodes need to be acquired either from regulators or third party businesses known as mobile aggregators. • Reverse Billing: This feature consolidates all charges to and from the number to one account. This makes the service toll free to the users of the phone number. Due to price sensitivity in rural communities, having the service be free to use is essential to adoption of the tool. Reverse billing needs to be negotiated independently per MNO. • Bulk Rates: SMS need to be purchased in pre-allocated bulk volumes that allow for significant discounts over street prices. The larger the volume of messages purchased, the lower the price per message. Bulk packages need to be negotiated independently per MNO. • Multiple MNOs: Due to the fact that the telecoms have incomplete and non-overlapping coverage of the country there is the need to work with multiple MNOs. Some regions only have access to a single provider. Users rarely have access to more than a couple providers requiring engagements with multiple MNOs to be critical • SMC and Communication protocols: The sending and receiving of the messages themselves is handled by the MNO’s towers and data centers. For SMS these are handled by the Short Messaging Center. Once the messages are on the MNO’s system they can be passed back and forth to the software that runs the ICT system over normal internet protocols called SMPP or HTTP. -3- May 2014 UTILIZATION OF CBOS IN TRAINING AND PROMOTION OF MYVOICE During the pre-pilot exercise, the team provided two types of trainings to mobilize community interest and participation in the MyVoice tool. One training was performed exclusively with Fadama PIU members, and the other worked with a CBO. The trainings’ intent was to engage with local organizations, but it was unclear how critical local partners outside of the Fadama PIUs would be. The chosen partner, the Federated FCAs of A Fadama member harvesting cassava. FCT (or “Federation”), appears promising for its ability to effectively support implementation for MyVoice in the medium term. The Federation is an independent CBO created to support the Fadama operation. The Federation has adequate capacity for program management tasks related to reporter recruitment, training and management and for community mobilization, and it has a strong understanding of the Fadama operation. During the pilot, the Federation was highly engaged in recruiting both FUGs/FCAs and reporters to participate. Additionally, the Federation showed great initiative: the Federation Chair joined Reports by date 20 Reports 10 22 Mar. 30 Mar. 11 Apr. 20 Apr. 29 Apr. Reporters Non-reporters -4- PERSPECTIVES ON SOCIAL DEVELOPMENT Briefing Notes From Nigeria the team in each session, helped with instruction and engaged a drama group and traditional singers and dancers to stimulate interest in MyVoice. The Federation maintained consistent communication with the team during the pre- pilot period. This included providing updates on the Federation’s activities related to reporter training, community mobilization, and progress monitoring. The activities they organized and ran included: • Running trainings for FUG members and returning to communities where there had been training but minimal usage; • Coordinating general meetings for all FCA members in each testing location in collaboration with the FCA Chairmen; • Using ‘Makawa’ (praise singers or traditional entertainers) in each of the selected communities to mobilize Fadama members and inform them about MyVoice. Additionally, the Federation monitored MyVoice’s progress in-person by visiting communities to, in their words: ’see how committed the reporters and the communities are to MyVoice, what are they doing with it, how imperative is it to them, and what do they think of it.’ The Messages before/after outreach Federation also monitored the reporters’ 70 progress with voice calls. This enabled the Federation to be in touch with reporters throughout the pre-pilot period, in order Number of messages to make sure they are committed to their role as reporters. Finally, the Federation 35 allowed reporters and FUGs to call them with any questions or issues they encountered with MyVoice. Looking at the utilization of the system, Two weeks Two weeks there was a large increase in usage after before outreach after outreach the Federation did their first outreach -5- May 2014 activity. Messaging during the weeks Box 2: Technical Challenges of System that followed the outreach was over Design double than before. Additionally, during During the duration of the pre-pilot many technical the second round of outreach by the insights were uncovered that would help refine the working of the software powering MyVoice. The Federation, there was a large uptick two most significant are outlined below: of use by the trained reporters, while SMS response time may lead to user independent reports remained steady. disengagement and uncompleted This implied that the CBO efforts had messages. Text messages sent to MyVoice were variable in response a larger impact on trained reporters time. In some instances, after a user that were encouraging fellow Fadama had sent a message to MyVoice, system messages were quickly returned (in less members to report than on individuals than 1 minute). In others, it took more self-reporting their own issues. This could than 5 minutes for a system response; sometimes the messages did not come be significant for empowering those who at all. This was primarily due to mobile themselves lack the knowledge or literacy network conditions outside the control of the project. On average, it took 20 skills to report issues they are having with minutes and as many as 9 messages to the Fadama program to the PIU. complete a decision tree response with substantial periods of waiting. This seems problematic as users may become Beyond the Federation or other CBOs, distracted and may not complete their session. Fadama’s own infrastructure has shown significant capacity to engage with FUGs The system was too complex for many users. Many users, both end-users and and FCAs. This is an important insight reporters, were not well versed in either since working with CBOs is difficult, the structure and processes of Fadama III or the process of sending and receiving if the program scales to more states SMS messages. By the end the training appropriate CBOs might not exist. sessions, most reporters were able to successfully send and respond to MyVoice system messages, but made errors with respect to the Fadama categorization scheme. PIU ADOPTION OF THE ICT TOOLS FOR DATA AND FOLLOW-UP Both of these challenges point to designing a simpler system, one with less interactions to bring down the total time of interacting with the A co-creative process with the PIU system and one with simpler content that may be less specific to how Fadama III works but more itself was used to develop the MyVoice understandable to the farmers using the system. dashboard that it uses to see and respond to messages from farmers. Throughout -6- PERSPECTIVES ON SOCIAL DEVELOPMENT Briefing Notes From Nigeria Homescreen of the dashboard of the Fadama PIU the development of the tool, the project team worked with the Fadama III PIU office in FCT to clarify functionality, roles and responsibilities with respect to their usage of the system and their responses to inputs. The team worked with the PIU officers to co-develop workflows and plans for responding to different types of incoming issues, and immediately prior to the launch of the pre-pilot, the PIU team met and agreed to follow up issues that were reported to the system. This cooperative design approach helped the team build and implement a system that respected the ground- level realities in which the PIU operated. To further improve the tool’s effectiveness, the PIU management also strongly encouraged its members to use the system to engage with Fadama beneficiaries. The dashboard tool works as follows. Each PIU member has a unique log-in to view a custom dashboard that shows all reports sorted geographically or by specific issues (e.g. issues of payment, service providers, information). The display has an alert on -7- May 2014 the homepage of the dashboard that indicates whether a colleague has ‘tagged’ or assigned an alert to another colleague. This allows issues that are the responsibility of a specific PIU team member to be flagged and responded to by that individual. The system also allows for a direct SMS response to be sent to the farmers, triggered from the web interface. An analysis of the PIU’s use of the dashboards during the pre-pilot reveals interesting trends. Out of the 314 messages sent by farmers to the PIU, only six were responded to using the system, and eighteen were tagged for follow-up. Additionally, of the eleven accounts created for the PIU, only three team members ever logged into the system - two of them only did so at the beginning of the pilot. All eighteen messages that were tagged were tagged by the same individual. Following a discussion with the PIU to learn why the system was used so little, a number of factors emerged, including: • It was unclear whether the data being reported was real or from trainings; • Many of the messages did not warrant follow-up; • Many of the messages were positive; • Some of the messages were not understandable; • Messages reported by proxy reporters on someone’s behalf were not ones that could be responded to. Additionally, it was suggested that a high number of messages were in fact followed up on, but this follow-up was not noted in the system. For example, one team member logged in and told another member about the issue, and then the farmer or FUG was contacted by phone. This suggests a valuable lesson for designing ICT tools in a way that supplements, rather than replaces, traditional or existing mechanisms and structures. It is important to note that follow-up can occur without any display within the system itself. The PIU members are not asked to independently log their responses - whether to an individual farmer, through facilitators or by making improvements to -8- PERSPECTIVES ON SOCIAL DEVELOPMENT Briefing Notes From Nigeria the Fadama program. Within the MyVoice dashboard, responses to issues and tags are, at best, proxies for actual follow-up to FUG member’s issues. Given that most interactions would not be visible to the system, it is not possible to accurately know the number of follow-ups – or indeed what constituted a follow-up. Nevertheless, the dearth of responses and tags, proxies as they may be, do demonstrate either a lack of motivation, time, capacity or interest to respond by the PIU. It is clear that more focus needs to be given in one-on-one trainings of PIU members and to active monitoring and management of the use of the site. The pre-pilot’s use by the PIU could also signal a different workflow in which one team member watches the site and then notifies other team members to follow up on issues. This might include a mechanism to alert PIU members via text or email with a message to which they need to respond. Fortunately, the team’s experimental approach of this MyVoice activity – namely, by testing and incubating it in this smaller pre-pilot exercise – enables the team to incorporate these lessons in future iterations as it scales to the pilot and beyond. Additionally, the pre-pilot reinforced the team’s appreciation of the need for a program coordinator to act as a liaison between the community-focused implementing partner, government partners, and the World Bank. This role could act as another layer of accountability and encourage PIU members to log in, respond through the site and flag serious issues to the PIU coordinator. CONCLUDING REMARKS The MyVoice pre-pilot provided the Fadama project an opportunity to explore some of the practical implementation aspects of deploying an ICT for social accountability tool in the field. The initial evidence shows there is enough demand that the program could work without local CBO backing, although it would be greatly enhanced by such support. Questions about the long-term sustainability of interest in the system remain, and CBO support would certainly provide a mechanism for continued engagement with Farmers Groups. -9- May 2014 Furthermore, engagement by the government officials was not strong, and follow-up around specific activities was hard to track. More targeted and one-on-one training with government stakeholders in the PIU should be attempted in the next generation of MyVoice. It is also recommended that a third-party monitor and follow-up with the PIU – especially for the first few months of the pilot’s launch. More broadly, institutions deploying ICT citizen and beneficiary engagement tools should not underestimate the level of commitment and institutional capacity required by governments to effectively respond to information obtained through such tools. Lack of response over time could damage government reputations and estrange communities from these programs. As demonstrated in many ICT projects, the tool itself is not a silver bullet, and it will only provide value if it is used both by the communities and government to foster dialogue, encourage understanding and target resources and knowledge. It is clear, however, that tools such as MyVoice do make it much easier for rural community members to initiate that dialogue - a significant empowerment in of itself. The MyVoice pre-pilot lays the foundation to continue experimenting with the tool’s integration in the Fadama project, and it reinforces the team’s aim to iteratively adapt and co- creatively design its activities to an ever-evolving local context. - 10 - Panthea Lee is a principal at Reboot, an advisor and implementer for institutions dedicated to social progress. At the time of writing, Merrick Schaefer was a Senior Innovation Specialist working in the ICT Unit at the World Bank. He is now the Lead for Mobile Data in the Global Development Lab at USAID. This note reflects on the work undertaken by a larger team from the TWICT and Social Development Units. Both units joined efforts to implement the activities planned for the project described in this note. The team was comprised of: Caroline Sage, Merrick Schaefer, Paula Andrea Rossiasco, Rosa Maria Martinez, Matthew McNaughton, Kanbak Christyn Labar and Priyanka Pathak. This Briefing Note was cleared by Ian Bannon, Sector Manager, AFTCS, Africa Region, The World Bank. This essay is one of several that comprise the series “Perspectives on Social Development: Briefing Notes from Nigeria”. These briefing notes highlight the World Bank’s cross-disciplinary, adaptive and context-specific approach to social development in Nigeria. Topics include gender, social inclusion, violence prevention, social accountability and community-driven development. They are written by members of the World Bank social development team working in Nigeria. Learn more at http://bit.ly/NotesFromNigeria Photography © Copyright Reboot, All Rights Reserved. This work is published under a Creative Commons Attribution 3.0 Unported (CC BY) license. You can share and adapt this content for any purpose, provided you give appropriate credit and identify any changes.