OnTrack: Review of pilot implementations Annex 1. Detailed findings per assessment area 1. Quality and relevance of design ............................................................................................ 2 2. Contribution to development results .................................................................................... 9 3. Process efficiency ................................................................................................................ 19 4. Monitoring of behavioral changes ...................................................................................... 22 5. Sustainability of OnTrack ..................................................................................................... 23 6. Replication of OnTrack ........................................................................................................ 24 This annex presents findings from the data collected for the indicators identified in each assessment area. 1. Quality and relevance of design Indicator 1.1: Relevance to broad WB goals Figure 1 WB staff survey: perceptions on OT alignment with WB goals In interviews and the survey of WB staff there is consensus that OnTrack’s objectives are broadly aligned with WB goals, particularly with President Kim’s drive to obtain beneficiary feedback for all WB projects where there is an identifiable beneficiary. Senior management in interviews expressed reservations about the relevance of OnTrack across the heterogeneous range of Bank projects and contexts for which citizen feedback is now being sought. Senior managers anticipated two main challenges:  Overselling the potential of “There is not much consensus or clarity about what it is and how to do it. The programs like OnTrack evangelists oversell and the audience under-listen [...] people may equate CE  The costs associated with CE with development goals and that’s not initiatives. At this stage of true.” development, there is a large “Currently OnTrack is perceived as time upfront cost to establishing an consuming, cumbersome and expensive OnTrack platform. There is no low while it should be the other way round: cost “toe in the water” option. light and flexible.” 2 Indicator 1.2: Criteria for selecting the pilot countries and projects The internal documents provided by the OnTrack team include few and mostly indirect references to the reasons or criteria used in the selection of pilot countries and projects. Interviews with WB staff, both in Washington, D.C. and in the country offices, point to the following:  The general perception is that the selection of the countries was mainly opportunistic, building on current relationships with the country teams, especially in the case of Bolivia. In Zambia, the choice was linked to the opportunity of funds being available through the E-ISR project. One member of the OnTrack team indicated that there was “a conscious choice to target very poor countries with lack of voice and failures in governance”.  For the choice of projects, in the absence of real demand, the team expressed a preference to identify and work with projects with a strong community engagement component. An FAQ document prepared for OnTrack indicates that suitable projects for the implementation of such a mechanism would be projects in remote or extensive areas, where the government has a program for improving the monitoring of quality of service, and where government and implementing agencies are keen to strengthen accountability mechanisms. It goes on to say that “sectors that are best suited for citizen feedback are Health and Education, Agriculture and Rural Development, Urban Development, Infrastructure, Water, and Community Driven Development” (OnTrack, FAQ, p.3). The selection of the OnTrack pilots is consistent with the conditions described in the document. Indicator 1.3: Mobile phone/Internet penetration in pilot sites In all pilot countries, OnTrack referred to existing data regarding mobile phone penetration and Internet use. Table 1 Mobile phone and Internet penetration Mobile cellular Internet Users 2013 subscriptions Bolivia 98% 39.5% Nepal 71% 13.3% Source: World Development Indicators While mobile phone penetration is high in all pilot locations, there was insufficient prior inquiry about the pattern of use of mobile phones by project beneficiaries. In Nepal, for example, research would have found that OnTrack’s SMS texting system for reporting by beneficiaries was ill suited to a population with low literacy levels and lack of familiarity with Roman script. This issue is 3 being addressed - after a long delay - by building a voice integration system to allow beneficiaries to call a toll-free number and leave a recorded message. The OnTrack team was aware that use of the Internet is low in the pilot areas. Accordingly, limited use of the online features of the system was expected. In focus groups discussions we conducted with project beneficiaries in Bolivia and Nepal, we were able to confirm the broad use of mobile phones, especially among younger people, who were also more likely to know how to send text messages. We found that while many men have personal mobile phones, women, with the exception of some younger women, usually do not. Rather, they have access to a mobile phone in their household. Beneficiaries who said they used the Internet regularly were exceptions. For example, a cooperative of coffee farmers in Bolivia claimed they use the Internet regularly to check for coffee trading prices. In Nepal, some beneficiaries expressed a hypothetical interest in the possibility of sharing experiences with other community organizations beyond their Village Development Committee (VDC). Partner organizations in Nepal and agricultural extension workers in Bolivia, who carry out the awareness raising activities for OnTrack, said there was potential value in the online aspect of the feedback system. They believe that exposure to the system will eventually lead to more engagement. During the orientation sessions they seemed excited by the idea of sharing their success stories and photos on the OnTrack website. Our review of the platform, however, shows that this excitement did not translate into concrete action by beneficiaries during the pilots. Findings from the PAR 1 beneficiaries survey in Bolivia The face-to-face survey carried with PAR 1 project beneficiaries in the rural areas of Santa Cruz, broadly confirmed the findings from our qualitative research. It provided a lot of nuance regarding the patterns of use of mobile phones and Internet by the target population. 4 Figure 2 PAR survey: access to technology devices In our survey, only 7% of respondents mentioned not having access to a cellular phone. However, only 39% have access to computers and 28% to a smartphone. Access to computers (67%) and smartphones (48%) is significantly higher amongst respondents that have received some kind of higher education (representing 21% of the beneficiary population). Figure 3 PAR survey: Weekly use of SMS Although there is a high penetration of mobile phones among the beneficiary population, 50% do not use SMS texting (with 22% stating they don’t know how 5 to). Use of SMS is directly related to the beneficiaries’ educational level. Some 63% of those who have not completed secondary education do not use SMS texting. They represent 65% of the beneficiary population. Figure 4 PAR survey: frequency of Internet use In line with data from World Development Indicators, the majority (60%) of PAR 1 beneficiaries report not using the Internet. This percentage goes up to 84% for those beneficiaries who have not completed secondary education. People who use the Internet do so mainly for reading the news (61%) and for communication through email (45%). Some 44% mention using it for obtaining information on market prices for their crops or cattle, while 34% mention that they have used the Internet to obtain information about PAR. Indicator 1.4: Sensitivity to process and culture In interviews, staff from both the World Bank staff and PIUs said there was a strong will on the part of the OnTrack team to adapt the program to the local context and culture. However, as they described it, the process “We did not consult first and happened the other way round with the then design, but first team proposing a standardized approach designed and then asked if it and then looking for ways to customize and would apply to the context”. adapt it. OnTrack team member The issue of adaptation to local context and culture is related to two elements:  The platform was created by a web developer in the USA. When the issue came up (in all projects), extensive engagement between the PIUs, OnTrack and the platform developers took place about customizing 6 aspects of the platform. Our interlocutors suggest this was time- consuming and did not always lead to a prompt and satisfactory solution.  The feedback system itself did not initially take into account cultural specificities such as language (particularly the use of local dialects in Kapilvastu, Nepal), the preference of communities for oral communications (both in Bolivia and Nepal), or existing mechanisms for dealing with grievances in the community. This was notably the case in the urban project in La Paz, Bolivia, where people are accustomed to make demands in person to local leaders. Indicator 1.5: Openness and inclusiveness of the platform Here we have examined the perceptions of the different constituents regarding the openness of the feedback mechanism in terms of access and ‘ease of use’. During focus group discussions in Bolivia, beneficiaries said that the SMS functionality of the system is useful. It allows their reports to be delivered quickly and at low cost – and they no longer need to take time off and bear the costs of going personally to the PAR offices to report issues regarding their projects. ‘Ease of use’:  In Bolivia, focus group participants indicated that the SMS system was fairly easy to use, although they indicated that older farmers did not know how to text. This was confirmed by the PAR beneficiaries survey results as discussed under indicator 1.3 above.  In Nepal, texting is more difficult amongst beneficiaries because, in addition to not knowing how to use the texting function, they find it hard to write in Roman script  An added complication for users is the need to indicate the project site when sending in a report, using a code for the Alliance (Bolivia-PAR) or the VDC (Nepal-PAF) to which they belong.  In both countries, beneficiaries as well as frontline workers indicated that the online function is a challenge for most people, as they are not accustomed to using computers or the Internet.  Furthermore, in our inquiries we discovered that the platform itself is considered slow. Specifically, the maps often took several minutes to load completely, even in countries with good Internet connectivity. The lack of a ‘search’ function also hindered its use. Inclusiveness: The limitations of the evaluation (see Annex 2) precluded an assessment of how the actual users of the mechanism in pilot activities relates to the entire project beneficiary community or to specific subsets of the beneficiary population who are of particular interest to the Bank (e.g., the poor, women, disabled, minority groups…). The project monitoring data, including website usage, are insufficient to complete this analysis and we were unable to obtain the necessary demographic data of OnTrack users after the fact. 7 Focus group participants both in Nepal and Bolivia, as well as people we interviewed, suggest that the system is more suited to young people than to the older generation who did not know how to text or use the Internet. Through the survey of PAR 1 beneficiaries in Bolivia we found, as previously discussed, a direct connection between the use of SMS and Internet with the educational attainments of beneficiaries, indicating a high level of exclusion of beneficiaries with lower educational levels. Furthermore, when asked about the likelihood of using a mechanism such as OnTrack, a significantly higher portion of respondents over 40 years old and those who have not completed secondary education say that they would never use such a mechanism. The gender of the respondents, however, did not seem to be an influencing factor here. During focus group discussions with PAR beneficiaries in Bolivia, participants indicated a preference for the representative/chief of the alliance to send in reports on behalf of all members. This would seem to show a lack of alignment between OnTrack, which is meant to enable easy, open and spontaneous feedback, with the prevailing pre-Internet forms of civic engagement. This may be explained by the fact that only two representatives from each alliance participated in the orientation session. Indicator 1.6: Directness of feedback process Our interviews in PAR and PAF indicated a consistent appreciation that OnTrack does overcome intermediary gatekeeper problems. OnTrack is a system that in theory permits beneficiaries, both individually and collectively, to send in reports directly to the PIU without passing through the local intermediaries. For example, partner organizations in Nepal noted that being able to report directly to PAF without passing through the intermediaries (the “social mobilizers”) provides a useful additional way to hold social mobilizers to account. Similarly, the regional coordinators for PAR in Santa Cruz, Bolivia, indicated that thanks to OnTrack they can hear directly from the alliances without the intermediation of the extension workers. The national coordinators for PAR, meanwhile, indicated that they could get direct feedback without the intermediation of the regional coordinators. PIUs tend to find OnTrack particularly relevant to projects with beneficiaries in remote areas. The reason given is that the affected communities would otherwise be even more excluded. Given lack of supporting evidence that this is the case -- compounded by the near absence of actual feedback captured through OnTrack pilots -- this assertion has to be considered more prospective than actual. Lastly, from conversations with both beneficiaries and frontline workers, it seems that in practice OnTrack does not enable feedback from individuals. As noted in the discussion of the inclusiveness of the system, it seems that reports made through the system are negotiated at the community/rural alliance level 8 before being sent in. In other words, the system is in fact ‘controlled’ by the chiefs/representatives of the community organizations. Indicator 1.7: Adaptation of OnTrack to ‘in country’ conditions A number of points relating to adaptation to in-country conditions are covered above, including issues relating to mobile phone/internet penetration in different country and cultural contexts (indicators 1.3 & 1.4). Regarding in-country conditions, a recurrent issue raised by people we interviewed relates to the openness of implementing organizations to beneficiary feedback. While beyond OnTrack’s remit, this speaks to the need for an analysis of the readiness of the World Bank’s partners to take on board feedback (a ‘readiness calculus’). It also underlines the importance of presenting the full range of elements required for citizen feedback to generate improved development outcomes, including data analysis, sense-making, further inquiry to understand the full import of feedback messages and, most important, course corrections by service implementers. These course corrections must then be communicated to service recipients so that their subsequent feedback is directly responsive to the course corrections. In the 3 main pilot projects (Bolivia and Nepal), OnTrack took advantage of a favorable political climate for establishing citizen feedback mechanisms. In the PBCV project, the municipality of La Paz already has various open communications channels with citizens (“la brigada cazabaches” and the mayor’s radio show). The central government of Bolivia, on which the PAR project depends, prides itself on promoting citizen participation. Similarly, in the case of Nepal, the PAF project is the direct responsibility of the office of the Prime Minister, which also demonstrates, according to interviewees at the country office, openness towards such approaches. In Zambia, however, this condition does not seem to be met and is cited by interviewees as one of the main reasons for lack of progress. A specific issue of lack of adaptation to in-country conditions was raised in Bolivia where national legislation does not allow the websites of public entities to be hosted abroad. The OnTrack platform is currently hosted in the US. 2. Contribution to development results Indicator 2.1: Level of awareness by project beneficiaries For all three projects where the OnTrack pilot was implemented, communication plans were developed but have not been fully implemented. Some material (brochures and posters) was distributed at all sites and there were various mentions in the press, particularly in relation to the launch of the platform. In Bolivia, a high level launch took place in Cochabamba with the participation of 9 President Kim and the Chief Secretary of the Nepalese government participated in the launch there. The main awareness-raising activities carried out at the pilot sites involved a single orientation session. The evidence suggests that this level of preparatory effort is insufficient to establish OnTrack as an effective citizen feedback mechanism. For PAR in Bolivia, all 142 rural alliances in the Santa Cruz area took part in one orientation session. In this session 2 people per alliance were invited to participate (generally, the head of the alliance and one member). For PBCV in La Paz, 2 sessions took place in 2 different neighborhoods with children and adults. In Nepal, orientation sessions took part with 30 community organizations. 5 representatives per community organization participated in the session. In all cases, facilitators gave an overview of the system and its objectives. Participants were told how to send reports using the SMS system and they sent test messages during the sessions. Regarding the online aspects, extension workers and social mobilizers report that on only a few occasions were they able to test the system ‘live’ when computers with an Internet connection were available. In Bolivia, the local Internet café was used on some occasions, while both in Bolivia and Nepal a community centre was occasionally used for that purpose. However, in most instances no computers with an Internet connection were available during the orientation sessions. Explanations were limited to showing participants brochures and describing the system orally. Information from orientation session participants was not passed on, as had been anticipated, to other beneficiaries. In focus group discussions, those that did not participate in training/orientation had not heard about it. In Nepal, some beneficiaries who had participated in orientation did not recall specific details about the session they attended. In a focus group meeting in Nepal, two women had an OnTrack brochure in their hands but could not read it because the brochure is only available in Nepalese, while they speak a Hindi dialect. Most women had not heard about OnTrack in the communities we visited in Nepal. 10 Figure 5 PAR survey: Awareness of existence of a reporting mechanism A similar picture is shown by the results of the survey of PAR beneficiaries in Santa Cruz. 69% of respondents affirm that they know of ways for reporting issues or sharing opinions and suggestions regarding the project. When asked to identify those ways, the majority point to in-person interactions with PAR representatives. Figure 6 PAR survey: Awareness of existing reporting mechanisms 11 Only a handful of beneficiaries (14%) in the area where OnTrack was piloted are aware of the existence of a digital engagement mechanism. Some 16% say they know that an SMS reporting mechanism is available and 9% that an online feedback platform exists. These percentages might be even smaller in reality if we take into account that 32% of respondents chose the option “call a free phone” which has not actually been made available to PAR beneficiaries in this area. Those who reported awareness of an SMS and/or online feedback mechanism were asked to say how they had heard about it. Some 48% (29 respondents) said that they had heard of OnTrack from a person who works with the project, 36% (22 respondents) that they participated in an OnTrack training, and 36% (22 respondents) through a friend, relative, neighbor or colleague. Indicator 2.2: Beneficiary usage rate The recorded usage rates of OnTrack are low. There have been only 250 tickets (registered reports) since the launch of the PAR platform in April 2013. Of this number, the majority were tests sent during training sessions by participants or trainers. Only 4 tickets seem to respond to real reports sent spontaneously by project beneficiaries outside the training sessions. In PAR, the explanation for the low use of the platform by all types of stakeholders was that OnTrack was implemented at the end of PAR 1 and PAR 2 is only just getting underway. There was a general sense that the system would have been used more if it had been implemented at the beginning of the project. The current plan is for the OnTrack system to be implemented as part of PAR 2 with the expectation that this will lead to an exponential increase in its use by beneficiaries. Since the launch of the PBCV platform, 58 tickets were received, all seemingly sent by users during training sessions.1 Since its launch in October 2013, some 32 tickets were received on the Nepalese platform. Almost all these were received during the orientation sessions. With the lack of a search function and an easily navigable back-end for the platform, it is difficult to know the exact number of photos, stories and suggestions contributed by beneficiaries to the online platform. However, by browsing the three platforms, it appears that there has been little use of these functions – and this was mainly during the training and orientation sessions. 1 For the Bolivian platforms, Keystone was given access to the back end and could verify this information directly in Spanish. For the Nepalese platform, Keystone could not verify the information in Nepalese language and we rely here on information reported by OnTrack staff and the PIU. 12 Table 2 Number of platform visitors Visitors of OnTrack platforms Bolivia Nepal (PAR and PAF PBVC)2 March 2013 (launch of PBVC 209 N/A platform) April 2013 (launch of PAR platform) 748 N/A May 2013 322 N/A June 2013 771 N/A July 2013 (official visit of President 1,028 N/A Kim in Bolivia) August 2013 647 N/A September 2013 536 N/A October 2013 (launch of PAF 626 259 platform) November 2013 528 48 December 2013 368 48 Average visitors per month 578 118 Total visitors from launch to end 5783 355 2013 Source: Monthly website analytics reports Table 3 provides data on website analytics through the end of 2013 and indicates minimal traffic on the 3 platforms. In Bolivia there is a spike in traffic around the time that World Bank President Jim Kim participated in an event for the launch of OnTrack in Cochabamba in 2013 and in Nepal around the time of the launch with the participation of the Chief Secretary. The survey of PAR beneficiaries in Santa Cruz, Bolivia offers some important insights on the likelihood of use of ICT enabled feedback mechanisms such as OnTrack and the motivations for the use of such mechanisms. 2 The data for the two Bolivian projects cannot be disaggregated because, while they have distinct user interfaces, the back end is shared. 13 Figure 7 PAR survey: Likelihood of reporting via mobile phone Although 66% of beneficiaries express the view that they would be either likely or very likely to use their mobile phones to report problems with a project, if this option was available to them, only 11% report having actually done so in relation to PAR (see figure 8). When asking this question, we tested whether the promise of anonymity influenced peoples’ readiness to use a reporting mechanism via mobile phone. Our experiment showed that for the beneficiary population in general, anonymity is not a primary consideration. However, it does seem to influence the likelihood of using such a reporting mechanism in the case of women and those who have not completed secondary education. 14 Figure 8 PAR survey: Types of participation Beneficiaries clearly favor offline and traditional participation modes. They show themselves however very open to digital engagement with only 22% and 24% respectively saying they would never use an SMS or online feedback mechanism. This proportion is significantly higher among beneficiaries who have not completed primary education (42% and 43% respectively). Figure 9 PAR survey: Preferred ways for reporting 15 Asked about what would be their preferred way to submit problem reports, the in-person option is significantly more popular amongst beneficiaries with a lower education level (76% of those who have not completed secondary education choose this option vs. 61% of those who have secondary or higher education). In the same way, online and SMS options are more popular with those beneficiaries that have secondary or higher education. Figure 10 PAR survey: Beneficiaries' self-efficacy While expressing openness towards using different channels of participation, including digital ones, the perception of self-efficacy by PAR beneficiaries is quite low. They tend to believe that they have little understanding of political issues and limited capacity to influence governmental actions. These perceptions link to their motivations for using a digital engagement mechanism. It is striking that among the reasons they mention that would potentially motivate them to use such a mechanism, very few people say they believe that engaging in the process will lead to actual improvements in the project. 16 Figure 11 PAR survey: Motivations for Digital Engagement Indicator 2.3: Responsiveness to feedback Since limited beneficiary feedback was received, it is hard to assess the level of responsiveness of the PIUs to the feedback they did get. It is important to note that the current OnTrack system sends an automatic reply to those submitting a report with a ticket number that can be used by the beneficiary for follow-up. Beneficiaries are informed whether the status of their ticket is open or closed, but there is no reporting back to the beneficiary about proposed actions to address the concerns they raise – nor is this contemplated in the current set up. However, it is contemplated in the customization requests presented to the platform developers and is included in the ToR for a local vendor to develop the platform for the PAR project in Bolivia. During focus group discussions in Nepal, at least 2 instances came to our attention of reports submitted via SMS by beneficiaries that received no response. The members of the PIU present in the discussion said technical issues accounted for the lack of response. When asked about their expectations about what should happen once they send a report, beneficiaries repeatedly stated that they expect to receive a response indicating how it will be addressed and a timeframe for resolution. 17 Figure 12 World Bank staff survey: perceptions of responsiveness to feedback Figure 12 shows the perceptions of surveyed World Bank staff on a scale of 0-10 on the following 3 statements:  The use of SMS and internet technology by OnTrack enables providers of public services to respond to user feedback in a practical and timely way  OnTrack’s design ensures that users who provide feedback on public services always receive a response  In my experience, providers of public services involved with OnTrack always respond to feedback received by users through OnTrack While World Bank staff involved in the OnTrack pilots tend to see ICT’s potential in enabling a timely response to feedback provided, they are skeptical about whether OnTrack’s design ensures a response. Those that express an opinion tend not to have much confidence in the PIUs’ responsiveness to feedback. Indicator 2.4: Timeliness of response to feedback Due to the low levels of use, it has not been possible to assess the timeliness of response. In the World Bank staff survey, respondents were asked to indicate whether they knew of at least one example where a public service provider has given a timely response to feedback provided by a user through OnTrack. Some 4 respondents said “yes” but were unable to provide a concrete example. Indicator 2.5: Fix rate Here too, due to the low levels of use of the feedback system, it is not possible to meaningfully assess this indicator. Only a single instance of a ‘fix’ reported by beneficiaries came to our attention through staff surveys and interviews. This positive example was in the PAR project where a beneficiary reported that forage for livestock was not arriving on time and the project team took action to resolve the issue. 18 3. Process efficiency Indicator 3.1: OnTrack quality of advice and support The PIUs were consistently appreciative of the responsiveness of the OnTrack team and the support received. Continuous support was provided by the OnTrack team, including a series of training sessions for PIU staff. An example of this responsiveness was the decision to restructure the OnTrack model to build and host the technology locally rather than in the US. By contrast, the CMUs/TTLs are less satisfied. The main complaint is that there have been too many surprises as the project unfolded with respect to hidden costs, unexpected length of OnTrack implementation process (for example, to customize for the local realities), and requirements for specialized staff to maintain the system. These are some of the main reasons that led to the decision to build the platforms locally in future. These findings apply to all 3 pilots. Indicator 3.2: PIU quality of service Under this indicator we intended to examine the quality of service provided by the PIUs in relation to the implementation of the OnTrack feedback system. However, due to the limited use of the platform in all the pilot sites, it is hard to assess this aspect. In previous sections we provided observations on two key aspects regarding this indicator: responsiveness to the feedback provided by users (Indicator 2.3) and the quality of outreach and awareness raising activities regarding OnTrack (indicator 2.1). Indicator 3.3: PIU/public service providers’ capacity When asked about what kind of capacity elements are necessary for the PIUs to effectively implement and manage the feedback system, both the PIUs and World Bank staff in all pilot sites focus on the technological issues. They mention that in-house technical capacity - i.e. a member of staff dedicated to handling the IT aspects - is needed to run and maintain the online platform and SMS system. Only PBCV in Bolivia said it has sufficient in-house capacity. In both PAR and PAF, where the decision was taken to localize the platform, intensive support is required by the OnTrack team to help the PIU draft the ToR and select the appropriate local vendors. Another capacity element brought forward in interviews is the need for specialist communications staff to focus specifically on outreach and awareness raising. During our visits we were able to confirm that all 3 pilot projects have in place a communications officer that can lead on this aspect. 19 One neglected issue is strengthening institutional capacity so that PIUs are able to analyze and integrate the feedback into their activities. There are no clear workflows at any of the project sites about how to deal with feedback. An analogy was drawn by a World Bank staff member regarding the PIU in the peri- urban sanitation project in Zambia. It has a customer relations person already in place — unrelated to OnTrack – but there is no mechanism to respond to customer complaints. Both PIUs and World Bank staff raise a hypothetical concern about the PIUs not having sufficient capacity to deal with large flows of feedback. As indicated, this is not the case at present but they anticipate it will happen if the system is completed or expanded. Indicator 3.4: New needs identified by the implementation of the program Figure 13 World Bank staff survey: perceptions on OT effectiveness for identifying new needs There seems to a belief among some World Bank staff that OnTrack can enable the identification of new needs by the beneficiaries of a project. While in theory this could be the case, we were unable to find any instances where such identification has taken place. This may be due to the limited use of the system by project beneficiaries. 20 Indicator 3.5: Human, material and financial resources invested by different constituents Table 3 OnTrack costs in USD July 1, 2012- Feb 28, 2014 OnTrack costs July 1, 2011 - Feb 26, 2014 OnTrack team Travel Platfor All m Oth FY Fiscal WB Short term OnTrack Boli Nep Zam Gha Oth travel Develop er total Year staff consultants team total via al bia na er total ment 66,1 3,34 12,4 11,0 123,1 FY12 70 14,287 80,457 6 49 - - 78 26,874 15,250 536 17 167, 29,3 39,6 20,5 25,2 1,30 1,46 465,0 FY13 309 157,340 324,649 07 13 34 10 8 115,972 22,968 2 50 121, 3,09 1,95 (736 1,74 95,8 506,4 FY14 129 229,080 350,209 4 8 - ) 5 6,061 54,297 74 42 Total 354, 35,7 54,0 20,5 24,4 14,1 97,8 1,094, s 609 400,706 755,315 47 20 34 74 32 148,907 92,515 72 609 The core OnTrack team comprised the TTL, the Technical Lead, and one Engagement Lead for each of the countries where OnTrack was piloted. A dedicated consultant provided communications and general support. Occasional support was also provided by temporary consultants. For the period for which we were provided with detailed financial information (July 1, 2011 - Feb 28, 2014), the total costs related to the OnTrack team were USD 755,315. During this period, approximately 26 international trips were undertaken. The total costs associated to travel were USD 148,907. The cost for the development and maintenance of the OnTrack platform was USD 92,515. The total cost for the program is USD 1,094,609. While not a measure against which a pilot should be assessed, it is interesting to note that the cost ratio per ticket generated by the system has been USD 3,219 and per website visitor USD 178. In the World Bank staff survey, only 2 out of 17 staff surveyed considered that the level of human and financial resources invested in OnTrack to be proportionate to results. In interviews with different levels of staff, the cost of maintenance of the platform (USD 60,000/year/project) came up repeatedly as a point of contention. No answers were forthcoming. We are unable to provide an estimate of the different resources invested by the PIUs, as this information was not available or forthcoming. In interviews, PIUs diplomatically indicated that they felt that the level of financial and human resources deployed was proportionate to the results obtained. They did volunteer, however, that the cost for the maintenance of the platform was too expensive and that this, along with the need to adapt to local specificities, was the reason for which they took the decision to develop and host the platform locally. An estimate provided for the local development of the platform for PAR in Bolivia was USD 40,000 (including management of the platform) and for PAF in Nepal USD 10,000. These costs reflect only the development of the platform. 21 Costs related to management, communications and engagement have not been calculated. 4. Monitoring of behavioral changes Indicator 4.1: Existence and implementation of a monitoring and learning system OnTrack does not yet have a system to monitor KPIs, or to assess and take corrective action on the basis of a reliable and continuing stream of indicators. During interviews we were informed by staff that neither the data on the number of tickets generated and responded to, nor website traffic has been systematically tracked or analyzed. As noted previously, much of this data cannot be easily manipulated because the platform does not provide the necessary functionality. Most analysis is currently carried out by manual extraction of the data. Indicator 4.2: Quality of relationships between different constituents (including management of expectations) The general feeling about relationships is mildly positive and it is reasonable to consider relationship quality as an asset for OnTrack, especially in relation to the PIUs. Surveyed World Bank staff give an average rating of 7.5 out of 10 on the extent to which relationships between different WB staff involved in OnTrack and the PIUs are based on mutual understanding and respect. PIUs believe that their opinions are taken into “Lots of communications at account, even if implementation of OnTrack first, but then no follow up.” produced surprises on a continuous basis. The Nepal PIU Nepal PIU called for more follow up. The more negative views expressed in Zambia and from TTLs and CMUs were all connected to the tendency to over-promise. In one case (PBCV in Bolivia) there was frustration about the inability to conclude a formal agreement with the Bank on OnTrack deliverables. 22 5. Sustainability of OnTrack3 Indicators 5.1 & 5.2: Buy in by PIU/public service providers and top-level government authorities/Perceptions on the value of project beneficiary feedback by PIU/public service providers In interviews with WB staff and from our observation in our interactions with the PIUs in Bolivia and Nepal, it became clear that OnTrack was effective in engaging clients and in creating genuine excitement about building a citizen feedback mechanism. The OnTrack team reports changes in the attitudes of the PIUs towards transparency. Notably, they point to the willingness of the PIUs to share all key project information, including financial aspects to populate the mapping element of the OT platform. Openness to sharing information publicly was not a given at first and has been the product of continuous engagement and encouragement by the OnTrack team. For Zambia we do not have enough information to make an assessment, however it has come to our attention that lack of buy-in from the PIUs (and the higher government authorities on which they depend) was the main reason the project ground to a halt. Certain nuances regarding the commitment of the PIUs to take action on the feedback relate to opinions expressed during interviews to the effect that not all feedback is valid. Interlocutors point to a high level of interpretation, with project beneficiaries not always in a position to understand the projects and provide ‘good quality’ feedback. In the PAR project, senior management indicated that they did not want OnTrack to become the “wailing wall”. Rather they saw it as a good platform for beneficiaries to share their stories and learn from each other, thereby minimizing the importance of the reporting and suggestion-making functions of the system. All PIUs and authorities consulted for this evaluation acknowledge finding great potential in OnTrack for monitoring the implementation of the projects/services and identification of issues that need attention from the provider. It is less clear that they understand their responsibilities to respond to feedback and take corrective actions. 3 In addition to the findings on the specific indicators presented in this section, indicators 2.2 (beneficiary usage rate), 3.1 (OnTrack quality of advice and support), 3.2 (PIU quality of service) and 3.3 (PIU/public service providers capacity), already analyzed in previous sections are also important for the analysis of the sustainability of a mechanism such as OnTrack. 23 Indicator 5.3: Buy in by TTLs and CMUs Operations staff consulted for this assignment expressed high interest in beneficiary feedback mechanisms and specifically in OnTrack’s potential. They find the concept compelling and see its potential in making a significant contribution to the outcomes of their projects in way that is complementary to traditional monitoring systems. This opinion was echoed in a staff meeting held in the WB office in La Paz, where the evaluators presented their approach on citizen engagement and reviewed together the associated opportunities and challenges. Regarding the specific commitment of the TTLs of the projects on which OnTrack was piloted, the picture is more nuanced. It was challenging to secure even a 45- minute interview with them to get their opinions on the pilot. They were unaware about the details of the pilot’s design and implementation, and all of them expressed frustration about the OnTrack team’s failure to provide accurate costing for the maintenance of the platform in advance. They expressed a commitment to the basic principles and logic of citizen feedback and to the success of OnTrack. However, as a senior manager at WBI said: “If there was true commitment, OnTrack would not need to be flying people around, the TTL would have done the work and gotten local consultants”. Indicator 5.4: Clear definition of OnTrack’s role going forward The OnTrack team conceived OnTrack as a complete program for enabling citizen feedback, not as a mere feedback mechanism and an online platform. However, this message has failed to get across. The point of view of other WB staff and PIUs is almost exclusively focused on the technological aspects. The fact that OnTrack itself took on the development of the platform on which the system is based has contributed to this perception. In the context of the restructuring of the World Bank into global practices, OnTrack is included under the “Digital Engagement” chapter of the Governance Global Practice. Its role is becoming more clearly defined as an advisory service to projects/PIUs for developing and implementing ICT-enabled citizen feedback mechanisms. This is a departure from the software development role previously assumed. 6. Replication of OnTrack Indicator 6.1: Interest expressed by other actors in WB WB staff indicated in interviews that one of the main achievements of OnTrack was to raise awareness and create excitement about CE in the WB. Some interlocutors indicated that there is broader – although non-specific -- interest in the approach from elsewhere in the World Bank. But, they say, cost issues are a ‘turn off’. They want to see the results of the pilots before taking things further. 24 At the country level, the PIUs indicate that they have not made a major effort to disseminate OnTrack. It is still seen as a pilot and although they report that the idea generates general interest in their countries, that interest has yet to lead to concrete uptake at this stage. Indicator 6.2: Completeness of feedback system as product/Level of readiness of feedback system As mentioned, while the rhetoric about OnTrack emphasizes that it is a program, so far efforts have been mainly concentrated on the technology – and even this aspect of the approach remains incomplete. Consultations with a range of different stakeholders and our own observations suggest the following technology issues still require attention:  Development of local platforms (underway)  General slowness of the platform  Making the user interface more user friendly  Large volumes of spam on Bolivian platform  Voice integration in Nepal (underway)  Lack of search function  Lack of notification mechanism when report is received Other aspects that remain incomplete:  A clear results framework and theory of change  A monitoring and evaluation plan  A strategy for engaging with TTLs and other operations staff, with clear roles and responsibilities  A strategy for engaging with the PIUs  Costing of the different aspects of the program  The design of the feedback loop itself. There is no clear description, workflow or guidelines about how to the received feedback should be treated, analyzed and used for taking corrective action and closing the loop with the users and project beneficiaries. 25