Performance Evaluation of GFDRR-UK Aid Challenge Fund Open-Source, DIY Remote Weather Stations in Sri Lanka February 28, 2018 TABLE OF CONTENTS Acknowledgements 2 I. Executive Summary 3 II. Background 4 III. Purpose and Evaluation Questions 5 IV. Methodology 6 V. Limitations 7 VI. Findings 7 VII. Conclusions 18 VIII. Recommendations 19 Appendix A: Challenge Fund Logic Model 20 1 ACKNOWLEDGEMENTS During my three years of close work on monitoring, evaluation, learning and design (MELD) of disaster risk information projects with GFDRR, I have learned that risk information comes in many forms and experiences differing degrees of acceptance and uptake in different environments. It has been truly fascinating to be able to delve deep into the issues surrounding why these differences occur, and apply that learning to Challenge Fund implementation. I have been amazed at the rapid spread of many risk information technologies supported by GFDRR and UK Aid as they innovate, inspire and influence others to increase disaster resilience around the world. In looking at the path from innovation to uptake of one such technology – open-source, DIY remote weather stations in Sri Lanka – I was fortunate to be aided by the kind involvement of many evaluation participants. Chief among them were Government of Sri Lanka officials in the Ministry of Irrigation and Water Resources Management’s Irrigation Department. I am deeply grateful to Irrigation Department officials throughout the country for their time, sharing of experiences and insight into the utility of such devices in their country. I am also grateful to staff at the International Water Management Institute (IWMI) who have been involved with the innovation and spread of this device since late 2013 for their time and willingness to share their experiences with remote weather stations in Sri Lanka. Academics at the universities of Moratuwa in Sri Lanka and Applied Sciences and Arts of Italian Switzerland, as well as private sector actors, NGOs, international donors, and additional Government of Sri Lanka officials in the Coordinating Secretariat for Science, Technology and Innovation, Department of Meteorology and Disaster Management Committee offered invaluable insight into the utility of remote weather stations in Sri Lanka, and beyond. Dr. Yann Chemin, who originally innovated open-source, DIY remote weather stations for use in Sri Lanka while working for IWMI, provided background and contact information for many people in Sri Lanka who currently employ similar technologies to increase resilience; without his assistance, this evaluation would not have discovered the depth of evidence it did. World Bank staff, both in Sri Lanka and around the world, provided valuable insight for this evaluation; I am grateful for their assistance and support. Of course the views expressed in this document do not reflect the official views of any person or organization. They are those of the evaluator only. All errors contained herein are mine alone. This publication was produced at the request of the Global Facility for Disaster Reduction and Recovery’s (GFDRR’s) Innovation Lab as a deliverable under the UK Aid Science for Humanitarian Emergencies and Resilience project (SHEAR). It was prepared by Tracy C. Thoman, Ph.D., Independent Consultant and GFDRR Senior MELD Specialist. 2 EXECUTIVE SUMMARY This performance evaluation using Case Study design and mixed methods focused on open- source, DIY remote weather stations (RWSs) in Sri Lanka. First innovated under a project implemented by the International Water Management Institute (IWMI), this risk information technology was subsequently the focus of a Challenge Fund (CF) project, also implemented by IWMI. The Challenge Fund is a joint initiative of GFDRR and UK Aid and seeks to bridge the gap between technology and on-the-ground user needs in the field of disaster risk identification, thereby building greater disaster resilience. The scope of this evaluation included, but was not limited to, the IWMI Challenge Fund project; it also looked at broader efforts involving this technology in Sri Lanka. It addressed the following five theory-based evaluation questions: 1. To what extent are CF Logic Model output-level results evident in Sri Lanka? 2. To what extent are decision makers regularly accessing remote weather stations in Sri Lanka? 3. To what extent is there increased investment in remote weather stations in Sri Lanka? 4. Are the results evident sustainable? Is yes, why? If not, why not? 5. Are there any unintended consequences – positive or negative – as a result of the introduction of remote weather stations in Sri Lanka? Findings showed that all five output-level results are clearly evident in Sri Lanka. Namely, the RWS experienced a process of continual development and improvement from its innovation to the time of this study; partnerships with six local organizations were formed or strengthened; the tool was co-developed with ten beneficiaries; the project was gender-informed because a gender analysis was conducted and gender gaps were identified and communicated; 144 beneficiaries were trained; and $741,900 was leveraged. Decision-makers are regularly accessing RWSs in Sri Lanka. Specifically, five Irrigation Department officials in the pilot location of Anuradhapura are regularly accessing these devices. In addition, officials in the Irrigation Department Headquarters, many regional offices, the Meteorology Department, Lanka Rainwater Harvesting Forum and its funder USAID, the Coordinating Secretariat for Science, Technology and Innovation (COSTI), the University of Moratuwa, and A&T Labs are investing in the further development and use of this device. There were 4,832 unique views of the IWMI website that contains a wealth of information about remote weather stations, including how to build and maintain them. The evidence gathered during this evaluation strongly supports the existence of a positive cycle of sustainability of logic model results surrounding RWSs in Sri Lanka. The following unintended positive results were discovered: increased coordination and confidence, and the development of local champions and communities of practice (COPs). Recommendations include support for a RWS COP workshop in Sri Lanka; conducting a hands- on training for Irrigation Department officials; following up with partners in Sri Lanka regarding identified gender gaps; making explicit the requirement to mention funding sources of CF projects; inclusion of both a strong local champion and communities of practice in the CF Logic Model; and requiring Implementing Partners to keep careful records. 3 BACKGROUND Challenge Fund The Challenge Fund aims to connect innovation to local contexts to help better identify changing climate and disaster risk and enable more effective decision-making to build resilience. There have been two phases of the Challenge Fund to date. Phase I funded 15 projects in over 20 countries throughout Africa, East Asia and the Pacific, and South Asia. These projects focused on new approaches to risk information, ranging from flood simulation technology, to mobile app building, to weather station pilots and film production.1 Phase II of the Challenge Fund supported seven of these 15 projects to further develop their risk information tools, with an overall goal of decision maker access and use of their tools in target countries. All Phase II projects are scheduled to close by the end of February 2018. This evaluation seeks to capture lessons learned from one of the reportedly successful projects in both phases, that of the RWS in Sri Lanka. The underlying logic of the Challenge Fund at the start of the initiative was: • If new data/tools/approaches to support disaster risk management decision making are developed in response to local demand; and • If there is improved capacity among the community to develop the new tool; and • If gender is considered in risk identification, outreach and capacity development plans; and • If there is improved capacity of local beneficiaries to apply the new tool; and • If resources are leveraged from the private sector; assuming • The Challenge Fund delivers high-quality, demand-led data/tools/approaches; then • New high quality and relevant data/tools/approaches will be accessed by decision makers. • If new high quality and relevant data/tools/approaches are accessed by decision makers, assuming • Challenge fund projects make products available to the right people who have the capacity and demand to use them, then • There will be greater and more effective investment in disaster resilience, preparedness, response and recovery in target countries. This theory of change is depicted in Annex A, by the current Challenge Fund Logic Model. The model has been adapted over the past two years as learning from Challenge Fund implementation has been integrated.2 Open-Source, DIY Remote Weather Stations in Sri Lanka Climate change-related weather irregularities and resultant extreme weather events are becoming increasingly common in Sri Lanka. For example, the past decade has shown an increased incidence of flooding. As recently as May 2017, more than 200 people died as the result of severe flooding in Sri Lanka. Decision makers in Sri Lanka are very much interested in 1 https://www.gfdrr.org/winners-2015-challenge-fund 2 Please see Annex A for the current CF Logic Model. 4 exploring technologies that can assist in early warning for disasters, with the hope to minimize damage. Recognition of Government of Sri Lanka demand for one such device, the open-source, DIY remote weather station (RWS), began in late-2013 during a project implemented by the International Water Management Institute (IWMI). Although the project was largely focused on water storage in two river basins in Sri Lanka, Irrigation Department officials expressed their deep concern to project managers that there was not enough warning about high-intensity rainfall for proper operation of reservoirs to avoid floods. One district that had this particular concern, and was also within the geographic area of the project, was Anuradhapura. Floods occur along river basins in Anuradhapura ever year. Dr. Yann Chemin, a scientist working for IWMI at the time, was tasked with creating a low-cost weather station to assist irrigation engineers in Anuradhapura with reservoir management.3 During that project, which ended in late 2015, two open-source, DIY remote weather station prototypes were developed and installed for testing; they were intended to be used to gather real- time data on rainfall, temperature and humidity upstream from the river basin. Training for irrigation engineers in Anuradhapura was also conducted during that project; and another three weather stations were assembled and passed to local irrigation department engineers. Phase I of the IWMI Challenge Fund project began in late 2015 as this project was ending. It is clear that existing strong and positive relationships between IWMI staff and Irrigation Department officials in Sri Lanka were important factors in the implementation of the IWMI Challenge Fund project, both during Phase I and Phase II, to further develop these remote weather stations. Another important relationship, started during the first IWMI project in Anuradhapura, was with Sri Lanka’s Coordinating Secretariat for Science, Technology and Innovation (COSTI) – which was established in 2013, just before the introduction of RWSs in Anuradhapura. COSTI hosted a roundtable discussion in November 2014 on developing a national climate observatory system in Sri Lanka, during which IWMI staff played a significant role. During the earlier (2013-2015) IWMI project, Dr. Chemin was also working as a senior lecturer at the University of Moratuwa, the highly-regarded and largest technical university in Sri Lanka today. During his tenure there, he taught many students and developed new curricula in open GIS, including open-source, DIY remote weather stations. Also during his spare time, he worked with a local NGO, Lanka Rainwater Harvesting Forum (LRWHF), to build capacity on RWSs for its work with rural schools across the country. PURPOSE AND EVALUATION QUESTIONS The purpose of this evaluation is to continue to learn from Challenge Fund implementation so that future projects involving disaster risk information can use this learning as part of the evidence base for project design. Given that this is a theory-based evaluation, the focus of the evaluation is viewing the case of the development and spread of RWSs in Sri Lanka through the 3 Existing weather stations in the country could cost USD $10,000 or more and maintenance was extremely difficult. 5 lens of the existing Challenge Fund Logic Model. Simply stated, the purpose of this evaluation is to determine the relevance of the logic model in the case of RWSs in Sri Lanka. In looking at why the model and this case are similar or different, modifications to the logic model for future projects will be suggested. The most immediate users of this evaluation report are GFDRR Innovation Lab Challenge Fund administrators, as well as UK Aid SHEAR officials. In addition, the report could be used in other disaster risk information project design efforts for projects that focus on access and use by decision-makers to increase resilience. This is a utilization focused-evaluation in that the funders requested this evaluation as one of many learning data points to improve implementation of efforts to bring risk information technologies to scale around the world.4 Its findings, conclusions and recommendations are intended to be practical and relevant for GFDRR and UK Aid Challenge Fund decision makers. The evaluation questions are: 1. To what extent are CF Logic Model output-level results evident in Sri Lanka? 2. To what extent are decision makers regularly accessing remote weather stations in Sri Lanka? 3. To what extent is there increased investment in remote weather stations in Sri Lanka? 4. Are the results evident sustainable? If yes, why? If not, why not? 5. Are there any unintended consequences – positive or negative – as a result of the introduction of remote weather stations in Sri Lanka? METHODOLOGY The above evaluation questions are based upon the theory which underpins Challenge Fund implementation. The design, chosen by the funders for this evaluation, is a performance evaluation using Case Study design and mixed methods. The case, for the purposes of this evaluation, is that of open-source, DIY remote weather stations in Sri Lanka. This is one of 15 cases funded by the Challenge Fund in Phase I and one of seven funded in Phase II. Methods employed were key informant interviews (KIIs) using semi-structured interview protocols, observation, document and website review, and thematic analysis of data gathered. Sampling of KIIs was based upon contacting all of the 45 names provided by stakeholders involved in RWSs in Sri Lanka. KIIs were conducted with 36 people involved in this technology in Sri Lanka during the months of November and December, 2017 via Skype and in person. This represents a high response rate of 80%. KIIs were conducted with Government of Sri Lanka officials in the ministries of Irrigation and Water Resources Management and Disaster Management, including the latter’s Department of Meteorology; multilateral and bilateral partner officials; NGO staff; private company employees; academic institution staff; and former staff of these organizations in Sri Lanka who were active in development and use of these devices. Project documents, open source remote weather station community of practice documents and 4 Please see other Challenge Fund learning documents: https://www.gfdrr.org/sites/default/files/publication/CF- NIWA-Case-Study.pdf; and https://www.gfdrr.org/sites/default/files/publication/Challenge-Fund-Phase-I-Results- and-Learnings.pdf. 6 websites, NGO and private sector documents and websites, academic publications, Government of Sri Lanka websites and the like were reviewed for this evaluation. LIMITATIONS Evaluation findings are only as good as the evaluation questions, design and methods allow them to be. For this evaluation, every effort has been made to minimize biases and threats to internal and external validity of the data. In a purposive sample such as that employed here, key informants were chosen because of their direct knowledge and experience with RWS in Sri Lanka. However, the evaluator did not have access to the entire population of key informants due to data limitations. In addition, those who responded to requests for an interview may not share the same views and experiences as those who did not respond to evaluator requests. Given the 80% response rate for this evaluation, and success in interviewing respondents from all training sessions and relevant communities of practice over the four years of work with these devices in Sri Lanka, these biases are somewhat lessened. It should also be noted that the open nature of the evaluation questions and semi-structured interview protocols did not unnecessarily limit the information provided for this study. FINDINGS The findings presented below are arranged by evaluation question for ease of reading. 1. To what extent are CF Logic Model output-level results evident in Sri Lanka? There are five outputs for projects to achieve in the CF Logic Model. Evidence showed that all five output-level results exist in the case of RWS in Sri Lanka. Namely: ü A new tool, the RWS, was developed in response to local demand in Sri Lanka; ü In terms of improved capacity among the community to develop the RWS, six partnerships were developed or strengthened with local partners – COSTI, Irrigation Department, University of Moratuwa, Lanka Rainwater Harvesting Forum, Meteorology Department and A&T Labs; and ten beneficiaries were involved in co-development and improvement of RWSs in Sri Lanka; ü Gender was considered through a gender analysis conducted by IWMI, which identified three gender gaps in the populations downstream from project reservoirs; these gaps were then communicated to Government of Sri Lanka counterparts; ü A total of 144 beneficiaries were trained in RWSs by both the IWMI project, as well as Dr. Chemin on his own time; and ü A total of USD $741,900 was leveraged by both IWMI and Dr. Chemin’s work on RWSs in Sri Lanka. When these results are viewed against the milestones for all 15 Challenge Fund projects over both Phases I and II, it is clear that this case of open-source, DIY remote weather stations in Sri Lanka is contributing far more than its fair share of Challenge Fund results.5 The above results 5 Challenge Fund milestones can be found in the Challenge Fund Logic Model in Annex A. 7 do not show why or how they were achieved, or what they mean for the lasting effect of Challenge Fund investment in RWSs in Sri Lanka. The remainder of this section on Findings seeks to address these issues to paint a fuller picture of RWSs in Sri Lanka. Similar with other successful Challenge Fund projects6, this study found that the key to success in achieving these expectation-exceeding output-level results for RWS in Sri Lanka was strong local relationships. Existing relationships and the strong partnerships that can result from these relationships were instrumental in the ability to achieve output-level results such as conducting the gender analysis, training 144 beneficiaries, leveraging USD $741,900, and innovation and development of the RWS together with beneficiaries in Sri Lanka over a period of four years. ü A new tool, the RWS, was developed in response to local demand in Sri Lanka. As mentioned in the Background section of this report, the RWS originated during a project funded and implemented by IWMI, beginning in late 2013. IWMI has had a strong and positive relationship with the Irrigation Department in Sri Lanka for many years. The first RWS in Sri Lanka was created in direct response to a request from an existing partner – in this case Irrigation Department officials in Anuradhapura – and experienced continual improvement through today. During the piloting of these devices in Anuradhapura over the past four years, many issues were discovered and addressed by IWMI, as well as its partners in the Irrigation Department. For instance, small animals found their way into the weather stations and disrupted proper functioning. The heat and humidity of Sri Lanka made some of the earlier wires brittle and non- functioning. Over time, many wires and boards were substituted with a simpler design, limiting these and other problems that had surfaced. Existing relationships and the strong partnerships that can Outside of the IWMI project, those influenced by result from these relationships Dr. Chemin – either while he was a lecturer at the were instrumental in the ability University of Moratuwa, working in his spare time to achieve output-level results with LRWHF, communicating with colleagues in such as conducting the gender academia around the world, or through the Open Monitoring Systems Working Group he helped analysis, training 144 establish in 2016 – also made continual beneficiaries, leveraging USD improvements to their versions of the device. For $741,900, and innovation and example, as a result of a long-term relationship he development of the RWS together had with another academic in the University of with beneficiaries in Sri Lanka Applied Sciences and Arts of Italian Switzerland over a period of four years. (SUPSI), Dr. Chemin helped draft the proposal that was funded for a project together with the University of Moratuwa to develop and pilot weather stations in several countries, including Sri Lanka. Although the remote weather station developed in that project differs from those currently being used in Anuradhapura, university professors first accessed RWS materials and resources found on IWMI’s website7 in order to further develop RWSs for their needs. 6 https://www.gfdrr.org/sites/default/files/publication/CF-NIWA-Case-Study.pdf 7 http://www.iwmi.cgiar.org/resources/mobile-weather-stations/ 8 While the word ‘failure’ often carries a negative connotation in everyday life, failures remain vital for learning. Those involved in the development of RWSs are happily transparent about the fact that it took a few years to work out bugs in the hardware to make this a working system that is sustainable. Many RWS stakeholders credit the Challenge Fund for supporting a project that allowed this tool to be continually adapted to the needs and environment in Sri Lanka. ü In terms of improved capacity among the community to develop the RWS, six partnerships were developed or strengthened with local partners – COSTI, Irrigation Department, University of Moratuwa, Lanka Rainwater Harvesting Forum, Meteorology Department and A&T Labs; and ten beneficiaries were involved in co-development and improvement of RWS in Sri Lanka. When asked open-ended questions about their experiences with RWS in Sri Lanka, 100% of the 31 respondents with direct knowledge of RWS interviewed for this evaluation credited Dr. Yann Chemin for building capacity among the community to develop this device. Dr. Chemin worked for IWMI during both phases of its Challenge Fund project; he maintains positive relationships with local partners and is credited for always being available to help others who are working on these DIY devices. As mentioned, IWMI maintains positive working relationships with the Department of Irrigation. It also has strong relationships with COSTI officials, which were instrumental in developing relationships with the Meteorology Department in Sri Lanka. Such inroads with local partners are key to eventual uptake and sustainability of RWSs in Sri Lanka. All partners were involved in the development of RWSs in Sri Lanka, either by modifying the piloted devices for use in Anuradhapura, or developing their own weather stations based upon the initial pilot device. IWMI’s and Dr. Chemin’s strong relationships with local partners were instrumental in this co-development of RWSs in Sri Lanka. Those who have been interested in, and worked to develop, these instruments are local champions, who continue to build capacity in others. For example, the Divisional Irrigation Engineer in Anuradhapura is now seen as a strong local champion for RWSs in Sri Lanka and often trains others in their assembly and use. Together with Dr. Chemin, a former student at the University of Moratuwa assisted a local NGO, LRWHF, to refine RWSs for use in schools in Sri Lanka. He subsequently worked as an intern with IWMI and is now building other risk information tools as a graduate student in Japan. LRWHF has very strong capacity in RWSs and contracts with former University of Moratuwa students whose technology firm, A&T Labs, builds RWSs for schools and others in the country. ü Gender was considered through a gender analysis conducted by IWMI, which identified three gender gaps in the populations downstream from project reservoirs; these findings were then communicated to Government of Sri Lanka counterparts. According to the 2016-2021 GFDRR Gender Action Plan, operations are considered to be “gender informed” if a gender analysis or gender impact is either taken into consideration during project design or is mentioned as an expected outcome. Operations are considered to have “gender actions” when analyzed as being “gender informed” while additionally having specific components/activities that seek to minimize mentioned gender gaps relevant to the project’s development objective(s). This project can be considered to be both gender informed and having gender actions. 9 With encouragement and guidance from the Challenge Fund team, IWMI successfully conducted a gender analysis in four out of the eight villages that were affected by severe flooding in December 2014 in Anuradhapura. According to IWMI project reporting, three gender gaps were identified: 1. Females may receive less food than males when food is scarce; 2. Females reported a loss of privacy and security at the camps while waiting for the water levels to recede; and 3. Respondents reported making their children—especially female children—skip school due to the longer commuting time, until the family moved back to its home in their village. Respondents highlighted that greater advance warning, compared to the 3-4 hours they received prior to the December 2014 flooding event, would help them to prepare for future severe flooding. They further agreed that text messages would be a helpful addition to loudspeaker announcements in advance of severe flooding. RWSs send texts to decision makers with information regarding rainfall upstream from villages and could be used in coordination with the Disaster Management Committee to send text messages to people living in vulnerable areas; this could provide a few additional hours of time. This information has been shared with the Irrigation Department in Colombo, and the Irrigation Department in Anuradhapura, to increase awareness of how males and females are affected differently by a disaster, and to demonstrate that there is demand at the civilian level for an advance warning system. No follow up has been noted at the time of writing this report. ü A total of 144 beneficiaries were trained in RWSs by both the IWMI project, as well as Dr. Chemin in his own time. Over the course of both phases of the CF project, IWMI conducted a total of six trainings for Anuradhapura Irrigation Department District Office staff, one formal trainings in Colombo for technical staff from several district offices that were nominated by the Irrigation Department HQ in Colombo; and one community of practice meeting. Outside of his work with IWMI, Dr. Chemin taught 100 students in Web GIS and GIS programming – including RWS design and use – during his time at the University of Moratuwa. He also worked closely with former students, local technology firms and LRWHF during that time. Twenty out of 20 (100% of) respondents that received training on RWSs who were interviewed for this study said that training provided by IWMI and Dr. Chemin was extremely useful in their work today. Reasons given for why they felt the training was useful focused around the need for such a device in early warning in Sri Lanka today. ü A total of USD $741,900 was leveraged by both IWMI and Dr. Chemin’s work on RWSs in Sri Lanka. Exposure through partnerships, training and classroom instruction to RWSs has resulted in widespread additional funding for these and similar devices throughout Sri Lanka. This evaluation found evidence of USD $741,900 in leverage for the furtherance of these open- source, DIY weather monitoring systems. This is a significant result, especially when viewed through the prism of the total intended leverage for all fifteen CF projects of GBP 1 million, or approximately USD $1.4 million. 10 As only one example of leverage, consider LRWHF, a noted leader in the field of remote weather stations in today’s Sri Lanka. An early adopter of RWSs as part of its project implementation, and home to a strong local champion of this technology for early warning for vulnerable populations, LRWHF got its start with these devices while attending a rainwater harvesting symposium at IWMI in September 2014, funded by USAID. Dr. Chemin’s presentation at this symposium is cited as being the inspiration for LRWHF, which subsequently included RWS pilots in its proposals for funding. USAID has funded LRHR in its efforts to integrate RWSs into its programs for many years and remains keen to support further use of this device for the purposes of education and early warning to vulnerable populations in Sri Lanka. Another example of leverage can be seen in a project among SUPSI and the University of Moratuwa. Funding is being provided to develop and test weather systems, as well as integrate a software system to house open data from the stations. Leverage is also visible from the Government of Sri Lanka’s Departments of Meteorology and Irrigation’s plans to assemble and place RWSs in schools and near reservoirs across the country, as well as COSTI and the National Building Research Organization’s (NBRO) plans to assemble devices and place them near existing stations to test for accuracy. 2. To what extent are decision makers regularly accessing remote weather stations in Sri Lanka? The Challenge Fund Logic Model contains three indicators to measure decision maker access to risk information tools. They are: 1. Decision-makers regularly accessing tools; 2. Tools are openly-available to the broader user community on an appropriate platform; and 3. Tools are being downloaded. Evidence gathered during this evaluation shows that all three indicators have been achieved by RWSs in Sri Lanka, and that this case is once again punching far above its weight when compared with intended results for all 15 CF projects in both phases. ü 15 decision makers in Sri Lanka today are regularly accessing RWSs. For example, the devices installed during the IWMI CF project are accessed directly by five decision-makers in the Anuradhapura Irrigation Department on a regular basis. Data from these devices is also accessed by at least one Irrigation Department official at the present time in Colombo. Three schools in vulnerable communities, as part of LRWHF projects, access data from these tools regularly, as does the Deputy Director at LRWHF. Officials in the Meteorology Department and COSTI access the RWS on the grounds of the Met Department. Professors and students at the university of Moratuwa access their RWSs several times each day. In addition, Department of Irrigation and COSTI officials have plans and budgets dedicated to building and placing many more RWSs across Sri Lanka. Due to the fact that the pilot devices in Anuradhapura are experimental in nature, the data from these devices are not yet being used systematically for early warning. They have been used in that way within the USAID-funded LRWHF project in more than one location and the results were saved livelihoods for vulnerable populations. Data from LRWHF RWSs is also transferred to www.wunderground -- an online platform for sharing weather information. Importantly, officials within COSTI, the Irrigation and 11 Meteorology Departments, and the Disaster Management Committee all stated that they feel RWSs can be instrumental in early warning to save crops, livelihoods and even lives in the very near future in Sri Lanka. ü RWS information is openly-available in several locations. RWSs are based on open source technology. Stakeholders surrounding RWSs in Sri Lanka have worked hard to locate locally-sourced parts for these devices so that today, all parts can be purchased locally. As reported earlier in this document, a great deal of comprehensive RWS information, such as RWS manuals, can be found on IWMI’s website8. RWS manuals can also be found on LRWHR’s website. Manuals are available in English, Tamil and Sinhala. In addition, RWS materials including technical files can be found on Dr. Chemin’s open mobile weather station GitHub site.9 The Community of Practice website for the Open Monitoring Systems Working Group10 also holds some useful information regarding RWSs, as does the Sri Lanka chapter of OsGeo website11. ü There have been 4,832 unique page views on the IWMI RWS site since the start of the CF project. Information on downloads of RWS materials housed on the abovementioned open source websites is unavailable. However, the active communities of practice and stakeholders involved in RWSs in Sri Lanka interviewed for this evaluation unanimously agree that access to RWS information is easily achieved. In addition, the costs of these devices have gone down over the past four years. At approximately 50,000 rupees, or just over USD $300, RWSs are relatively low-cost and therefore practical for Sri Lanka government counterparts. Although these three measures show tremendous access to RWSs in Sri Lanka, government officials An official within the Disaster hope to see even more in the near future; a true Management Committee offered, testament to the fit-for-purpose nature of these “This is what we need to devices, demand for RWSs in Sri Lanka remains coordinate all of the emergency strong. One highly-placed Irrigation Department official noted, “We need more weather stations to responders and all other ground- monitor water levels so we can advise in crisis level responders in times of situations…in the future, we must have this kind of emergency….in May 2017, we water management in our country.” Echoing this had more than 200 deaths; these sentiment, an official within the Disaster instruments would have Management Committee offered, “This is what we definitely helped with that. Early need to coordinate all of the emergency responders and all other ground-level responders in times of warning and detection are very emergency….in May 2017, we had more than 200 important for us.” deaths; these instruments would have definitely helped with that. Early warning and detection are very important for us.” 88 http://www.iwmi.cgiar.org/resources/mobile-weather-stations/ 9 https://github.com/YannChemin/MWS 10 https://wiki.osgeo.org/wiki/Open_Monitoring_Systems_Working_Group 11 https://wiki.osgeo.org/wiki/Sri_Lanka 12 Similar versions of these statements were provided to the evaluator during open-ended interviews by 83% of key informants. Across many Government of Sri Lanka ministries, NGOs, private and public sector institutions, plans for continued local investment in RWSs are brisk. 3. To what extent is there increased investment in remote weather stations in Sri Lanka? The Logic Model for the Challenge Fund measures investment in risk information tools by the number of local policies, plans or investments informed by the tools. Evidence shows numerous plans and investments being informed and greatly influenced by RWSs in Sri Lanka. ü Department of Irrigation officials, both within Headquarters in Colombo and in all four field offices visited for this evaluation, have plans to further develop, assemble and use RWSs for early warning. All four field offices which took part in this study have plans to request funds from Headquarters to assemble their own devices and begin to use them. They also have plans to train others in this open-source technology. The Department of Irrigation Headquarters in Colombo plans to build an RWS on its premises for educational and testing purposes; and upon completion of successful testing, it plans to deploy RWSs in Irrigation offices throughout the country. ü The Department of Meteorology plans to assemble and deploy 200 RWSs in schools for climate and technological education purposes. It intends to replace manual rain gauges with automatic stations in the future. ü COSTI has plans to develop four RWSs, based partly upon RWS information on open source websites, including IWMI’s site and Dr. Chemin’s GitHub site. The four stations will be placed by NBRO; and data collected will be compared with data from existing stations. Once testing is complete, NBRO plans to identify a model for deployment in landslide-prone areas across Sri Lanka. ü SUPSI and the University of Moratuwa have plans, based initially on IWMI website information, to measure nine parameters in their RWSs: temperature, rainfall, wind speed, wind direction, pressure, humidity, soil moisture, river level and visible light. This project has completed its prototype and is now in its testing phase. They plan to deploy 30 devices throughout Sri Lanka. This project will connect data from these devices to a central server, which it intends to make open. SUPSI worked with IWMI to install software in its office for data management purposes. ü A&T Labs is a small private company established by former University of Moratuwa students in 2014. Their first exposure to RWSs was through Dr. Chemin and continued with IWMI and LRWHF. Responding to a request from Dr. Chemin, they were able to create an RWS with all locally-sourced parts. They are now building more complex RWSs for government insurance and other purposes. ü Lanka Rainwater Harvesting Forum invests in RWSs as part of its projects in rural community schools. This DIY technology has been a successful tool for LRWHF to use 13 in educating rural community school children and teachers about changes in their environment. In addition to using RWSs as instructional tools, teachers and students monitor data regularly and use that information for early warning purposes to protect livelihoods of farmers in that community. The total investment amount represented by all of the above efforts is USD $741,900 toward further development, assembly and use of these weather monitoring systems. It is clear that the stage is set in Sri Lanka for continued investment in RWSs into the future. 4. Are the results evident sustainable? If yes, why? If not, why not? The Challenge Fund Logic Model does not contain indicators or associated milestones for sustainability of risk information tools. However, for the purposes of this evaluation, sustainability is defined as CF Logic Model results for the case of RWSs in Sri Lanka lasting into the future. Based upon all data gathered and analyzed for this evaluation, all results at the output, outcome, purpose and goal levels of the Logic Model appear very likely to continue well into the future, with the possible exception of gender. Evidence supporting this determination is provided below. Sustainability was built into RWS efforts for Sri Lankan use from the beginning. Those involved in the initial development of this open-source, DIY device emphasized local sourcing of parts so that they could be more realistically funded, maintained, repaired and replaced in the local context. The fact that they were envisioned as a solution to the Irrigation Department’s stated rainfall data collection challenges, and co-developed with several local partners also bodes well for their future prospects in the country. This deliberate strategy by IWMI and Dr. Chemin started in late 2013 and continues today. The philosophy behind these devices has always been open source, in conjunction with others in the open source community, and a continual process of development and improvement. In the case of RWSs in Sri Lanka, this philosophy has been instrumental in achieving all results in the CF Logic Model. It all began with a very strong local champion of the utility of such a device to address local partner demand. That local champion – together with IWMI staff and local officials – over a period of four years while working for IWMI and as an independent scientist, influenced many others who are now local champions for RWSs. And taken together, these local champions comprise a very active RWS Community of Practice (COP) that is creating new versions of RWSs for varied uses. In this way, the story of RWS development and uptake in Sri Lanka is one of a mutually reinforcing cycle, which continually serves to produce CF Logic Model results; therefore, this cycle is one of sustainability – not in the traditional sense because the device won’t look the same in the future as it does now, but the results of training, co-development, access, etc. will continue. Dr. Chemin, in reference to many of the organizations that have been created around, or began to incorporate different versions of RWSs into their operations, stated, “We are like grandparents looking at grandchildren growing up.” 14 Former students at the University of Moratuwa, for example, established A&T Labs; after first being introduced to RWSs by Dr. Chemin, the company now produces its own versions of weather stations for government and other uses. LRWHF quickly adopted RWSs in its programming as mainly educational devices in schools, but also for early warning in surrounding communities. Local champions exist in both organizations. The Irrigation Department in Anuradhapura now has several champions of this open-source, DIY device for use in early warning, one of whom has been looked to by the Irrigation Department Headquarters and other divisional offices to train others in the assembly and use of these devices. Today this local champion has successfully built capacity in his staff to train others as well. LRWHF has trained 200 students, teachers and rural electricians in Sri Lanka on RWS assembly and use. It is clear that local champions have emerged, and are creating other local champions, who are in turn innovating new (albeit related) devices. The cycle is set to continue. When asked to assist a strong local champion to address the problem of measuring river levels in Anuradhapura, IWMI quickly responded to this local demand by assembling a new device for testing. And now investment by the Irrigation Department is scheduled to be made in testing these devices. This new innovation is the result of the positive cycle of mutually-reinforcing results. From innovation, to access to investment, and back again to innovation, etc. The cycle has already repeated itself in many ways over the past four years and is set to continue to do so. One strong local champion summed his views of RWS sustainability as follows, “One way or the other, we will succeed in creating a working remote monitoring system…We will continue after the project ends; we have no choice, no options.” Evidence of a mutually-reinforcing system, he continued, “We need this system and we need our people to be empowered. Right now, we have two people who are purchasing equipment from local markets to implement knowledge from the training. The Irrigation Department is funding these costs and will continue to do so as others are trained by officials who have tested what they learned in IWMI trainings.” In addition, there are plans to install an RWS at Irrigation Department Headquarters in Colombo and use it as a training tool and pilot device. This local champion from Anuradhapura is thinking of many more innovations using the same board in RWSs for other applications. He is working with other divisional engineers as well as staff in Colombo to continually test and develop monitoring systems, creating what can be viewed as an internal COP within the Irrigation Department. Over time, many interested people in Sri Lanka have communicated with each other regarding their experiences in using and testing these devices. They have learned from their experiences and are able to innovate further, with the benefit of that knowledge. This active COP, then, is a result of output-level achievements such as partnerships and training, but also leads to investment and further innovation. One example could be seen in the University of Moratuwa learning from some of the challenges faced by the Challenge Fund project, which resulted in its use of different batteries, among other things, for its RWSs. One local champion there referred to RWSs he developed as his “babies”. Local champions for this technology in Sri Lanka are clearly impassioned and dedicated. 15 If we follow the path of one student who was at the University of Moratuwa during the time Dr. Chemin was a lecturer there, we can see the sustainability cycle in action. This student credits Dr. Chemin with being the person who introduced open source geo-spatial software and hardware to students and staff at the University of Moratuwa’s Department of Town & Country Planning. This student was greatly influenced by Dr. Chemin and wanted to learn about this technology to such an extent that he accompanied him to volunteer with LRWHF. During his time there, he proved to be a quick study and took part in training school students, teachers and local electricians about RWSs. He then became an intern at IWMI where he continued to work on the devices in Anuradhapura. Upon graduating, he worked as an instructor at the University of Moratuwa. He then taught others how to assemble and use RWSs, while continuing to volunteer with LRWHF in schools. Crediting Dr. Chemin with influencing him throughout, he successfully completed a Master’s degree in geospatial science in Japan, and will begin his Ph.D. later this year. This champion of these open-source, DIY devices has gone on to innovate a crowdsourcing system that inserts data through mobile technologies. This innovation can be seen as the result of all of the other results before it. The Sustainability Cycle below shows the path from innovation to inspiration to influence, which then leads to more innovation, inspiration and influence regarding RWSs in Sri Lanka. Challenge Fund RESULTS Sustainability Cycle LRWHF is another clear example of this cycle. Through strong relationships, staff there were shown the innovation, their capacity was built and they were inspired. They quickly became involved in the COP and accessed openly-available information about RWSs. They then invested in this technology in their programming in rural community schools. Over a period of 16 three years, they modified and improved their RWSs, and taught 200 school children, teachers and rural electricians how to assemble and use these open-source, DIY devices. That data is openly-available on weather underground, etc. The cycle continues, and argues strongly for sustainability of RWSs in Sri Lanka. These examples of the sustainability cycle show the key roles played by all results in the CF Logic Model except gender. Although women are very much involved in COPs and in some cases are in leadership roles, the only deliberate gender consideration discovered in this evaluation is the gender analysis conducted by IWMI. It remains unclear how local officials view gender and whether gender gaps will be addressed moving forward. In addition, this cycle shows that local champions and COPs are results in the case of RWSs in Sri Lanka, even if not made explicit in the CF Logic Model. 5. Are there any unintended consequences – positive or negative – as a result of the introduction of remote weather stations in Sri Lanka? This evaluation made intentional use of semi-structured interview protocols, allowing respondents to bring up issues of importance to them without biasing their thinking. Thematic analysis and coding of these interview responses showed results which were not overtly intended by the CF Logic Model. These results are referred to as “unintended consequences”. Increased coordination among local partners was discovered through this study. Respondents at the University of Moratuwa, for example, explained that prior to their project with SUPSI, two different departments within the university had never worked together on a project. Their joint work on RWSs had the unintended effect of increased intra-departmental coordination. Such coordination can also be seen within several divisional locations of the Department of Irrigation. Many regional engineers interested in RWSs know each other from IWMI trainings and are now talking with, and learning from, each other. Their communication and frequent contact can be viewed as the start of a COP within that Department. Increased confidence of former trainees was a clear result of the formal and informal training efforts surrounding RWSs in Sri Lanka. One divisional Irrigation Department engineer said, “After the two-day training, I had confidence that I understood, could teach others and can contact instructors”. A senior divisional engineer noted, “We have to train divisional engineers and assistants to the point where they have confidence to assemble and use RWSs, as well as modify them to their needs.” Confidence is not a stated output in the CF Logic Model, but it was highlighted by respondents as important to achieving higher-level results. Impassioned local champions are evident throughout the RWS landscape in Sri Lanka. Twenty-one people, all of whom were trained by IWMI or Dr. Chemin outside of the time he worked for IWMI, are passionate about RWSs in Sri Lanka today. They work with other interested parties to provide open access to RWSs and the data they produce, invest in their development and use, and innovate, inspire and influence others. They can be viewed as local champions of this technology, and were not explicitly planned within the framework of the CF Logic Model. 17 Communities of Practice can be seen as a logical extension of partnerships, capacity building, local champions, training, leverage and gender integration; they are in evidence surrounding RWSs in Sri Lanka, yet are not a current part of the CF Logic Model. The core members of the COP in Sri Lanka discovered in this study are: the University of Moratuwa, LRWHF, the Department of Irrigation, IWMI, COSTI, the Department of Meteorology, A&T Labs, and the Disaster Management Committee. There have been many meetings to facilitate learning throughout this community over the past four years held by several members of the community. IWMI focused on building a COP during Phase II of its CF project. Its staff met with COP members and convened a COP meeting in May 2016, which Government of Sri Lanka officials mention as instrumental to their understanding of RWSs and their potential for future use. Even though COP actors are familiar with each other, there has not yet been a COP gathering where representatives from all organizations mentioned in the paragraph above have been present. CONCLUSIONS The technology involved in open-source, DIY remote weather stations is not overly-complex if you are a student of such things. However, the true innovation in the case of RWSs in Sri Lanka was the application of this technology to help irrigation engineers monitor the levels of their water tanks and therefore be able to operate their reservoirs to limit flooding downstream, to limit crop damage and loss of life. This much-needed early warning device is low cost and reasonable to sustain, given government budgets in a middle-income country such as Sri Lanka. Innovation of this appropriate technology would not have occurred if local demand was not present and relationships were not strong. Such strong partnerships allowed training to be effective, capacity to be built, leverage to be created and a gender analysis to be conducted. The GFDRR-UK Aid Challenge Fund was the catalyst needed by the Government of Sri Lanka to further test the use of these devices in one pilot district; development and refinement of such devices takes time, and two phases of CF support helped tremendously with the learning process. Challenge Fund projects are intended to produce results related to innovating, inspiring and influencing risk information access and use for increased disaster resilience. The case of RWSs in Sri Lanka not only shows evidence of all of the results occurring, it shows that the results continue in a mutually-reinforcing cycle called the sustainability cycle. The results feed off of each other, guaranteeing the existence of some form of RWS access and use in Sri Lanka in the future. 18 RECOMMENDATIONS Challenge Fund Implementation: 1. The Challenge Fund should consider support for a workshop to bring together all interested parties in remote weather stations in Sri Lanka – governments, private sector and NGOs to share experiences and future plans regarding these open-source, DIY devices. 2. The Challenge Fund should consider responding to Government of Sri Lanka requests for additional and hands-on training in these innovative devices by supporting a training in Sri Lanka for Irrigation Department officials from around the country. 3. If possible, follow up with partners such as Irrigation Department and Disaster Management Committee officials in Sri Lanka regarding identified gender gaps in vulnerable populations downstream from reservoirs in an effort consider ways to address these gaps.12 4. Make explicit at the beginning of CF project implementation the requirement to mention funding sources of projects in relevant circumstances such as media coverage, trainings or workshops. Logic Model: 5. Consider whether a strong local champion for the risk information tool is a necessary output- level condition for the higher-level Logic Model results to occur. If so, recommend including local champion development as an output, and not an assumption, of a revised CF Logic Model. 6. The CF should consider whether communities of practice may be a necessary condition for eventual uptake. Strongly recommend including a strengthened community of practice as an outcome in the Challenge Fund model, to accompany the current outcome of increased access. 7. Consider removing indicator at outcome level # of downloads or risk information tools, as it could viewed be redundant with other measures such as the tools being openly-available. 8. Strongly suggest depicting the CF Logic Model as a sustainability cycle of mutually-reinforcing results. 9. Require Implementing Partners to keep careful records of easily-monitored outputs such as meetings, workshops, seminars and training sessions at the very start of project implementation. These will add to the evidence base of future theory-based evaluations which look at higher-level results such as the outcomes of access and use. 12 As this evaluation has found, this risk information tool has already achieved the highest level result of the CF logic model. Implementing these recommendations could help sustain these results for several years to come. This would help cement use within the Department of Irrigation – the original requestor/partner of this rapidly-spreading device. 19 ANNEX A: CHALLENGE FUND (CF) LOGIC MODEL Results Indicators Milestones at end of Milestones at end of Phase I Phase II Goal: Greater and Number of policies, N/A At least 6 local policies, more effective plans or investments plans or investments investment13 in disaster informed by CF-funded informed by CF-funded resilience, risk information risk information preparedness, response data/tools/approaches; data/tools/approaches and recovery Purpose: 14 Number of documented N/A 10 documented instances Data/tools/approaches instances of high-level of high-level decision utilized by decision decision makers using makers using CF-funded makers CF-funded risk risk information information data/tools/approaches data/tools/approaches Outcome: New high (a) Number of decision (a) 20 decision- (a) 30 decision-makers quality and relevant makers accessing CF- makers regularly regularly accessing CF- data/tools/approaches funded accessing CF-funded funded accessed by decision data/tools/approaches; data/tools/approaches data/tools/approaches; makers (b) # of tools available (b) 5 tools are openly (b) 6 tools are openly on open platform; (c) # available to the available to the broader downloads of broader user user community on an data/tools/approaches community on an appropriate platform; (c) appropriate platform; 70 recorded downloads (c) 50 recorded of data/tools/approaches downloads of data/tools/approaches Output: New toolkit Number of new Phase I CF yields at In total, at least ten new of demand-led, open datasets/tools/approaches least 10 new datasets datasets and tools data, tools and developed and tools. Phase II developed, 50% in DFID approaches for CF proposals priority countries resilience, evaluated and funded preparedness, response and recovery Output: Improved (a) Number of (a) 5 new (a) 5 new partnerships capacity of the partnerships developed partnerships developed with local community to deliver with local partners developed with local partners. (b) 15 demand-led, co- across disciplines that partners. (b) 10 beneficiaries involved produced products support the development beneficiaries co-development of tools of demand-led products; 13 Investments at the goal level include partner country budget allocations. 14 This purpose-level result was added by GFDRR in its “Challenge Fund: Phase I Results and Learnings” document, https://www.gfdrr.org/en/publication/challenge-fund-phase-1-results-and-learning. 20 (b) Number of projects involved co- developed with development of tools beneficiaries15 Output: Role of Number of project plans 10 Phase I CF All Phase II projects gender16 considered in considering gender in projects considered considered the role of project design and risk identification, gender in project gender in delivery outreach and capacity implementation implementation development plans Output: Improved Number of targeted 10 beneficiaries A cumulative total of 20 capacity of target beneficiaries that receive trained during Phase beneficiaries trained beneficiary users to training during the I over the life of project apply data, products project and tools in practice New evidence on what Evidence on what works 15 Phase I 1 case study evaluation works in developing communicated through monitoring self- of a successful Phase I demand-led data, tools progress, final and assessments CF project and 2 case and products to support evaluation reports completed. Regular study evaluations of resilience and MEL reporting Phase II projects using demonstrating the completed. rigorous evaluation value of risk data and methods completed. tools for decision Phase I learning making document submitted. Final report communicating examples of value of information for decision making Output: Leverage17 British pound leverage £500,000 cash and £1million cash and in- from other sources in from other sources in-kind contributions kind contributions CF and related activities leading toward same overall goal 15 Beneficiary refers to a person or group that derives advantage from something. For the purposes of the CF logic model, the term refers to those government officials within partner countries that take part in this project. – typically mid-level government decision-makers. 16 Gender, according to the 2016-2021 GFDRR Gender Action Plan, can be looked at as follows. Operations are considered to be “gender informed” if a gender analysis or gender impact was either taken into consideration during project design or mentioned as an expected outcome. Operations are considered to have “gender actions” when analyzed as being “gender informed” while additionally having specific components/activities that seek to minimize mentioned gender gaps relevant to the project’s development objective(s). 17 Leverage at the output level of the logic model is meant to include cash and in-kind contributions from public or private sectors for activities that assist in achieving CF results, whether directly requested by the project or not. 21