from EVIDENCE to POLICY Learning what works for better programs and policies December 2015 102710 HAITI: Can Smartphones Make Schools Better? Over the last decade, millions of children have been able The World Bank is committed to making sure all chil- to go to school because countries made free primary school dren have the chance to get a primary school education. Suc- education a priority. What happens now that they are in ceeding requires helping governments discover innovative school? Are they learn- solutions to problems that might keep kids from learning or ing? A lot needs to keep them out of school. In Haiti, World Bank researchers happen to make sure worked with the Ministry of Education to evaluate a pilot that children can get a program to use digital technology to help keep track of quality education. To teachers in school and enable more effective monitoring start, education officials of schools. The plan was to use smartphones to take pho- EDUCATION must be able to manage tos in school of teachers and send the photos to a central schools effectively and server that could be accessed by school inspectors and monitor what’s going ministry officials. The problem was that implementing on, especially whether the program was difficult and compliance was low. In the teachers are showing up for work and what they’re doing end, there was no effect on teacher absenteeism or stu- once they are in school. The availability of new technolo- dent learning, in part because of the problems getting the gies, such as smartphones, offers new routes for monitoring program up and running. The evaluation highlights the teacher absenteeism and for improving communication be- challenge of using technology in countries with limited tween schools and central authorities. But can information infrastructure and serves as a cautionary tale that tech- and communications technologies, known as ICTs, really nology alone cannot solve the complex challenges many change how schools—and school officials—act? school systems face. Context Haiti has made substantial improvements in educational don’t have a car or enough gasoline to go to the schools. access over the last decade and 90 percent of children of They also each get about 35 schools to monitor, making it primary school age now are enrolled in school. Most of hard to visit all on a regular basis. the schools are privately run, often through a foundation The Ministry of Education was interested in improv- or religious institution. Although the government doesn’t ing its approach for monitoring schools. A World Bank-led administer these private schools, the Ministry of Educa- team worked with the government to develop a pilot to test tion assigns inspectors to visit and monitor both private whether technology could improve monitoring by allow- and public schools. Nonetheless, inspections don’t occur ing for “virtual” inspections. Haiti seemed like a particularly regularly. The country’s poor road infrastructure makes it good candidate for this technology-based solution since it has hard for inspectors to get to the schools, and they often high overall usage of mobile phone technology, with nearly 70 cell phones per 100 people. The plan was to use this about which of their teachers were and were not turning technology to complement in-person visits by inspectors up. An impact evaluation, funded in part by the World and make it simpler for the education ministry to moni- Bank’s Strategic Impact Evaluation Fund, was built into tor schools, while giving school directors clear information the program to measure the effect. Evaluation Researchers randomly assigned 200 public and private detailed data on the school and the school director, and ad- primary schools either to a control group or to partici- ministered an Early Grade Reading Assessment (EGRA) pate in the program. In the 100 schools assigned to re- to a sample of 15 students at each school. After the pro- ceive the program, directors received a smartphone with gram was launched in January 2014, the research team a built-in system that allowed them to send information collected data on absenteeism and other basic informa- about the school to a centralized server, including dai- tion from all the schools. In May and June of 2014, as ly photographs of teachers to verify they were at work. the program ended, the research team collected endline School inspectors could access the server in real time for data from all the schools and administered the EGRA to efficient supervision. Both the inspectors and the school another random sample of students in all 200 schools. EDUCATION directors received training from the research team and Researchers also conducted an in-depth qualitative study the cell phone company on how to use the phones and in a small sample of schools after the program ended, software. interviewing 37 teachers, 14 directors, and four inspec- Baseline data collection started in November 2012. tors about the operational challenges of the program and During this initial round, the research team collected their perceptions of it. Results The program faced challenges from the start, Once the program was launched, school officials including delays and technical problems that reported difficulties using the new technology, made it hard to implement it as planned. despite having received training beforehand. There were so many problems getting schools ready for School directors were given cell phones with digital cam- the pilot that the program ended up starting months late. eras, a data connection subscription and a software pro- Instead of running for the whole of the 2013–2014 school gram for uploading the photos and sending information year, the pilot was only in effect for four months. This about teacher absences to the central server, which could short implementation period reduced the chance of seeing be accessed by administrators who had the password. The any change in teacher behavior or student learning. cell phone company provided training to school directors The problems included administrative and procurement on how to use the phones to take picture and how to up- ones, such as difficulties identifying and hiring survey load the photos to the central server. School inspectors, firms and unrelated teacher strikes. who were in charge of checking that the uploaded photos This policy note summarizes the World Bank paper, forthcoming, “Can Information Technology Improve School Effectiveness in Haiti? Evidence from a Field Experiment,” Melissa Adelman, Moussa P. Blimpo, David Evans, Atabanam Simbou, Noah Yarrow. Support for the evaluation was provided by the Strategic Impact Evaluation Fund and the Bank Netherlands Partnership Program. were of the teachers who were supposed to be teaching Thirty-five percent of the schools in the treatment group (the uploaded photos were checked against a pre-existing had not received any inspector visit six months into the database of teacher photos), were trained in how to retrieve school year, while only 29 percent of schools in the control the information and match the photos. group had not received any inspector visit. This may sug- But school directors and inspectors still weren’t always gest that inspectors use the program as a substitute for ac- able to use the new technology and directors reported that tual school visits, rather than as a complement. However, they had difficulty keeping the phones charged, uploading it’s also possible that the inspectors used the technology to the data, and using the software. In interviews, all but one identify the schools in greatest need of support visits and school director reported technical difficulties with the pro- then concentrated their efforts on those specific schools, cess of taking and uploading the photos. Inspectors also but researchers were not able to verify if that was the case. reported significant technical difficulties on the receiving More research will be done in this area. end that led to their failure to use the database of pho- tos. And when school directors asked for a technician to come out and help—which was supposed to be available as part of the program—they usually didn’t get a response. It turned out the technical team, which was supposed to be provided by the cell phone service provider, wasn’t ready in time to handle the questions. In the end, very few schools actually used the program with any regularity. Schools in the pilot program only used the technology 30 percent of the time during the four-month implementa- tion period. Part of the reason was clearly due to the tech- nical difficulties with the actual technology, but it’s also possible that teachers viewed it as unfair. In Haiti, teachers’ The program had no impact on student learning— salaries are often delayed or not been paid altogether, so nor on teacher absenteeism. there’s little incentive for teachers to allow themselves to be held accountable when the system itself doesn’t hold Students’ test scores did not improve as a result of the pro- itself accountable. Teachers also didn’t get any extra com- gram, which was unsurprising given the program was im- pensation for taking part. The promised phone credits plemented for just four months and use of the technology never materialized because of logistical and administrative was limited during that period. The program didn’t improve problems. management practices such as record keeping, likely also be- The program also took time—whether it was checking cause so few schools used the technology, nor did it have an that the smartphone worked or taking and uploading photos impact on teacher absenteeism. —and school directors who managed more than one school The real lesson learned from the evaluation was that were less likely to participate, as compared with directors things are never as simple as they seem. The new technology who managed just one school. Similarly, school directors was more complex than it seemed for school directors and who also had other jobs were less likely to use the program. inspectors. Teacher absenteeism, also, turned out to be not as much of a problem as initially assumed. The surveys done The technology seemed to have introduced for the evaluation found that the rate of absence was usually additional problems, including a drop in between five percent and 10 percent. That’s actually quite inspections among schools that received the low for a low-income country, where rates can range from mobile technology. around 11 percent in Peru to 25 percent in India. Conclusion Across the developing world, technological solutions are tial for success must be rigorously tested on the ground often viewed as a panacea, capable of solving some of to ensure that the local infrastructure can absorb and the most critical problems developing countries are fac- sustain them. In addition, as this evaluation and others ing. In the education sector, many school systems are show, infrastructure is only part of the equation: Mak- looking to technology to increase accountability and ing sure that teachers, school managers and inspectors improve student learning. Although there have certainly are on board with the program—and willing to carry it been successes with technological innovations in the ed- out—is equally important. Without care, technologies ucation sector, governments should be aware that tech- like smartphones, swipe cards, and time clocks could nology-based programs that work in one context may use up precious government resources, without any ben- not work in another, and innovations with the poten- efit to schools. EDUCATION The Strategic Impact Evaluation Fund, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health and job opportunities for people in developing countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. The Evidence to Policy note series is produced by SIEF with generous support from the British government’s Department for International Development. THE WORLD BANK, STRATEGIC IMPACT EVALUATION FUND 1818 H STREET, NW WASHINGTON, DC 20433 Produced by the Strategic Impact Evaluation Fund Series Editor: Aliza Marcus Writer: Laura Burke