from EVIDENCE to POLICY Learning what works for better programs and policies September, 2015 101159 UGANDA: Does Information Matter? In many poor countries, the quality of education, health care Results have been mixed and even when there is a positive impact, and other public services is low. Figuring out how to fix that is there hasn’t been a lot of follow-up work to measure whether gains a key development goal. The World Bank and other donors of- are maintained over the long term. ten encourage community The World Bank works closely with governments to improve involvement as a way to the quality of services critical for reducing extreme poverty and improve the delivery of improving shared prosperity. Understanding the impact of differ- public services. This ap- ent initiatives is crucial for successful programs and policies. In proach, known as Com- Uganda, researchers supported by the World Bank evaluated a pro- munity Driven Devel- gram that sought to boost the quality of healthcare by giving com- opment, seeks to create munity members a voice in creating action plans with clinics for opportunities for people what needed to be done and how to do it. The evaluation found to participate in the that when communities also received information about how well planning, oversight, and a clinic performed in areas such as wait time and provider absen- HEALTH implementation of public teeism, there was more community involvement and an improve- services such as health care and education. In practice, this often ment in care. The findings are important for governments and means encouraging meetings between community members and development groups looking to community involvement as a providers, and having them work together on a plan for improving way to improve delivery of public services. As this evaluation delivery of services. How effective is this approach? It’s not clear. shows, information is critical in order to make a difference. Context Uganda has made important progress toward meeting the Mil- In 2005, a World Bank supported team decided to test the lennium Development Goals, including halving its poverty rate impact of a program to encourage rural residents to get in- to around 20 percent from more than 50 percent in the early volved in local health care delivery. Community members and 1990s. But healthcare, especially in the rural areas, is generally of health staff were given report cards grading the quality of local poor quality. Small clinics, known as dispensaries, provide care to clinics, including information about specific clinic operations, Uganda’s rural residents. The clinics offer preventive, maternity, absences and the quality of care. Meetings were facilitated be- and outpatient care, as well as lab services. All services are sup- tween community members and health facilities to allow them posed to be free. Six to 10 people staff most clinics, including a to draw up a shared vision of what was needed and make a trained medical worker, nurses, nursing aides, and others. But plan to achieve this. A year later, the evaluation* found that roughly 50 percent of the staff are absent on a typical day and the quality of healthcare improved, as reflected in lower child patients’ average wait time exceeds two hours. Uganda’s health mortality and improved child weight. In 2007, the team ex- sector is decentralized and Health Unit Management Commit- panded the project in order to test whether just bringing people tees are supposed to be the link between the community and the together with health providers for meetings and encouraging facility. In practice, there’s little action on the part of the commit- community monitoring processes­ —without providing report tees in terms of supervision or support. cards—could be as effective. *Björkman Nyqvist, Martina and Jakob Svensson, “Power to the People: Evidence from a Randomized Experiment on Community-Based Monitoring in Uganda,” Quarterly Journal of Economics, 124:2 (2009): 735–769 Evaluation The initial evaluation was implemented in 2005 in 50 rural Twenty-five new communities were identified for this phase. communities in nine districts, covering all of Uganda’s four re- Using the original study’s procedures, researchers randomly as- gions. Twenty-five communities were randomly selected for the signed 13 to a treatment group and 12 to a control group. Com- treatment group and the other 25 were assigned to a control munities in the treatment group were asked to attend meetings group. The treatment group received report cards that con- to identify the priorities for the local health clinic and how to tained detailed information about the quality of care and ac- improve quality. Separate meetings were held for community tivities of their local health center, and then meetings were held members and health facility staff, and then they were brought between community members and health clinic staff to draw together in a third meeting to agree on a joint action plan. The up a list of problems, goals and a plan of action. The control communities in this second phase were surveyed at baseline in group didn’t receive anything. In each treatment community, 2007 and then two years into the program, in 2009. short follow up meetings were held between community mem- For the purposes of the evaluation, a community was defined bers and health facilities in mid-2005, 2007 and 2008. Com- as all households living within a five kilometer radius of the lo- munities in the treatment and control groups were surveyed in cal health center. On average, there were 2,500 households per 2006 and after four years to see whether the positive gains that community. The communities were stratified by location and had been reported in the treatment communities after the first then population. In each location, half the communities were year were sustained. randomly assigned to the treatment group and the remaining to In the second phase, starting in 2007, a separate account- the control group. About 100 households were surveyed in each ability program was put in place in a different set of communi- community. Researchers also reviewed health records. In order HEALTH ties and evaluated. This new program sought to mimic the ear- to measure how important the data was to changing quality of lier program but without giving people detailed information on care, researchers reviewed health outcomes for households whose health facility performance. Researchers wanted to understand communities took part in the first participation-information whether the information, which had been expensive to collect, evaluation and compared this with the control group. They then was necessary to improve quality of care or whether it was enough looked at health outcomes of households whose communities to bring the community together to meet with health clinic staff took part in the second, participation-only, phase, and compared in order to create an effective community engagement program. the outcomes with those for the corresponding control group. Findings Giving people information specific to the performance In terms of height-for-age, an important accumulative mea- of their local health facility led to better health sure of proper nutrition and health care, children who had lived outcomes, especially for young children. for at least three years in communities that had received and dis- cussed score cards were 10 percent taller than children in the Between the years 2006 and 2009, the mortality rate for chil- communities without the intervention. dren under age five dropped by 23 percent, when compared with communities where no program was put in place. Similarly, in- In communities where report cards were distributed fant mortality dropped by about 28 percent and the neonatal and discussed, health facilities were in better death rate for infants under the age of one month dropped by condition and health workers appeared to do more for almost 44 percent. pregnant women and infants. This policy note is based on “Information is Power: Experimental Evidence on the Long-Run Impact of Community Based Monitoring,” Martina Björkman Nyqvist, Damien de Walque, Jakob Svensson, World Bank, Policy Research Working Paper, 7014; August 2014 available at http://documents.worldbank. org/curated/en/2014/08/20144947/information-power-experimental-evidence-long-run-impact-community-based-monitoring The general condition of the clinics—the floor, walls and fur- the communities in the second evaluation were brought together niture, as well as the clinic’s smell—was much better even four with health facilities to draw up a joint plan, there was no impact years after the initial intervention. Health workers were more ac- on local residents’ use of clinics and little difference in health tive when it came to running routine tests and doing check-ups outcomes when compared with the communities where there and they followed clinical guidelines more closely. Midwives were were no facilitated meetings. The treatment group showed little more likely to examine pregnant women, check their weight, draw difference from the control group in the following areas: under-5 mor- a blood sample, check the fetus and tell the women about po- tality, infant mortality, neonatal mortality, the number of births, tential pregnancy complications. Post-delivery, the rate at which and the number of pregnancies. newborns were checked in the first two months rose by 24 percent. Not surprisingly, in these communities there was also increased use of health services. Four years after the project began, participants in the treatment group that received report cards and discussed them in commu- nity meetings continued to make use of local health clinics more often than those in the control group. Depending on which mea- surement tool the researchers used, the increase in use of out- patient services rose between 16 percent to 27 percent and the increase in use of services after delivering a baby varied from 21 percent to 25 percent. The increase in pregnant women going to health facilities to deliver their babies rose by around 50 percent. Some of this increased use came from people who stopped going to traditional healers—or trying to heal themselves and instead turned to the health facility. Giving local residents and health staff information up front about the functioning of the health clinics—and comparisons with other clinics, along with the national standard for care—led them to draft an effective, Treatment and management practices in these health long-term plan to solve local problems and improve clinics didn’t change, which accounts for the lack of health care service. improvement in health outcomes. The information in the report cards allowed residents and health In communities where facilitated meetings were held, but report staff to focus on problems that could be solved locally, including: cards weren’t distributed on the quality of care, there was no sign absenteeism, opening hours, waiting time, and patient-clinician of increased exchange of information between residents and health interactions. Residents and health workers also could address these staff. Similarly, there was no evidence that residents took a larger issues themselves. In short, the report cards provided key informa- role monitoring health staff. Management of the clinics stayed the tion allowing the two sides to create an effective reform agenda. same, as did the degree of staff adherence to clinical guidelines. Steps that improved service included having a suggestion box, numbered waiting cards, a staff duty roster, and posters notifying Communities without report cards were less successful patients about their rights and that services were free. at drafting effective plans to solve local health care problems. However, there was little improvement in health out- comes in communities where meetings were held but Health staff and residents in these communities identified issues information wasn’t first distributed on health facility that mainly required help from outside parties—such as more quality and health indicators. financial and other support from senior authorities and non- governmental organizations, as well as more timely delivery of Efforts to spur more local participation in health care services medicine. They didn’t come up with plans that focused on local had little impact on health workers’ behavior and health care de- issues that facility staff and users could possibly resolve on their livery when this wasn’t combined with giving people information own, such as long waiting times, opening hours and absenteeism on the functioning of the local health facility. Two years after (something that communities that received report cards did do). Baseline data showed that there was a gap between rate at around 50 percent. Staff also never mentioned other what community members reported as being problems, problems, such as mistreatment of patients or lack of adher- and what the real problems were, and this gap may be ence to clinical guidelines, as problems that could be harming why communities that received report cards were able health care quality and usage. Instead, staff would usually point to improve services. to outside factors, like limited funding, as the main problem behind poor quality of care and health outcomes. When com- Staff at health facilities, for example, would say the wait time munities didn’t have any other information, they ended up fol- was usually two minutes, when the survey showed it was closer lowing what the health staff said was the problem; when they to two hours. They wouldn’t say there was a problem with ab- received the report cards with data, they were better able to senteeism, although unannounced surveys put the absenteeism pinpoint what needed to be fixed. Conclusion Information turns out to be a powerful tool in community- problems are. Relying on anecdotal evidence isn’t sufficient. based monitoring programs aimed to fixing local service de- While it’s costly and time consuming to gather such data, it livery problems. The findings of these two, related evaluations may be necessary to avoid trying to implement even costli- indicate that to ensure effective community participation, er community-driven interventions that fail because people everyone needs to understand what the real service delivery don’t have information on what the real problems are. HEALTH The Strategic Impact Evaluation Fund, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health and job opportunities for people in developing countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. The Evidence to Policy note series is produced by SIEF with generous support from the British government’s Department for International Development. THE WORLD BANK, STRATEGIC IMPACT EVALUATION FUND 1818 H STREET, NW WASHINGTON, DC 20433 Produced by the Strategic Impact Evaluation Fund Series Editor and Writer: Aliza Marcus Study researchers: Martina Björkman Nyqvist (Stockholm School of Economics) Damien de Walque (World Bank’s Development Research Group). Jakob Svensson (Stockholm University) The original had problem with text extraction. pdftotext Unable to extract text.