from EVIDENCE to POLICY Learning what works for better programs and policies December 2016 PAKISTAN: Does Sharing Test Scores with Parents Improve Student Learning? Education is a driver of development around the world. The World Bank, as part of its commitment to quality But as many countries have found, getting children into education for all, works with policymakers to develop inno- school is only the first step. Not all schools provide the vative ways to improve school quality. This requires gather- same level of teach- ing evidence on what works and why. In Pakistan, research- ing, and children in ers sought to measure the impact of providing families with especially poor coun- information about their child’s test scores and about the tries may finish pri- test scores in the other schools of the village. The evalua- mary school without tion found that giving parents information led to improved basic reading and test scores, lower fees in the private schools in the village EDUCATION math skills. With the and higher primary school enrollment. The results indicate renewed focus by de- that when parents know how well their children are doing velopment groups and in school—and know how well other children are doing governments on what in different schools—it can spur better learning. Impact children learn when they’re in school, it’s important evaluation evidence increasingly is showing that people to understand some of the factors that might motivate are motivated to demand more and better services when schools to do better and motivate parents to successfully they have information that allows them to correctly demand better quality. judge the situation. Context Private schools have become increasingly common in a good education and that parents may not fully realize low and middle-income countries, where state run edu- what their children can or should be learning. cation systems can suffer from poor governance leading Researchers wanted to test what might help improve to overcrowding, teacher absences and poor facilities. In school quality and student achievement in both private Pakistan, a third of all primary school age students are and government-run schools. Focusing on Punjab, the enrolled in private school. These schools, which charge largest province in Pakistan with a population of over tuition and are aimed at low-income families, keep costs 100 million, the research team decided to measure the down mainly because their teachers aren’t civil servants impact of providing information on student test scores but are instead hired directly at lower wages. Families of- to parents. The question was whether this (schools whose ten prefer private schools because they believe the quality average score was below the 40th percentile in the base- of education is higher. However, while private schools in line test) would spur parents to push for better quality Pakistan follow a curriculum similar to that of govern- education, and whether this would also make the private ment schools, they aren’t as closely regulated. This has school market more competitive because parents could raised concerns that students may not always be getting now choose based on clear measures of quality. Evaluation The research team carried out the evaluation across three other third graders in the school, and average scores in the districts in Punjab province, each district picked to repre- other schools in the village. Parents received the scores in sent a different socioeconomic area. Within these districts, the form of a “report card” at a school meeting, during 112 villages were randomly picked from among the vil- which the information was explained. A year later, the lages that each had at least one private school. On aver- process was repeated. The initial testing of the students age, each village contained 631 households and had seven and the baseline surveys took place in the spring of 2004. schools, one of which was government-run. These schools Parents received the test scores in September 2004, right each enrolled an average of 166 children in grades one after the summer break. The process was repeated in 2005. through five. Household surveys were conducted with 1,800 house- There were on average 18 children in each third grade holds in the villages at baseline and at the end of the pro- and all the third graders in the villages—about 12,000 gram. The schools and teachers themselves were surveyed children in total—were tested. Half the villages then were as well. For all tested grades, researchers also administered randomly assigned to receive information about student a short questionnaire to 10 randomly selected children, for and school performance on a standardized test. The infor- a total of 6,000 children. These three data sources—from EDUCATION mation included how well the specific child tested in Eng- students, school, and households—allowed researchers to lish, math and Urdu, how well the child ranked against corroborate self-reported data from multiple sources. Results Test scores among private school students whose But improvement depended on the initial performance parents received the report cards improved, but of the school. The test scores of children at low perform- this was only for students in lower-performing ing private schools (schools whose average score was below schools. the 40th percentile in the baseline test) rose substantially, while those in at high achieving schools (schools whose av- In villages where families were given information about erage score was at or above the 40th percentile in the base- how well their child had done on standardized tests— line test) didn’t change. This suggests that schools where and data on how this score compared with scores at other students tested poorly worked to improve their scores, schools—test scores in private schools improved and this while those where students were already doing well didn’t gain continued two years after the report cards were first feel the need to make changes, at least on the quality front. handed out or two years after the last report card was handed out. The increase was about 42 percent of the av- Test scores at government-run schools improved, erage yearly gain over student test scores in villages where too, regardless of whether schools were already no information was provided. doing well or not. This policy note is based on World Bank Policy Research Working Paper 7226, “Report Cards: The Impact of Providing School and Child Test Scores on Educational Markets,” Tahir Andrabi, Jishnu Das, Asim Ijaz Khwaja, Development Research Group, March 2015. http://documents.worldbank.org/curated/en/2015/03/24220579/report-cards-impact-providing-school-child-test-scores-educational-markets American Economic Review (forthcoming) https://www.aeaweb.org/articles?id=10.1257/aer.20140774&&from=f Students in government schools whose parents received compared with fees among schools in villages where report information on test scores were doing better on the tests a cards weren’t handed out. year later, when compared with students in villages where families didn’t get the report cards. The gain was the same Tuition fees dropped because now information regardless of whether the government school was ranked as about quality was clear and widely available. high performing or not before the evaluation was launched. This result is interesting because it shows that the in- Before the report cards, parents had some information centives to improve weren’t just about schools doing better about schools, but it was far from perfect. The only way for fear of losing paying customers. Rather, it could be that that parents could get schools to produce high quality was providing information motivates everyone—including by giving them high fees. This provided the incentive that parents, students, and schools—to try harder. schools needed, since they knew that if they produced low quality and were discovered in the process, parents would Giving parents report cards was just one part of withdraw their children, leading to substantial losses in making sure they were informed. At the meetings revenue. As parents’ information improved as a result of where report cards were handed out, the scores the report cards, these additional inducements were no were explained and parents were told what helps longer required and prices declined. improve test scores. EDUCATION Private school tuition in these villages ranged from less than U.S. $1 The meetings, which were run by members of the Learn- a month to about double that, but in any case less than the daily rate ing and Education Achievement in Punjab Schools for unskilled labor at the time. Between private school fees and other Project, weren’t designed to assign blame or to promote schooling costs, families spent about three to five percent of their monthly budget on education. one school over another. Instead, the meetings offered a chance for parents to learn what can help improve test scores. Giving parents the information verbally was im- As prices dropped and school quality increased, portant because the adult literacy rate in these villages more children enrolled in school. was about 37 percent, so many parents couldn’t even read the reports. In 2004, before the study began, 76 percent of boys and 65 Every meeting started with a 30-minute open dis- percent of girls ages five to fifteen were enrolled in school. cussion on what can influence test scores, including the After report cards were distributed, enrollment for primary teacher, the school environment, the home environment school children in private and government schools rose by and the child’s own behavior. Presenters were careful not 4.5 percent, or about 40 children in each village, compared to offer advice to parents or blame the child for a poor with enrollment in schools that were in villages that didn’t test score. get report cards. The increase in enrollment may have been because costs went down and quality improved. With increased competition, tuition rates at private schools dropped. The lowest performing private schools in villages where report cards were handed out were also Giving out report cards increased parents’ information more likely to shut down. about schools and raised student test scores in low per- forming private schools. However, this was not the only In villages where report cards were handed out, private impact. Better information also helped push down tuition schools that had low test scores after one year were 12.5 fees. Schools where test scores were substantially higher percentage points more likely to shut down, a sign that at baseline saw a bigger drop in their fees, by about 25 students were changing schools. As for public schools, the percent, compared with initially low-performing schools. report cards didn’t cause any closures, which was expected On average, fees at private schools dropped by 17 percent because their survival wasn’t dependent on school fees. In Punjab, children rarely go to school outside ciding factor in whether or not a child goes to school, their village, making it important to think locally especially for girls. Increasing the distance to school by when focusing efforts to improve education. 500 meters reduces enrollment for boys by 1.5 to three percentage points, and for girls by nine to 11 percentage Children rarely attend school outside their village. In points, according to a study of education in Punjab.* fact, studies have shown that distance is usually the de- *Andrabi, T., J. Das, A. Khwaja, T. Vishwanath, and T. Zajonc (2007). The Learning and Educational Attainment in Punjab Schools (LEAPS) Report. World Bank, Washington, DC. Conclusion Much of the debate about how to improve learning fo- provide, costing just U.S. $1 per child, which is much cuses on public schools. Yet, increasingly, parents face cheaper than other approaches that are proven to im- many choices when it comes to where to send their prove student performance. The importance of meet- EDUCATION children to school. When so many choices are avail- ings, given that the majority of parents were illiterate, able, information can be a powerful tool. These results can’t be ignored as a possible route for raising parental show that approaches that improve performance in the involvement and helping them understand what is go- private sector can simultaneously strengthen the public ing on in the schools. This is one of the areas that im- sector, which may have implications beyond education. pact evaluation researchers focused on education may Sharing test scores also turned out to be a fairly inex- want to do further work. pensive approach. The report cards were inexpensive to The Strategic Impact Evaluation Fund, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health and job opportunities for people in developing countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. The Evidence to Policy note series is produced by SIEF with generous support from the British government’s Department for International Development and the Children’s Investment Fund Foundation. THE WORLD BANK, STRATEGIC IMPACT EVALUATION FUND 1818 H STREET, NW WASHINGTON, DC 20433 Produced by the Strategic Impact Evaluation Fund Series Editor: Aliza Marcus Writer: Laura Burke