73367 from EVIDENCE to POLICY Learning what works, from the Human Development Network October, 2012 PAKISTAN: Can Low-Cost Private Schools Improve Learning? Education is central to giving children the building blocks The World Bank is working hard to help countries meet the for a life free of poverty. When schools fail to deliver quality United Nations Millennium Goal of universal primary educa- education, children are left without the skills and knowledge tion, and to ensure that schools teach effectively and students can they need to realize their capabilities and become productive learn. To understand whether low-cost private schools can im- adults. This isn’t just a prove access to education and promote student learning—espe- problem of insufficient cially in cases where public schools aren’t succeeding—the World supplies or poor facili- Bank carried out an evaluation of a new public-private education ties. Policymakers and partnership in Pakistan at the request of the government. Private education experts in de- schools in the program receive a per-student monthly subsidy veloping countries often in exchange for waiving tuition for all students and meeting a grapple with the prob- minimum pass rate in a standardized academic test administered lems of accountability: to students. By linking the subsidy to student learning standards, It can be hard to create the program aims to push schools to perform better. The evalu- mechanisms for hold- ation found that the threat of losing the subsidy worked, suc- ing schools responsible cessfully pushing schools to ensure better learning so that they for student achievement, but across the world, promising in- could keep their funding. To encourage schools to do even bet- education novations are being introduced. Impact evaluations are being ter, the minimum pass rate schools must reach to receive the used to test different models for promoting better teaching and subsidy is being progressively increased. Given the discussion more successful schools, including through support for low- underway in many countries about holding schools to minimum cost private schools, which can offer an alternative to poorly learning standards, this study shows that the risk of losing incen- functioning public schools. tives can be a powerful motivator for schools to do better. Context Pakistan’s overall education levels are among the lowest in the South the World Bank and other researchers as part of the Learning and Asia region and, in fact, in the world: Adult literacy was 50 percent Educational Achievement in Punjab Schools study found that, by in 2005, the year the private school program was launched, com- the third grade, less than 20 percent of students in rural Punjab pared with a literacy rate of 58 percent across the region, according could comprehend a simple paragraph in Urdu and that few could to World Bank figures. Although around 95 percent of children start tell time or subtract double-digit numbers. primary school on time, the completion rate is less than 70 percent. Increasingly, low-cost private schools, especially in rural areas Student learning is also generally low. A test conducted in 2003 by and poor urban neighborhoods, are an attractive option for families looking for better educational opportunities for their children. The Did you know: percentage of children enrolled in private schools in Punjab province School participation in Punjab for children aged 6-15 was… jumped 36 percent between the 1998-1999 school year and 2004- 65.7 percent overall 2005, right before this program was launched. The schools’ popular- 61.2 percent for girls ity was likely due, in no small part, to their low fees and perceived 60.9 percent for children from rural areas better quality education. These schools often boast lower teacher 48.7 percent for children from the poorest households absentee rates, better facilities, and higher levels of student learning, (Pakistan household survey data, 2004/05) even though teacher salaries tend to be lower than in public schools. For the Pakistani government, these low-cost private schools offered the minimum pass rate two times in a row are dropped from the an opportunity to reach students from economically-disadvantaged program. The monthly subsidies were initially fixed at about $3 per families with a potentially higher quality of education by channeling student, or about half the cost of what the government was spend- some public funds for education to these schools. ing to educate students in public schools. Schools are free to use the To expand access to such low-cost private schools – and promote money as they see fit. The schools must follow the national curricu- better quality education -- the Pakistani government created a special lum and to promote further uniformity, government textbooks are accountability-based public-private education partnership program handed out free of charge to the schools.To be initially considered for Punjab province, where more than half of the country’s 182 mil- for the program, two-thirds of the school’s students have to meet a lion people live. The Foundation Assisted Schools program, which minimum passing grade in a basic entry screening test. started as a pilot in 2005 and is administered by the government’s The program began in November 2005 with 54 schools in Punjab Education Foundation, targets low-cost private schools. seven districts in Punjab. As of June, 2010, when the evaluation was Schools admitted into the program receive monthly, per-student completed, the program covered 798,000 students in 1,779 pri- cash subsidies. In return, schools must waive tuition for all students, mary, middle, and secondary private schools in 29 of Punjab’s 36 which helps attract poor families, and ensure that a minimum per- districts. In 2010, the program had a budget of about $29 million. centage of their students pass a biannual standardized academic test, Currently, the program covers one million students. called the Quality Assurance Test (QAT). Schools that fail to reach Evaluation education World Bank researchers set out to evaluate the effectiveness the impact of the threat of expulsion from the program. To of pushing schools to meet minimum learning standards by measure impact, researchers used a regression discontinuity linking continued program eligibility to student test scores. design, comparing the average student QAT scores in schools They relied on data from five consecutive rounds of semi- just below the minimum pass rate cut-off point in the previ- annual standardized testing, beginning with the 2007-2008 ous QAT round to those in schools just above the minimum. school year. This corresponded to the first major expansion of (For a full description of methods used by the researchers, the program, which occurred during Phase 3, when the num- please see their Policy Research Working Paper #5465.) The ber of schools increased to 676 from 194. This large increase, evaluation looked at the causal effects on student learning of coupled with what was an unexpectedly high rate of failure the threat of expulsion of schools from the program as well as to meet the minimum pass rate in the November, 2007 QAT the causal effects on learning of cash bonuses to teachers for round, gave researchers an adequate sample size to evaluate achieving a pass rate of at least 90 percent in the QAT. Findings Accountability works. Schools threatened with meet the minimum pass rate, at least 67 percent of students losing access to subsidies almost always managed tested needed to score 40 percent or better on the QAT. In to raise student scores to meet the minimum pass the November testing round, 51 percent of schools failed rate needed to keep the funds. to meet the minimum pass rate. In order to maintain the subsidies, these schools had to raise student achievement to The schools that joined the subsidy program in Phase 3 took meet the minimum pass rate in the Spring 2008 round of their first QAT test in November, 2007, about two months testing. In March, 2008, among the schools that didn’t meet into the school year. The QAT is a curriculum-based, multi- the minimum pass rate the previous round, almost 100 per- subject, written test created for this program. For a school to cent passed. This policy note is based on the research paper, “Short-run learning dynamics under a test-based accountability system: Evidence from Pakistan,� Felipe Barrera-Osorio and Dhushyanth Raju. November 2010. In many cases, these low-cost schools rely on offering to keep pushing themselves and their students. Early results free tuition in order to attract students, making the from the tests show that schools have successfully met the threat of losing the subsidies a serious one. challenge, as very few are forced to drop out because of miss- ing the minimum pass rate two times in a row. Losing access to the subsidies can seriously hurt a school’s op- erations, since families may have enrolled their children to take The program shows that it is possible for schools advantage of the free tuition. Schools that have been disquali- to produce higher learning even with less money. fied have trouble holding on to their staff and keeping students, Parents seem to agree. and many end up closing. This live-or-die approach for schools in the program puts added pressure on schools to succeed. A LEAPS survey (http://www.leapsproject.org/assets/ publications/LEAPS_Report_ExecSummary.pdf ) of private To protect against cheating the system, multiple and government schools in Punjab found that parents versions of the same test are prepared, the test ranked 60 percent of teachers in private schools as “above versions and grades tested are varied across schools average� or excellent, compared with 45 percent of teach- and over testing rounds, and all students are expected ers in government-run schools. to be in school on testing day. In primary schools, the QAT is given to two grades, and in middle and secondary schools, three grades are tested. Grades tested in one round are not necessarily tested in the next round. Starting with the November 2007 round of QAT test- ing, the administration of the tests was done by independent agencies. Strict guidelines cover how the tests are to be trans- ported to the schools and require hand-scoring of each test. A staff member of the foundation in charge of the program is present at each test. Schools are told on which day they will be tested, but to avoid teachers sending home weaker students, schools must show 100 percent attendance on the day of the test. (Although in practice the test is given if the attendance rate is at least 80 percent.). To prevent schools from focusing teacher attention on grades that will be tested, schools don’t know until the day of the test which grades will be picked. There are eight versions of the test for each grade and these are brought to schools in sealed packets and randomly distributed. While government-run schools do spend more than these private schools, the extra money does not necessarily However, once schools met the minimum pass rate, raise the quality of lessons or result in better infrastructure. they did not rise any higher. Instead, the higher costs appear linked to higher teacher salaries—a function of teachers being government employ- The program was successful at pushing schools to do better, ees. Low-cost private schools pay their teachers less, but get or at least as well as necessary for staying in the program. But more out of them. researchers did not see any signs that schools that met the mini- mum pass rate in the QAT after failing the first time managed The program makes other demands of schools to to keep raising their student learning levels even higher. raise quality. To counter this, the Pakistani foundation in charge of the program has begun regularly raising the minimum pass rate Schools have to accept a range of conditions. Apart from after each testing round. The plan is to encourage schools waiving tuition and fees for all students and meeting the minimum pass rate in the QAT, they can conduct only one of expulsion played on raising learning levels in schools. But class at a time in the classroom, they must provide adequate this mechanism is just one of many that may be at work. For infrastructure, furniture, and teaching tools, ensure a maxi- example, schools want to ensure that they aren’t in danger of mum teacher-student ratio of 1:35, and commit to not using falling below the minimum pass rate in the QAT, so they may the school for after-hours classes or tutoring. invest more to make sure student learning improves to give themselves a bigger “cushion� in test results. Second, start- A separate study* conducted by the World Bank ing in November 2008, schools were required to display their found that schools that initially tested just above QAT pass rate—and that of other schools in the district—in the minimum pass rate in the screening test to enter a prominent place. This likely leads to pressure from parents the Punjab-based program, later showed improved on the school to do even better. Third, because parents don’t school facilities, lower teacher-student ratios and have to pay for schooling, this may free up resources for other education better student learning compared with schools that spending on children, such as books or better food. Students tested just below the minimum pass rate and did not also may be able to spend more time in school and on school- enter the program. work, because their schooling costs are covered. Fourth, the per student subsidy and the ability to rely on it—as opposed Unpacking the “black box� of why something works isn’t to possibly intermittent payments by parents in a fee-charging easy, and researchers who worked on both projects suggest school—may make it possible for a school to do more long- these changes could be due to a number of factors. The re- term planning and invest more in student learning. searchers were able to identify the important role the threat Conclusion It’s common to focus on incentives for encouraging better and not charging anyone tuition. Because they had to agree not learning by motivating teachers. Certainly, incentives have to charge anyone tuition, the subsidies became their operat- their place and many evaluations, such as the World Bank- ing budget and enabled schools to attract even more students. sponsored impact evaluation of a teacher incentive program in But schools also faced losing these subsidies if they couldn’t Andhra Pradesh (Evidence to Policy note, September 2010), reach minimum achievement standards. The threat of losing show clear benefits. the money pushed schools to ensure better learning, as mea- The presence of low-cost private schools in mainly poor, sured by the minimum pass rate on the achievement test. As rural areas allowed Pakistan to focus on the flip side of incen- the Pakistan evaluation shows, sometimes the “stick� works. tives. Schools received full subsidies for taking in poor students *Felipe Barrera-Osorio and Dhushyanth Raju, “Evaluating a test-based subsidy program for low-cost private schools: regression-discontinuity from Pakistan,� Manuscript (2009). The Human Development Network, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health and job opportunities for people in developing countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. This Evidence to Policy note series is produced with the generous support of SIEF. THE WORLD BANK, HUMAN DEVELOPMENT NETWORK 1818 H STREET, NW WASHINGTON, DC 20433 Produced by Office of the Chief Economist, Human Development Network, Communications/Aliza Marcus amarcus@worldbank.org