77639 Managing for Results in Primary Education in Madagascar: Evaluating the Impact of Selected Workflow Interventions ´ rard Lassibille, Jee-Peng Tan, Cornelia Jesse, Ge and Trang Van Nguyen Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 The impact of speci�c actions designed to streamline and tighten the workflow pro- cesses of key actors in Madagascar’s primary education sector are evaluated. To inform the strategy for scaling up, a randomized experiment was carried out over two school years. The results show that interventions at the school level, reinforced by interventions at the subdistrict and district levels, succeeded in changing the behavior of the actors toward better management of key pedagogical functions. In terms of education outcomes, the interventions improved school attendance, reduced grade rep- etition, and raised test scores ( particularly in Malagasy and mathematics), although the gains in learning at the end of the evaluation period were not always statistically signi�cant. Interventions limited to the subdistrict and district levels proved largely ineffective. JEL codes: I21, I28, J24 Ge´ rard Lassibille (corresponding author; gerard.lassibille@u-bourgogne.fr) is a research director at the Centre National de la Recherche Scienti�que in the Institut de Recherche sur l’Economie de l’Education, Dijon, France. Jee-Peng Tan ( jtan@worldbank.org) is advisor in the Education Department of the Human Development Network at the World Bank. Cornelia Jesse (cjesse@worldbank.org) is operations of�cer in the Africa Human Development Department at the World Bank. Trang Van Nguyen (tnguyen16@worldbank.org) is an economist in the Poverty Reduction and Economic Management Sector Department in the East Asia and Paci�c Region at the World Bank. The authors are deeply indebted to Esther Duflo for invaluable help and guidance, particularly in setting up the experimental design for this impact evaluation. They thank their counterparts at the Madagascar Ministry of Education, led at the time of this study by Tahinarinoro Raza�ndramary and Paul Randrianirina, and the staff of Aide et Action for help in conceptualizing the workflow tools, implementing the experiment, and collecting the data. Among close collaborators on this work, the authors especially appreciate Pierre-Emmanuel Couralet and Erika Strand for on-the-ground supervision during the experiment and Mathieu Laroche and Muriel Nicot-Guillorel for technical support to the Malagasy counterparts. The authors also thank colleagues and friends who provided input, advice, and comment to guide the design of the experiment, among them Sajitha Bashir, Benu Bidani, Robert Blake, Deon Filmer, Elizabeth King, Arianna Legovini, Robert Prouty, Lina Rajonhson, Patrick Ramanantoanina, and Venkatesh Sundararaman. The authors thank three anonymous referees and the editor of the journal for insightful comments. Finally, they acknowledge the support of the World Bank and the governments of France, Ireland, Madagascar, and Norway, as well as that of the donor partners of the Education for All Fast Track Initiative through the Education Program Development Fund. THE WORLD BANK ECONOMIC REVIEW, VOL. 24, NO. 2, pp. 303 –329 doi:10.1093/wber/lhq009 Advance Access Publication August 6, 2010 # The Author 2010. Published by Oxford University Press on behalf of the International Bank for Reconstruction and Development / THE WORLD BANK. All rights reserved. For permissions, please e-mail: journals.permissions@oxfordjournals.org 303 304 THE WORLD BANK ECONOMIC REVIEW Over the last 10 years, low-income countries in Africa have made striking pro- gress in expanding coverage of primary education. However, in many of these countries the education system continues to deliver worse results than expected, putting at risk the goal of universal primary school completion. Weak adminis- tration, inadequate focus on results, and poor governance structures are thought to be some of the reasons for the meager results. Better management of workflow processes at each point along the service delivery chain might therefore improve productivity and provide a useful tool for raising the per- formance and ef�ciency of education systems. Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 This article reports on the �rst known attempt at a randomized impact evalu- ation of interventions to improve management of the teaching process in an African setting. It adds to the growing literature on randomized impact evalu- ation of education programs in developing countries. Kremer and Holla (2009) group such evaluations into �ve broad categories, with the interventions typically aiming to broaden access or improve learning outcomes: lowering private costs, increasing subsidies, or providing students with health care (examples include Bobonis and Finan 2009; Evans, Kremer, and Ngatia 2008; King and Orazem 1999; Kremer, Moulin, and Namunyu 2003; Miguel and Kremer 2004; Tan, Lane, and Lassibille 1999; and Todd and Wolpin 2006); enhancing teacher inputs or other resources such as textbooks or flipcharts (Barnejee and others 2006; Duflo, Dupas, and Kremer 2007; Glewwe and others 2004; Glewwe, Kremer, and Moulin 2009); reforming pedagogy through radio- or computer- assisted instruction (Barnejee and others 2007; Jamison and others 1981); redu- cing teacher absenteeism through better incentives (Barnejee and Duflo 2006; Glewwe, Ilias, and Kremer 2008); and informing and involving local commu- nities and transferring to them control of school management in areas such as teacher hiring and �ring (Barnejee and others 2006; Duflo, Dupas, and Kremer 2007; Gertler, Patrinos, and Rubio-Codina 2008). The experiment reported in this article targets Madagascar’s public primary school system, which enrolls 80 percent of students at this level. Thanks to increases in public spending since the late 1990s, the country has been making good progress in expanding coverage in primary education. At the same time, the government put in place important reforms to strengthen administration of the education system. The technical staff of the ministry developed tools to streamline and tighten the workflow processes of all actors along the service delivery chain, focusing on measures to make explicit the functional responsi- bilities of teachers, school directors, and district and subdistrict administrative staff through a coherent and detailed manual of operations and to increase information flows and accountability through report cards at the school and higher administrative levels. These tools are discussed in detail in section II. To evaluate the impact of the workflow-enhancing interventions on the be- havior of service providers and on school outcomes, a randomized experiment was carried out over two school years (2005/06–2006/07) in a sample of school districts. The experiment was designed to answer three main questions: Lassibille, Tan, Jesse, and Van Nguyen 305 How much do the workflow-enhancing interventions alter the behaviors of service providers and schooling outcomes? How do the interventions differ in impact when aimed at the school, subdistrict, and district levels? Do the inter- ventions have spillover effects? The results after two years of experimentation suggest that only when com- bined as a package of school-level interventions reinforced by interventions at the subdistrict and district levels do the workflow-enhancing tools improve the management practices of service providers. School attendance rose and grade repetition fell in the intervention schools by statistically signi�cant amounts; Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 test scores also rose, though the gain was not always statistically signi�cant. Interventions limited to the subdistrict and district levels were largely ineffec- tive, probably due to weak mechanisms for monitoring and control and to the lack of a true leadership culture among these actors. These �ndings are sugges- tive and potentially useful for other Sub-Saharan African countries facing the same management issues in education. They can help inform similar work to clarify the options for addressing these issues. The remainder of the article proceeds as follow. Section I describes the edu- cation sector in Madagascar. Section II explains the interventions. Section III focuses on the design of the experiment and the implementation of the impact evaluation. Section IV presents the empirical results. And section V concludes. I . MA D A G A SCA R ’ S E D U CAT I O N S E C TO R IN CONTEXT Madagascar has not always prioritized education in the allocation of public spending, but in recent years the government has begun to channel more resources to the sector as part of its commitment under the Heavily Indebted Poor Countries Debt Initiative and in its poverty reduction strategy. As a result, public recurrent spending on education rose from 2.2 percent of GDP in 2000/ 01 to 3.3 percent in 2006/07. In tandem with the increase of funding, the gov- ernment introduced several important reforms (Government of Madagascar 2004b). Among the keys reforms are eliminating school fees for primary edu- ´ coles) and using school cation, launching a system of capitation grants (caisse e grants to incentivize performance, providing kits scolaires and textbooks to primary school students, creating school boards in all primary schools, provid- ing public subsidies to supplement the pay of non–civil service teachers in public schools who in the past have been hired and paid entirely by parent associations, restructuring the primary and secondary cycles of schooling, and introducing new pedagogical approaches.1 The most visible signs of progress are the large increase in coverage in primary education in recent years. In 2006/07, the education system enrolled some 3.8 million students in both public and private schools—more than twice 1. The new government that came into power in early 2009 is reviewing these reforms to determine their future direction. 306 THE WORLD BANK ECONOMIC REVIEW the enrollment in 1996. As a result of this increase, the gross enrollment ratio in primary education rose from 83 percent in 1996 to 123 percent in 2006, and thus the number of primary school teachers in the public sector also increased dramatically, from some 28,000 to 59,000. More than 4,000 new primary public schools have been created during the last 10 years. While the progress in coverage has been impressive, enormous challenges and problems remain for system performance. Entry rates to grade 1 are high, but less than half of each cohort reaches the end of the �ve-year primary cycle. Despite government interventions, grade repetition has not improved as rapidly Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 as expected. In fact, repetition rates are still uniformly high throughout the primary cycle, averaging about 18 percent. With regard to student learning, the 2005 Programme d’Analyse des Syste ` mes Educatifs de la Confemen survey indi- cates that Malagasy students performed better than their peers in other low- income African countries (for example, Benin, Cameroon, Chad, and Mauritania). But their scores were still low in absolute terms: about 30 percent on the test in French and about 50 percent on the tests in Malagasy and math- ematics (PASEC 2007). They also performed worse than the 1998 cohort of Malagasy students in grade 5, who scored 48 percent in French and 59 percent in mathematics (World Bank 2002). Several factors explain the poor performance of the primary education system. Many demand-side conditions are impervious to policy interventions, particularly in the short run, such as household poverty, community character- istics, the opportunity cost of children’s time, and parental perceptions of the value of education (see for example, Haveman and Wolfe 1995). Supply-side factors under the control of policymakers and managers in the education sector (see for example, Hanushek 1997) include the allocation of teachers across schools and the management of the pedagogical process. In Madagascar the degree of randomness in the allocation of teachers across primary school has diminished to a large extent over the last 10 years (Government of Madagascar 2008). Assigning teachers more consistently across schools clearly signals better administrative management of the system. Within schools, however, many aspects of the pedagogical processes are poorly managed, and tasks essential for student learning are neglected. A survey of workflow processes conducted as part of the impact evaluation reported in this article offers particularly telling revelations.2 The data show that student absenteeism is poorly monitored by teachers, with attendance taken an average of 13 days a month, and poorly supervised by school direc- tors, with 10 percent neglecting the task entirely and only a third signing off on the attendance records kept by teachers. Essential pedagogical tasks are 2. The workflow details presented in the following paragraphs are based on data codi�ed from about 850 workflow artifacts collected from about 150 teachers in 40 schools that provide a record of their work over a full school year (see section III); the results discussed here pertain to teachers in schools that did not receive any of the interventions associated with the experiment reported in this article. Lassibille, Tan, Jesse, and Van Nguyen 307 often neglected: 20 percent of teachers do not prepare daily lessons plans, only 15 percent consistently prepare daily and biweekly lessons plans, and a third of school directors never discuss daily lesson plans with teachers. Student academic progress is also poorly monitored. The results of tests and quizzes are rarely recorded, if at all, and 25 percent of teachers do not prepare individual student report cards. Communication from teachers to parents on student learning is often perfunctory, and student absences are rarely communi- cated to parents. School directors seldom follow up on student performance: 75 percent do not discuss learning outcomes with their teachers, and only Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 20 percent sign off on test results and student report cards. The same goes for teacher absences, which average nearly 10 percent3—hardly a negligible �gure. Only 8 percent of school directors monitor teacher absences by taking daily attendance or tracking and posting a monthly summary of absences, and more than 80 percent fail to report teacher absences to subdistrict and district admin- istrators. The general impression that emerges from these observations is one of obvious lack of organization, control, and accountability—all of which can compromise the performance of the system and the chances of success of the many ongoing reforms. II. DESCRIPTION OF THE INTERVENTIONS To put the evaluated interventions in context, this section discusses a few note- worthy features of the administrative arrangements in Madagascar’s primary education system. The system consists of a large number of public schools, funded and managed by centrally appointed civil servants, and a small number of mostly urban private schools, funded largely by student fees and managed by religious and other entities. In 2006/07 the network of some 18,000 public primary schools enrolled about 3.1 million students and employed 18,000 school directors and 59,000 teachers (table 1). These schools were supervised by 1,500 subdistrict adminis- trators (chefs ZAP), who provide the �rst line of administrative and pedagogi- cal support to the schools under their care and who in turn report to one of 111 district administrators (chefs CISCO), who in turn report to one of 22 regional administrators (DREN)4 and the ministry of education. The subdis- trict and district administrators channel resources to schools, supervise teaching and learning practices and the collection of school statistics, and administer the annual national examination at the end of grade 5. Their work also includes distributing paychecks to teachers; organizing school building and maintenance projects; overseeing the distribution of books and small grants to schools; 3. This estimate compares reasonably well with others studies conducted on the topic in Madagascar (see World Bank 2008). 4. The regional administrators are new. They were introduced when the impact evaluation was already well under way, but were not (and still are not) yet operational. 308 T A B L E 1 . Administrative Structure of Madagascar’s Public Primary School System, 2006–07 School Subdistrict District Regional Ministry of Actors Students Teachers directors administrators administrators administrators Education Total number of actors 3,103,000 59,000 17,600 1,544 111 22 1 Number under each actor: THE WORLD BANK ECONOMIC REVIEW Students 53 177 2,010 27,900 141,000 3,103,000 Teachers 3 38 530 2690 59,000 School directors 11 160 800 17,600 Subdistrict administrators 14 70 1,544 District administrators 5 111 Regional administrators 22 Source: Authors’ analysis of the 2006/07 Fiches d’Enque ˆ tes Rapides. Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Lassibille, Tan, Jesse, and Van Nguyen 309 designing and implementing in-service training for teachers, school directors, and other staff; providing in-service support; and ensuring timely returns on the school census questionnaires. The key actors in the service delivery chain are responsible for only a few actors at the level below them. Each district administrator manages an average of 14 subdistrict administrators, and each subdistrict administrator is responsible for about 10 school directors. Each school director manages three teachers and 177 students on average. The interventions being considered by the Madagascar Ministry of Education seek to tighten the management of pedagogical workflow processes Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 at each point along the service delivery chain described above and to increase the focus on results by making explicit to the actors their responsibilities and supporting them with tools and procedures to accomplish their tasks, inserting supervision and follow-up at key points in the administrative hierarchy, and facilitating school-community interactions and promoting accountability for results around school report cards. Each set of actors performs many tasks, but a few activities at each point along the service delivery chain are central to the mission of managing for results and must therefore receive routine attention by the relevant actor. After intensive consultation and discussion, the ministry of education identi�ed the main responsibilities of the various actors. Out of this exercise emerged the core job description for each service delivery agent in the education system. Based on this job description, ministry of�cials developed the corresponding operational tools and processes, which focus on six broad domains of activity: pedagogy, student learning and follow-up, management of instructional time, administration, school statistics, and partnership with the local community.5 Typically, the set of products for each actor is a rationalization and simpli�- cation of existing templates that have become unwieldy, incoherent, and out- dated over time. More than 30 tools were developed for tasks considered essential for a well functioning system, among them 7 designed for use by tea- chers, 8 by school directors, 8 by subdistrict administrators, and 9 by district administrators. Each operational tool encourages superiors to pay closer atten- tion to the work of their direct reports (for example, requiring school directors to review information supplied by a teacher and suggest speci�c follow-up actions in case of need) and is accompanied by a simple and clear guidebook that explains how and when each tool should be used. Training modules have been speci�cally customized and offered to all the relevant actors (see section III). Depending on the actor and task, the templates and accompanying work- flow processes are used daily, weekly, monthly, or at a speci�c time in the year. As noted above, they replace tools that had become increasingly incoherent and idiosyncratic for lack of upkeep and updating. Teachers and administrators 5. For details on the tools, their actual presentation, their purpose, and how they are to be used by the various actors in the education system, see Government of Madagascar (2004a) and www.education. gov.mg. 310 THE WORLD BANK ECONOMIC REVIEW use the existing tools only sporadically, if at all, and the tools themselves have all but lost their power to reinforce the reporting relationships among teachers, school directors, and subdistrict and district administrators. While it is important to rationalize and tighten workflow processes through the operational tools described above, these processes may still be ineffective without an explicit focus on results. Report cards with customized information for each school, subdistrict, or district, one way to provide better accountability and address this concern,6 are part of the interventions created by this impact evaluation. The report cards are prepared using data from the Ministry of Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Education’s annual school census. To make them easy to understand for illiter- ate parents and members of poor rural communities, the content is kept simple. At the school level they contain a small set of performance indicators, information on enrollments and resources endowments, and selected compara- tive data that show a school’s performance and resource endowment rank rela- tive to other schools. Schools fall into one of four performance categories, each marked by a relevant icon: excellent, satisfactory, in dif�culty, and disappoint- ing.7 At the subdistrict and district levels the report cards contain aggregate information on the same indictors as the school report cards, indicate the per- formance category to which the subdistrict or district in question belongs, and lists the schools or subdistricts in each performance category. At the district level report cards can serve as an instrument for self- evaluation and as a basis for designing and implementing an action plan for improvement. At the school and subdistrict levels report cards can likewise help focus attention on results and encourage action toward better outcomes. At the school level recent policy changes have been, until the coup in early 2009, strengthening the prospects in this regard. In the second phase of the Programme National pour l’Ame ´ lioration de l’Education (Government of Madagascar 1997), the government decided to encourage greater participation in school-based management by various actors through the nationwide contrats-programmes initiative, which set out explicit responsibilities for parents, teachers, and school directors. It essentially sought to foster commit- ment by the various parties to shared targets for increases in enrollment and examination pass rates and decreases in teacher absenteeism and to explicit contributions of materials and labor for classroom renovation and construc- tion. School report cards can provide key information to clarify and motivate 6. School report cards have been used extensively in other countries; for an analysis of their impact on student performance, see, for example, Banerjee and others (2008) and Hanushek and Raymond (2004). 7. A school is “excellent� when it achieves above average results despite receiving a smaller endowment of resources than average, “satisfactory� when it achieves the expected above average result with an above average endowment of resources, “in dif�culty� when it achieves below average results with a below average endowment of resources, and “disappointing� when it achieves below average results despite being endowed with more than the average amount of resources. A sample school report card is shown in �gure A1. Lassibille, Tan, Jesse, and Van Nguyen 311 decisionmaking by the relevant actors. For this reason, one of the interventions in impact evaluation reported below included structured school meetings with the staff of the school, parents, and community members in order to engage them in purposeful discussion about the school improvement plan for their school in light of the information in its report card, thus promoting parental monitoring of the school and accountability. To summarize, the interventions under consideration by the Madagascar Ministry of Education to tighten management of the primary education system consisted of providing workflow templates or tools, report cards, and related Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 instruction guidebooks; facilitating meetings between school staff and the com- munity to develop and agree on a school improvement plan; and structuring training sessions to follow a set agenda that informs and motivates teachers, school directors, and subdistrict and administrators toward better performance (the Ame ´ lioration de la Gestion de l’Education a` Madagascar [AGEMAD] interventions). III. DESIGN AND I M P L E M E N TAT I O N OF THE I M P A C T E VA L U A T I O N Because of the diversity of actors and the large number of workflow processes and operational templates involved, many possibilities exist for implementing the interventions presented above. Because the operational tools by themselves might not be suf�cient to improve productivity unless combined with account- ability measures to focus attention on schooling outcomes, the evaluation team, in discussion with counterparts in the Madagascar Ministry of Education, packaged the interventions and targeted them to two main sets of actors: midlevel bureaucrats (subdistrict and district administrators) and front- line service providers at the school level (teachers and school directors). Figure 1 provides an overview of the intervention design of the experiment,8 which addresses three important questions. How much do the AGEMAD inter- ventions affect the way schools function and their performance in terms of key outcome indicators? Would subdistrict- or district-level interventions produce the desired impact on service provider behaviors and on schooling outcomes, or would such interventions be ineffective unless reinforced with direct follow-up and training at the school level? Would all schools in a subdistrict require direct follow-up and training to achieve the desired impact, or would it be suf�cient to target some schools to serve as demonstration projects for repli- cation in the remaining schools? These questions are addressed with an eye to obtaining results that inform the strategy for scaling up the interventions by implementing a nested random- ized experiment (see, for example, Boruch 1997; Dennis and Boruch 1989; and 8. For a more complete picture of the experimental design, see �gure A2, which details the precise nature and type of interventions in each treatment group and the population and sample sizes involved. 312 THE WORLD BANK ECONOMIC REVIEW F I G U R E 1. Stylized Description of the Experimental Design Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Source: Authors’ construction. Duflo, Glennester, and Kremer 2007). First, 30 relatively accessible and similar districts were selected as the universe for the experiment,9 with 15 districts ran- domly chosen to receive the interventions and the remaining 15 chosen to be managed as usual (as control districts). The 30 districts contain 573 subdis- tricts and 6,488 schools. For practical reasons, the most remote and inaccess- ible subdistricts and schools were removed (107 subdistricts and 2,714 schools), leaving 466 subdistricts and 3,774 schools, with 259 subdistricts and 2,053 schools in the intervention districts and 207 subdistricts and 1,721 schools in the control districts. From the 3,774 schools, a sample of 1,212 schools in 179 subdistricts was randomly selected to evaluate the impact of the interventions. With 303 schools in each intervention group, the sample size is large enough to detect the impact of the interventions on behaviors and indi- cators of school performance. The relevant administrators in the intervention districts received oper- ational tools, corresponding guidebooks explaining the use of the tools, and training relevant to their tasks, along with a district report card. Within the intervention districts, some subdistricts were randomly selected to receive 9. Power calculations show that this sample size is suf�cient to detect a 0.25 standard deviation effect in school-level average test score, assuming a 0.1 correlation across schools within a district. To save space, baseline data on the pro�les of the experimental and nonexperimental schools are not shown here but are available on request from the authors. To summarize, the data indicate that the 30 experimental districts are comparable to the 81 nonexperimental districts in number of students and teachers, percentage of repeaters, and number of schools managed by each subdistrict administrator. However, on average, the district administrators in the experiment are responsible for more subdistrict administrators and therefore for more schools than the district administrators in the nonexperimental districts. Lassibille, Tan, Jesse, and Van Nguyen 313 operational tools relevant to subdistrict-level functions, corresponding guide- books, speci�c training, and a subdistrict report card. The other subdistricts received only training and a report card, schools in these subdistricts received only a report card. A sample of schools in the intervention subdis- tricts was randomly selected to receive the operational tools designed for use by school directors and teachers, corresponding guidebooks, training, and a school report card to be discussed in meetings with the community. For the other schools, subdistrict administrators received school report cards for onward distribution and templates of the school-level tools for replication. Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 These tasks were left to the initiative and discretion of these administrators.10 To summarize, the nested design of the experiment creates four types of schools: schools in the control districts, which receive no intervention at any level in the system (control group), schools in intervention districts and in nonintervention subdistricts with no school-level interventions (group 1), schools in intervention districts and in intervention subdistricts with no school-level interventions (group 2), and schools in intervention districts and in intervention subdistricts that directly received school-level interventions (group 3). The diffusion of interventions to schools in groups 1 and 2 is left to the initiative of the subdistrict or district administrator, following the training-of-trainers model extensively used in Madagascar ( formation en cascade). By contrast, schools in group 3 receive the interventions directly, and these school-level processes are reinforced at the subdistrict and district levels.11 Since intervention assignment was random, comparing these four groups of schools gives a consistent estimate of the impact and makes it possible to answer key questions about the interventions. Comparing group 1 and the control group reveals the effect of a training-of-trainers or “cascade� model that involves only the district-level administrators to implement the interven- tions. Comparing group 2 and the control group reveals the impact of a more intensive form of the cascade model that involves both subdistrict- and district-level interventions. Comparing group 3 with the control group reveals the maximum impact of the full set of interventions at the school, subdistrict, and district levels. Comparing groups 2 and 3 reveals the marginal impact of the most intensive package of interventions that directly target teachers and school directors along with subdistrict and district administrators over the 10. Schools and subdistricts in intervention districts were randomly assigned to one of the interventions after the sample was strati�ed by school size and repetition rate. Within the largest districts, 40 intervention schools were selected from each of about 10 subdistricts, which were themselves randomly selected; and within the smaller intervention districts, 13 intervention schools were selected at random from each of about 4 subdistricts, which were themselves randomly selected. 11. The interventions received by schools in group 3 are inherently more costly than those received by schools in groups 1 and 2 A central issue in the impact evaluation is to determine whether they achieve greater impact than the less costly packages. 314 THE WORLD BANK ECONOMIC REVIEW second-most intensive package that directly targets only subdistrict and district administrators.12 Because schooling processes take time to evolve and produce results, the experiment took place over two school years. In the �rst year, changes, if any, were expected mostly in the behaviors of the various actors, and in the second year, these changes would have had suf�cient time to translate into changes in at least some of the schooling outcomes. The experiment started in September 2005 and ended in June 2007. A team comprising central-level ministry staff was formally constituted to oversee the Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 experiment’s implementation. It was reinforced by technical experts and facili- tators (the latter supervised by Aide et Action, a nongovernmental organization that assisted in implementing the school-level activities). In total, more than 50 people were involved in the impact evaluation. Some 200,000 operational tools and 11,000 guidebooks were distributed to the actors, and some 10,000 report cards were produced for school directors and subdistrict and district adminis- trators by the Ministry of Education using school census data. Training modules, role-playing games, and supporting material for each set of actors were provided to nearly 4,000 participants. In the �rst year, central-level minis- try staff involved in the experimentation organized and delivered a four-day course for the subdistrict and district administrators, who in turn delivered a two-day course for the teachers and school directors under their supervision. In the second year, training for the various actors was shortened to two days for the subdistrict and district administrators and to just one day for teachers and school directors. And over the two years, 1,500 meetings between schools and their communities were organized with the support of specially recruited facili- tators to discuss and prepare the school’s improvement plan. Data on the behavior of school personnel were collected from December to May during each of the two school years using a school questionnaire adminis- tered following random unannounced visits to the schools in the experiment. The questionnaire yielded detailed information on the pedagogical and admin- istrative organization of the school, on the personal characteristics of the tea- chers and their quali�cations, and on teacher and student absenteeism. It also gathered data on various aspects of teacher behavior and how teachers do their jobs. The dataset contains information on some 4,000 teachers in the 1,200 schools involved in the impact evaluation.13 Besides the school-level data, information was also collected from subdistrict and district administrators during the second year. And at the end of the exper- iment, some 850 administrative and pedagogical tools that were used by tea- chers and school directors in 2006/07 were collected from 40 schools 12. The comparison thus measures the impact of the most intensive package of interventions after netting out the direct and indirect effects of a package centered on the administrators in which a cascade of bene�ts flowing through the relevant administrators to the school-level personnel under their supervision. 13. Five teachers were randomly selected in each school. Lassibille, Tan, Jesse, and Van Nguyen 315 randomly selected from those in the control group and in group 3. Because this information is codi�ed directly from the artifacts actually used by teachers and school directors in the course of their work, it provides an independent exter- nal check on the data reported by school personnel.14 As indicated below, there is a high degree of consistency between the two sources of information, a feature that strengthens the analytical results based on the reported data. In terms of schooling outcomes, the indicators used to evaluate the impact of the interventions are based on school census data (Fiches de Fin d’Anne ´ e and Fiches d’Enque ˆ tes Rapides) on the number of repeaters and dropouts and on Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 pass rates on the primary school leaving exam (Certi�cat d’Etudes Primaires Ele´ mentaires, CEPE) for grade 5 students. Some schools have no candidates sitting for the CEPE exams; so to compare the schools across the full spectrum of outcomes, achievement tests were administered in both years of the exper- iment. The baseline test was administered in February 2006 to about 25,000 stu- dents in grade 3; the post-intervention test was administered in May 2007 to about 22,000 students in the same cohort (in grade 4). The test instrument is based on test items from the 2005 Programme d’Analyse des Syste ` mes Educatif de la Confemen survey; in both years, a maximum of 25 students per school were tested in French, Malagasy, and mathematics. To minimize selection bias in the data, assiduous effort was made to administer the tests to all students in the baseline sample. The survey enumerators were thus instructed to identify stu- dents who were absent from school on the day of the test, seek out these students in their homes, and administer the tests at home whenever feasible.15 I V. E M P I R I C A L R E S U L T S The empirical results are presented here in two parts. The �rst pertains to the direct impact of the interventions on the behavior of school personnel. The second pertains to the indirect effect of the interventions on students and their schooling outcomes. Because the experiment is randomized, the impact of a 14. Detailed data on artifacts, such as those collected in this experiment, are extremely rare— unsurprising, given that teachers may be unwilling to part with a source of information that provides such a complete a record of their work. In The Gambia, for example, Adekanmbi, Blimpo, and Evans (2009) report on an ongoing impact evaluation in which the enumerators collected data on only one artifact—the lesson plan—and then only by asking teachers to show it to them during the visit. The study found that 35 percent of the teachers surveyed declined to show their lesson plan while claiming to have it, 17 percent did not have a lesson plan at all, and 48 percent said they had a lesson plan and were able to show it. 15. The baseline characteristics of schools in the intervention and control groups are summarized in table A1. The data show that schools in the various groups are comparable in terms of size (about 250 students), number of teachers (four), repetition rate (about 18 percent), CEPE pass rate (about 61 percent), percentage of correct answers on the test administered in year one (about 61 percent), and loss of students from the sample between the two school years spanning the AGEMAD experiment for whom test scores were collected (about 11 percent). An important feature in the resulting database is that differences in baseline test scores between students who dropped out of the sample and those who remained were small and comparable across intervention and control groups. 316 THE WORLD BANK ECONOMIC REVIEW particular intervention on the behavior of school personnel and on student out- comes ( y) is estimated using the following regression in year 2: X 3 ð1Þ y¼aþ bi gi þ e i¼1 where gi is an indicator for whether the school is in an intervention group i (with i ¼ 1, 2, or 3), with the control group being the omitted category; a measures the average value of y in the control group; and bi estimates the Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 impact of each intervention on y. A similar formulation is used to evaluate the pairwise impacts of the three interventions relative to each other. Direct Effects on Service Providers’ Behaviors Alternative methods exist for evaluating and presenting the direct effects of the interventions because many actors are involved, each with multiple responsibil- ities. This article focuses on the school personnel closest to the students’ daily learning activities, the teachers and school directors. It further focuses on the seven tasks for each group that Malagasy educators deem essential to the mission of managing for results (table 2). For teachers these tasks include taking daily roll call, preparing the lesson of the day, monitoring student learning, helping lagging students, and the like. For school directors the tasks include keeping a register of enrollments, analyzing student absences on a regular basis, following up lesson planning with teachers, reviewing student performance, and so on. To keep the analysis tractable, a “good� teacher (or more accurately, a minimally conscientious teacher) and a “good� school director are de�ned here as one who performs all seven workflow tasks that Malagasy educators consider essential to the role. In the same vein, a well managed school is one where the school director and all the teachers perform all their essential tasks.16 With regard to teacher absenteeism, the interventions had no signi�cant impact.17 Absenteeism was about 9 percent in the control group, compared with 9–10 percent in the three intervention groups (table 3). Teacher absentee- ism is much lower in Madagascar than in other developing countries such as Bangladesh, India, and Uganda (Chaudhury and others 2006). It is linked mostly to the fact that many teachers must travel to a central location, often 16. An alternative approach is to follow the methodology of Kling, Katz, and Liebman (2007), who analyze program effects using a seemingly unrelated regression for each task—either separately or in clusters of tasks. This approach was considered for this article, but ultimately the judgment of Malagasy educators that the individual tasks are an integrated package of closely connected actions required for managing the teaching and learning process was accepted. 17. Data on teacher absenteeism are based on information supplied by the school directors in the sample schools to the enumerators, who make unannounced visits following a random schedule. Teachers are considered absent if they were absent, for whatever reason, for at least one day during the week preceding the visit. Lassibille, Tan, Jesse, and Van Nguyen 317 T A B L E 2 . Tasks Considered by Malagasy Educators to be Essential for Teachers and School Directors Teachers School directors Takes daily roll call Keeps a register of enrollments Prepares daily lesson plan Signs off on daily roll call Prepared bimonthly lesson plans Analyzes student absences on a monthly or bimonthly basis Monitors student learning Reviews student test results Has tested students during the past two Takes stock of teacher absences months Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Helps lagging students Informs subdistrict or district administrator of teacher absences Discusses student learning issues with Follows up with teachers on lesson planning school director Source: Authors’ construction based on Government of Madagascar 2004a. far from their place of work, to collect their salaries. In this context, absentee- ism cannot be tackled simply by tightening supervision to ensure that teachers report to work. The problem may require action outside the education sector— for example, to replace the current method of paying teachers with a more con- venient and secure system. With regard to task execution, 42 percent of teachers in the control group performed all the tasks deemed essential for good classroom management, and 24 percent of schools saw all teachers perform all the essential tasks (see table 3). The corresponding values rise to 63 percent and 43 percent, respect- ively, for schools in group 3, and the differences are statistically signi�cant. In groups 1 and 2 about 53 percent of teachers performed all the tasks, and 30– 36 percent of schools had all teachers perform all tasks, but the differences between these values and those for the control group are not statistically signi�- cant. These results show that the interventions changed behavior only when targeting the entire chain of service delivery, with schools bene�ting directly from the interventions and indirectly through the interventions at the subdis- trict and district levels. In schools where the interventions cascade down to schools indirectly through subdistrict and district administrators, the impact on the extent to which teachers perform their essential tasks is limited. The results for well managed schools are similar. The share of well managed schools in group 3 exceeds the corresponding share in the control group by 22 percentage points, and the difference is statistically signi�cant. The correspond- ing difference for groups 1 and 2 is also positive but not statistically signi�cant. Even for schools in group 3, however, there is no room for complacency. After bene�ting from the interventions for two years, the share of well managed schools in this group is still only 37 percent, perhaps reflecting the fact that it takes time to change behavior. Comparisons across the three intervention groups can be used to evaluate the existence of spillover effects that might arise from the diffusion of practices T A B L E 3 . Impacts of the Interventions on Providers’ Behaviors and Schooling Outcomes (percent) 318 Differences between intervention Diffusion of the programa Differences with respect to control groupb groupsc Direct to Direct to teachers and Via subdistrict teachers and Via subdistrict school and district Via district school and district Via district Group 2 Group 3 Group 3d Control directors administrators administrators directors administrators administrators versus versus versus Indicator groupa (group 3d) (group 2) (group 1) (group 3d) (group 2) (group 1) group 1 group 1 group 2 Impact on service provider behaviors Teacher absenteeism 9.2 8.7 9.9 10.1 –0.5 0.7 1.0 ratee –0.2 – 1.4 – 1.2 (1.3) (1.4) (1.4) (1.8) (1.7) (1.5) Teachers performing 42.4 63.0 53.6 53.4 20.6** 11.2 11.1 0.2 all tasksf THE WORLD BANK ECONOMIC REVIEW 9.6*** 9.4*** (9.7) (9.6) (9.5) (4.1) (4.2) (3.3) Schools with performing 23.9 42.8 36.4 30.0 18.9** 12.5 6.1 6.4 all all tasksg teachers 12.9*** 6.4 (9.0) (7.7) (7.5) (5.3) (5.3) (6.4) Well 14.6 36.7 23.5 22.0 22.2*** 9.0 7.5 1.5 14.7*** 13.2*** managed (8.5) (8.0) (7.2) (4.4) (5.0) (4.7) schoolsg Impact on students’ schooling outcomes and learning Attendance 86.6 90.7 88.1 89.6 4.1** 1.5 3.0 –1.5 1.1 2.6 rateh (1.9) (2.3) (2.1) (1.1) (0.9) (1.4) Repetition 22.6 17.5 20.0 18.1 – 5.1** –2.6 –4.5** 1.9 – 0.6 – 2.5 ratei (1.9) (1.9) (2.0) (1.2) (1.3) (1.5) Dropout 6.1 5.5 5.5 4.3 – 0.6 –0.6 –1.8 1.2 1.2 0.0 ratej (1.6) (1.5) (1.5) (1.3) (1.4) (1.4) Pass rate at 69.1 73.0 76.3 76.7 3.9 7.2 7.6 –0.4 – 3.7 – 3.3 CEPEk (5.9) (5.9) (5.8) (2.9) (3.2) (2.4) Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Year two test score l French 29.9 30.5 29.1 29.7 0.6 –0.7 –0.7 0.0 1.3 1.4 (2.5) (2.5) (2.5) (1.1) (1.2) (0.8) Mathematics 50.0 51.7 49.2 48.9 1.6 –0.8 –1.1 0.3 2.7** 2.5** (2.9) (2.9) (2.9) (1.5) (1.5) (1.1) Malagasy 49.8 52.1 50.1 49.8 2.3 0.3 –0.3 0.3 2.3 2.0** (2.4) (2.3) (2.4) (1.3) (1.3) (0.9) All three 42.7 44.3 42.3 42.2 1.5 –0.4 –0.6 0.2 2.1 1.9** subjects (2.5) (2.4) (2.5) (1.2) (1.2) (0.9) *** Signi�cant at the 1 percent level; ** signi�cant at the 5 percent level. Note: Numbers in parentheses are standard errors. a. Average; observations are weighted by the probability of selection of the school. b. Standard errors are clustered at the district level. c. Standard errors are clustered at the subdistrict level. d. Interventions at the school level are combined with those at the subdistrict and district levels. e. Absenteeism during the week before the school visit; information provided by the school director; the unit of observation is the school. f. The unit of observation is the teacher. g. Schools with up to �ve teachers only; the unit of observation is the school. h. Attendance during the month before the school visit; information provided by the school director; the unit of observation is the school. i. Percentage of repeaters in the total enrollment; the unit of observation is the school. j. Rate of net exit; impacts on this rate are evaluated at the end of the �rst year of the experiment; the unit of observation is the school. k. Percentage of students who pass the Certi�cat d’Etudes Primaires Ele ´ mentaires (CEPE); impacts on this rate are evaluated at the end of the �rst year of the experiment; the unit of observation is the school. l. Raw percentage of correct responses; the unit of observation is the student. Source: Authors’ analysis of the 2006/07 Ame ´ lioration de la Gestion de l’Education a ` Madagascar (AGEMAD) school survey, the 2007 AGEMAD posttest, the 2005/06 Fiches de Fin d’Anne ´ e, and the 2006/07 Fiches d’Enque ˆ tes Rapides. Lassibille, Tan, Jesse, and Van Nguyen 319 Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 320 THE WORLD BANK ECONOMIC REVIEW and knowledge by administrators with responsibility for multiple schools in different locations. Schools in group 2 were not better managed than those in group 1, however, suggesting that the interventions that bene�ted subdistrict administrators did not reinforce the impact of the interventions that bene�ted the district administrators. Comparing groups 2 and 3 also shows no evidence of diffusion of the package of practices in schools that bene�ted directly from the interventions to other schools in the same subdistrict and districts under the supervision of the same administrators. These results imply that targeting some schools to serve as demonstration project for replication by the subdistrict Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 and district administrators in the remaining schools under their care is unlikely to be effective.18 Among the possible reasons: lack of interest and motivation by subdistrict and district administrators, which may be linked to their heavy work load and also to their low level of technical competence; weak mechan- isms for monitoring and supervising schools by the subdistrict administrators, who are most directly responsible for this part of the workflow in the service delivery chain; and absence of a true leadership culture among both the subdis- trict and district administrators. In subdistricts and districts where only the administrators received the interventions, practically none of the schools received the workflow templates designed to tighten management of the peda- gogical processes or training or support,19 and in 75 percent of the schools report cards were never discussed with the communities. A legitimate concern about the conclusions thus far is that the underlying information is based on data reported by teachers and school directors during unannounced school visits by survey enumerators and may be biased since the enumerators did not directly examine relevant workflow artifacts (such as teacher attendance records or lesson plans). The latter approach was proposed but rejected by government counterparts as being impractical for two reasons: it would have put the (relatively junior) enumerators in the same position as school inspectors, and it would have required a overly costly commitment of time to capture all the details of the workflow processes managed by each teacher and school director in the sample. Instead, an external check on the data collected by the survey enumerators—by gathering and codifying the workflow artifacts (cov- ering an entire school year) from a sample of randomly selected schools, 20 in the control group and 20 in intervention group 3—was used (see section III). This unique database reveals how teachers and school directors in each group performed their duties throughout the school year, in particular regard- ing following actions: monitoring and following up on student absenteeism, preparing lesson plans, and tracking progress in student learning (table A2). The data unequivocally indicate that after two years of bene�tting from the 18. The �nding that interventions at the subdistrict and district levels have no signi�cant impact on teachers’ behavior does not rule out the possibility of an impact beyond the two-year time frame of the experiment reported in this article. 19. Only 3 percent of the schools in group 1 and 2 received a copy of the tools from the subdistrict administrators. Lassibille, Tan, Jesse, and Van Nguyen 321 interventions, teachers in group 3 were more conscientious in executing their duties than teachers in the control group, a �nding that con�rms the conclusion based on self-reported information. To illustrate, compared with teachers in the control group, those in group 3 monitor student absences for an average of 12 more days during the year; prepare lessons for a larger portion of the school year, covering an average of one more bimester; and record their work in log- books more than 1.5 times as often and for twice as long during the school year. Teachers in group 3 appear to have a better grasp of their students’ edu- cation progress and learning dif�culties, and the information they communicate Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 to parents via student report cards is more comprehensive and exhaustive. School directors in group 3 are also more conscientious in exercising their supervisory and monitoring duties. Indirect Effects on Students’ Schooling Outcomes and Learning The �rst of the schooling outcomes pertains to student attendance at the school level, which is based on the number of students present at school during the month before the survey. This information was provided by the school directors during the unannounced school visits. Other measures of schooling outcomes include dropout and repetition rates and pass rates on the CEPE exam taken at the end of the primary cycle—all of which are computed from administrative data maintained by the Ministry of Education.20 Finally, data on learning are from the achievement test scores administered at the end of the experiment. Interventions at the school level reinforced by those at the subdistrict and district levels signi�cantly improved student attendance compared with the control group, boosting the rate by about 4 percentage points (see table 3). The interventions also had a positive and signi�cant impact on grade repetition, reducing it by 5.1 percentage points. By contrast, interventions left to the initiative of subdistrict and district administrators had no signi�cant impact on either student attendance or grade repetition. While the package of interventions targeting school personnel as well as the subdistrict and district administrators that supervise them helped reduce student absenteeism and repetition, it did not reduce dropout rates or improve pass rates on the CEPE exam. Likewise, the interventions targeting only administrators failed to alter any schooling outcomes apart from lowering the repetition rate in intervention group 1. As noted earlier, however, the time frame for evaluation was relatively short, particularly in view of the processes involved in altering beha- viors; the results observed so far may still be partial or incomplete at this time. As for student learning, students in the schools receiving interventions throughout the whole service delivery chain had better scores in all three tested subjects than their peers in the control schools did, but the gains were not 20. For practical reasons, the data on dropout and CEPE pass rate are for 2005/06. 322 THE WORLD BANK ECONOMIC REVIEW statistically signi�cant.21 The comparisons between group 3 and groups 1 and 2 are also interesting: students in group 3 achieved higher scores than those in group 1 (students in schools that received the interventions through district administrators only), with the gain in mathematics being statistically signi�- cant. The students in group 3 also outperformed those in group 2 (students in schools that received the interventions through subdistrict and district adminis- trators), with statistically signi�cant gains in Malagasy and mathematics but not in French. The absence of a pattern of unambiguous gains in test scores across all three subjects is not surprising given the short duration of the evalu- Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 ation period. And the absence of gains in French in any of the comparisons is also consistent with the fact that most Malagasy primary school teachers have a poor grasp of the language themselves, an impediment that better workflow processes can hardly be expected to address. In sum, the results on schooling outcomes are consistent with the earlier �ndings. In its most direct and intensive form, the interventions changed the behavior of all actors toward better management. These changes translated immediately into increases in student attendance and sizable reductions in dropout rates. However, changing service providers’ behavior takes time and effort, and a two-year time frame was probably too short to produce clear-cut impacts on student test scores. V. C O N C L U S I O N Inadequate funding does not appear to be the only reason for the poor per- formance of Madagascar’s primary schools. A detailed analysis of how edu- cation is delivered in schools reveals that many aspects of the pedagogical process are poorly managed and that far too many school personnel and administrators neglect tasks deemed essential for student learning. As men- tioned earlier, 20 percent of teachers do not prepare daily lessons plans, school directors rarely follow up with their teaching staff on student performance, and in only 15 percent of the sample schools do all the teachers and school direc- tors consistently perform the package of seven tasks considered essential by Malagasy educators. There is thus substantial scope to improve the manage- ment of the pedagogical process as part of the country’s effort to raise the per- formance and ef�ciency of public primary schools. The randomized experiment to evaluate selected interventions to streamline and tighten the workflow processes of keys actors yields some interesting results. A package of intensive and direct interventions involving school-level personnel and subdistrict and district administrators changed the behavior of teachers and school directors toward more conscientious execution of the tasks for which they are responsible. It also improved school attendance, reduced grade rep- etition, and raised test scores, although the gain in scores was not statistically 21. Larger sample sizes might have detected statistically signi�cant gains. Lassibille, Tan, Jesse, and Van Nguyen 323 signi�cant. A laissez-faire (and less costly) version of the interventions, targeting only the subdistrict and district administrators—in the hopes that they would in turn disseminate and supervise implementation of the interventions to improve workflow processes at the school level—proved largely ineffective. Because the experiment did not test the impact of simply providing resources without the managerial changes, the relative roles of these factors cannot be separated out. Beyond their intrinsic interest, the results also offer a basis for exploring the implications for policy development in Madagascar. Scaling up the most effective interventions among those evaluated—that is, interventions directly targeted to Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 teachers and school directors and reinforced by support from subdistrict and dis- trict administrators—by simply replicating the arrangements used during the experiment would almost certainly cost too much. The reason is that during the experiment, the option of relying on existing institutions to carry out the training of teachers and to facilitate community-school meetings was unavailable. Thus, a separate team had to be hired and trained to implement the AGEMAD interven- tions; this approach was also the best way to adhere to the strict requirements of the experiment for implementation and data collection. In the scaling up phase, a potentially more sustainable strategy would be to integrate the AGEMAD inter- ventions into existing structures and programs. For example, ongoing training programs for teachers, school directors, and subdistrict and district administrators could include short modules to familiarize participants with the AGEMAD tem- plates and processes for workflow management and train them in using the tools. Similarly, the procurement and distribution of the workflow templates and the facilitation of school meetings could be integrated into ongoing initiatives to decentralize funding and decisionmaking to schools, districts, and regions through the system of localized school improvement grants. More broadly, scaling up the AGEMAD interventions effectively means motivating a large number of actors in the system to change their behavior at work. The task is daunting at best and will almost certainly require sustained effort to foster a culture of results-oriented management and leadership throughout the system. Finally, beyond the implications for Madagascar, the impact evaluation reported in this article also enriches discussion of promising approaches for improving educational outcomes in low-income countries. The results suggest that packaging an increase in resources to schools in the form of support for improving workflow processes and accountability measures (that is, teachers’ personal engagement in their work, proper monitoring and supervision of their work, involvement of parents and the community, and the like) is worth con- sideration. Such changes are probably especially relevant in dysfunctional schools—where teachers neglect their essential pedagogical duties, where school directors routinely fail to support and supervise the teachers, and so on. Future research could therefore be directed at enhancing knowledge about the nature of the problems and promising interventions in this regard. Such knowl- edge would inform the design of policies and programs for education advance- ment in low-income countries. 324 THE WORLD BANK ECONOMIC REVIEW APPENDIX F I G U R E A1. School Report Card Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Source: Authors’ construction. Lassibille, Tan, Jesse, and Van Nguyen 325 F I G U R E A2. Experimental Design and Description of the Interventions Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Note: Size of the follow-up sample in parentheses. Source: Authors’ construction. T A B L E A 1 . Baseline School Characteristics and Attrition in End Line Test Scores Differences with respect to control group Direct to Via subdistrict and teachers and district Via district Control school directors administrators administrators Characteristic group (group 3a) (group 2) (group 1) Baseline school characteristics Number of students 239.5 – 4.8 – 25.3 – 39.3 (28.2) (24.9) (23.3) Number of sections 5.7 – 0.0 – 0.4 – 0.4 (0.3) (0.3) (0.2) Number of teachers 4.4 0.0 – 0.4 – 0.2 (0.4) (0.3) (0.3) Repetition rate (percent) 17.7 3.7 2.2 2.9 (2.0) (2.0) (2.0) Pass rate at CEPEb 60.7 1.7 1.4 1.7 (percent) (3.0) (3.0) (3.0) (Continued ) 326 THE WORLD BANK ECONOMIC REVIEW TABLE A1. Continued Differences with respect to control group Direct to Via subdistrict and teachers and district Via district Control school directors administrators administrators Characteristic group (group 3a) (group 2) (group 1) Year one test scoresc 60.9 0.7 – 1.0 – 1.0 (percent) (1.9) (1.8) (1.9) Loss of sample for year Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 two test Loss rated (percent) 11.1 3.1 1.4 3.4 (2.2) (1.5) (2.2) Difference in year one test – 0.68 2 2.7 2 0.2 2 2.0 scores between quitters (1.3) (1.6) (1.6) and stayers in year 2e (percentage points) Note: Numbers in parentheses are standard errors clustered at the district level. a. Interventions at the school level are combined with those at the subdistrict and district levels. b. Percentage of students who pass the Certi�cat d’Etudes Primaires Ele ´ mentaires (CEPE). c. Raw percentage of correct responses in French, Malagasy, and mathematics. d. Percentage of students who did not take the year two test scores. e. Differences in the raw percentage of correct responses in French, Malagasy, mathematics in year one test scores between students according to their status in year 2 as quitters or stayers. Source: Authors’ analysis of the 2005/06 Fiches d’Enque ˆ tes Rapides, the 2004/05 Fiches de Fin d’Anne´ e, and the Ame´ lioration de la Gestion de l’Education a ` Madagascar pretest. T A B L E A 2 . Task Execution Rates by Teachers and School Directors in a Subsample of Schools (percent, unless otherwise indicated) Control Direct interventions to teachers Task schools and school directors (group 3a) Roll call (average over the school year) Number of days task was performed per month 13.0 14.5 Monthly sheets with a recapitulation of rate of 21.0 76.0 student absenteeism Monthly sheets not signed by the school 67.0 46.2 director Weekly lesson plans Teachers not using the tool 13.6 1.5 Teachers using the tool at least once in the year 86.4 98.5 Frequency of use (average number of 2.8 4.2 bimestrial segments) Teachers using weekly lessons plans 21.1 55.1 throughout the year (Continued ) Lassibille, Tan, Jesse, and Van Nguyen 327 TABLE A2. Continued Control Direct interventions to teachers Task schools and school directors (group 3a) Teachers not reporting the subtitle of the 74.1 33.3 lessons Teachers not annotating their observations on 60.5 30.3 the lesson plans Weekly lesson plans not signed by the school 49.0 35.2 director Daily lesson plan Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Teachers not using the tool 52.4 21.7 Teachers using the tool at least once in the year 47.6 78.3 Average number of days used in the year 34.1 62.7 (maximum 150) Weekly sheets with annotated observations 11.5 30.8 Teachers who have followed the weekly 38.8 51.3 lesson plans Weekly sheets not signed by the school 71.9 60.2 director Record of tests Evaluation reports with no annotation on the 62.7 52.1 number of students missing their tests Evaluation reports with no annotation on the 74.4 14.9 number of students performing above the class average Evaluation reports not signed by the school 80.0 59.0 director Student report cards With missing information on student’s class 11.7 5.0 rank With no information on the average score of 94.4 33.9 students in the class With no observations on student’s attendance 46.9 28.1 Not signed by the teacher 16.7 13.2 Not signed by the school director 80.2 62.8 Teacher’s travel pass for authorized trips b Teachers not using the tool 92.0 74.0 Teachers using the tool at least once in the year 8.0 26.0 Pass is signed on teacher’s arrival at 13.8 68.8 destination by the relevant authority Pass is signed on teacher’s leaving the locality 20.7 81.3 by the relevant authority a. Interventions at the school level are combined with those at the subdistrict and district levels. b. Teachers are routinely granted permission for travel to collect their salaries at designated localities. These trips may result in excessive absences from the classroom unless adequately supervised. Source: Authors’ analysis of the 2006/07 Ame ´ lioration de la Gestion de l’Education a ` Madagascar tools survey. 328 THE WORLD BANK ECONOMIC REVIEW REFERENCES Adekanmbi, A., M. Blimpo, and D. Evans. 2009. The State of The Gambia Lower Basic Education. Banjul, The Gambia: Department of State for Basic and Secondary Education. Banerjee, A., R. Banerji, E. Duflo, R. Glennerster, and S. Khemani. 2006. Can Informational Campaigns Spark Local Participation and Improve Outcomes? A Study of Primary Education in Uttar Pradesh, India. Policy Research Working Paper 3967. Washington, D.C.: World Bank. ———. 2008. Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India. Policy Research Working Paper 4584. Impact Evaluation Series 21. Washington, D.C.: World Bank. Banerjee, A., S. Cole, E. Duflo, and L. Linden. 2007. “Remedying Education: Evidence from Two Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 Randomized Experiments in India.� Quarterly Journal of Economics 122(3): 1235–64. Banerjee, A., and E. Duflo, 2006. “Addressing Absence.� Journal of Economic Perspectives 20(1): 117 –32. Banerjee, A., S. Jacob, M. Kremer, J. Lanjouw, and P. Lanjouw. 2005. “Promoting School Participation in Rural Rajasthan: Results from Some Prospective Trials.� Working Paper. Massachusetts Institute of Technology, Cambridge, Mass. Bobonis, G., and F. Finan. 2009. “Neighborhood Peer Effects in Secondary School Enrollment Decisions.� Review of Economics and Statistics 91(4): 695– 716. Boruch, R. 1997. Randomized Experiment for Planning and Evaluation: A Practical Guide. London: Sage. Chaudhury, N., J. Hammer, M. Kremer, K. Muralidharan, and H. Rogers. 2006. “Missing in Action: Teacher and Health Worker Absence in Developing Countries.� Journal of Economic Perspectives 20(1): 91 –116. Dennis, M., and R. Boruch. 1989. “Randomized Experiments for Planning and Testing Projects in Developing Countries: Threshold Conditions.� Evaluation Review 13: 292– 309. Duflo, E., P. Dupas, and M. Kremer. 2007. “Peer Effects, Pupil-Teacher Ratios, and Teacher Incentives.� Working Paper. Massachusetts Institute of Technology, Cambridge, Mass. Duflo, E., R. Glennester, and M. Kremer. 2007. Using Randomization in Development Economics Research: A Toolkit. Discussion Paper 6059. London: Centre for Economic Policy Research. Evans, D., M. Kremer, and M. Ngatia. 2008. “The Impact of Distributing School Uniforms on Children’s Education in Kenya.� Working Paper. Harvard University, Cambridge, Mass. Gertler, P., H. Patrinos, and M. Rubio-Codina. 2008. Empowering Parents to Improve Education: Evidence from Rural Mexico. World Bank Policy Research Working Paper 3935. Washington, D.C.: World Bank. Glewwe, P., N. Ilias, and M. Kremer. 2008. “Teacher Incentives.� Working Paper. Harvard University, Cambridge, Mass. Glewwe, P., M. Kremer, and S. Moulin. 2009. “Many Children Left Behind? Textbooks and Test Scores in Kenya.� American Economic Journal: Applied Economics 1(1): 112– 35. Glewwe, P., M. Kremer, S. Moulin, and E. Zitzewitz. 2004. “Retrospective vs. Prospective Analyses of School Inputs: the Case of Flipcharts in Kenya.� Journal of Development Economics. 74(1): 251–68. Hanushek, E. 1997. “Assessing the Effects of School Resources on Student Performance: An Update.� Educational Evaluation and Policy Analysis 19(2): 141– 164. Hanushek, E., and M.A. Raymond. 2004. Does School Accountability Lead to Improved Student Performance? NBER Working Paper Series 10591. Cambridge, Mass.: National Bureau of Economic Research. Haveman, R., and B. Wolfe. 1995. “The Determinants of Children’s Attainment: A Review of Methods and Findings.� Journal of Economic Literature 33(4): 1829– 78. Lassibille, Tan, Jesse, and Van Nguyen 329 Jamison, D., B. Searle, K. Galda, and S. Heyneman. 1981. “Improving Elementary Mathematics Education in Nicaragua: An Experimental Study of the Impact of Textbooks and Radio on Achievement.� Journal of Educational Psychology 73(4): 556– 67. King, E.M., and P.F. Orazem. 1999. “Evaluating Education Reforms: Four Cases in Developing Countries.� World Bank Economic Review 13(3): 409– 13. Kling, J.R., J.B. Liebman, and L.F. Katz. 2007. “Experimental Analysis of Neighborhood Effects.� Econometrica 75(1): 83– 119. Kremer, M., and A. Holla. 2009. “Improving Education in the Developing World: What Have We Learned from Randomized Evaluations?� Annual Review of Economics 1: 513– 42. Kremer, M., S. Moulin, and R. Namunyu. 2003. “Decentralization: A Cautionary Tale.� Working Paper. Harvard University, Cambridge, Mass. Downloaded from wber.oxfordjournals.org at International Monetary Fund on September 15, 2010 ´ lioration de l’Enseignement – Madagascar, Government of. 1997. Programme National pour l’Ame PNAE2 du MinESEB. Antananarivo: Government of Madagascar. ———. 2004a. Ame ´ lioration de la Gestion de l’e ` Madagascar : approche strate ´ ducation a ´ gique et plan d’actions. Antananarivo: Government of Madagascar. ———. 2004b. Rapport sur le suivi des objectifs du mille ´ naire pour le de ` Madagascar. ´ veloppement a Antananarivo: Government of Madagascar. ———. 2008. Ele ` me e ´ ments de diagnostic du syste ´ ducative ´ ducatif malagasy. Le besoin d’une politique e ´ naire et la re nouvelle pour l’atteinte des objectifs du mille ´ . Antananarivo: ´ duction de la pauvrete Government of Madagascar. Miguel, E., and M. Kremer. 2004. “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities.� Econometrica 72(1): 159– 217. PASEC (Programme d’Analyse des Syste ` mes Educatifs de la Confemen). 2007. Quelques pistes de ´ flexion pour une e re ´ ducation primaire de qualite ´ pour tous. Rapport Madagascar. Dakar: Confe ¸ ais en Partage. ´ rence des Ministres de l’Education des Pays ayant le Franc Tan, J.P., J. Lane, and G. Lassibille. 1999. “Student Outcomes in Philippine Elementary Schools: An Evaluation of Four Experiments.� World Bank Economic Review 13(3): 493– 508. Todd, P., and K.I. Wolpin. 2006. “Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling.� American Economic Review 96(5): 1384–1417. World Bank. 2002. Education and Training in Madagascar: Toward a Policy Agenda for Economic Growth and Poverty Reduction. Washington, D.C.: World Bank. ———. 2008. Service Delivery in the Education and Health Sectors in Madagascar. Washington, D.C.: World Bank.