75722 DIME BRIEF The transformative effect of managing for results in primary education in Madagascar The Development Impact Evaluation Initiative is a AGEMAD broad-based World Bank Over the past decade there has been impressive progress in access to program to generate primary education in Madagascar. However, enormous challenges remain knowledge on the to improve pedagogy. Within primary schools, many aspects of the effectiveness of pedagogical processes are poorly managed and tasks that are essential for government programs. It student learning are neglected. Results from a 2005 survey confirmed that supports government agencies adopt a culture in many schools student absenteeism and student academic performance of real time evidence- are poorly monitored and that teachers fail to conduct essential based policy-making on pedagogical tasks such as preparation of lesson plans. the basis of rigorous impact evaluation. By The Malagasy Ministry of Education assembled a team of educators to testing how to make examine the options for renewing and rationalizing the tools for school policies work, it management. The team worked with sustained support from the Bank contributes to improving policy performance. team to elaborate a new set of tools for educational management focused mainly at the school and classroom levels. They consulted extensively with DIME works with 300 their colleagues in regional offices and with school heads and teachers, a agencies in 72 countries process that yielded a new set of tools that was both validated and owned across 15 thematic by practitioners in the system. programs to generate knowledge, improve The Ministry of Education thus moved to implement AGEMAD quality of operations and (Amélioration de la Gestion de l’Education à Madagascar), a far reaching strengthen country capacity for evidence- program to improve workflow processes in the school system, and increase based policy- making. results focus. The program stressed responsibilities and accountability processes through provision of management training, tools and guides. It This impact evaluation is also facilitated school-community interactions and promoted part of the Africa Program accountability for results through school report cards. Report cards for Education Impact became the basis for school meetings’ discussion for developing school Evaluation. improvement plans. Impact evaluation The Ministry of Education was keen to test the efficacy of the new tools before scaling them up country-wide. Thus, the idea of a rigorous impact evaluation took root. In preparation for the impact evaluation, the Bank facilitated the Malagasy team's training in impact evaluation, and ensured their active engagement with external technical experts to conceptualize the design of the evaluation and the questionnaires for data collection. The impact evaluation was designed to understand whether management and accountability interventions altered the behaviors of service providers and schooling outcomes. In addition, understanding how intensive the treatment would need to be was of great policy interest. DIME Brief on Madagascar’s AGEMAD 1 Impact of AGEMAD (%) Control schools AGEMAD schools 9.2 Teacher absence 8.7 42.4 Teacher completes all key tasks * 63 23.9 All teachers in the school perform their key tasks * 42.8 14.6 Well managed schools * 36.7 86.6 Student attendance * 90.7 Repetition * 22.6 17.5 6.1 Drop out 5.5 69.1 Success rate at CEPE exam (Certificat d'Etudes Primaires Elémentaires) 73 50 Math test (correct answers) 51.7 49.8 Malagasy test (correct answers) 52.1 29.9 French test (correct answers) 30.5 42.7 All subjects combined score 44.3 * Statistically Significant DIfference 0 10 20 30 40 50 60 70 80 90 100 To this end, a randomized experiment was performing their key tasks rose by 20 carried out over the course of school years percentage points. These tasks include 2005/06 and 2006/07 in a representative monitoring student attendance, preparing sample of school districts. School districts were lessons plans, and evaluating student randomly assigned to control and three performance. treatment arms. In each treatment arm the intervention was targeted towards different In turn, teacher behavior seems to have levels of the school administration chain— positively impacted students’ behavior. Student district level only, district and sub-district levels, attendance improved by 4 percentage points and district, sub-district and school levels. The and grade repetition fell by 5 percentage design made it possible for the evaluation to points. There is also indication that student separately identify the effectiveness of the drop out decreased and test scores rose, different levels of intensity and test whether although the gains were not statistically training directly only the district or both the significant by the time of the evaluation. district and sub-district level administrators Thus, AGEMAD, implemented intensively would be sufficient to affect school level across administrative levels, succeeded in behaviors. changing the behavior of school actors towards better management and learning outcomes. Impact evaluation results This contrast with interventions limited to the After two years of implementation and when the district and sub-district levels which were interventions encompassed all administrative mostly ineffective. District and sub-district levels (district, sub-district and school), the levels officials were not able to play a provision of management training, tools, and the transformative role with the schools perhaps use of accountability mechanisms significantly due to a combination of weak mechanisms for changed the behavior of service providers and monitoring and control, lack of leadership, improved school management. interest or motivation and/or high work load at the district level. AGEMAD interventions had statistically significant impacts on school functioning. They contributed to a greater focus on key pedagogical tasks by teachers and principals. In treatment schools the proportion of teachers DIME Brief on Madagascar’s AGEMAD 2 Policy implications The Africa Program for Education Impact Evaluation (APEIE) In 2008, when the results of the impact evaluation APEIE was established in 2007 to improve the ability were shared and discussed with the Malagasy team, of countries to meet the MDG goals in the context of the Education for All Fast Track Initiative, which the government decided to scale up the program to has been financing the program. APEIE objectives the rest of the school system. The AGEMAD are to: management tools were integrated into the normal (i) Build technical and organizational curriculum of teacher training programs and schools capacity in the education sector to were encouraged to use the tools to train the large conceptualize and implement rigorous impact evaluations through a learning-by- number of community teachers funded by the doing approach; Madagascar's grant from the Education for All Fast (ii) Build country-level evidence on the Track Initiative Catalytic Fund. effectiveness of education policy and programmatic alternatives; and In addition, the AGEMAD impact evaluation gave Malagasy policy makers an appreciation of the (iii) Provide cross-country venues for dialogue, networking support and benefits of impact evaluation. The Ministry of publication outlets to foster cross- Education launched another impact evaluation in fertilization and peer-to-peer learning 2008 focused on school feeding. regarding the efficacy of education interventions. Beyond Madagascar, the study highlights the APEIE has been supporting 14 country-specific importance of pedagogical processes in securing impact evaluations and convening regular cross- country meetings to build capacity and discuss good learning outcomes. It also shows how a policy lessons. program like AGEMAD can affect teachers’ behavior  In Abuja in 2007, the country delegates and and cause a positive ripple effect on students’ the team task leaders received training on behavior. The provision of training, tools, and impact evaluation and international evidence information; the supervision and follow-up at key on education policies. Each team applied their new knowledge to the development of points in the administrative chain; and the the impact evaluation for their education facilitation of school-community interactions around program. Ghana was interested in evaluating school management committees and school report cards can help enhance the functioning Senegal was interested in school grants. of schools and improve student outcomes.  In Dakar in 2008, countries presented their baseline results and discussed challenges This is the first study that has put a precise value on moving forward. Already, the workshop was in the hands of the participants sharing notes to improving neglected school tasks. It has also and experiences. It was a good opportunity to shown that there is no quick fix. Improving school hear from clients and team task leaders alike.  In Accra in May 2010, the original countries management requires a concerted effort from all and new recruits into the program discussed administrative levels in the education system to how the evidence guided their education support the work of teachers and school heads at the programs and what new policy innovations they are considering moving forward. front lines of service delivery. Source: Gérard Lassibille, Jee-Peng Tan, Cornelia Jesse and Today APEIE is the first program to be fully Trang Van Nguyen (2010). “Managing for Results in integrated into a sector unit and mainstreamed Primary Education in Madagascar: Evaluating the Impact into the operations. Two economists are based in the education unit and provide full-time support to of Selected Workflow Interventions.” The World Bank operations to incorporate a results-based Economic Review, Vol 24(2). approach in the Africa education sector portfolio. http://wber.oxfordjournals.org/cgi/content/full/lhq009?ijkey=a Arianna Legovini, Muna Meky, and Jee-Peng-Tan q1zMhsZHXn8jOU&keytype=ref#content (2007) www.worldbank.org/dime To contact the authors email jtan@worldbank.org and cjesse@worldbank.org DIME Brief on Madagascar’s AGEMAD 3