Report No: ACSXXX . Myanmar. M&E Activities for Grants and Stipends (P156298) Subtask for Monitoring School Grants and Stipends (P151137) Summary of Outputs . May 19, 2016 . GED02 Education Global Practice East Asia and Pacific Document of the World Bank Standard Disclaimer: This volume is a product of the staff of the International Bank for Reconstruction and Development/ The World Bank. The findings, interpretations, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Copyright Statement: The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/ The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, http://www.copyright.com/. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. Summary of Outputs for Myanmar M&E Activities for Grants and Stipends (P156298) Subtask for Myanmar Monitoring School Grants and Stipends (P151137) Table of Contents 1. Introduction 2. Objectives and Progress 3. Activities a. Component 1 : Qualitative Assessment (Process Monitoring) b. Component 2 : Quantitative Survey (Household and School Surveys) c. Component 3 : Early Grade Reading/Math Assessment (EGRA/EGMA) d. Component 4 : Spot-checks for verification of Disbursement-Linked Indicators e. Component 5 : Just-in-Time Technical Assistance f. Dissemination 4. Reflections and Ways Forward 5. Annex: List of Outputs 1. Introduction This report summarizes outputs which have been delivered under the Myanmar M&E Activities for Grants and Stipends (P156298), which is a subtask of Programmatic AAA on Monitoring School Grants and Stipends in Myanmar (P151137). The PAAA is designed in parallel with the World Bank financed project (P146332) to provide necessary technical assistance on the monitoring and evaluation (M&E) of the school grants and stipends programs. Australian government (Department of Foreign Affairs and Trade) has been our close partner in Myanmar education sector, and has funded this PAAA. The simultaneous roll out and expansion of two government programs, notably school grants and stipends, started in June 2014, and the Ministry of Education (MoE) has been successfully implementing the programs and made a great progress over the past two years. In particular, the real time M&E data, which were generated through this activity, has supported the successful program expansion, through informing the government decision making and have empowered the MoE to lead and derive evidenced-based improvements. The initial duration of this PAAA was four years, and this subtask was initially envisaged to be delivered by October 2018. However, it has been agreed to close this subtask by May 2016, with a set of outputs which have been produced to date. This report consists of three chapters. Chapter 2 describes the objectives of the subtasks, together with achievements towards the development objective. Chapter 3 summarizes all M&E, with the specific outputs and outcomes from each activity, followed by the last chapter on reflections and ways forward. 2. Objectives and Progress The development objective is to generate evidence and promote evidence based policy making to support improvements in the implementation of the government’s school grants and stipends programs (supported under an ongoing World Bank-financed project, P146332). The following intermediate outcomes are used to track progress in achieving the program’s development objectives: (i) At least one revision is made to the operational guidelines and training manuals that reflect evidence from this task; (ii) Ministry of Education staff undertake at least one independent assessment of the program using the questionnaires developed through this task;(iii) Evidence put forward through each component of this task - the school survey, the household survey, the Early Grade Reading Assessment and the Qualitative Assessment - is cited in public fora by Ministry of Education officials and development partners. The achievement of the current PDO is on track, along with progress with intermediate indicators. The activities supported under this PAAA have generated a wealth of information on how the programs are implemented, providing critical inputs to the review of school grants and stipends programs which are nascent and the first national programs implemented by the Ministry of Education using clear criteria and elaborate operational guidelines in Myanmar. Based on lessons learned from the initial years of program implementation, an annual review has been conducted to review the program policy, leading to significant adjustment to make the programs more inclusive and efficient. For example, major improvements included (i) expansion of school grant programs to 5.2 million students in primary grades and 8.2 million students in all grades; (ii) the inception of a school grant program in monastic schools, which are attended by many poor children; and (iii) improved targeting mechanisms for the stipends program to identify the poorest children in the poorest areas, using a combination of objective indicators and participatory local consultations. In addition, client capacity, both in terms of implementing and monitoring the programs, has been increased over time. The activity provided handholding and guidance to a team of 30+ Myanmar counterparts who lead the implementation and monitoring of the programs. Such hands-on technical assistance was critical for the program success and continuity, especially during the drastic change in key counterparts due to Ministry-wide restructuring in 2015. Counterparts are demonstrating stronger leadership in all aspects of M&E, such as conducting their own field visits and consolidating key findings from various M&E activities. 3. Activities This task aimed to support a set of M&E activities in order to generate data and information for the government to evaluate how successful the stipends and school grants have been in achieving their objectives and to further improve the programs. It consisted of main five components: 1) A qualitative assessment which assesses the design and implementation of the two programs to provide recommendations on how to improve them; 2) Evaluations of the two programs through quantitative surveys at school and household levels; 3) Early Grade Reading/Math Assessments (EGRA/EGMA); 4) Spot-Checks to verify administrative reporting of Disbursement Linked Indicators; and 5) Just-in-time Technical Assistance to the Ministry of Education counterparts. The progress and deliverables under each component are described in this chapter. Component 1: Qualitative Assessment (QA) Qualitative Assessment (QA) aims to support improvements in the design and implementation procedures of the programs through regular and timely feedback collected from school/community and township levels. Save the Children has been contracted since August 2014 and has worked closely with the Ministry of Education to carry out the activities. Over the last two years, 35 townships (out of 330 total townships) have been covered by the QA, and nearly 400 schools have been visited. There has been continuous dissemination1 of QA findings to the Ministry of Education to support program implementation, and the QA has been playing an instrumental role to bring Myanmar’s schools and people into the decision making process. Specific outputs  Report: Qualitative Assessment of School Grants and Stipends Programs, Year 1 and Year 2, respectively  Presentations to the MoE (M&E Working Groups, Technical Working Groups, and Steering Committees)  QA Instruments (including key questions, training manuals, focus group discussions, key informant interview, document review) Component 2: Quantitative Survey for Evaluating the Programs Quantitative data collection included a school and household survey, aimed at assessing whether the program has managed to achieve its objectives as well as to support the implement of program design and implementation changes. Piggy-backing on the initial design, a teachers’ module is due to be incorporated to the school survey. The first data collection was conducted in September to October 2015 to collect the baseline from 800 schools in 80 townships and 1,800 households. The preliminary result from school survey was presented to the Steering Committee in February 2016 and to the major development partners in May 2016, to seek feedback and advice for the second round of data collection which is slated for September 2016. Specific outputs  Presentation on preliminary findings from the school survey  Instruments (training manuals, school survey questionnaires, household survey questionnaires)  Data (to be made publically available as public goods) Component 3: Early Grade Reading/Math Assessment (EGRA/EGMA) 1 Quarterly meetings with monitoring and evaluation working group, four monthly technical working group, and biannual steering committee meetings Early Grade Reading Assessment aims to capture learning outcomes in the early grades. Building upon the experience from the first round2 of EGRA, the MoE has internalized the EGRA. Led by the Department of Myanmar Research, the MoE has conducted two more rounds of EGRA, mobilizing enumerators from teachers’ colleges, using the instrument to collect similar information from schools in regions outside of Yangon and for the first time is administering the learning assessment in ethnic languages. Specifically, round 3 was conducted in 2016 in 14 townships in Bago, covering 1,600 students in 60 sample schools. Based on the findings from EGRA, reading interventions have been designed through a series of design workshop, reaching training to teachers from 30 schools (50 percent of total schools that were involved in the first round of EGRA) in Yangon. In parallel, an Early Grade Math Assessment (EGMA) is underway to measure foundation skills in numeracy and mathematics in the early grades. The Bank team has been closely working with three education colleges so that the teacher educators can adapt the EGMA toolkit into the Myanmar context. Launch of EGMA is anticipated in January 2017. The Bank team has been providing support and quality control to further adapt the survey instruments, to train enumerators and to help analyze and interpret data. Specific outputs  Report “Myanmar EGRA for Yangon Region (EGRA Report from the first round)�  EGMA toolkit Component 4: Spot-check The spot-checks aim to assess the veracity and integrity of the administrative data reported by the Ministry of Education, in order to give external validation to support the verification of the Disbursement Linked Indicators (DLIs). Save the Children has carried out spot-check activities for the first two years, and its verification showed that DLIs have been met. The first spot-check was conducted in 30 townships (9 percent sample of townships) from February to March 2015, supporting the disbursement amounting USD 23 million in June 2015. The second spot-check was conducted in 50 townships (15 percent sample of townships) from December 2015 to February 2016, to support the next disbursement of USD 27 million which is scheduled in June 2016. Timely conduct of the spot-check has contributed to rapid project disbursement. The disbursement rate of the project is expected to reach around 70 percent by the end of this fiscal year, within less than two years from the project effectiveness. Specific outputs  Spot-check report (as supporting document to the disbursement)  Spot-check instruments Component 5: Just-in-time Technical Assistance This component enabled the team to address emerging needs and requests to support evidence based dialogue, including drawing links between the M&E activities for the program and the broader data need and to support the use of the data in the MoE. It supported hands-on technical assistance to the MoE counterparts, who have been increasingly engaged and have been broadening their knowledge of evaluation The resources allocated to this component of the programmatic task have allowed the team to be responsive to requests from the government and other development partners to support evidence based dialogue. Activities supported under this component are broadly classified into the following three categories. a) Supporting the generation of administrative data. During the first year of project implementation, data management of basic information within the MoE proved to be a major challenge. Based on this finding, excel training was conducted for the key MoE personnel responsible for budget and program planning and monitoring. For the stipends program, a simple management information system was developed and introduced to help the Township Education Offices and the Department of Basic Education monitor the stipends students. b) Supporting the analysis and translation of data. The Bank team has consolidated a wealth of information collected from M&E activities, qualitative assessment in particular, and prepared issues and recommendation notes which the MoE technical staff have used to distill the main challenges facing the project and to discuss recommendations in order to report to their management. The Bank team also 2 With the technical support of the World Bank and Save the Children Myanmar, and financial support by Australia, MoE has successfully adapted an Early Grade Reading Assessment to Myanmar language and administered it in DBE3’s Yangon schools (in January 2014). provided concrete and detailed technical inputs on how to translate the program changes and recommendations into revised operational guidelines. c) Flexibly responding to emerging needs. The Bank team has leveraged this PAAA to accommodate two specific assessments, notably a pre-assessment for stipends and a rapid assessment for school clusters. These assessments were previously not foreseen but emerged as a result of the development of the dialogue. In August 2015, a pre-assessment (preparatory feasibility study) was designed for the stipends program to examine whether the stipends program could enter into three conflict affected states (Rakhine, Kachin, and Shan North). The M&E Working Group welcomed this proposal, and the activity was jointly conducted by the Township Education Offices, representatives from the M&E WG, Save the Children, and the Bank/Australia team. The joint team reported back the issues and recommendations from the pre-assessment to the Steering Committee, influencing the roll out of the program in conflict sensitive areas. In January-February 2016, rapid assessment of school cluster was designed to gather inputs for new teachers mentoring program. Save the Children has kindly agreed to conduct this assessment as a rider question with ongoing spot checks. The findings from 600 schools in 30 townships provided useful inputs to the design workshop which was held in March 2016. Specific outputs  Excel training for Ministry of Education staff (Finance and Department of Basic Education)  Stipends Students Payment Management System (software, user guide manuals, database of stipends students)  Issues and Recommendation Notes, based on the first and second year implementation, respectively  Inputs to the revision of operational guidelines and other program materials  Presentation of Pre-assessment (for stipends’ roll out in conflict affected areas)  Presentation of main findings from rapid survey on school clusters (for teachers mentoring program) Dissemination: Dissemination to government is continuous and ongoing, since the activities are done in close collaboration with government counterparts and have been feeding into the programs through three working groups which meet regularly (i.e., M&E Working Group meet quarterly, Technical Working Group and Steering Committee meet biannually.). In fact, government has been involved in all M&E activities throughout the process, not only limited to the dissemination of results and findings from these activities. Dissemination is not limited to the central level, and the Ministry of Education and Save the Children have been conducting joint sharing meetings at a decentralized level (e.g., township level). Furthermore, external dissemination has been conducted to share key findings and to seek advice from development partners who have been working on related issues and to reach a wider audience3. 4. Reflections and Next Steps The PAAA has been on track and successful in achieving its objective. The PAAA allowed the team to support the MoE in implementing and monitoring the school grants and stipends program based on the evidence generated through multiple M&E activities. The introduction of monitoring arrangements and progress reports using data gathered through monitoring forms represents a substantial break from previous experience within the government. Over the past two years, the Ministry of Education has been demonstrating stronger engagement and gradually internalizing this M&E system. The Ministry of Education has improved the programs, based on lessons learned from the initial years of program implementation. This is the Program Development Objective of this PAAA, and the information and evidence gathered through various M&E activities led to the introduction of annual review process for the government to make major policy changes to continuously improve the programs over time (as elaborated in Chapter 2. Objectives and Progress.). The respective M&E component satisfactorily yielded the expected outputs, including the following: 1) qualitative assessment gathering feedback on the design and process of the programs; 2) quantitative survey collecting the baseline; 3) EGRA findings leading to reading intervention; 4) spot-check 3 For example, joint presentation by Save the Children, Ministry of Education, and the World Bank was held in April 2015 to share the Qualitative Assessment findings from the first year and the program changes for the second year. In May 2016, the World Bank and Australian embassy jointly hosted a presentation on the school survey to get feedback from key development partners who are working on education in Myanmar. verifying the administrative reporting for the timely disbursement, and; 5) just-in-time Technical Assistance supporting the launch of Stipends Management Information System, the conduct of pre-assessment in conflict sensitive states as well as rapid assessment of school clusters. Close collaboration has been a critical and essential factor for the successful institutional arrangement. With hands on technical assistance, the use of evidence/data to build an understanding of project implementation issues has helped to build trust between the Ministry of Education and the World Bank/Australia team, which in turn led to in-depth dialogue and a greater sharing of data. The three layered institutional set-up within the Ministry (i.e., M&E Working Group, Technical Working Group, and Steering Committee) has become a backbone to elevate main issues and recommendations through internal reporting mechanism for the management decision. Furthermore, Save the Children has played an invaluable role in bringing timely feedback as well as building capacity at local levels. The flexible financing from Australia allowed the Bank team to maintain and deepen the engagement while being responsive to the government’s requests. This level of engagement, strong focus on M&E, and the responsiveness would not have been feasible with the Bank’s administrative budget alone. The Bank team will continue to take full advantage of the grant flexibility in order to effectively respond to emerging priorities and needs from the government. Moving forward, the focus of the PAAA shall evolve and be aligned with the maturity of ongoing programs as well as with the launch of a new teachers mentoring program. This PAAA was initially designed to support the ongoing financing project, while the Bank team is currently preparing additional financing to also support a new teachers mentoring program. Moving forward, the PAAA needs to align these developments with the projects, by adjusting activities to meet expanded Program Development Objective. More specifically, greater focus should be directed towards further strengthening Ministry’s internal capacity as well as in-depth analysis on outcomes. While the Ministry of Education has been making great progress in generating program information through its own administrative reporting, there is a greater scope to increase the volume of information coming through their internal system and to improve its quality. In addition, with maturity of the programs, the M&E focus needs to move from basic process monitoring and collecting the baseline towards in-depth analysis on specific emerging issues (e.g., the heterogeneity of program perception and implementation across areas) and assessment on outcomes (e.g., dropout, improving teaching and learning) whether the programs are achieving its intended objectives. The team will continue to leverage the M&E activities under this PAAA to extend support to the Ministry of Education to further improve the programs. Annex: List of Outputs produced under Myanmar PAAA M&E Produced by Title of Outputs Date Fiscal Year Just in Time TA World Bank, MoE Issues and Recommendation Notes (Annual Program Review) Jan 2015 2015 Dec 2015 2016 World Bank Hands-on technical assistance to the MoE, including throughout 2015, 2016 preparation for monitoring meetings, program monitoring, administrative data, generating information and reports World Bank Inputs to the revision of operational guidelines, posters and Apr 2015 2015 the development of training module for Year 2 roll out AcePlus (Stipends Development of Stipend Management System (software, user May 2015 to 2015, 2016 Management guide manuals, database of stipends students), conduct of Feb 2016 Information System) training to MoE staff (township computer staff) KMD Company Excel training for MoE staff (finance and DBE) – training Aug 2015 2016 manual, training plans MoE Presentation from field visits by MEWG Aug 2015 2016 MoE Synthesis and presentation of Issues and Recommendations Dec 2015 2016 MoE Presentation of summary of Program Implementation Dec 2015 2016 MoE Revised Operational Guidelines, training modules, program March 2015 2015 materials (Inputs provided by the World Bank and Save the March 2016 2016 Children) MoE Annual Progress Report April 2016 2016 MoE, Save the Presentations of summary of findings and recommendations Dec 2015 2016 Children, WB/DFAT from pre-assessment in Year 3 stipends states Save the Children, Rapid assessment of school clusters Mar 2016 2016 World Bank Qualitative Assessment World Bank Design and development of metrologies and instruments Aug 2015 2015 Save the Children Various presentations to the MoE (central levels (MEWG, 2015 2015, 2016 TWG, and SC) and township levels) Save the Children Reports of Qualitative Assessment (executive summary and Mar 2015 2015 full report) (Annual) Mar 2016 2016 Save the Children, Presentation to the Development Partners Apr 2015 2015 MoE, World Bank Mar 2016 2016 Spot-checks World Bank Design and development of methodologies and instruments Jan 2015 2015 Save the Children Spot-check data and findings Apr 2015 2015 Spot-check data and report May 2016 2016 Quantitative Survey (School and Household Surveys) World Bank Design and development of methodologies and instruments Aug 2015 2016 MSR Collection, entry, and cleaning of data Sep 2015 to 2016 Dec 2015 World Bank Presentation of preliminary findings Jan 2016 2016 World Bank Presentation to the Development Partners Mar 2016 2016 EGRA and EGMA (Early Grade Reading and Math Assessment) World Bank Myanmar EGRA for Yangon Region (EGRA Report from the Mar 2015 2015 first round) MoE, WB EGMA toolkit Jan 2016 2016