Findings from the study: Quality of Education in Madrasah (QEM) 2010 60961 school uniforms to ensure that these are not made more important than provision of resources for learning and home study. their goals. Schools should review their practices in grade repetition as this gen- erally has more negative than positive im- condition of their school through regular building maintenance and systematically increasing resources. This process should Measuring Student Learning Achievement 4. The QEM study highlights the relationship pacts on performance. be guided by school self-evaluation against the requirements for school accreditation in Madrasah between higher student performance 2. Principals can take a stronger education- and district plans for meetings the MSS. This policy brief arises from a national study which measured the learning outcomes of Year 9 Madrasah students in Maths, Science, and the provision of good infrastructure, al leadership role within their schools by: School supervisors have an important role English and Bahasa Indonesia together with the influence of student, family and school background factors. qualified personnel and learning resour- making regular visits to teachers in their in supporting this planning process. DIAGRAM 1 : QUALITY OF EDUCATION IN MADRASAH ces. This has implications for how schools, classrooms; observing and providing feed- local and national governments allocate back on teaching practice; assisting teach- What can the local and national Quality of Education in Madrasah and manage resources. It also underscores ers to develop effective lesson plans and government do to lift the performance of the importance of monitoring and target- regular assessment including provision of madrasah? Measured By Sample ing funds to areas of need. meaningful feedback to students, to estab- 1. Commit to regular monitoring of Madrasah lish good homework habits and to develop school performance and school conditions Students' view of their Schools meeting 5. Efforts to ensure the provision of quali- positive classroom environments. by supervisors and district-level officials, Student Achievement Tests schooling experience Minimum Service Standards 50 Madrasah 50 Madrasah 50 Madrasah Tsanawiyah in Tsanawiyah in Tsanawiyah in fied teachers and principals must become supplemented by annual or bi-annual na- Western Java Eastern 1. Mathematics achievement test 6. School Life Questionnaire 7. Principal Interview Schedule Indonesia more urgent and more strategic. The high 3. Teachers can enhance boys' and girls' tional sample based monitoring of Ma- 2. Science achievement test 8. School Inventory Indonesia wages cost and the inefficient distribution variable performances in English, Bahasa drasah. 3. Indonesian achievement test 4. English achievment test of qualified teachers are difficult to address Indonesia and Science. Through the school 5. Student background characteristics assessment 6.233 nal year students in a decentralised environment. Continuing level curriculum development process, 2. Prioritise and support school-based in-ser- positive dialogue and policy development teachers can investigate gender differ- vice professional learning which focuses on * Madrasah Tsanawiyah - Islamic Junior Secondary School are needed on redeployment, multigrade ences in performance and develop activi- classroom practice. teaching and dual subject specialisation ties such as extra reading time with appro- Why was the QEM study conducted? sessment (PISA), Trends in Mathematics and School Life Questionnaire. This measures as being efficient and educationally sound priate and interesting materials for boys, 3. Strengthen leadership programs for prin- Madrasah constitute an established, and still Science Study (TIMSS) and the International students' perceptions and attitudes about strategies for ensuring that students have debating and public speaking in English as cipals to ensure they can provide effective growing, sub-sector in education. Because of Benchmark Tests of Mathematics. The Eng- schooling ­ relationships with teachers, rel- qualified teachers. well as Bahasa Indonesia, develop science pedagogical leadership in their schools. their history, they are more prevalent in ru- lish test was adapted from the Competency evance of schooling to their future, sense of activities related to girls' interests and ral and disadvantaged areas ­ the very areas in English as a Foreign Language Assessment achievement, feelings of self-worth and the What can principals and teachers do consider establishing single sex classes in 4. Review the plan for upgrading teachers' where the most strategic effort will be re- (CEFLA) which is a reading comprehension test social integration of the school. The question- right now, without extra financial these subjects for a period of time. and principals' qualifications and ensure quired to achieve national goals for participa- in English. The Bahasa Indonesia test was spe- naire is widely used in international programs resources, to lift the performance of that all Madrasah have a critical mass of tion and quality. These are also the areas with cifically developed for the study. as a measure of the affective domain of educa- Madrasah students? 4. Principals and School Committees can qualified teachers and all schools have a the lowest achievement on both international tion. 1. Principals and teachers can promote high discuss and prioritise BOS expenditure for qualified principal. tests and national exams and therefore in expectations. Through parent meetings educational equipment and teaching and most need of education intervention. Summary of Findings and formal and informal communications learning resources, especially library books 5. Review teacher deployment and appoint- a Across the 3 regions, student achievement in Maths, Science and English was well below the with community they can continually raise and science equipment. This may be more ment practices to ensure Madrasah in dis- Furthermore, in an era of increased account- international average on items drawn from international tests such as the Programme for In- awareness of the role of high expectations effective than hiring additional teachers. advantaged areas have sufficient qualified ability, parents, community and government ternational Student Assessment (PISA) and the Trends in International Mathematics and Sci- and home study resources for success in Teachers may need support in the effective teachers. want information about the academic out- ence Study (TIMSS). learning. In regular school assemblies use and management of new resources. comes of Madrasah which they expect to de- and class meetings, teachers can promote 6. Identify regional and district patterns in liver high quality education, both religious a Positive correlations were found between student achievement and home-background factors and discuss role models and "heroes" to 5. Principals and School Committees can education resourcing and expenditure and and academic. such as level of resources, including books and study resources in the home, students' aspira- encourage students to have high expec- develop an annual and longer term deve- formulate intervention plans to equalise tions and parents' level of education. tations and believe that they can achieve lopment plan (3 ­ 5 years) to improve the learning opportunities. Overview of the study The purpose of the study was to investigate the a School factors associated with higher performance included the qualifications and profession- The main QEM study was a collaborative venture led by AusAID and co-funded by the Basic Education Capacity Trust Fund administered through the quality of education using international-style al practice of teachers and principals, the number of resources in the school and the amount World Bank, which also provided logistical and communications support. The study was overseen by the Director-General for Islamic Educa- tion in the Ministry of Religious Affairs. All technical aspects of test development, sampling, test procedures and analyses were the tests of achievement and to examine the rela- of instruction and homework. While larger school size appeared to be associated with higher responsibility of the Australian Council of Educational Research. Implementation, field monitoring, cultural and educational advice was tionship between achievement and school and achievement, it was not school size per se that made the difference, but the association of in- provided by the Indonesian University of Education (UPI) at Bandung. Preparation of this document received partial funding from the European Commission and the Kingdom of the Netherlands under the home background factors. creased size with factors such as more resources, more qualified teachers and better facilities. supervision of the World Bank. The findings, interpretations and conclusions expressed in this paper do not necessarily reflect the views of the Government of Indonesia, the World Bank, the Government of the Netherlands or the European Commission. For more information please contact Mae Chu Chang, mchang @worldbank.org; or Sheila Town, stown@ Instruments a There were some gender differences in achievement which are worldbank.org. Four tests of academic achievement. The common across many countries: girls performed better than Maths and Science tests each comprised 30 boys in the language tests (English and Bahasa Indonesia) Human Development Sector World Bank Office, Jakarta Indonesia Stock Exchange Building Tower 2, 12th Floor multiple choice items and included link items and boys performed better than girls on Science. Jl. Jenderal Sudirman Kav. 52-53 from Program for International Student As- There was no gender difference in Maths Phone: (021) 5299 3000 Fax: (021) 5299 3111 performance. Prepared by the Education Unit, World Bank Indonesia Based on Mohammad Ali, Julie Kos, Petra Lietz, Dita Nugroho, Furqon, Asmawi Zainul, Emi Emilia, 2010 "Quality of Education in Madrasah: Main Study", joint publication of AusAID and World Bank Indonesia. Student/home background questionnaire plore the relationship of English and Maths can be used for regular sampling in both MONE related with achievement in English and Sci- DIAGRAM 3: CORRELATES OF STUDENT CHARACTERISTICS AND ACHIEVEMENTS, OVERALL (adapted). This questionnaire is widely used with student and school background factors. and MORA systems. ence than in Maths and Bahasa Indonesia. 0.25 in conjunction with achievement testing to Factors that Correlate with Level of MSS, by Region gather data about home background factors Potential Value of the Study 1. Home background: These student and home background factors 0.2 which are generally thought to be associated The study is the first international-style as- Socio-economic status brings a powerful ad- had a greater impact on performance in the 0.15 with achievement ­ eg. language spoken at sessment of learning outcomes in Maths, Sci- vantage: In each of the regions, but especially East and West than in Java where overall stan- 0.1 home, parents' level of education and level of ence, English and Bahasa Indonesia for a na- in the East and West, students with the most dards of living may be higher. resources in the home. Students filled out both tionally representative sample of Madrasah. resources at home, both educational (books, 0.05 Age of student in years Grade repetition questionnaires with their test papers. It provides a benchmark of Madrasah student magazines, study materials) and general ma- Both parents have an influence: Both moth- 0 performance and the factors which are associ- terial resources (such as electricity, TV, radio, ers' and fathers' education level was associ- Books at home Number of home Number of study Mother's education Father's education -0.05 Principal and School Inventory. Principals resources materials available ated with student achievement. A legacy of piped water, and motorbike) performed better ated with high performance in English but were interviewed about school and teacher the study is the high quality instruments and on all the tests. The number of books students over all the tests, it was fathers' level of edu- -0.1 characteristics such as teacher qualifica- methodology that, with minimal refinement, had access to at home was more strongly cor- cation that had the strongest association. The -0.15 Gender (boy=1; girl=2) tions, student enrolment, school governance language spoken at home, whether students and management practices, teaching and as- DIAGRAM 2: PERFORMANCE WITHIN EACH REGION ON MATHEMATICS, SCIENCE, INDONESIA AND ENGLISHT TEST boarded or lived at home, how many meals -0.2 Mathematics Science Indonesian English sessment practices and school resources. The Mean % Correct they had per day, whether students did some 20 Performance on tests in Percent School Inventory covered an abridged list of 18.1 work for the family, whether they had some- who marked the set homework, generally had West also scored better in schools where prin- 17.2 the items in the draft Minimum Service Stan- 15.6 16.2 16.2 16.7 one to help with homework or whether they more positive attitudes towards schooling. cipals made regular visits to classrooms and 14.8 15.4 15.8 15.2 14.8 15 14.7 14.3 dards (MSS) including aspects of curriculum, 14 received private coaching, did not appear to Students in schools that were in poor condi- gave feedback to teachers about their perfor- 12.5 12.2 teaching process, teacher qualifications, faci- make a significant difference to performance tion and had never been visited by a supervisor mance. In the overall sample, 90% of princi- lities, infrastructure, management and evalu- 10 on the tests. had more negative feelings about schooling. pals were male; the average age was 44 years; ation. 70% held a bachelor's degree, and around 5 2. Student performance and 3. School factors: 90% of principals taught 9 lessons per week. Methodology attitudes: Teachers make a difference: Across all schools, Year 9 was selected for testing, being the fi- 0 Overall, gender differences were not signifi- but especially in the East and West, students School resources make a difference: School nal year of junior secondary education. The Overall Java West East cant but girls scored better in the language scored better in schools with more highly resources were positively correlated with stu- subjects to be tested were the core academic Mathematics Science Indonesian English tests and boys scored better in Science. Im- qualified teachers. Students also scored higher dent achievement, even after the impact of subjects and matched those tested by national portantly, students who had high aspirations in schools where there were more lessons per higher socio-economic status was taken into exams and the international testing programs performed better on all tests. Conversely, stu- week, where teachers had lesson plans and account. Some resources were more important in which Indonesia participates. The 8 instru- How big a difference can schools dents who had repeated a grade at some point objectives for the class, where teachers gave in some regions than others ­ for example, ments above were developed or adapted and in their schooling performed more poorly weekly assessments and feedback to students in the East, students in schools with science piloted to ensure their reliability and validity. make to student achievement? on all tests. Absenteeism did not appear to and set homework frequently. Whether or not laboratories and science equipment scored Testing was conducted by trained education be correlated with performance. Across all teachers marked all the set homework did not higher than others. In the West, achievement researchers mainly drawn from post-graduate In education research, differences in students' achievement scores are generally attributed schools, students generally felt positively seem to be related to test performance. differences were associated with the school level in a university Education faculty. Teams to 3 main sources - student level factors, classroom level factors and school level factors. Ac- about the things they were learning at school having a separate, furnished principal's office. spent 2 days in a school to complete the test- curate measures of the relative contribution to achievement scores from these 3 sources help and the relevance of these things to their fu- Principals make a difference: In the West, In Java, the overall physical condition of ing, interviews and observations. policy-makers to understand what influences learning and where their interventions can be ture lives. Students who were able to borrow schools with more highly educated principals school buildings was correlated with achieve- focused. The higher the level of variation attributable to within-school (classroom level) or books from their library and who had teachers performed better than others. Students in the ment. Sample between-school differences, the more the scope for improvement. A nationally representative random sample was designed to allow separate analyses for For example, on the QEM Maths results, 64% of the variance was attributed to individual stu- RECOMMENDATIONS the 3 major geographic areas of Indonesia. The dent factors and 36% to school factors. This 36% is both good and bad news. The bad news 12,396 Madrasah which had a Year 9 class con- is that schools are not equal in their impact on student learning. Some schools are produc- Key Policy issues for consideration 2. The QEM study identified significant cor- teaching practice without the significant stituted the sampling frame. Schools were di- ing higher outcomes, regardless of the ability of students or the type of home background. 1. Regular and reliable monitoring of stu- relates of achievement which can be disruption and cost of off-site meetings. vided into the 3 main regions and then sorted This finding is similar to the 2006 PISA results for Indonesia on which 33% of the variation dent achievement is an important step in readily addressed by schools now, at little according to province, public/private schools, in Maths scores was due to between-school differences. The good news is that the study has efforts to improve quality of education. additional cost, but these actions require 3. Principals and teachers can influence achievement group (based on previous year's clearly identified some of the school level factors that make a positive difference to achieve- Indonesian policy makers cannot rely on leadership from principals and supervi- parents to support their children's learn- exam scores) and school size. 150 schools ment. national exams and international testing sors. Qualified and experienced principals ing by prioritising their limited financial were then randomly selected for the study. programs to provide the kind of informa- can make a difference by regularly visiting resources for education to books and Table: Student level vs school level sources of variation in student achievement scores on QEM and PISA results. tion that is useful for improvement, espe- classrooms, giving feedback and advice home study materials. A recent study Types of Analyses PISA cially for a sub-sector. A well designed and to teachers. Principals themselves should and much anecdotal evidence suggest that Student achievement scores were aggregated QEM PISA PISA OECD PISA PISA managed sample testing program is an af- be supported by supervisors who are well since school fees have been largely abo- % variance Non-OECD Indonesia Indonesia countries Thailand Australia to school level and a range of descriptive and and source countries fordable solution and when implemented qualified, experienced and have a bud- lished, parents' education expenditure is 2009 2006 2006 2006 2006 2006 correlational analyses undertaken. In recog- with collection of background data on get for travel and education resources. A primarily allocated to provision of uniforms nition of the complex relationships involved Student 64 67 63.2 60 71.2 80.2 school and home factors it enables policy planned program and a cooperative climate and snacks. This is especially worrying in factors in student achievement, Hierarchical Linear makers to prioritise programs and expendi- which supports continuous professional disadvantaged areas. Schools and parents Between- 36 33 36.8 40 29.8 19.8 Modelling (HLM) was applied to further ex- school factors ture to make the most difference. learning within the school is a highly ef- should be encouraged to review their local fective and affordable way of enhancing school policy on the number and costs of