RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE VIETNAM Identifying Reliable Predictors of Learning for Results-Based Financing in Education MAY 2018 REACH funded the development of a machine-learning predictive modeling tool in Vietnam to identify which variables are reliable predictors of student learning outcomes. Directly financing learning The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on outcomes can be problematic the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH for many reasons: grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. 2a+4=c Many education systems around Conditional cash transfers (CCTs), 2a+4=c 3ax3=9 5a-5=10 the world have reached nearly teacher performance pay systems, Teaching to the test. universal access to schooling, but and disbursement-linked indicators ensuring high-quality learning for (DLIs) are all examples of RBF that all students has proven to be more have been shown to be effective at difficult to achieve. Results-based improving learning outcomes at the financing (RBF) has the potential to student, parent, teacher, and school transform the way in which education district levels. However, directly Students have widely systems improve by incentivizing financing learning outcomes can diverse abilities. students, parents, teachers, be problematic for many reasons— school administrators, and other because it can add such distortions to stakeholders to achieve better results. real learning as “teaching to the test,” RBF mechanisms work by linking because it is difficult to set targets financial incentives to measurable for learning for all students with Stakeholders don’t know how to improve learning. results such as school attendance, widely diverse abilities, and because dropout rates, student test scores, or teachers, students, and policymakers other indicators of education quality. may not know how to improve This note was adapted from Musso, Mariel F., Eduardo C. Cascallar, Neda Bostani, and Michael Crawford. (2018). “Are Teaching Practices and School Characteristics Reliable Proxies for Learning Gains? A Results-based Approach to Financing of Education through Machine-Learning Predictive Modeling.” The World Bank Group. Unpublished Manuscript. 2 RFB EDUCATION | EVIDENCE learning. Therefore, as a precondition to establishing RBF systems, it is first necessary to identify the intermediate drivers of student learning to shed light on the mechanisms through which learning is achieved. The Results in Education for All Children (REACH) Trust Fund at the World Bank funded the development of a machine-learning predictive modeling tool in Vietnam to identify which variables are reliable predictors of student learning outcomes. This predictive model was used to analyze over 2,000 variables from the Young Lives (YL) longitudinal survey to predict which ones are the key drivers of student performance in language and math among grade five students in Vietnam. Using this approach revealed that the most important variables were related to students’ characteristics, including their raw or school administrators could be learning, such as testing to the test cognitive abilities (as measured by a incentivized to hire teachers with the or cheating. The predictive model Cognitive Development Assessment most effective characteristics. developed in this study has helped to and other tests administered fulfill this requirement by identifying early in childhood), physical Vietnam has achieved learning a series of key indicators to which characteristics and health, school outcomes that are on a par with disbursements could be linked. This and pre-school trajectory, and their much wealthier countries and study will help the Government of routines and habits. In addition, that are higher than those of most Vietnam and education stakeholders teacher characteristics and parental countries at similar income levels. to understand in detail both the expectations are also important However, to continue to build on drivers of the country’s educational predictors of student performance. these learning gains, the government success and the factors that limit Effective RBF mechanisms could be is moving away from donor aid some students from sharing in that designed to take into account many based on “more of the same” inputs success. However, just as incentives of these categories of variables. For and towards a new emphasis on conditioned on test scores can example, parents and teachers could financing results. In light of this, distort learning, providing incentives be given incentives to implement the Ministry of Education intends to achieve intermediate indicators interventions known to promote to implement disbursement-linked can also cause distortions. children’s cognitive and physical financing and other RBF approaches, Therefore, when implementing these development, by parents before which will make it necessary to RBF approaches, it will be necessary school age and by teachers during identify practical indicators that can to monitor teachers, students, and the school years. Also, parents could be linked to financing in order to parents carefully to ensure that be given incentives to become more avoid the common pitfalls involved the intermediate indicators are not being manipulated. involved in their children’s education, in directly incentivizing student VIETNAM 3 CONTEXT Over the last 20 years, Vietnam In addition to high levels of access The Government of Vietnam has has been a development success to education, Vietnam has also had also enacted a series of reforms and story, and education has played significant success in achieving investments in education that have a significant role in this success. good education quality. Based on contributed to its success. In 2005, According to the World Bank, the the results of the 2012 and 2015 the Education Law committed the total poverty rate in Vietnam fell rounds of the Programme for government to providing primary from 57 percent in 1992–1993 to International Student Assessments and lower secondary education 37 percent in 1997–1998.1 The (PISA), Vietnam’s 15-year-olds for all students, and this law was primary school gross enrollment rate performed as well in math, reading, amended in 2010 to extend universal has reached a high of 96 percent, and science as their peers in much education to pre-primary school. The while the secondary school gross richer countries and much better enrollment rate grew from 32 percent than those from other developing Government of Vietnam spent 18.5 in the early 1990s to 85 percent in countries. Although Vietnam has percent of its public expenditure and 2010.2 In addition, there is only a one of the lowest income levels of 5.7 percent of the country’s GDP on small gap in enrollment between all PISA participating countries, its education in 2013—well above the boys and girls at both the primary performance has been better than global average.4 and lower secondary school levels. the international average.3 The government WHY WAS THE linked financing approaches, including both domestic and international intends to adopt RBF interventions INTERVENTION sources of funding. The Ministry of Finance recently embraced the use that incentivize CHOSEN? of disbursement-linked approaches in major donor-assisted investments, while borrowing in the social sector is intermediate indicators To extend the progress that has being tied to performance and results already been made in student such as teachers’ learning, the Government of rather than inputs. For example, the Enhancing Teacher Education characteristics or Vietnam is continuing to pass ambitious reforms. It has piloted Program uses financial incentives children’s cognitive new pedagogical approaches to strengthen teacher education through the Vietnam: Escuela Nueva institutions using DLIs such as abilities, which lead to program, co-financed with the World the satisfaction rates of teachers positive learning gains Bank. In this program, students learn and principals with continuous professional development programs. through group discussions with their but are not as easy to peers while their teachers act as To facilitate the shift from input- manipulate. facilitators, which is very different from the traditional learning approach based aid to results-based financing, the Vietnamese government has in Vietnam. The government is also recognized the critical need to rapidly moving away from donor aid gather accurate and reliable based on “more of the same” inputs information on what works to and towards innovative disbursement- improve its education outcomes. 4 RFB EDUCATION | EVIDENCE The government’s ultimate goal is to improve student learning outcomes, However, before such an RBF intervention can be designed, HOW DID THE but providing direct incentives to students or teachers to improve test it is critical to identify which of these intermediate indicators of INTERVENTION scores can be problematic for several reasons. First, paying students or education quality are the best predictors of learning outcomes. WORK? teachers for improvements in test Doing this accurately and reliably To identify the key drivers of scores may introduce distortions requires robust statistical analysis student learning in Vietnam, into real learning, such as “teaching of all potential determinants of analysis was conducted on the to the test” or cheating. Second, learning. Studies using a regression largest and most detailed dataset it can be difficult to aggregate approach have shown that a related to student success—the students who have diverse learning unique combination of resources, Young Lives dataset. Using these abilities into a single learning target investments in education, and data, six distinct predictive models that is neither too difficult nor too cultural factors in Vietnam were developed using an artificial easy to achieve. Third, even though have resulted in disciplined and all stakeholders may be properly focused students, hard-working motivated to improve learning, they teachers with close supervision Predicts high and low may not know how to translate from principals, and committed performers in math their inputs and effort into higher and Vietnamese. academic achievement. Therefore, and involved parents with high HIGH rather than financing either basic expectations for their children. PERFORMERS inputs or learning outcomes directly, While these conditions might help which often leads to distortions, to explain Vietnam’s impressive DATA the government intends to adopt RBF interventions that incentivize learning gains, statistical analysis has found that all these factors can ANN intermediate indicators such as explain only 50 percent of Vietnam’s teachers’ characteristics or children’s positive academic performance at cognitive abilities, which lead to most.5 Therefore, further analysis LOW PERFORMERS positive learning gains but are not as is needed to identify the additional easy to manipulate. drivers of learning in Vietnam. VIETNAM 5 neural network (ANN) approach. The predictive models produced Table 1: Examples of Independent ANN predictive modeling can a prioritized list of variables and Variables Used to Predict Student find patterns within the complex categories of variables that were the Outcomes relationships between inputs and most accurate and reliable predictors Variable category Example of variables outputs, which makes it an ideal of high or low student performance tool to demystify the “black box” of as well as predictive weights that Children’s Raw score in Cognitive cognitive which inputs contribute to learning quantified the strength of the Development Assessment factors gains. This methodology uses prediction for each variable. This list Children’s Weight-for-age z-score machine-learning techniques to was the basis of the investigation physical factors Health compared to peers improve the quality of prediction of how these factors contribute to over time as more data are learning outcomes. It is important to Children’s Hours of sleep routines and introduced and the relationships note that the variables may interact habits Hours spent studying between variables become clearer. differently between high performers Children’s Age started school The models were trained using data and low performers. school trajectory Years of pre-school on a subset of students to establish the relationships between various To test the value of the ANN Attitudes and perceptions Children’s predictors and learning and were approach, it was contrasted with non-cognitive Trust then tested using the remaining a statistical analysis using a more factors Motivation data to measure the predictive traditional technique—logistic regression. Six logistic regression Parents’ language power of these relationships. Parents’ models were built using the same background Ethnic group The ANN models were built to independent variables and the same Household Household durable goods predict two different outcome outcomes, and the results were socioeconomic status On list of poor households variables—which students would compared to those produced by the be high performers (in the top 33 ANN models. Household Father’s and mother’s education years of education percent of the student distribution) and which students would be One important caveat that must be Parenting Frequency of father-child interaction low performers (in the bottom 33 noted when interpreting the results of the predictive modeling is that Child’s level of education percent) on grade five examinations Parental there is a fundamental difference expectations in math and Vietnamese. The First job independent variables used in the between prediction and causation. Frequency of homework model to predict student outcomes Predictive models can identify Teaching practices Whether a calculator is were drawn from the Young Lives which variables predict learning, used in the classroom longitudinal study, which followed but this does not necessarily mean Years of experience Vietnamese students for 15 years. that changing those variables will Teachers’ Attitudes; This survey collected over 2,000 improve learning. For example, characteristics variables in 15 categories, including owning a textbook might be Teaching awards students’ cognitive, non-cognitive, predictive of higher test scores, but Principals’ Years of experience and physical characteristics, parents’ this could be only because richer characteristics Highest level of education and household characteristics, families are the only ones that can Number of pupils per teachers’ and principals’ afford textbooks, and children of School classroom characteristics, teaching practices, rich families are more likely to have information Condition of classroom school characteristics, and the better learning outcomes for other characteristics of the broader school reasons. Further investigation is Quality of health facility community. Examples of variables in needed to determine whether the Community Existence of adult each category are listed in Table 1 to use of the textbook or some other education classes the right. variable contributes to learning. 6 RFB EDUCATION | EVIDENCE WHAT WERE THE RESULTS? The predictive models achieved a goods; (e) students’ school trajectory, significant role in their academic high level of accuracy in predicting including years of pre-school performance. However, variables student performance. The six and age at starting school; and that are more within the control predictive ANN models were able (f) teachers’ characteristics, such of education policy-makers—such to achieve between 95 percent and as years of experience, attitudes, or as teachers’ characteristics—were 100 percent accuracy in terms of teaching awards. second in importance to students’ predicting which students would characteristics. While teachers’ be in either the top third or bottom These findings confirm that characteristics are more amenable third of the student population. students’ characteristics—which to change through interventions, Furthermore, the ANN models cannot be easily influenced by the expected effect sizes of these were able to predict high and policymakers—continue to play a changes on learning would be low performers in both math and language more accurately than the logistic regression models, Table 2: Predictive Weight of Variables by Category for each Subject-Outcome which were between 77 percent Combination (%) and 97 percent accurate. Subject Outcome Most predictive variables Predictive weight (%) Children’s cognitive factors 18.22 Student characteristics (cognitive Children’s physical factors 15.90 ability, physical factors, routines High ______ Children’s routines and habits 10.49 and habits, and school trajectory) and teacher characteristics were Top Household socioeconomic status 9.74 found to be the most predictive 33% Children’s school trajectory 8.76 categories of variables in Teachers’ characteristics 6.52 Math predicting student performance. Although the most predictive Children’s cognitive factors 18.46 variables for each subject (math Low Children’s physical factors 17.13 and language) and outcome (high- ______ Children’s routines and habits 10.26 performing and low-performing) Bottom Household socioeconomic status 9.93 varied somewhat, the most 33% consistently predictive categories Children’s school trajectory 8.93 of variables were the following: Teachers’ characteristics 5.94 (a) students’ raw cognitive ability Children’s physical factors 16.67 (as measured by a Cognitive High Children’s cognitive factors 16.64 Development Assessment and ______ Children’s routines and habits 12.26 other tests administered early in childhood), which was predictive Top Household socioeconomic status 9.53 33% of academic performance later Children’s school trajectory 9.04 Language in school; (b) students’ physical Teachers’ characteristics 6.07 factors and health, such as Children’s cognitive factors 18.74 birth weight and weight-for-age; Low Children’s physical factors 17.19 (c) students’ routines and habits, ______ Children’s routines and habits 11.80 such as how much time they spend Bottom Household socioeconomic status 10.08 sleeping or studying; (d) household 33% Children’s school trajectory 8.49 socioeconomic status, such as the possession of certain durable Teachers’ characteristics 5.40 VIETNAM 7 lower than the effect sizes if it were categories of variables for each weight to the overall models. In possible to change variables that are subject-outcome combination are each of the six models, the top 20 more fundamental to learning, such listed in Table 2, with the percentage variables contributed between 21 as students’ key cognitive abilities predictive weight shown for each percent and 22 percent predictive like working memory. category of variables. weight. Therefore, it is the cumulative effect of the entire set of variables This is consistent with previous However, there is no “silver bullet” that makes it possible to accurately research that has demonstrated variable to predict learning as many predict performance rather than any the crucial role played by working variables are needed to accurately one variable or set of variables. This memory and attention in academic predict students’ academic insight has important implications achievement.6/7 This suggests that performance. While the predictive for policymakers when choosing interventions aimed at improving models were able to identify those between a broad or a targeted set of children’s cognitive abilities and variables that best predict student interventions and when managing physical health from an earlier performance, each variable on its expectations of the potential age, increasing children’s access own has a relatively small predictive impact of any one intervention. to pre-school, or hiring teachers weight of between 0 percent and 3 These results suggest that RBF with optimal characteristics could percent. Each of the 15 categories mechanisms would need to address positively impact students’ academic of variables contributed between a broad set of indicators in order to performance. The most predictive 1 percent and 19 percent predictive have a significant effect on learning. WHAT WERE to identify children and families at have been shown to be effective in risk using indicators like children’s boosting children’s cognitive ability THE LESSONS health at birth. Third, it highlights the need for interventions that focus and physical health. Similarly, RBF mechanisms could give teachers LEARNED? on factors that are more closely under the control of education in the classroom an incentive to maximize their students’ cognitive policymakers. Teachers should be stimulation, particularly in pre-school. By providing a clearer picture of carefully selected and trained to Second, school administrators the inter-relationships between the maximize the teacher characteristics could be incentivized to improve variables that lead to different levels that were found to be most predictive their teacher selection and teacher of student performance, these results of good student performance. training processes to enhance the can help policymakers to design teacher characteristics that best targeted interventions to foster In all three of these policy areas, predict student performance or the most effective determinants the results of this study could be teacher training institutions could of student learning. This study has used to inform the development be incentivized to foster these identified three important priority of RBF mechanisms to incentivize areas for policymaking. First, the various stakeholders to improve analysis reconfirms that policies to student performance. First, parents’ The results of this study promote children’s cognitive ability and physical health early in life investments in their children’s could be used to inform cognitive ability and physical health should be prioritized, such as early could be incentivized, for example, the development of RBF access to pre-school programs, early through CCTs aimed at encouraging mechanisms to incentivize cognitive stimulation, and pre-natal pre-school attendance, pre-natal and and post-natal medical care. Second, post-natal medical care, childhood stakeholders to improve it points to the need for interventions nutrition, and other actions that student performance. characteristics. Third, parents could be incentivized to be more CONCLUSION cognitive ability and physical health and teacher characteristics as the involved in their children’s education RBF mechanisms can improve most predictive of good academic and to actively encourage their student learning outcomes by performance in Vietnam. Although children to work hard and perform providing financial incentives to the study found these categories of well in school. Lastly, officials at students, parents, teachers, school variables to be the most predictive intermediate levels of government administrators, or local governments of student learning, there is no single could also be incentivized through to encourage them to work towards variable that predicts test scores, carefully targeted disbursement- improving learning results. However, and understanding the complex linked indicators to improve any of as a precondition to establishing inter-relationships between all of the indicators that have been shown these RBF mechanisms, it is critical the drivers of learning is important to predict student learning. to identify those indicators and for achieving better results. interventions that are most likely to These findings suggest that RBF While incentivizing intermediate improve student learning. Using a mechanisms should be designed to indicators rather than learning machine-learning predictive modeling focus on a broad set of education outcomes can have the benefit of approach with a dataset of over 2,000 quality indicators rather than any removing distortions to learning, they variables, this study achieved high single indicator. These findings will can cause distortions themselves. levels of accuracy in predicting both help the Government of Vietnam to Therefore, the implementation of any high-performing and low-performing design and implement RBF incentives RBF approach needs to be carefully grade five students on math and and other effective interventions monitored by the relevant stakeholders language exams in Vietnam. This to enable it to build on its progress to ensure that any manipulation of the modeling methodology effectively in improving learning for the next indicators is minimized. identified variables related to student generation of students. 1 The World Bank Group 2 UNECSO Institute for Statistics (UIS) 3 PISA 2012 and 2015 (OECD) 4 UNECSO Institute for Statistics (UIS) 5 Parandekar, S., and E. Sedmik (2016). “Unraveling a secret: Vietnam’s outstanding performance on the PISA test.” Policy Research Working Paper 7630, 1-43. 6 Musso, M. F., E. Kyndt, E.C. Cascallar, and F. Dochy (2012). “Predicting mathematical performance: The effect of cognitive processes and self-regulation factors.” Education Research International. Vol. 12. 7 Boekaerts, M. and E.C. Cascallar. Unpublished manuscript. PHOTO CREDITS: Cover: “Language barrier might widen gaps in learning” by World Bank/Chau Doan, license: CC BY-NC-ND 2.0 Page 2: “Students in primary school” by World Bank/Simone D. McCourtie, license: CC BY-NC-ND 2.0 Page 4: “Vietnam: Poor and ethnic minority students face persistent lower education performance” by World Bank/Chau Doan, license: CC BY-NC-ND 2.0 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.