Finance & PSD Impact DECEMBER 2017 The Lessons from DECFP Impact Evaluations ISSUE 45 Our latest note shows how big data can help combine experimental with non-experimental approaches in impact evaluations when take-up is low. Learning the Impact of Financial Education When Take-Up is Low Gabriel Lara Ibarra, David McKenzie and Claudia Ruiz Ortega Low levels of financial literacy are pointing at the lack of power to detect any pervasive in both developed and developing effect. countries, leading to many governments, non-profits, and banks offering financial education programs. However, voluntary Figure 1. Treatment vs Control Time Paths participation rates in these programs are often very low. This was the case with a recent experiment we implemented in Mexico. We collaborated with the Mexican bank BBVA Bancomer in an experiment to measure the impact of Adelante con tu futuro (Go ahead with your future), a large-scale financial education workshop that BBVA Even if this program helps those who Bancomer conducts in Mexico since 2008. participate, low take-up rates dramatically As of 2016, about 1.2 million participants reduce our ability to detect such an effect. have received the training. This by no means is a unique situation. Over 100,000 credit card clients Despite their large number of users, the participated in the experiment. Of 73,654 response to many financial product clients who were assigned to the treatment marketing campaigns such as those offering group, only 583 attended it. That is, take-up credit cards or selling insurance products are was 0.8%. In a second experiment that we also incredibly low. designed to test personalized financial coaching, again take-up was low (only 6.8% How Big Data Can Help of clients in the treatment group received the So, what can we do to credibly estimate coaching sessions). the effect of financial education on the clients With take-up rates this low, it is not that did take-up the program? surprising that we are unable to detect any Our solution is to use the richness of big effect of financial education using the pure data. As part of the study, we have access to experimental approach. As only a handful of a large administrative data set (of 660 MB), clients received treatment, the trajectories of which follows the monthly financial the treatment group follow closely those of indicators of each client for up to 18 months the control group in the months after the prior to the intervention and 6 months after it. intervention (see Figure 1). Thus, the Moreover, from the experimental approach experimental method ITT estimates (which we also had a large pool of clients randomly measure the effect of being offered the assigned to the control group. program) are close to zero. The LATE This data enables us to obtain credible estimates (which are the experimental estimates by combining the experiment with treatment effects for those who actually two non-experimental approaches. We first receive treatment) are not statistically use propensity score matching to find, among significant, with wide confidence intervals Do you have a project you want evaluated? DECRG-FP researchers are always looking for opportunities to work with colleagues in the Bank and IFC. If you would like to ask our experts for advice or to collaborate on an evaluation, contact us care of the Impact editor, David McKenzie (dmckenzie@worldbank.org) the clients in the control group, a subset of Monthly credit card spending increases by clients that best mimics the pre-intervention 63.7 percent, and the likelihood of owning a financial trajectories of clients in the deposit account with our partner bank also treatment group that received treatment. increases by 2.7 percentage points. To show that our results are robust to the choice of counterfactual, we conduct five Figure 2: Trajectories of financial outcomes of different approaches to obtaining a non- those receiving workshops compared to nearest neighbor matched control group experimental counterfactual, changing the variables used in the matching and using all matches in the common support vs just the nearest neighbor matches. Importantly, we are able to overcome a common challenge of propensity score matching, referred to as selection on unobservables. That is, if individuals in the control and treatment groups are so similar, why don’t individuals in the control group participate in the intervention? In our case we know why: clients in the control group were randomly not invited. The two financial education interventions With this matched control group, we then help clients reach the minimum payment and estimate the impact of attending the pay their bills on time more often, without workshop or receiving coaching using reducing their credit card spending. Both difference-in-differences. As our control interventions increase the likelihood that group matches by construction the trajectory clients are profitable for the bank. of variables of the treated clients over 18 months before the intervention, we thus have Policy Recommendations a credible test of common trends, a usual Our results have the following policy concern with difference-in-differences. implications: 1) While many clients do not want to Results participate in financial education, such The effects of the workshops on the programs can still offer some benefit to treated clients are summarized in Figure 2 those who do decide to take these (the coaching intervention had similar programs when offered – and doing so results). Under our preferred specification, can be profitable for the bank. we find that participating in the workshop 2) Access and usage of big data (i.e., in the increases by 11 percentage points the form of administrative information) can likelihood of paying more than the minimum help experimental evaluations with very payment, and reduces by 3.4 percentage low take-up. points the likelihood of delaying payment. For further reading see: Lara Ibarra, Gabriel, D. McKenzie and C. Ruiz Ortega “Learning the Impact of Financial Education when Take-Up is Low”, World Bank Policy Research Working Paper 8238, November, 2017. Recent impact notes are available on our website: http://econ.worldbank.org/programs/finance/impact