79805 June 2013 Financial capability in low- and middle-income countries: measurement and evaluation A REPORT ON THE WORLD BANK’S RESEARCH PROGRAM AND THE KNOWLEDGE FROM THE RUSSIA FINANCIAL LITERACY AND EDUCATION TRUST FUND Financial capability in low- and middle-income countries: measurement and evaluation A REPORT ON THE WORLD BANK’S RESEARCH PROGRAM AND THE KNOWLEDGE FROM THE RUSSIA FINANCIAL LITERACY AND EDUCATION TRUST FUND ROBERT HOLZMANN FLORENTINA MULAJ VALERIA PEROTTI with support from RICHARD HINZ, ELAINE KEMPSON, MATTIAS LUNDBERG, and KINNON SCOTT © 2013 International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org The findings, interpretations, and conclusions expressed here do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because the World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Cover photos: World Bank Cover design/layout: Nita Congress Contents Preface — vii Acknowledgments — ix Abbreviations — xi Summary — xiii 1 Study motivation, approach, and structure  —  1 1.1 Introduction — 1 1.2 Lessons and issues from high-income countries — 4 1.2.1 Broadening the concept of financial literacy and measurement — 4 1.2.2 Establishing an evaluation results framework and testing for financial education programs — 6 1.2.3 Financial education, behavioral finance, and alternatives to impact outcomes — 7 1.3 The background for financial literacy in low- and middle-income countries — 10 1.3.1 Definition of LICs and MICs and common relevant characteristics — 11 1.3.2 Idiosyncratic characteristics that may matter — 13 1.4 Addressing knowledge gaps: selection, conceptualization, and implementation — 16 1.4.1 Selection of the World Bank’s RTF work program: measuring financial literacy and measuring the effectiveness of financial education — 16 1.4.2 Conceptual alternatives to measuring financial literacy/ capability — 17 1.4.3 Implementation of the measurement and evaluation toolkit approach — 21 References — 22 2 Measuring financial capability in low- and middle-income countries  —  29 2.1 Motivation and overview of the measurement program — 29 2.2 Conceptual options and selected framework — 32 2.3 The FSA approach and lessons learned from HICs — 34 v Financial capability in low- and middle-income countries: measurement and evaluation 2.4 Key issues in LICs and MICs — 37 2.4.1 Access — 37 2.4.2 Poverty — 38 2.4.3 Education — 38 2.4.4 Location — 38 2.4.5 Informality — 39 2.4.6 Risk management — 39 2.5 Overview of selected approach  — 40 2.6 Implementing the approach in LICs and MICs — 44 2.6.1 Overview of implementation — 44 2.6.2 Focus groups: identifying manifestations of capability — 45 2.6.3 In-depth interviews: designing and testing the questions — 47 2.6.4 Pilot testing — 50 2.6.5 Key questionnaire features — 50 2.6.6 National surveys — 54 2.6.7 Factor analysis: constructing components  — 55 2.6.8 Factor analysis: identifying domains — 63 2.6.9 Cluster analysis: identifying vulnerable groups (intervention targets) — 64 2.7 Main outputs, lessons learned, and next steps  — 68 References — 72 3 Measuring the effectiveness of financial capability-enhancing interventions  —  77 3.1 Motivation and overview of the evaluation program — 77 3.2 Issues and options: from financial education to broader financial capability interventions — 79 3.2.1 Conceptual development — 80 3.2.2 Implications for program design and evaluation — 82 3.2.3 Paucity of the existing evidence base — 85 3.3 Selected approach: field evaluations and a toolkit — 86 3.3.1 State of knowledge and identification of research gaps  — 87 3.3.2 Filling the gaps through field evaluations — 89 3.3.3 Individual project designs — 93 3.3.4 A toolkit for M&E — 104 3.4 Lessons and policy implications — 105 3.4.1 Process of Implementation — 106 3.4.2 Research methodology — 107 3.4.3 Identification of impact — 109 3.5 Conclusion and proposals for next steps — 110 References — 116 vi Contents 4 Conclusions: from the main results to the next steps   —  121 4.1 Main results  — 121 4.2 The RTF results in an international context — 126 4.3 Key open issues — 128 4.3.1 Conceptual issues — 128 4.3.2 Empirical issues — 128 4.3.3 Costing and financing of interventions — 129 4.3.4 International networking and knowledge sharing — 129 4.4 Proposed next priority steps — 130 References — 131 Boxes 1.1 The normative/cognitive approach — 20 2.1 From basic concept to questionnaire — 41 2.2 From questionnaire replies to financial capability scores — 43 3.1 RTF case studies using traditional financial education — 95 3.2 RTF case studies using nontraditional financial education — 98 3.3 RTF case studies using combined interventions — 101 Figures 1.1 Account penetration around the world — 12 2.1 Factor analysis associates variables with components and domains — 56 2.2 cumulative distribution of component scores across pilot countries — 58 2.3 Cumulative distribution of domain “controlled budget� — 70 2.4 Cumulative distribution of domain “making provisions for the future� — 70 Tables 1.1 Account-holding around the world by income group and region — 14 2.1 Content of the main questionnaire — 51 2.2 Mapping of capability concepts with specific questions — 53 2.3 Results of pooled regression of component scores across pilot countries — 60 2.4 Mexico: average characteristics of the five clusters — 66 3.1 Priority areas guiding the selection of RTF field evaluations — 90 3.2 Summary of impact from the RTF evaluations — 111 vii Preface F inancial capability—the capacity to effectively manage financial resources over the life cycle and engage constructively with financial products and services—is now an essential skill for individuals in all walks of life and a central public policy concern throughout the world. The catalytic role of financial sector development and financial inclusion in development imposes myriad new challenges on the citizens of low- and middle-income countries to engage with banks, insurance products, and savings vehicles. The erosion of traditional forms of family and community support that accompanies development, and innovative social protection programs such as conditional cash transfers, necessitates increased responsibility and new skills. As the recent global financial crisis has illustrated, exercising these skills in a respon- sible manner is an important contributor to broader economic stability and shared prosperity as well. Recognizing the rapidly emerging importance of financial capability in low- and middle-income countries, the Ministry of Finance of the Russian Federation in October 2008 established the Financial Literacy and Education Trust Fund at the World Bank. The Trust Fund finances the development of resources and knowledge to support policy makers and practitioners as they struggle to create strategies and programs to enhance the financial capability of their diverse populations. This effort grew out of the challenges that Russia has faced in making its own transition to a market economy, which led to the designation of financial literacy as a key priority under the Russian presidency of the G8 in 2006. The Trust Fund provided substantial resources for the World Bank and the Organisa- tion for Economic Co-operation and Development (OECD) to undertake two distinct but coordinated programs of work. The two institutions began this effort with consul- tations to develop an overall strategy that reflects their respective characteristics and expertise. The OECD, in conjunction with the International Network for Financial Education (INFE), focused its efforts on the review and stocktaking of policy devel- opment and national strategies to facilitate the sharing of experience and formulate principles, guidelines and best practices. The World Bank focused its efforts on devel- oping diagnostic and measurement tools and undertaking research on the results achieved by a wide range of programs designed to improve financial capability. Both ix Financial capability in low- and middle-income countries: measurement and evaluation institutions developed new survey instruments to measure and monitor the knowl- edge, attitudes, skills, and behavior of individuals. In combination, these efforts make an important and lasting contribution to the base of knowledge on these issues and serve as important resources for policy makers and practitioners. This report provides a summary and overview of the work undertaken by the World Bank that was supported by the Trust Fund. It focuses on the two main elements of this effort: the development, testing, and application of a new survey instrument designed to measure a broader, behaviorally oriented definition of financial capability specifically designed for use in low- and middle-income settings; and the documen- tation of methods for the evaluation of financial education and capability enhance- ment programs in conjunction with a carefully selected suite of research studies that both illustrate these innovative methods and add to the base of evidence about what works and what does not. The work of the OECD is similarly summarized and documented in a set of its own publications that are referenced throughout the report. All of these resources are available on the Trust Fund website, www.finlitedu.org, and the OECD website www.financial-education.org. x Acknowledgments T he genesis, development, and completion of the project owe unlimited thanks and recognition to many people without whom this intellectual voyage would not have been possible. This begins with the staff of the Ministry of Finance of the Russian Federation who brought the theme of financial literacy to the agenda of Russia G8 Presidency. They had faith in the World Bank and OECD promise to deliver an innovative contribution on this essential issue to fulfill the interest and ambition of the Russian Federation as an emerging sponsor of development aid and policy thinking. Special appreciation is afforded to the Vice Minister of Finance, Sergey Storchak, who took the visionary decisions to initiate the enterprise, and the Director of the Department of International Relations within the Ministry of Finance, Andrey Bokarev, who was essential in guiding the process. Others in the ministry who deserve recognition include Anna Valkova, Elena Ilina, and Anna Zelentsova. The crit- ical role played by our World Bank colleague Andrei Markov as an effective interloc- utor and facilitator throughout the entire project also requires recognition. Critical intellectual support for the effort was provided by the participants in the November 2009 workshop in Washington, D.C., who were willing to spend their time and energy to shape the content and process of the research agenda. Some of the participants became highly valued members of a group of experts that have provided guidance throughout. These include Gerrit Antonides, Sharon Collard, Olga Kuzina, Annamaria Lusardi, and Christian Poppe, all of whom provided important expert advice to teams undertaking the research projects. Others making important contributions through meetings of the International Network on Financial Education where the project results at various stages were presented and discussed include Diana Crossan, Jason Fichtner, Sue Lewis, and José Alexandre Cavalcanti Vasco. In addition, the work benefited from a joint learning event on methods for impact eval- uations of financial capability enhancement programs undertaken with Alyna Wyatt from Genesis Analytics in Cape Town, South Africa, in the fall of 2011. The staff of the RAND Corporation, which developed the Toolkit of Evaluation Methods, in particular Angela Hung, Arie Kapteyn, and Joanne Yoong, made a significant contribution to the overall effort. And last but not least, we enjoyed the company and intellectual xi Financial capability in low- and middle-income countries: measurement and evaluation interactions with our OECD colleagues and comanagers of the Trust Fund program, in particular Adele Atkinson, Andre Laboul, and Flore-Anne Messy. Most importantly, the program could not have been successfully completed without the dedicated effort of the teams working on the development and implementation of the financial capability measurement surveys and the program impact evaluation. This was an outstanding example of a productive team effort by the 12 country survey and 17 impact evaluation task teams consisting of the respective country teams, the regional World Bank task team leaders, members of the Development Economics Research department of the World Bank, and the team managing the administration of the Trust Fund in Washington, comprising more than 100 individuals whose contributions can be found in the reports and publications summarized in this document. Last but not least, we would like to thank our colleagues from the social protection and financial sectors for their collaboration on a cross-sector topic that promises to contribute so much to the World Bank’s vision of development. While all authors take responsibility for the full report, the drafting and the support was distributed across chapters. Robert Holzmann is the senior advisor to the Trust-funded project, in charge of the overall report; he drafted chapters 1 and 4 and appendix A. Valeria Perotti drafted chapter 2 and appendix B; Florentina Mulaj drafted chapter 3 and appendix E, and for appendixes C, D, and F she built on other work linked with the Trust Fund. Richard Hinz is the Program Manager of the Russia Financial Literacy and Education Trust Fund and offered important guidance and editorial support. Elaine Kempson, the lead external consultant for the project, helped critically at every stage of the project and provided important review, feed- back, and guidance for the final report. Mattias Lundberg and Kinnon Scott, who served as World Bank internal task team leader and technical advisor for the M&E and measurement components, respectively, provided similar internal technical support. Amy Gautam did the copy editing, and Nita Congress did the design and layout work. Without this joint effort, input, and support, the project and this report would not have been possible. xii Abbreviations BRIC Brazil, Russia, India, and China CAPI computer-assisted personal interviews CC Coordination Committee (of the RTF) CCT conditional cash transfer CGAP Consultative Group to Assist the Poor DEC Development Economics (of the World Bank) DFID Department for International Development FEF Financial Education Fund (of DFID) FL&E financial literacy and education FPD Financial and Private Sector Development (Network of the World Bank) FSA Financial Sector Authority GIZ Gesellschaft für Internationale Zusammenarbeit (of Germany, formerly GTZ) GNI gross national income G2P government to person HDNSP Human Development Network Social Protection and Labor (World Bank) HIC high-income country IBRD International Bank for Reconstruction and Development ICT information and communications technology IDA International Development Agency INFE International Network on Financial Education LIC low-income country M&E monitoring and evaluation MIC middle-income country MIS management information system NS National Strategy xiii Financial capability in low- and middle-income countries: measurement and evaluation OECD Organisation for Economic Co-operation and Development PCA principal component analysis RCT randomized controlled trial RTF Russia Financial Literacy and Education Trust Fund SC Steering Committee (of the World Bank) SRM Social Risk Management All dollar amounts are U.S. dollars unless otherwise indicated. xiv Summary s trong and rising interest in improving financial capability has moved to the fore- front of public policy concerns worldwide during the past decade. Originating in concerns of high-income countries (HICs) about vulnerable populations and the potential impact of low levels of capability on stability these issues have rapidly expanded to middle income and developing countries as they consider the relation- ship between financial capability, financial sector development and inclusion and broader economic development. The common assessment across all countries is that the level of understanding of financial issues by individuals is too low and asso- ciated with behavior that has consequences for individuals (such as undersaving or wrong product choices) and for economies (such as less efficient financial markets and over-indebtedness of households). Providing financial education has tradition- ally been seen as the key intervention to improve skills, attitude, and behavior, thus leading to better financial outcomes. A number of countries have established comprehensive national financial literacy and education (FL&E) strategies and many others are thinking about doing so. Many more countries want to do likewise and are looking for guidance and support. While real progress has been made in bringing countries and key public actors together to exchange information and experiences, there has been less progress on the concep- tual and empirical side. Uncertainty remains in relation to the conceptual definition and measurement of financial literacy and capability; and there is considerable empirical uncertainty about the effectiveness of traditional forms of education rela- tive to other types of interventions. Even if more conceptual and empirical certainty were to develop in higher income settings, its translation into low- and middle-in- come environments is far from straightforward. Conceptual and empirical uncertainty is normal for any new topic, particularly one that involves the many different psychological, social, economic and other influences that affect financial behavior. To reduce uncertainty and move toward a broadly shared conceptual framework calls for analytical rigor, empirical testing, and high- quality knowledge exchange. It also requires addressing the inevitable differences in perspective and interpretations of the evidence. In addition, translating the framework that is now emerging from HICs to the very different conditions in low- and middle-in- xv Financial capability in low- and middle-income countries: measurement and evaluation come countries (LICs and MICs) calls for careful review and adjustment of concepts, measurements, and interventions to the particular circumstances of these countries. The World Bank’s interest in the topic of Financial Literacy, Capability and Education is closely linked with its development mandate and the critical role of effective finan- cial intermediation to support poverty alleviation and shared prosperity. Successful intermediation requires a well-developed financial sector, sound regulation, and a financially capable population. Only then will intermediation be able to deliver on critical economic development outcomes such as inter-temporal resource exchange and risk diversification, as well as the social policy objectives of managing income and consumption allocation and addressing risks across an individual’s life cycle. While this is important for countries of all levels of development, the content and measurement of financial capability and the interventions to achieve it are likely to differ across the development spectrum. This report provides an overview of the conceptual foundations and work program implemented by the World Bank under the Russia Financial Literacy and Education Trust Fund (RTF) generously supported by the Ministry of Finance of the Russian Federation beginning in October 2008. The overall objective of the Trust Fund was to support implementation of the December 2006 summit statement from the Russian G8 Presidency, which greatly advanced the topic of Financial Literacy and Education within the international policy discussion. The specific objective of the overall effort was to extend the knowledge base to help LICs and MICs prepare and implement national strategies and programs in this area. This report positions the results of the RTF work program undertaken by the World Bank within the broader realm of international knowledge activities. It highlights the contributions of this work program to the conceptual development and measure- ment of financial capability and the evaluation of the results achieved by programs directed towards its enhancement. A main feature of this effort is a broadening of focus from a narrow concept of knowledge (i.e., the traditional definition of “financial literacy�) toward a concept of “financial capability� that incorporates a wider range of issues related to attitudes and behavior. In parallel, the realm of promising interven- tions has been broadened from more traditional method of classroom education and seminars to include other financial capability-enhancing methods such as entertain- ment education and social marketing. The elements of the work program led by the World Bank (the OECD work is addressed separately in a number of complementary publications) focused on two measurement-related topics: (1) how to measure financial capability in a way that is applicable to diverse levels of economic development and across individuals of different income levels; and (2) how to measure the effectiveness of interventions to improve financial capability including, but extending well beyond, formal financial xvi Summary education programs. The choice of this focus reflects a number of considerations, hypotheses, and assumptions that led to the specific design of the work program. The most important of these are: „„ Effective national strategies require a measurable concept of what they seek to improve that needs to be grounded in analysis that can be reliably under- taken in LIC and MIC settings. „„ A workable concept needs to go beyond financial knowledge and skills (i.e., the traditional financial literacy concept) to include attitudes, behavior, and outcomes. This is embodied in the emerging concept of financial capability. „„ Financial capability should be measured with an approach based on both measurable outcomes and behavior that does not prejudge the types of inter- ventions that might be effective in improving financial capability. „„ Rigorous and comprehensive monitoring and evaluation (M&E) is required to identify effective interventions to improve financial outcomes; this is rarely done throughout the world and is especially limited in LICs and MICs. Rigorous refers here to the use of the best counterfactual available, while comprehen- sive refers to the use of both quantitative and qualitative methods: the first to establish causality, the second to shed light on the underlying change mecha- nism—the why and how of program impact. Measuring financial capability in low- and middle-income countries The selection of the financial capability measurement concept began with the identification of two key dimensions: normative versus positive and cognitive versus agnostic. The normative approach uses as a benchmark the optimal behavior of indi- viduals typically derived by assuming that individuals maximize their inter-temporal utility over their lifetime; the positive approach takes as a benchmark the outcome of financial decisions judged to be superior. The cognitive approach assumes that any deviations from optimal financial behavior are due to a lack of information and/or training, implicitly making financial education the preferred intervention. The agnostic approach has no expectations regarding what prevents individuals from optimal behavior and is consequently open to any kind of intervention for improvement. The normative/cognitive approach builds on sequential information and learning constraints; the positive/agnostic approach focuses on what matters empirically, but makes no prior assumptions about the why and how. Both approaches focus on the broader concept of financial capability for good financial behavior but with a different emphasis. xvii Financial capability in low- and middle-income countries: measurement and evaluation For conceptual, empirical, and policy reasons, the World Bank project team selected the outcome-driven, positive/agnostic approach to measure financial capability. This approach identifies individuals' financial capability via behavior or manifestations (such as keeping track of daily expenditure) that are considered to matter for good outcomes. The assessment is based on the judgment of one’s peers (vox populi) and what they consider to be good outcomes, and on the characteristics that lead to those outcomes, such as good behavior, but also attitudes, skills, and knowl- edge. However, there is no logical chain of assumptions on how to progress toward improved patterns of behavior or assumptions about what intervention will positively impact desirable financial characteristics. An important impetus for this approach is that it implicitly takes account of people's effective constraints (e.g., limited access to financial products), which can affect their judgment of what constitutes good behavior. The approach is a priori agnostic about why and how good outcomes are achieved. This conceptual approach to defining and measuring financial capability was initi- ated by the United Kingdom’s Financial Sector Authority (FSA) in the mid-2000s, and initially focused on knowledge, skills, and attitudes. Focus groups, however, made it clear that many people had a different notion of capability that they defined in behavioral terms. Specifically, the focus groups identified a range of attributes that, when combined, denote financial capability. These were grouped into four domains: managing money (i.e., living within one’s means and tracking one’s expenditures); planning ahead (i.e., coping with unexpected events and making provisions for the long term, including education, health, and old age); making choices (i.e., being aware of the options available and being able to choose the one most appropriate for one’s circumstances); and getting help (i.e., becoming self-reliant through information gath- ering and knowing where and when to turn for advice and help). The project team, however, did not simply adopt the four domains and copy the FSA’s scoring methods. Instead the team undertook a complete bottom-up approach to identify the domains and manifestations of financial capability in LICs and MICs and to develop a method for measuring these attributes that would provide compa- rable measures across the widely varying country settings. The objective of this effort was to construct indicators of financial capability that were as culturally and income neutral as possible, and that would be valid and consistent across different environ- ments. Without assuming the content and range of the concept of financial capability, an inductive approach was used to develop the survey questionnaire. This was based on focus groups and cognitive testing in eight countries (Colombia, Malawi, Mexico, Namibia, Papua New Guinea, Tanzania, Uruguay, and Zambia). The questionnaire was then applied in national surveys in seven countries (Armenia, Colombia, Lebanon, Mexico, Nigeria, Turkey, and Uruguay) and in a pilot survey in Papua New Guinea. The resulting findings and guidance on the use of the instrument and analysis of the data, xviii Summary effectively comprising a financial capability measurement toolkit, are now available for public use. The key assumption of this approach was that financial capability is an underlying set of abilities and behaviors that cannot be measured directly. The focus group results revealed a set of behaviors, attitudes, and skills that denoted a financially capable (or incapable) person that was remarkably consistently across the very different coun- tries in which the research and survey development work was undertaken. Multiple questions were developed, tested and included in the survey instrument to measure each of the manifestations of capability that were defined by the focus groups and in-depth interviews. To accommodate people with low levels of literacy and education, the questions were simplified by splitting difficult topics into a sequence of questions which were then combined as a derived variable in the analysis. Also, because of the complexity of the financial capability concept, responses to several questions were combined to measure each manifestation of financial capability. Several groups of questions were analyzed together to understand the relationships among the different mani- festations. As a consequence, while answers to single questions (or small groups of questions) can be used to analyze specific issues (such as “How many people plan how to use their money?�), any study aiming to use these data to assess the level of financial capability and to provide policy recommendations should use this more complex type of analysis of survey responses. The work on the financial capability measurement instrument and the analyses of the survey data collected in the seven countries offer a rich set of information that leads to a number of conclusions. The most important of these are: „„ Through the application of the positive/agnostic approach and use of the vox populi method, it is possible to identify a range of common attributes of finan- cial capability that apply across very diverse low- and middle-income settings. Most, but not all, of the identified topics resonate with the findings from similar efforts in HICs, including the United Kingdom (where the approach was originally developed), Ireland (where the results were broadly replicated), and other countries where this method has been tested. „„ For LICs and MICs and specifically for lower-income groups, the relevant elements of financial capability are oriented to managing resources on a day-to-day basis and to planning for the future. These have also been identi- fied in HICs. However, two other topics found to be most relevant in HICs—the capability to choose among alternative financial products and finding and assessing information, help, and advice—found only limited resonance in the other settings. xix Financial capability in low- and middle-income countries: measurement and evaluation „„ From the coded replies to the questionnaire, it is possible to create scores for individual components of financial capability that are robust and mean- ingful across different countries. The statistical process using the pooled data suggests that two domains of capability are relevant—“controlled budgeting� and “making provisions for the future.� However, it is not statistically mean- ingful to collapse these into a single score for a uniquely comparable level of financial capability across different settings. While these two domains emerge in all investigated countries, cross-country comparisons need to be taken with a grain of salt as they may not be statistically robust given the subtle differences in the composition of these two domains across countries. While such analyses can be conducted at the individual country level, the number of domains needed to capture all the components of financial capability may differ from two to four. „„ Critical for guiding and focusing national strategies and financial capability interventions, the results demonstrate that the populations of individual coun- tries can be segmented into groups with varying levels of capability across 12 components of financial capability. Each group's strengths and weaknesses for each component can be determined—as can its demographic, social, and economic characteristics. Thus, it is possible not only to identify vulnerable groups that show low scores of financial capability, but also to offer, through the identified components, first indications of the most appropriate interven- tion: financial education in some cases, behavior-oriented interventions in others. Whether such interventions are truly effective, however, needs to be settled through rigorous evaluation. „„ The results are based on the design and use of a questionnaire that works across different income groups and quite different cultures. This promises to capture relevant manifestations of financial capability accurately without introducing an income or culture bias. Clearly, this is result that needs further testing and elaboration. Evaluation of Financial Capability Enhancement Programs To promote rigorous and comprehensive program impact evaluation in LICs and MICs and to garner knowledge about the effectiveness of specific financial capability interventions, the World Bank's program evaluation effort under the Trust Fund had three main elements: (1) development of a toolkit providing methodolog- ical guidance on the design and implementation of evaluations of financial capability enhancement programs; (2) a stocktaking exercise to identify knowledge gaps to xx Summary guide the selection of interventions to be assessed; and (3) a process for soliciting and financially supporting program evaluations for 17 competitively selected inter- ventions in LICs and MICs. The scarcity of such evaluations in LICs and MICs motivated the development of the impact evaluation toolkit. The RTF set out to develop a toolkit specifically targeted to these types of environments. Its development was led by the RAND Corpora- tion, contracted by the World Bank through a competitive procurement. Through this effort, the RTF aimed to make evaluation expertise available to stakeholders operating in resource-scarce environments and to illustrate technical aspects of evaluation in simple terms, using as case studies the evaluations supported by the Trust Fund. While many handbooks address different aspects of M&E, the RTF toolkit differs in a number of ways: (1) it provides an overview of all evaluation and research methods, including impact and process evaluation and quantitative and qualita- tive techniques; (2) it is designed specifically for the evaluation of financial capa- bility programs, including the challenges of evaluating different types of programs, especially those that use media mechanisms and social marketing (which usually require special attention in an evaluation); (3) it entails the development of outcome measures specific to financial capability; and (4) it includes the details of the inter- ventions themselves, not just the evaluations. The toolkit is publicly available and can be downloaded at no cost from the RTF website. To better understand the state of knowledge in terms of which programs work and which don’t and to identify gaps in the existing knowledge to help guide decisions on the funding of evaluations, the RTF undertook a stocktaking exercise. Past reviews of financial capability programs were limited in terms of the information they provided about the reviewed evaluations, such as program characteristics and the rigor of the evaluations. In the first step of the review a set of evaluations related to the topic of financial education and broader financial capability (both completed and in progress) was systematically collected. From over 1,000 evaluations found a sample of 129 was selected and through their review areas of over- and underrep- resentation were identified to guide the selection of the evaluation studies to be funded. The results of this review are described in detail in the in depth discussion of the program and is available on the Trust Fund website. The most intensive element of the work program (in terms of staff and financial resources) was the process for soliciting and financially supporting impact eval- uations for 17 competitively selected programs in LICs and MICs. The large alloca- tion of resources to this element of the work program was motivated by the limited number of rigorous impact evaluations in this field. This is likely due to: (1) the high cost of rigorous M&E for new interventions, which can exceed $100,000, in addition to costs related to the design and implementation of the intervention itself; (2) the xxi Financial capability in low- and middle-income countries: measurement and evaluation public good nature of rigorous M&E, which produces knowledge for worldwide public consumption at the expense of the intervention's provider; (3) the lack of human capital and expertise in M&E; and (4) resistance from program providers (often NGOs) to evaluation, which may reveal unfavorable results and/or limited impact, poten- tially jeopardizing funding from financiers (often financial institutions, foundations, or public bodies). Conditions for the competitively selected interventions included: (1) (co-) financing the M&E part of the intervention, but not the intervention itself; (2) announcing which interventions would qualify for financing, thereby addressing knowledge gaps iden- tified in the stocktaking exercise; (3) committing recipients to participate in peers reviews and workshops where the evaluation would be reviewed and discussed; and (4) offering hands-on advice by a panel of international experts with experience in financial capability programs and evaluation methods. The impact evaluation toolkit, the preparation and implementation of the 17 M&E pilots in Asia, the Pacific, Africa, and Latin America, and the preliminary results from these studies offer a wealth of information that will be further enriched as the remaining results emerge and longer-term evaluations are undertaken. However, some initial conclusions can already be drawn. The most valuable of these include: „„ Supporting rigorous and comprehensive M&E of financial capability inter- ventions works very well and makes a lot of sense from a public policy perspective: it provides firsthand knowledge on promising interventions; the interventions themselves can be solicited or at least influenced; the results allow an early understanding and correction of what works and what does not; the results save valuable financial resources over the medium term; and the approach is highly incentive-oriented, or can be structured to this end. However, to make this happen requires some innovative thinking and manage- ment. „„ Getting the most out of M&E, and at times making it worthwhile at all, requires upstream and downstream efforts. The lessons from the M&E support under the RTF work program suggest that one should: (1) design the evaluation in conjunction with the program design upstream to be very clear about the objectives and hypothesis to be tested; (2) clearly identify knowledge gaps among interventions to target financing for priority areas; (3) wherever feasible, adopt mixed methods for evaluation to answer not only if there is a measurable outcome but also why and how; (4) attempt to isolate the specific factors responsible for the change—an approach particularly needed for outcomes and financial behaviors subject to many influences; (5) conduct comparative examinations to measure the relative impact of programs and projects delivered in different settings or across different target groups; xxii Summary (6) explore variations in the delivered quality and intensity or level of the service (e.g., education and information); and (7) incorporate and test the insights from psychology and behavioral economics into intervention design and implementation. „„ The knowledge creation aspect of the program did very well in reviewing existing, methodologically acceptable evaluations to identify gaps in inter- ventions. This exercise was the first of its kind and was extremely helpful in guiding the solicitation and selection process for the evaluation studies. The program also did very well in promoting and evaluating new and promising interventions beyond financial education. The 17 impact evaluation pilots include the use of entertainment based education (edutainment) through a TV soap opera in South Africa and a feature film in Nigeria, comparative evalua- tions (e.g., comic books, radio, social networks versus classroom teaching) in Kenya and India, mixed interventions (e.g., classroom teaching plus nudging using SMS) in India, and the use of debit cards and separate accounts in conjunction with education workshops in Malawi and the Dominican Republic. „„ Promotion of the exploration of alternative interventions is fostered by the outcome-based/agnostic approach, which does not have a fixed vision regarding how better financial capability can be achieved, and by the comple- mentary view that lessons from behavioral economics offer indications as to why the traditional assumed pathway from knowledge and skills to attitude and behavior may not work as envisaged. It is too early to say whether these alternative interventions are truly successful: the short-term results are just being realized and the long-term effects may be different. Nevertheless, the results support further probing of edutainment, social marketing, and behav- ioral economics-based approaches as potentially very effective and cost-effi- cient interventions. „„ Finally, regarding the aspiration to substantially move the knowledge agenda of what works or does not in traditional areas of interventions, the results to date are both sobering and encouraging. The limited evidence suggests that enhancing financial capability among poor households is extremely difficult to achieve, particularly with regard to planning for the future. Some program evaluations indicate that financial education may work for both knowledge and savings outcomes with high-quality interventions that “get everything right,� but this promising result needs to be confirmed under more varied circumstances. One general observation seems to be that the more rigorous the evaluation, the less likely the program is to demonstrate a positive impact. This is a demonstration of the importance of rigorous design and evaluation rather than an encouragement to evaluate less rigorously. xxiii Financial capability in low- and middle-income countries: measurement and evaluation To better appreciate the RTF work program results, the report puts its knowledge contributions in the context of the international discussion about issues, gaps, and priorities and highlights their addition. The most critical contributions are that: „„ The measurement of the improvement in financial literacy or capability and the rigorous evaluation of related interventions remain top priority topics. They were identified and selected by the RTF as critical items in the FL&E area in 2006/08, and they remain priority topics in 2013 for formal financial institu- tions and NGOs alike. This is documented in the recommendations by the 2012 Citi Foundation commissioned “Report on Bridging the Gap� and in the conclu- sions of the 2011 MasterCard Foundation commissioned report on “Taking Stock: Financial Education Initiatives for the Poor.� „„ The RTF results are based on the first conceptually consistent and empir- ically rigorous work on the measurement of financial capability and of the effectiveness of the related interventions in low- and middle-income settings. The work is promising and ready for full replication in other countries. Two publications of in-depth methodological guidance, A Toolkit for the Evaluation of Financial Capability Programs in Low- and Middle-Income Countries and Measuring Financial Capability: Questionnaires and Implementation Guid- ance for Low- and Middle-Income Countries can be downloaded from the Trust Fund website, applied, and used to compare new results with those of the RTF pilots. The processes underlying the development and production of this technical guidance is fully documented and available on the website and summarized in the remainder of this report and its appendixes, which are also available on the website. „„ The financial capability measurement results under the RTF are open for full comparison with other national measurement efforts by various countries (including the OECD/INFE survey also supported by the RTF) This allows for a comparison at the conceptual level, but also for testing whether results are similar both within and across countries. „„ The approach and results of the RTF M&E pilots can be compared with other similar efforts. For example, the impact evaluation sponsoring approach was used by DFID’s Financial Education Fund, which sponsored 15 evaluation proj- ects in Africa. Lastly, an increasing number of LICs and MICs-oriented research are focusing on rigorous impact evaluation and are now extending the evalu- ated interventions to financial education. The World Bank–led RTF work program has significantly advanced the knowledge agenda on financial capability measurement and related program evaluation issues. xxiv Summary Despite this progress, many questions remain unanswered and new issues have been raised, including: „„ Conceptual issues, such as: –– Developing a broadly shared vision, definition, indicators, and measure- ment for financial capability. –– Developing a conceptual framework regarding what kind and form of interventions work best for each financial domain and situation (country, individual characteristics, etc.). For example, teaching individuals to draw up a budget may well be amenable to financial education and learning, while teaching them to plan for retirement may require a combination of interventions including education, choice architecture (i.e., nudging), and advocacy. –– Defining the role of financial sector actors in financial education and clearly delineating financial education from product promotion. „„ Empirical issues, such as: –– Empirically establishing what matters for good financial outcomes. What is the importance of financial knowledge and cognitive skills (i.e., the original financial literacy idea) and for what kind of outcomes? What is the rele- vance of financial capability as defined by the vox populi method and for what kind of outcomes? –– Merging the measurement of financial capability and the effectiveness of interventions into one empirical framework. The indicators to measure the effectiveness of an intervention would also be part of the financial literacy/ financial capability indicator set. –– Exploring further the measurement of the effectiveness of non-cognitive interventions (such as advocacy, edutainment, etc.) on financial behavior. The outcome will have a major impact on the content and process of any national financial strategy. „„ Costing and financing issues, such as: –– Identifying the cost effectiveness of financial education programs and other interventions. Addressing and solving the measurement and evalua- tion issues helps to identify effective interventions but does not yet iden- tify cost-efficient ones. –– Closing capability gaps where they exist. It is claimed that such gaps have widened in recent years due to the increase in access to financial xxv Financial capability in low- and middle-income countries: measurement and evaluation products and services in LICs and MICs, while financial education and skills enhancement is lagging well behind. This creates a unsustainable and potentially dangerous situation for the poor and for development outcomes, and is costly to eliminate. –– Evaluating the cost burden and externalities of interventions and achieving a consensus on how these should be allocated. The questions of who profits from the interventions and who should pay for them have received limited consideration to date. In the abstract, costs should be borne by those who profit from the intervention. However, acknowledging the initial costs and externalities of interventions on development and economic growth provides space for the role of public subsidies, mandated inter- ventions, and structured cost-sharing between the government, financial sector providers, and individuals. „„ International networking and knowledge sharing issues, such as: –– Integrating private sector financial services providers (such as formal financial institutions, MFIs, or NGOs offering financial services, financial education, or both) into existing international networks. –– Including private sector providers, who claim a shared (conceptual) frame- work and a knowledge infrastructure, to avoid duplicative or contradictory efforts. –– Placing detailed, rigorous, and constantly updated information and knowl- edge in the public domain. At present there is no established and reliable means to make the emerging information fully available to policy makers, especially those in developing countries. „„ Proposed next steps, such as: –– Strengthening the knowledge platform for all actors by: ƒƒ Encouraging countries to apply the financial capability survey tool (enhanced by modules on financial knowledge or other topics as they deem useful), undertake the analyses on domains and scores, repeat the survey periodically, and share their findings internationally. ƒƒ Motivating providers of financial capability interventions to engage with M&E teams early on and to prepare for the discussion, prepara- tion, and implementation of comprehensive program evaluations that make use of the methodological guidance provided through the RTF and other sources. xxvi Summary ƒƒ Exploring cost-effective ways to more systematically collect the results and lessons of impact evaluations across the world, particularly in LICs and MICs. –– Using national financial (education) strategies more strategically by: ƒƒ Establishing the priority areas where more knowledge about the effec- tiveness of traditional interventions is needed and promising inno- vating interventions are expected. ƒƒ Motivating the application of rigorous and comprehensive M&E by co-financing program evaluations and offering technical support from the very beginning of projects. ƒƒ Encouraging peer learning among the providers of financial capability interventions through facilitation, technical support, and knowledge dissemination. –– Reviewing and expanding financial (education) strategies by: ƒƒ Finding financial and administrative means to expand impact evalu- ations from short- to medium- and long-term effects to gain a much- needed understanding about the time profile of their effectiveness. ƒƒ Exploring promising and cost-effective interventions such as entertain- ment based education, choice architecture and social marketing to improve financial behavior on a large scale. ƒƒ Making the search for cost-effectiveness of financial capability inter- ventions a guiding principle of strategy and knowledge generation. xxvii S CHAPTER 1 tudy motivation, approach, and structure 1.1 Introduction The last 10 years or so have experienced strong and rising interest in the level of financial literacy worldwide and in educational interventions to improve it. While this interest was initially concentrated in high-income countries (HICs), the enthusiasm has expanded to the poorer parts of the world. A common theme across all countries is an assessment that the level of understanding of financial issues by individuals is too low, with negative consequences for individuals (such as undersaving or wrong product choices) and for economies (such as less efficient financial markets and over-indebtedness of households). Providing financial education has traditionally been seen as the key intervention to reduce ignorance and improve skills, attitude, and behavior, thus leading to better financial outcomes. A number of richer countries have established comprehensive national financial literacy and education (FL&E) strategies and many others are thinking about doing so. Many poorer countries want to do likewise and are looking for guidance and support. While real progress has been made in bringing countries and key public actors together to exchange information and experiences on ideas and practices, there has been less progress on the conceptual and empirical side. Conceptual uncertainty concerns the objectives, definition, and measurement of financial literacy; empirical uncertainty concerns the effectiveness of financial education compared to other interventions to improve financial outcomes. Even if more conceptual and empirical certainty did exist in HICs, the translation into low- and middle-income environments may be far from straightforward. Conceptual and empirical uncertainty is normal for any new topic and typical of the pre-paradigm phase in any new discipline, particularly one that covers aspects from so many different disciplines. To reduce uncertainty and move toward a broadly shared conceptual framework calls for analytical rigor, empirical testing, and high- quality knowledge exchange with the occasional strong academic dispute that should be embraced, not avoided. Translating a not yet fully established framework from HICs to low- and middle-income countries (LICs and MICs) calls, in addition, for 1 Financial capability in low- and middle-income countries: measurement and evaluation careful review and adjustment of existing concepts, measurements, and interven- tions to the specific circumstances of poorer countries. The World Bank’s interest in the topic of FL&E is closely linked with its development mandate and the critical role of functioning financial intermediation for economic and social development. Successful intermediation requires a well-developed financial sector, sound regulation, and a financially capable population. Only then will intermedi- ation be able to deliver on critical economic aspects such as intertemporal exchange and risk diversification, as well as social policy aspects such as managing income and addressing risks across an individual’s lifecycle. While this is important for countries at all levels of development, the content and measurement of financial capability and the interventions to achieve it are likely to differ across the development spectrum. This report presents the approach for supporting FL&E in LICs and MICs developed and implemented by the World Bank under a trust fund generously provided by the government of the Russian Federation. The overall objective of the Russia Finan- cial Literacy and Education Trust Fund (RTF) was to support implementation of the December 2006 summit statement from the Russian G8 Presidency, which intro- duced the FL&E topic into the international discussion. Its specific objective was to develop knowledge that would help LICs and MICs in the preparation and implemen- tation of their national FL&E strategies. This report positions the results of the RTF work program within the realm of inter- national knowledge activities in this area and highlights its critical contribution. A main feature is the broadened focus from a narrow concept of knowledge (i.e., the traditional definition of “financial literacy�) toward a concept of “financial capability� that captures a range of financial issues around financial behavior. In parallel, the realm of promising interventions has been broadened from simple financial educa- tion to include other financial capability-enhancing measures such as edutainment and social marketing. The World Bank–led activities focused on the development of methodologies and tools for the measurement of: (1) financial capabilities; and (2) the effectiveness of associated interventions in LICs and MICs. These were supple- mented by parallel stocktaking and other activities by the International Network on Financial Education (INFE) in OECD and non-OECD member countries, also supported by financial resources from the RTF. The disseminated results of both activities should help countries better design, implement, monitor, and evaluate national FL&E strategies and interventions. To motivate the selected approaches and implementation of the World Bank-led work program, the remainder of chapter 1 summarizes lessons learned from the FL&E discourse in HICs; it then suggests special circumstances to consider when translating FL&E concepts and approaches to low- and middle-income environments; and lastly, it outlines the background of the RTF and the key considerations for 2 CHAPTER 1. Study motivation, approach, and structure selection of: (1) the specific measurement approach of financial capability piloted in LICs and MICs; and (2) the related impact evaluation support programs piloted. Both work program elements resulted in knowledge management tools for public use in countries across the world, including: a rigorously developed and tested survey tool to measure financial capability in varied settings; and an impact evaluation toolkit to facilitate the application of rigorous monitoring and evaluation (M&E) of financial education and other interventions. Chapter 2 presents the considerations, choices, selections, and implementation issues around the measurement of financial capability in LICs and MICs. This chapter attempts to make the thought and implementation process around the development of the financial capability survey instrument as transparent as possible to allow for replication, but it also provides some external critique and improvements to further progress in this critical area. While significant progress has been made with the application of the emerging capability concept and toolkit in LICs and MICs, further work needs to be done to measure capability in more countries and over more time periods. Chapter 3 presents the considerations, choices, selections, and implementation issues around the measurement of the effectiveness of financial education and other financial capability-enhancing interventions in LICs and MICs. Again, this chapter makes the thought and implementation process around the three key elements of the RTF work program transparent: the selection and financing of M&E pilots; the development and application of an M&E toolkit on financial education and other financial capability-enhancing interventions in some of the pilots; and the empirical results and policy conclusions with regard to the effectiveness of such interventions. Chapter 4 summarizes the key conclusions from the RTF work program, puts them into the context of international developments in this area, highlights issues, and offers suggestions for next steps for the international community and for countries undertaking or preparing national strategies.1 1  Six supplementary appendixes, available on the Trust Fund website (www.finlitedu.org), offer a brief overview of the RTF-related work program of the World Bank and the OECD (appendix A), brief summaries of the financial capability survey methodology and the individual results of the country pilots (appendix B), an international stocktaking of impact evaluations of financial capa- bility interventions (appendix C), a description of the U.K. Department for International Devel- opment’s Financial Education Fund program (appendix D), individual impact evaluation projects financed under the RTF (appendix E), and the chapters of the M&E toolkit (appendix F). 3 Financial capability in low- and middle-income countries: measurement and evaluation 1.2 Lessons and issues from high-income countries Over the last decade or so, major initiatives on FL&E have been undertaken by a number of HICs, and much progress has been made in sharing that experience under the leadership of the OECD.2 Agencies created in New Zealand (Retirement Commission 1995, renamed Commission for Financial Literacy and Retirement Income), the United Kingdom (Financial Service Authority 2000), Canada (Financial Consumer Agency of Canada 2001), the United States (Financial Literacy and Educa- tion Commission 2003), and Australia (Financial Literacy Foundation 2005, since transferred to the Australian Securities and Investments Commission) have taken the lead on financial literacy issues and their websites provide a wealth of national and international information, particularly on innovative studies and tools. The OECD’s 2003 Financial Education project started an international assessment of how much people know, and its 2005 study on “Improving Financial Literacy: Anal- ysis of Issues and Policies� (OECD 2005) was the first stocktaking conducted at the international level. In 2008, the OECD created the International Network on Financial Education (INFE), which serves as an international meeting and clearinghouse on financial education and national strategies.3 In parallel, the academic literature on FL&E has increased significantly (e.g., see the websites by OECD and World Bank referenced below), and major contributors include the three research centers spon- sored by the U.S. Financial Literacy and Education Commission: he University of Wisconsin, Boston College, and the RAND Corporation.4 A review of this policy and the academic discourse on FL&E for the purpose of appli- cation in LICs and MICs identified a number of lessons and issues, as follows. 1.2.1 Broadening the concept of financial literacy and measurement While financial literacy continues to be an issue of concern in many countries and many research studies, its original concept has expanded to include financial skills and competences, attitudes, and behavior. To emphasize this broader definition, the United Kingdom coined the term “financial capability,� which is increasingly being 2   The opening chapter draws on Holzmann (2010 and 2013). 3   See the OECD website for events and documents, www.financial-education.org. For an over- view of the current status of national strategies for financial education, see Grifoni and Messy (2012). For links to the three centers and their related research output, see www.socialsecurity.gov/ 4  retirementpolicy/retirement-security.html. 4 CHAPTER 1. Study motivation, approach, and structure used worldwide and is the concept used predominantly in this report.5 The term “financial literacy� is used only in the narrow knowledge sense in this report. This change in content and definition is important as it has a major bearing both on how the objectives of financial capability are defined and measured and on the choice of interventions to improve them. The consensus nowadays is that the broader concept is more relevant for financial outcomes, but how to best define, measure, and influ- ence financial capability are still topics of investigation. The concept of financial capability proposes that financially capable individuals should demonstrate financial behavior that is considered desirable, such as drawing up budgets, controlling daily expenses, and planning and saving for old age. Getting to that behavior is traditionally conceptualized as the move over time from knowl- edge to skills to attitudes to behavior (see sections 1.3 and 2.1 for a discussion). “Knowledge� includes understanding the purpose of saving and how to do it, “skills� the capacity to make a saving plan, “attitude� the willingness to save, and “behavior� the actual action of putting money into savings. In this conceptualization, information and cognitive understanding form the basis of the ultimate desired financial behavior. This information-based cognitive route is also the underlying concept of much of financial education. Most academic studies take course participation (as an input) to measure the impact on cognitive skills (as intermediate outputs) or on actual behavior (as an outcome). But this process may not be needed to achieve desired financial outcomes, nor may it work as conceptualized. In an alternative, outcome-oriented approach to operationalizing financial capability, the key concern is with behavioral outcomes and four content domains identified by the United Kingdom’s National Financial Capability Survey 2005, namely: managing money (keeping track, making ends meet), planning ahead, choosing prod- ucts, and staying informed (see FSA 2005 and 2006 a-c; and section 1.3). Developed bottom-up through focus groups and exploratory studies, this approach seems to have emerged as the consensus for measuring financial capability and identifying capability gaps and target groups. It has been applied with adjustments in Ireland (in 2008), and has been applied in or informed surveys undertaken in Australia, Austria, Canada, the Netherlands, and the United States. Other countries are thinking about its application. While these five domains seem to have universal appeal and application, the key challenge is how to translate them into questions that incorporate national and local circumstances, particularly the situations of people with low incomes in LICs and 5   Other terms such “financial competences,� “financial understanding,� “financial insights� (in Dutch), and “financial culture� (in Italian and Spanish) have been used, but “financial capabili- ties� has seen the most acceptance and use. 5 Financial capability in low- and middle-income countries: measurement and evaluation MICs. Ideally, survey questions should be universal, but country- and situation-spe- cific adjustments are likely to be required. To do so well and to achieve comparable results across countries requires substantial preparatory work and coordination. 1.2.2 Establishing an evaluation results framework and testing for financial education programs The country work on financial literacy strategies and the intensified exchange across countries have underscored the importance of a full results framework that investigates the key questions: why and what, for whom, and how?6 Ideally, such a framework would provide the detailed links between national as well as individual program objectives, inputs, delivery, outputs, outcomes, and impacts. Establishing such a framework requires: (1) a clear formulation of the objectives of a national strategy (NS) and its individual components; (2) a clear hypothesis of how proposed interventions (type and delivery) are conjectured to influence outputs and outcomes; (3) a clear understanding of the priorities for interventions, identified by surveys of countries’ financial capability; and (4) an approach developed ex ante for the qualita- tive and quantitative monitoring and (impact) evaluation. In practice, key elements of such a framework are still missing in many countries. The objectives for enhanced financial understanding are relatively well artic- ulated and range from the increased supply and complexity of financial market instruments to individuals’ needs to take better care of their own finances against the background of perceived low levels of financial literacy to the consequences for individuals and society (e.g., see Orton 2007; Deb and Kubzansky 2012). However, achieving these objectives requires a much more detailed breakdown and selection of appropriate indicators, for which there is currently no broad consensus. More specific objectives often reflect country-specific concerns that vary across settings and change over time. Last but not least, while specific objectives and indicators are more easily established for individual programs, bringing them together in a consis- tent NS is still more of an art than a science. The link between objectives and proposed interventions should be based on a structured model grounded in theory (economic, physiological, and/or other) and should reflect established hypotheses based on prior measurement of intermediate and final outputs and outcomes. Such a structured model of financial capability interventions is still missing, yet this combination of measurement and evaluation with theory would make much better use of empirical results, including those from randomized control (RCT) trials. A narrow focus on a few treatment parameters 6   For elements of such a framework, see Microfinance Opportunities (2006 and 2009), Kempson (2008), Kempson and Atkinson (2009), O’Connell (2009), Mundy (2009), MasterCard Foundation (2011), and Deb and Kubzansky (2012). 6 CHAPTER 1. Study motivation, approach, and structure using a social experiment approach not underpinned by theory, as proposed by the “randomistas,� will not suffice. Much has been said on the limitations of the latter.7 Two limitations are stressed here: as a hypothesis can never be verified, only falsified (Popper 1935), a theory-less measurement approach requires a very high (essential infinite) number of experiments to gain confidence in the results. This is both imprac- tical and wasteful. Second, focusing only on direct treatment effects provides little policy relevant information on policy design, reform, and scale-up unless supported by a broader qualitative assessment of the delivery mechanism and context. Simply put, researchers with a theory and some hypotheses beforehand regarding the chan- nels of impact will have a better chance of designing a financial intervention that will help clarify its causal impacts. Combined with mixed-methods evaluation, this will yield a much better understanding of how one might have gotten from point A to point B. The development of hypotheses and fully structured models requires high-quality quantitative and qualitative evaluations, but few have been done to date. While the number of interventions to improve financial literacy has increased dramatically (mostly financial education), rigorous M&E of such interventions is still the exception rather than the rule, particularly with regard to measuring impacts. Furthermore, the impact evaluations done have been predominantly ex post, not considered in the overall intervention design, which limits their quality and value. There seem to be many reasons for this, ranging from a lack of understanding of the importance of M&E for its own sake to program sponsors’ reluctance to provide substantial funds in addition to financing the intervention. As M&E results are a public good by nature, program sponsors have limited incentive to finance them. Finally, impact evaluations often focus more on quantitative impacts and less on the qualitative process. The latter is particularly important for establishing the most effective modes of delivery. 1.2.3 Financial education, behavioral finance, and alternatives to impact outcomes Many attempts to increase levels of financial capability have focused on diverse methods of financial education, albeit with limited empirical evidence that they are very effective. Results from the burgeoning literature on behavioral economics offer an explanation as to why education may have limited effects on improving financial capability, but also provide hints for new and innovative interventions on how to improve behavioral outcomes. If both types of such interventions were to largely For a discussion, see Ravallion (2008 and 2009), who tirelessly hammers these important 7  points inside and outside the World Bank. For more on integrating qualitative and quantitative approaches in program evaluations, see Rao and Woolkock (2003). 7 Financial capability in low- and middle-income countries: measurement and evaluation fail, at least for critical target populations, other and more direct measures to affect behavior—such as advocacy or innovative consumer protection—would need to be explored and tested. The limited empirical evidence does not lend strong support that financial education is effective; i.e., it has not documented consistent and sizable positive impacts on financial knowledge and/or behavior. Most international reviews of the sparse evidence come to a conclusion similar to that of Atkinson (2008, p. 5), who stated, “…there is little in the way of robust evidence to show the overall effect of financial training.�8This conclusion is valid across different types of interventions, from more formal academic training in schools to more ad hoc training at the work- place. It is also broadly echoed by the few evaluations in poorer countries, mostly undertaken around microfinance projects. However, this lack of evidence is not sufficient proof that financial education has not worked and never will. It is more likely the case that there have been: limited studies available; inappropriately chosen results indicators; data and estimation issues; little attention to the type and quality of the delivery mechanism; lack of a control group; and a predominance of ex post evaluations. This situation emphasizes the need for more rigorous impact evaluations as part of overall program design rather than dismissal of the notion of financial education altogether. Some empirical results, mostly from academic studies of retirement saving in the United States, directly link measured levels of financial literacy with financial outcomes and claim that this link is statistically significant and causal.9 The literacy is measured around the understanding of interest rates and the effects of inflation along with more nuanced concepts of risk diversification. On these topics, a knowl- edge shortfall is found to be particularly concentrated among women, minorities, and the least educated, as is the low level of financial planning and saving. This contrasts with the knowledge of the financially savvy, who are more likely to plan and succeed in their planning, and who are seemingly less subject to choice avoidance and framing effects (Lara-Ibarra 2012). Some econometric evidence on the link between financial knowledge and decisions has been given causal interpretation, claiming that “targeted financial education efforts are likely to filling these knowledge gaps� (Lusardi and Mitchell 2011b, p. 3). Whether this optimistic interpretation of statistical 8  For other recent reviews on the effectiveness of financial education see Orton (2007), Atkinson (2008), Cole and Fernando (2008), Mundy (2009), O’Connell (2009), and Agarwal et al. (2011). 9  See Lusardi and Mitchell (2011a) for an overview article that presents results for the United States and seven further studies in HICs across the world, and Lusardi and Mitchell (2011b) for further references, including on the effectiveness of targeted retirement education interven- tions. 8 CHAPTER 1. Study motivation, approach, and structure significance, causality, and targeted education effects holds is still under discussion, but even if it does, the results may not easily translate into a LIC environment. Of course, there is also the possibility that financial education may actually do very little for financial capability, at least for activities such as planning ahead. While good classroom-based financial education may increase financial knowledge, and linked with hands-on training may actually improve financial skills, there may be little impact on attitude, and even less on behavior. Attitude issues may be linked to a lack of trust in financial institutions (in some countries) or to cultural norms that may need interventions outside of financial education. Even if attitude issues can be overcome by financial education, and a desire instilled in consumers to, say, plan their finances, there may be other powerful impediments to changing their behavior to actually follow through. Behavioral finance literature provides many examples of cognitive biases with regard to attitude as well as behavior, such as procrastination, regret and loss aversion, mental accounting, status quo, and informal overload.10 This has led some authors to question the role of financial education in enhancing finan- cial capabilities and to claim that psychology, not knowledge, is the main driver of people’s actual behavior (e.g., Willis 2008 and de Meza, Irlenbush and Reyniers 2008). If this were broadly true, then the types of interventions to improve behavioral outcomes would need to be revised and could be guided by the results of behav- ioral finance and the broader field of psychology.11 This is already happening to some extent. For example, the lack of planning and following through for retirement saving has led to changing the default option in private pension plans (such as in New Zealand, the United States, and very recently in the United Kingdom) using inertia and the status quo bias to overcome behavioral shortcomings. Information over- load, which creates indecision, can be addressed by reducing the number of options available to consumers, such as the number of pension funds from which to choose. More broadly, the design of the (financial) choice environment can be adjusted to “nudge� individuals toward desired behaviors (Thaler and Sunstein 2008). But more direct approaches to changing behavior are also gaining importance. Social marketing for health issues (HIV/AIDS) and road safety (seat belts) has been successful at changing behavior without going through the knowledge-based cogni- tive route. There are older examples from some Central European countries that find that saving behavior is positively influenced by the annual “World Saving Day� on October 31; its related information campaign in schools exerts peer pressure on children to have their piggy bank full when they bring it to school on this day. More 10  For a recent excellent survey on psychology and economics, see DellaVigna (2009).   For reviews and suggestions of how to use behavioral economics/economic psychology to 11 make financial education more effective, see De Mello Ferreira (2010) and Yoong (2010). 9 Financial capability in low- and middle-income countries: measurement and evaluation recent attempts to directly influence financial attitude and behavior include “edutain- ment� interventions, with key messages on behavior delivered in TV soap operas, TV clips, or street theater.12 To summarize, the recent but short experience in HICs provides rich, albeit incom- plete, material to guide the introduction of financial literacy policies and programs in LICs and MICs. It provides critical lessons on how to proceed, including the need for clear objectives, a focus on measurement, and experimentation in interventions. But the practice needs to be adapted for low-income communities in LICs and MICs. 1.3 The background for financial literacy in low- and middle-income countries The last few years have seen a rising interest in financial literacy issues in both LICs and MICs. This can be documented by the number of countries with financial literacy initiatives (see OECD’s Financial Gateway website), by the number of conferences and workshops held on the topics and the rising number of countries participating, and by the specific initiatives of BRIC countries as well as regional initiatives for LICs. The latter include the “Partnership on Making Finance Work for Africa,� established in 2008, which includes a focus on financial access and capability, and the related September 2009 Accra and September 2011 Addis Ababa conferences that brought together over 200 and 300 participants, respectively, from most African countries on the issue of financial capability and consumer protection.13 This rising interest in financial literacy in the less affluent part of the world has many different motivations. Three notable ones are: (1) concerns about the perceived low level of financial capa- bility; (2) concerns about the low level of financial access or use; and (3) the recogni- tion that finance is a critical element for innovation and growth. This section outlines both common and idiosyncratic characteristics of LICs and MICs conjectured to be important to consider when measuring the level of financial literacy and designing interventions to raise it. It starts with a definition of those countries and their relevant characteristics in common with HICs. There are a few empirical studies that show an impact of TV soaps on fertility and divorce 12  rates (e.g., see Chong, Duryea, and La Ferrara 2008; and Chong and La Ferrara 2008) but they do not qualify as measuring the impact of a purposeful message on behavioral outcomes. Such behavioral effects have been measured for health and other messages included in radio and TV programs with reported positive effects but such quantifications have been limited so far (e.g., Vaughan et al. 2000; and Paluck and Green 2009). For a review of experiences and measure- ment issues with mass media and social marketing, see Mulaj and Jack (2012). 13  For further information, see www.mfw4a.org and www.facebook.com/MFW4A/info. 10 CHAPTER 1. Study motivation, approach, and structure 1.3.1 Definition of LICs and MICs and common relevant characteristics The World Bank’s definition of LICs and MICs is related to countries’ access to financial services of the World Bank Group, which is linked to income thresholds measured in gross national income (GNI) per capita. LICs, with GNI per capita in 2011 below $1,026, are eligible for grants and subsidized loans from International Develop- ment Agency (IDA is the soft lending arm of the World Bank Group). MICs, with a GNI per capita in 2011 between $1,026 and $12,746, can access loans under the terms of the International Bank for Reconstruction and Development (IBRD is the World Bank Group’s market-based lending arm). While these limits are admittedly somewhat arbitrary, they broadly reflect countries’ financial needs and opportunities, such as their access (or lack thereof) to the international capital market. LICs share a number of common characteristics relevant for the purpose of measuring financial capability and designing financial capability policies and inter- ventions. Six interrelated characteristics are of particular relevance: access, poverty, location, informality, education, and risks. „„ Access. In LICs, access to financial services is very limited for a very large share of the population. Measured as the percentage of households with an account in a formal financial institution, the distribution of account use in figure 1.1 reflects the distribution of LICs, MICs, and HICs. In LICs, account use is 24 percent on average, and 89 percent for HICs; MICs fall somewhere in between (see table 1.1 for details). From a measurement point of view, it is not easy to differentiate financial capability from financial access or use, as these are interrelated but not the same. In a HIC, there is rather less overlap between the people who are financially excluded (who tend to be small in number and concentrated among those with low income) and those with low levels of financial capability (who are more numerous and span all income levels). In a LIC, where the majority of the population makes no use of formal financial services, the overlap is far greater and low financial capability (in terms of awareness of financial services provision) may often constrain demand. „„ Poverty. LICs not only have a lower income per capita but typically also a much larger share of poor in the population, whether measured as absolute poverty (e.g., those living on $2 or below a day) or as relative poverty (e.g., the share of individuals having, say, less than 60 percent of the median income), as income inequality is typically also higher. Absolute poverty induces special behavior as physical survival takes priority. This behavior is often akin to that seen in those lacking financial capability albeit individuals may behave differ- ently if not poor (see Bertrand, Mullainathan and Shafir 2004 and Mullainathan 2011). 11 Financial capability in low- and middle-income countries: measurement and evaluation Figure 1.1 Account penetration around the world Source: Demirguc-Kunt and Klapper 2012. „„ Location. Most of the population in LICs: live in rural and often sparely popu- lated areas with limited exposure to financial institutions and products; have more limited cash needs; hold assets predominantly in land, cattle, seeds, or gold; and live in large families and tightknit communities. This creates special behavior for planning and saving that may be insufficiently appreciated if behavioral outcomes are assessed only in monetary terms. „„ Informality. This is the vastly dominant form of “employment� in LICs, where formal employment (i.e., having a labor contract or license, and paying social security contributions and income taxes) is restricted to very small share of the population (often 10 percent or less); the large majority work on their own account. As a result, managing money and other resources for a large part of the population means jointly managing one’s personal accounts and that of one’s microbusiness. „„ Education. The populations living in developing countries typically have lower levels of literacy and educational achievements compared to those in HICs. Thus a crucial challenge for financial capability measurement in these countries is to design questions that can be easily understood by everyone, including people who cannot read or write, but that can still apply to and be relevant for higher-educated respondents. „„ Risks. LICs are characterized by high levels of natural, security, economic, and other risks given their limited access to formal (public and private) risk management instruments, incomplete financial markets, and limited social 12 CHAPTER 1. Study motivation, approach, and structure transfer programs. Key risks are those related to agricultural production and health, managed by informal risk management arrangements (family, commu- nity). Hence, short-term weather and other insurances, where they exist, are of primary interest. As short-term risks dominate, long-term planning and saving (e.g., for retirement) are not priorities for most individuals. As such, this behavior does not necessarily signal a lack of financial capability or even myopia; it can be perfectly rational (see Holzmann and Jørgensen 2001). These characteristics are bound to influence: the priorities of policy makers with regard to the objectives of financial literacy programs and their target groups; the way individuals behave “financially� and how they react to interventions to change their behavior; and how financial capability can be measured. In addition, LICs exhibit other and more idiosyncratic characteristics that may differ in importance across countries and regions but that are likely to influence the measurement of financial capability and the success of capability improvement interventions (discussed below). MICs can be described as countries that share characteristics of both LICS and HICs, but exhibit higher heterogeneity. Hence part of the population in a MIC will exhibit characteristics and financial behavior similar to those found in a LIC, while another part will be much closer to a HIC. Table 1.1 presents the variation of account-holding across four income groups. The richest quintile of the upper-middle-income group has an access rate of 76 percent, while the lowest has a rate of 36 percent. This type of situation needs to be taken into account when measuring financial capability and designing interventions. Such issues also exist to a more limited extent in HICs that have a strong presence of ethnic minorities or other subgroups. 1.3.2 Idiosyncratic characteristics that may matter Besides their common characteristics, a number of more idiosyncratic characteristics are likely to influence the why and what, for whom, and how of financial capability programs, particularly in LICs, such as the following. Why and what While policy statements on the objectives of financial capability interventions in LICs often mirror those of MICs and HICs (and may have been copied from them), a common objective is to facilitate and increase the use of financial services, if only the most basic one (i.e., a bank account). There are three other common objec- tives, albeit with different emphasis: providing basic business education; preventing over-indebtedness; and avoiding scams and unscrupulous providers. „„ Basic business education. As most individuals in LICs are informal workers, as mentioned above, they typically do not separate personal from business finances, with sometimes detrimental effects on both. Hence financial capa- 13 Financial capability in low- and middle-income countries: measurement and evaluation Table 1.1 Account-holding around the world by income group and region Income Group Region World Dev. LIC LMIC UMIC HIC EAP ECA LAC MENA SAR SSA All 50 41 24 28 57 89 55 45 39 18 33 24 Gender Male 55 46 27 34 62 92 58 50 44 23 41 27 Female 47 37 20 23 53 87 52 40 35 13 25 22 Age group 15–24 37 31 16 21 49 76 50 32 26 13 25 17 25–64 55 46 29 31 61 93 58 51 44 20 36 29 65+ 54 35 18 26 43 89 38 35 43 20 32 19 Within-economy income quintile Poorest 38 25 16 16 36 85 33 32 21 7 21 12 Q2 45 35 17 25 49 90 46 41 30 10 31 16 Q3 52 42 21 28 58 92 54 44 42 14 35 22 Q4 57 50 29 32 69 93 70 52 47 15 36 31 Richest 67 62 39 47 76 91 76 58 61 25 51 45 Education level Primary or more 37 35 15 23 52 74 50 30 30 14 28 12 Secondary 62 49 35 33 62 91 62 46 42 19 45 38 Tertiary or more 83 72 54 63 82 97 84 71 69 43 70 56 Residence Rural 44 38 22 26 54 88 50 39 35 9 31 21 Urban 60 50 35 34 63 89 69 53 43 19 37 38 Source: Demirguc-Kunt and Klapper 2012. Note: Income groups: Dev. = developing economies; LIC = low income; LMIC = lower middle income; UMIC = upper middle income; HIC = high income. Regions: EAP = East Asia and the Pacific; ECA = Europe and Central Asia; MENA = Middle East and North Africa; SAR = South Asia; SSA = Sub-Saharan Africa. Regions exclude high-income economies. See the methodology section for regional and income group classifications. Data by education level exclude Zimbabwe; data by income quintile exclude Morocco: and data by residence exclude Germany, Guatemala Morocco, and the United Kingdom. bility programs frequently seek to strengthen basic business education. However, the issue is how to define, measure, and separate a consumer’s business capability from his financial capability. „„ Over-indebtedness. A specific overriding concern in many but not all LICs is the level of debt of major subgroups of the population to formal and informal lenders. As in HICs, this indebtedness is linked to individuals’ low level of financial capability and high poverty, but may also reflect cultural issues; these have been little explored. Some claim that it may be the outcome of a finan- 14 CHAPTER 1. Study motivation, approach, and structure cial capability gap, reflecting rising access to financial services for the poor in poor countries in recent years while individuals’ financial capabilities have not been lifted concurrently (Deb and Kubzansky 2012). This may also hold for some groups in HICs. „„ Avoiding scams and unscrupulous providers. In some countries, the regula- tion of financial services and consumer protection legislation are rudimentary and can put consumers at risk of scams and unscrupulous (if not criminal) providers of financial services. A poorly regulated formal financial services sector can also undermine consumers’ confidence in and use of its services. For whom As few national financial literacy/capability surveys have been undertaken in LICs (and MICs), the key target groups are not yet appropriately identified. However, other surveys and studies have found that gender may require a special focus and treat- ment in capability surveys as well as in education and other intervention programs. Gender-specific differentiation with regard to financial decision making ranges from an almost complete exclusion of women from key budgeting and planning decisions to having a majority of women running businesses and managing the day-to-day budget as well as the precautionary saving budget. How to do it There are various indications that financial capability surveys and interventions in LICs will have to take account of country specificities with regard to both the content and the delivery mechanism; this is already happening in part (e.g., see Deb and Kubzansky 2012; MasterCard Foundation 2011). Some examples include the following. „„ Event-specific interventions. In many countries, remittances play a major role in household resources; this may affect the measurement of financial capability, as those receiving remittances will have more exposure to finan- cial instruments but may not necessarily exhibit improved financial behavior. However, the receipt of remittances may create a “teachable moment� for budgeting and planning. Conditional cash transfers (CCTs) are also gaining importance in both LICs and MICs, with similar challenges and opportunities. „„ Financial access. Access to microfinance in many LICs and MICs may influ- ence financial behavior and its availability has already been instrumental in financial education. Similarly, mobile phones are becoming a popular tool for providing financial access, particularly to the very poor, and may create oppor- tunities for financial education. „„ Providers of financial education. Life in most LICs is still structured around a tight-knit community and trusted persons. This is likely to influence people’s 15 Financial capability in low- and middle-income countries: measurement and evaluation financial decisions but can also be an opportunity to provide information as well as to help change financial behavior. „„ Cultural differences. Some striking cultural norms may explain some of the differences in saving behavior (and possibly measured differences in finan- cial capabilities). For example, in many African societies, it is difficult to keep liquid resources away from the demands of the extended family. There are good (historic) economic and anthropological explanations for such requests to share available resources (Platteau 1996), which may have a continued impact on savings-type decisions and returns (Jakiela and Ozier 2012). But this may also lead to low holdings of liquid assets in cash or accounts and prefer- ences for illiquid assets for medium- and longer-term needs. This will impact measured financial capability, but financial education may be able to do little to change this behavior. In summary, LICs and MICs exhibit numerous common and idiosyncratic character- istics that are likely to influence financial behavior differently than in HICs. This has a bearing on how to measure financial capability and on the effectiveness of financial education and alternative interventions. 1.4 Addressing knowledge gaps: selection, conceptualization, and implementation The prior two sections explain the motivation behind the thinking regarding the establishment of the Trust Fund. The overall program and financing was divided into two separate work programs, one that was managed by the World Bank and a complementary program managed by the OECD. This section briefly outlines the considerations behind the World Bank’s work program topics, the concepts underlying the approaches, and their implementation. The refinements, details and most importantly, the results are provided in chapters 2 and 3. The OECD’s work program is addressed separately in a variety of reports and documents (see OECD publications in the references at the end of this chapter). 1.4.1 Selection of the World Bank’s RTF work program: measuring financial literacy and measuring the effectiveness of financial education “What you cannot measure you cannot manage.� This well-known and universally valid statement applies to all policy areas and was the basis of the World Bank-led work program on FL&E. It is closely linked to the questions: (1) What are the objectives of “financial literacy� or “financial capability�?; and (2) Which interventions, such as 16 CHAPTER 1. Study motivation, approach, and structure financial education, would most effectively help improve the situation? The questions are closely linked and hence should ideally be dealt with in an integrated approach. At the time of the establishment of the RTF in 2008, only a few surveys (mostly finan- cial literacy-oriented) had attempted to measure the effects of a diverse set of ad hoc variables on concepts, products, and processes critical for good financial decisions and outcomes. These knowledge-focused surveys generally recorded a low level of financial literacy (see OECD 2005 and Kempson and Atkinson 2009). But the relevance of financial knowledge for good financial decisions and outcomes was not clear. As mentioned above, a behavior-oriented type of survey measuring “financial capabili- ties� was pioneered by the United Kingdom and replicated in a few HICs, but it was not yet apparent how relevant and replicable the approach would be for LICs and MICs, or whether it would produce comparable results across countries. However, it was clear that a rigorously developed and tested survey instrument that offered comparable results, ideally over time and across countries, was required; i.e., a finan- cial capability measurement survey toolkit should be developed as a public good. Also at the time of the establishment of the RTF, the international knowledge about effective interventions to increase financial knowledge or to improve savings outcomes was extremely limited. The lack of knowledge covered essentially all dimensions of financial education, including the relevant objectives, target audience, content, and delivery channels. This lack of knowledge was not surprising, as very few education interventions across the world had ever been subject to rigorous monitoring and impact evaluation (quantitative, qualitative, or, best, both). Thus the choice of the second World Bank work program element was easy—promotion of M&E for financial education and other promising financial capability-enhancing interventions in LICs and MICs; development of an M&E toolkit to facilitate the appli- cation; and financial encouragement of the evaluation of specific interventions for knowledge management. Determining the most cost-effective trust-funded interven- tions given the large existing void required a bit more reflection, as discussed below. 1.4.2 Conceptual alternatives to measuring financial literacy/ capability Two polar views can be used to conceptualize financial capabilities: the normative/ cognitive approach and the positive/agnostic approach. This polar selection is actu- ally a short-cut for the choice between two dimensions (normative versus positive and cognitive versus agnostic), and can aid the selection of the most useful combina- tion of these. The normative approach uses as a benchmark the optimal financial behavior of indi- viduals typically derived by assuming intertemporal optimization across lifecycles; the positive approach takes as a benchmark the outcome of financial decisions by 17 Financial capability in low- and middle-income countries: measurement and evaluation individuals as judged by other individuals and their assessment of good or bad deci- sions. In its strictest form, the cognitive approach assumes that any deviations from optimal behavior are due to a lack of information and/or training, making financial education the preferred intervention. The agnostic approach, in its strictest form, has no expectations regarding what prevents individuals from behaving best (as judged by others); as a result, it is open to any kind of intervention for improvement. The normative/cognitive approach builds on sequential information and learning constraints but may not work; the positive/agnostic approach is focused on what empirically matters but makes no prior assumptions about the why and how (and may not be able to do so). Both approaches focus on the broader concept of financial capability for good financial behavior but with different emphasis, as discussed next. The normative/cognitive approach This approach implicitly takes as the normative benchmark the behavior of a fully rational and intertemporal utility maximizing individual who operates under complete financial markets; it explicitly assumes that a lack of information and learning is the (only) binding constraint for good behavior (and outcomes) and that this constraint needs to and can be eliminated in a sequential and cognitive manner. It builds on knowledge and skills as the key determinants of traditional financial literacy compo- nents, but over time attitude and capability were also added (see box 1.1). Actually, financial capability (at times called financial competence) is a summary descrip- tion, as it refers to the ability to apply knowledge, skills, and attitude in a sustained manner to achieve the relevant financial behavior (and outcome). This approach takes financial knowledge (i.e., an analytical understanding of financial concepts and products14) as the basic requirement that needs to be augmented by skills, such as the ability to apply knowledge in life to achieve financial literacy. This was the original concept, thought to be sufficient for good behavior and outcomes. However, it has become clear that knowledge and skills are not sufficient to, say, save for rainy days—this also requires the attitude to do so. Even attitude is not suffi- cient, as the behavior to follow through is also needed. But in this conceptualization, all of these steps are open to learning; one only needs to expose individuals to these concepts through financial education. With this conceptualization, an important move from financial literacy (with its focus on knowledge and skills) to capability (with its focus on behavior and results) took 14  See, for example, the Wikipedia definition of financial literacy: “Financial literacy is the ability to understand finance. More specifically, it refers to the set of skills and knowledge that allows an individual to make informed and effective decisions through their understanding of finances.� 18 CHAPTER 1. Study motivation, approach, and structure place, and both notions are now often used interchangeably.15 While the extension can be considered important progress, the focus on information and learning gaps ignores at least two other main constraints to achieving good behavior; i.e., those due to: „„ The enabling environment (e.g., restricted access to finance, lack of trust in financial institutions, etc.); and „„ The behavioral limitations of individuals that are not amenable to change through learning but require different interventions. These and other constraints will impact individual behavior compared to the coun- terfactual of their absence. Financial capability without these constraints thus becomes the idealistic/normative benchmark under perfect (financial) markets and fully rational and utility-maximizing individuals. The distance between knowl- edge-constrained and -unconstrained results can then be measured, normalized, and analyzed. However, this approach works only if good financial behavior is driven by knowledge, skills, and attitudes that can be influenced by cognitive action, and in particular, by financial education. If behavior is largely driven by psychology, peer effects, or other noncognitive drivers, the approach fails. The positive/agnostic approach This approach identifies financial capability via financial manifestations considered to matter for good outcomes. The assessment is based on the judgment of one’s peers (vox populi) and what they consider good outcomes, and on the features that lead to those outcomes, such as good behavior, but also good attitude, skills, and knowl- edge. However, there is no logical chain of assumptions on how to progress, and no prior assumptions about what intervention to use to positively impact the financial features needed for good outcomes. This approach implicitly takes account of effective constraints (e.g., limited access to financial products), as they are part of the individual’s judgment of what constitutes good behavior. The approach is a priori agnostic about why and how good outcomes are achieved. This approach to conceptualizing financial capability was initiated by the United Kingdom’s Financial Services Authority (FSA), and first started around the notion For example, as of 2010, the OECD/INFE defines financial literacy “a combination of financial 15  awareness, knowledge, skills, attitude and behaviours necessary to make sound financial deci- sions and ultimately achieve financial wellbeing� and uses this definition also as they base for their financial literacy measurement pilots (Atkinson and Messy 2012). This definition is also echoed and embraced by the European Insurance and Occupational Pension Authority (see EICOPA 2011). 19 Financial capability in low- and middle-income countries: measurement and evaluation Box 1.1 The normative/cognitive approach Knowledge  Skills  Attitude  Behavior ƒƒ Knowledge: cognitive understanding ƒƒ Skills: ability to apply knowledge ƒƒ Attitude: willingness to apply knowledge and skills ƒƒ Capability: ability to apply knowledge, skills, and attitude in a sustained manner to achieve the relevant financial behavior (as output) and policy objectives (as outcome) Source: Adapted from Kempson, Collard, and Moore 2005. of knowledge, skills, and attitudes (see Kempson, Collard and Moore 2005). Focus groups, however, made it clear that everyday people had a different notion of capa- bility that they perceived in behavioral terms. Specifically, the focus groups identified a range of attributes that, combined, denote financial (in-)capability; these were grouped into four domains: „„ Managing money (i.e., living within one’s means and tracking one’s expendi- tures); „„ Planning ahead (i.e., coping with unexpected events and making provisions for the long term, including education, health, and old age); „„ Making choices (i.e., being aware of the options available and being able to choose the one most appropriate for one’s circumstances); and „„ Getting help (i.e., becoming self-reliant through information gathering and knowing where and when to turn for advice and help from a third party, including for protection from market abuse and redress when abuse occurs). When probed, the focus groups in HICs agreed that skills, attitude, and person- ality were relevant for each of these domains, but they did not have a clear view of the role and causality of these factors. Surprisingly, knowledge was hardly ever mentioned; the focus groups specifically said that knowledge did not necessarily ensure capable behaviors—with participants using real life experiences to illustrate this. Selecting between the two approaches The two conceptualizations outlined above reflect the basis on which surveys can be undertaken to establish a baseline of financial capability in a country and to compare it with that of other countries and over time. The comparison over time may serve as 20 CHAPTER 1. Study motivation, approach, and structure a broad measure of the effectiveness of national financial literacy/capability strate- gies, including financial education and other promising interventions. For conceptual, empirical, and policy reasons, the World Bank project team selected the outcome-driven, positive/agnostic approach of capability, and tested the applica- bility of the United Kingdom’s FSA approach in LICs and MICs and the comparability of the results with those already undertaken in HICs. Specifically, the team followed the FSA’s empirical approach to determine the parameters of financial capability and to develop methods for reporting the results, rather than simply adapting the content and copying the scoring methods. 1.4.3 Implementation of the measurement and evaluation toolkit approach The implementation of both work program components—development of a financial capability measurement tool based on the outcome-driven approach and the devel- opment of an M&E toolkit for financial education and all other promising interven- tions—required decisions on how to best proceed. In view of the objectives for LICs and MICs, it was evident that a country-based approach was required and feasible given the operational experience and support of the World Bank. To allow for learning across different countries and regions, the use of voluntary pilots across a set of countries and approaches was a natural deci- sion. Furthermore, capacity building in the countries and with the teams was a key aspect in design and implementation. All of this was based on a shared and joint concept and definition of financial capability and the measurement of the effectiveness of financial education and other financial capability-enhancing interventions. The following was used as a starting point: „„ “Financial capability� constitutes the financial behavior demonstrated by an individual that is considered by his peers to be desirable as it leads to good financial outcomes. „„ The domains considered relevant to benchmark such behavior may differ across countries and income levels and may reflect differences in enabling environments, but are conjectured to comprise, in descending order of impor- tance: managing money, planning ahead, making choices, and getting help. „„ Financial education and any other financial capability-enhancing interventions are considered effective if they are able to affect components in at least one of these domains. 21 Financial capability in low- and middle-income countries: measurement and evaluation References Agarwal, Sumit, Gene Amromin, Itzhak Ben-David, Souphala Chomsisengphet, and Douglas D. Evanoff. 2011. “Financial Counseling, Financial Literacy and Household Decision-Making.� In O. Mitchell and A. Lusardi, eds., Financial Literacy: Implications for Retirement Security and the Financial Marketplace, chapter 10. New York: Oxford University Press. Antolín, P., and D. Harrison. 2012. “Annual DC Pension Statements and the Communications Challenge.� OECD Working Papers on Finance, Insurance and Private Pensions, No. 19, OECD Publishing. doi: 10.1787/5k97gkd06kth-en. Atkinson, Adele. 2008. “Evidence of Impact: An Overview of Financial Education Evaluations.� Consumer Research No. 68. FSA: London. Atkinson, A., et al. 2012. “Lessons from National Pensions Communication Campaigns.�OECD Working Papers on Finance, Insurance and Private Pensions, No. 18, OECD Publishing. doi: 10.1787/5k98xwz5z09v-en. Atkinson, A., and F-A. Messy. 2012. “Measuring Financial Literacy: Results of the OECD/ International Network on Financial Education (INFE) Pilot Study.� OECD Working Papers on Finance, Insurance and Private Pensions, No. 15, OECD Publishing. Atkinson, A., et al.  (2012), “Lessons from National Pensions Communication Campaigns�, OECD Working Papers on Finance, Insurance and Private Pensions, No. 18, OECD Publishing. doi: 10.1787/5k98xwz5z09v-en. Bertrand, M., S. Mullainathan, and E. Shafir. 2004. “A Behavioral-Economics View of Poverty.� American Economic Review, Papers and Proceedings 2004 (May): 491–523. Chong, Alberto, Suzanne Duryea, and Eliana La Ferrara. 2008. “Soap Operas and Fertility: Evidence from Brazil.� CEPR Discussion Papers 6785. London: Centre for Economic Policy Research. Chong, Alberto, and Eliana La Ferrara. 2009. “Television and Divorce: Evidence from Brazilian Novelas.� Journal of the European Economic Association 7(2–3): 458–68. Cole, S., and N. Fernando. 2008. “Assessing the Importance of Financial Education.� Manila: ADB.Deb, A., and M. Kubzansky. 2012. “Bridging the Gap: The Business Case for Financial Capability.� A report commissioned and funded by the Citi Foundation. Cambridge, Mass: Monitor, March. DellaVigna, Stefano. 2009. “Psychology and Economics: Evidence from the Field.� Journal of Economic Literature 47 (2): 315–72. De Mello Ferreira, Vera Rita. 2010. “Financial Education—Can Economic Psychology and Behavioral Economics Help Improve it?� Paper presented at the OECD-Bank of Italy Symposium on Financial Literacy: Improving Financial Education Efficiency, 9. June. Rome. de Meza, David, Bernd Irlenbush, and Diane Reyniers. 2008. “Financial Capability: A Behavioural Economics Perspective.� Consumer Research No. 69. FSA: London. Demirguc-Kunt, A., and L. Klapper. 2012. “Measuring Financial Inclusion: The Global Findex Database.� Policy Research Working Paper 6025, World Bank, Washington, DC. European Insurance and Occupational Pension Authority (EICOPA). 2011. “Report on Financial Literacy and Education Initiatives by Competent Authorities.� Frankfurt/Main: EICOPA. https://eiopa.europa.eu/fileadmin/tx_dam/files/publications/reports/Report_on_Financial_ Literacy_and_Education__EIOPA-CCPFI-11-018_.pdf. 22 CHAPTER 1. Study motivation, approach, and structure Grifoni, A., and F. Messy. 2012. “Current Status of National Strategies for Financial Education: A Comparative Analysis and Relevant Practices.� OECD Working Papers on Finance, Insurance and Private Pensions, No. 16. Paris: OECD Publishing. doi: 10.1787/5k9bcwct7xmn-en. Financial Service Authority (FSA). 2005. “Measuring Financial Capability: An Exploratory Study.� Consumer Research Study No. 37. London: FSA. —. 2006a. “Financial Capability in the U.K.: Establishing a Baseline.� Consumer Research Study No. 47. London: FSA. —. 2006b. “Financial Capability Baseline Survey: Methodological Report.� Consumer Research Study No. 47a. London: FSA. —. 2006c. “Financial Capability Baseline Survey: Questionnaire.� Consumer Research Study No. 47b. FSA: London: FSA. Gale, William, Benjamin Harris, and Ruth Levine. 2012. “Raising Household Saving: Does Financial Education Work?� Social Security Bulletin 72 (2). www.ssa.gov/policy/docs/ssb/v72n2/ v72n2p39.html. Holzmann, Robert. 2010. “Bringing Financial Literacy and Education to Low and Middle Income Countries: The Need to Review, Adjust, and Extend Current Wisdom.� IZA Discussion Paper No. 5114, http://ftp.iza.org/dp5114.pdf. —. 2013. “Participation in Mandated and Voluntary Social Risk Management Arrangements: The Role and Limits of Financial Education and Other Interventions. In Social Insurance and Labor Markets: How to Protect Workers While Creating Good Jobs, ed. M. Froelich, D. Kaplan, C. Pages, J. Rigolini and D. Robalino. Washington, DC: Inter-American Development Bank, Institute for the Study of Labor, and World Bank, in print. Holzmann, Robert, and Steen Jørgensen. 2001. “Social Risk Management: A new conceptual framework for social protection, and beyond.� International Tax and Public Finance 8 (4): 529–56. Hung, A., J. Yoong, and E. Brown. 2012. “Empowering Women Through Financial Awareness and Education.� OECD Working Papers on Finance, Insurance and Private Pensions, No. 14, OECD Publishing. doi: 10.1787/5k9d5v6kh56g-en. Jakiela, Pamela, and Owen Ozier. 2012. “Does Africa Need a Rotten Kin Theorem? Experimental Evidence from Village Economies.� Policy Research Working Paper WPS 6085. Washington, DC: World Bank. Kempson, Elaine. 2008. “Financial Education Fund: Monitoring and Evaluation Policy and Procedures.� Financial Education Inception Report. London: DFID. Mimeo, November. Kempson, Elaine, and Adele Atkinson. 2009. “Measuring Levels of Financial Literacy at an International Level.� OECD: Paris. Mimeo. Kempson, Elaine, Sharon Collard, and Nicholas Moore. 2005. “Measuring Financial Capability: An Exploratory Study.� Consumer Research Report 37. London: Financial Services Authority. Lara-Ibarra, Gabriel. 2012. “Information Framing and Retirement Management Decisions: Evidence from a Field Study in Mexico.� Paper presented at the LACEA conference in Lima, Nov 1–3. Lewis, S., and F. Messy. 2012. “Financial Education, Savings and Investments: An Overview,� OECD Working Papers on Finance, Insurance and Private Pensions, No. 22, OECD Publishing. doi: 10.1787/5k94gxrw760v-en. 23 Financial capability in low- and middle-income countries: measurement and evaluation Lusardi, Anna-Maria, and Olivia Mitchell. 2011a. “Financial Literacy around the World: An Overview.� Journal of Pension Economics and Finance 10: 497–508. —. 2011b. “The Outlook on Financial Literacy.� In O. Mitchell and A. Lusardi, eds., Financial Literacy: Implications for Retirement Security and the Financial Marketplace, chapter 1. New York: Oxford University Press. MasterCard Foundation. 2011. “Taking Stock: Financial Education Initiatives for the poor. A report.� Toronto: MasterCard Foundation. Martin, Matthew. 2007. “A Literature Review on the Effectiveness of Financial Education.� Working Paper Series WP07-03. The Federal Reserve Bank of Richmond. Richmond: FRB. Messy, F., and C. Monticone. 2012. “The Status of Financial Education in Africa,� OECD Working Papers on Finance, Insurance and Private Pensions, No. 25, OECD Publishing. doi: 10.1787/5k94cqqx90wl-en Microfinance Opportunities. 2006. “Assessing the Outcomes of Financial Education.� Working Paper #3. Available at: www.microfinanceopportunities.org. —. 2009. “Can Financial Education Change Behavior? Lessons from Bolivia and Sri Lanka.� Working Paper #4. December. Available at: www.microfinanceopportunities.org. Mitchell, O., and A. Lusardi. 2011. Financial Literacy: Implications for Retirement Security and the Financial Marketplace. New York: Oxford University Press. Mulaj, Florentina, and William Jack. 2012. “Evaluating the Efficacy of Mass Media and Social Marketing Campaigns in Changing Consumer Financial Behavior.� Social Protection Discussion Papers No. 2012. Washington, DC: World Bank. Mullainathan, S. 2011. LACEA’s Albert Hirscheman Lecture. http://www.youtube.com. Mundy, Shaun. 2009. “Financial Education Programmes in Schools: Analysis of Selected Current Programmes and Literature—Draft recommendations for best practices.� OECD. Mimeo. O’Connell, Alison. 2009. “Evaluating the Effectiveness of Financial Education Programmes.� OECD: Paris. Mimeo. OECD (Organisation for Economic Co-operation and Development). 2005. “Improving Financial Literacy: Analysis of Issues and Policies.� OECD: Paris. —. 2011. “Detailed Guide to Evaluating Financial Education Programmes,� OECD, Paris. —. 2011. “Guide to Evaluating Financial Education Programmes,� OECD, Paris. —. 2011. “Improving Financial Education Efficiency: OECD-Bank of Italy Symposium on Financial Literacy,� OECD, Paris. —. Forthcoming. “Addressing Women’s Needs for Financial Education,� OECD, Paris.   —. Forthcoming. “Financial Education for Youth : Rationale, Challenges, Illustrative Practices and Policy Guidance,� OECD, Paris. —. Forthcoming. “OECD/INFE Toolkit to Measure Financial Literacy and Inclusion: Guidance, Core Questionnaire and Supplementary Questions,� OECD, Paris. —. Forthcoming. “Financial Literacy and Inclusion: Results of OECD/INFE Survey across Countries and by Gender,� OECD, Paris. —. Forthcoming. “The Role of Financial Education in Financial Inclusion: OECD/INFE Evidence, Policies and Illustrative Case Studies.� Working paper, OECD, Paris. 24 CHAPTER 1. Study motivation, approach, and structure OECD/INFE. 2012a. High-Level Principles On National Strategies For Financial Education. OECD, Paris. —. 2012b. OECD/ INFE Guidelines for financial education in schools. OECD, Paris. Orton, Larry. 2007. “Financial Literacy: Lessons from International Experience.� Canadian Policy Research Network CPRN Research Report. September. Paluck, Elizabeth Levy, and Donald P. Green. 2009. “Deference, dissent, and dispute resolution: An experimental intervention using mass media to change norms and behavior in Rwanda.� American Political Science Review 103 (4). Platteau, J.-P. 1996. “Mutual Insurance as an Elusive Concept in Traditional Rural Societies.� Journal of Development Studies 23 (4): 461–490. Popper, Karl. 1935. Die Logik der Forschung. Wien. [The Logic of Scientific Discovery. Routledge Classics, 1968]. Ravallion, Martin. 2008. “Evaluation in the Practice of Development.� Policy Research Working Paper 4547. World Bank: Washington, DC. —. 2009. “Should the Randomistas Rule?� The Berkeley Electronic Press—Economists Voice. www.bepress.com/cgi/viewcontent.cgi?article=1368&context=ev. Rao, Vijayandra, and Michael Woolkock. 2003. “Integrating Qualitative and Quantitative Approaches in Program Evaluation.� In The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools, ed. Francois Bourguignon and Luiz Pereira da Silva. New York: Oxford University Press, pp. 165–90. Thaler, Richard, and Cass Sunstein. 2008. Nudge: Improving Decisions about Health, Wealth and Happiness. New Haven: Yale University Press. Vaughan, Peter W., Alleyne Regis, and Edwin St. Catherine. 2000. “Effects of an Entertainment- Education Radio Soap Opera on Family Planning and HIV Prevention in St. Lucia.� International Family Planning Perspectives 26 (4): 148–57. Yoong, Joanne. 2010. “Making Financial Education more Effective: Lessons from Behavioral Economics.� Paper presented at the OECD-Bank of Italy Symposium on Financial Literacy: Improving Financial Education Efficiency, 9. June, Rome. Willis, Lauren E. 2008. “Against Financial Literacy Education.� Iowa Law Review 94. 25 M CHAPTER 2 easuring financial capability in low- and middle-income countries 2.1 Motivation and overview of the measurement program Over the past few decades, financial literacy and financial education have obtained increasingly more attention in the policy agenda in both HICs and LICs. The recent financial crisis has reinforced the view that individuals need to be better equipped with knowledge and skills to be able to make informed and effective financial deci- sions due to the high level of individual responsibility for lifelong income planning and risk management, and due to the number and complexity of the products now available in financial markets. To date, however, very limited tools and evidence have been available to help policy makers determine the size of the problem and the key weakness areas, to identify specific target groups in the population, and to select the most effective and efficient type of intervention. As noted already, the objective of the RTF is to contribute to the advancement of financial literacy and capability by developing measurement and program evaluation tools that can be used fill in these knowledge gaps. Chapter 2 presents the measurement tools developed by the RTF in pursuit of this objective by providing an overview of the conceptual framework, the methodological approach, the implementation process, and the results obtained. The overall objective of the RTF measurement work was to develop a survey instru- ment that could be used to measure financial capability in a way that was both comparable across countries and independent of the level of income and educa- tion—because financial capability is not necessarily related with being wealthy or holding a graduate degree. The book by Collins et al. (2009) is just one testimony to how sophisticated the financial lives of poor households can be. Developing a measure of financial capability that suits the context of developing countries, where educational achievements and access to financial services are lower and poverty rates are higher, was the key challenge for the RTF measurement work. As discussed in more detail in the next section, the early literature focused on knowledge as the key element of financial literacy, based on the idea that rational 29 Financial capability in low- and middle-income countries: measurement and evaluation individuals can make optimal financial choices if they are knowledgeable about the key financial concepts that typically affect these decisions (interest rates and interest compounding, inflation, risk diversification). The international debate on financial literacy was already moving to an extended concept that included not only knowl- edge but also skills, attitudes and most importantly, behavior. Another conceptual issue was how to identify the specific concepts that should be included in the questionnaire and considered as evidence of financial capability. The first option was to rely on standard economic theory to determine what kind of knowledge, skills, and behaviors should be included and to assume a cognitive-based causal chain from knowledge to behavior (i.e., knowledge enables the individual to turn skills into behavior). An alternative option consisted of identifying mani- festations of financial capability through peer judgment about what elements are considered to be conducive to good outcomes and therefore to denote a financially capable person. This approach was selected for application in developing countries because it seemed more able to take into account the imperfect market conditions in which individuals operate. As it does not assume that individuals are fully rational utility-maximizers, this approach was also considered desirable given the emerging behavioral economics evidence that explains “suboptimal� choices by considering psychological traits and biases not accounted for in the cognitive-based approach. Implementing this peer-judgment-based conceptual approach required a long and structured process to develop an operational definition of financial capability. Following guidelines provided by the RTF, country teams in a selected group of LICs and MICs conducted extensive qualitative research to identify the key manifesta- tions of financial capability in terms of behaviors, skills, and attitudes that denoted financially capable/incapable people. These qualitative findings were compared across countries to identify a common core set of manifestations that seemed to be relevant and most frequently cited everywhere. Questions around these manifes- tations were then designed and tested to ensure that they were easily understood, were informative, and had the same interpretation across countries. Lastly, nationally representative surveys using the same questionnaire were conducted in each partic- ipating country. The selected approach was based on the idea that while financial capability cannot be measured directly, its key manifestations can be. The next question was then how to combine the information about these manifestations into a measure of financial capability. Similar research conducted in HICs seemed to suggest that financial capa- bility is a multidimensional concept. For example, four key dimensions (or domains) emerged in the U.K. study. The advantage of this approach was that it did not impose an a priori structure on the measure to be developed. Appropriate statistical methods were used to identify both the number and the nature of the domains. 30 CHAPTER 2. Measuring financial capability in low- and middle-income countries The process produced strikingly similar results across the participating countries. The manifestations of financial capability most frequently cited by the general public in LICs and MICs were similar to those that emerged in the U.K. study. Aspects of day-to-day money management and planning for the future were mentioned most frequently, while aspects relating to choosing products and staying informed were much less frequently cited across these countries (if cited at all). Psychological traits (which were not assessed in the United Kingdom) such as impulsiveness, attitude toward the futures, and achievement orientation/being enterprising also appeared to be relevant. It proved possible to design questions that produced components that mirrored the manifestations of financial capability identified and that provided robust scales across countries. It was also possible to segment the populations in the partici- pating countries using these components. Collapsing these into a smaller number of domains in the United Kingdom proved more complex. The statistical analysis iden- tified two key domains in the cross-country analysis: controlled budgeting and plan- ning for the future. However, while similar domains were found in all countries, there were subtle differences between them, such that a robust scoring system could not reliably be applied across countries. The results suggested that knowledge of financial concepts and products is a sepa- rate issue not strictly relevant for financial capability in LICs and MICs. Similarly, while the concepts of saving, borrowing within limits, and paying off debt were central in the discussions among peers, there was little mention of the choice and use of specific financial products. Section 2.2 presents in more detail the available options and the selected conceptual framework. Section 2.3 is an overview of the approach and lessons learned from the United Kingdom. The key measurement issues in the context of LICs and MICs are described in section 2.4. Section 2.5 contains an overview of the selected method- ological approach, while section 2.6 explains the step-by-step implementation of the approach in LICs and MICs. Lastly, section 2.7 provides an overview of key results and lessons learned from the process. Appendix B (available online at www.finlitedu.org) contains details of the analysis and high level results from the seven country survey pilots completed at the time of writing. For more details about methodology, process, and analytical results of the measurement project, the full project report should be consulted (Kempson, Perotti, and Scott 2013). 31 Financial capability in low- and middle-income countries: measurement and evaluation 2.2 Conceptual options and selected framework The concept of financial literacy/capability has evolved from the initial narrow defini- tion adopted in the early literature1 to a broader concept that now includes “a combi- nation of financial awareness, knowledge, skill, attitude and behavior necessary to make sound financial decisions and ultimately achieve financial wellbeing� (OECD/ INFE 2011). The great importance initially attributed to knowledge was based on the traditional theory of intertemporal choice. To make optimal financial decisions, individuals need to know what the inflation and interest rates are, how to calculate compound interest rate, and how to manage risk by diversifying their portfolios. When these concepts are not known, the individual cannot make an optimal deci- sion. On the other hand, if one assumes that people are perfectly rational and maxi- mize utility according to theory, then all they need is correct information. Based on this conceptual framework, most policy interventions have focused on providing financial education to try to increase the level of knowledge and awareness around financial concepts and the use of financial products. However, while some evidence has been found that knowledge of financial concepts is associated with various desirable financial behaviors,2 not much is known yet about the causality of these relationships, and more importantly, whether financial education can increase this knowledge and through it affect behavior that determines good outcomes.3 A clear limitation of the early approach to financial literacy is therefore that it does not take into account the combination of behaviors, skills, and attitudes that need to complement knowledge to achieve good financial outcomes. As a result, the concept of financial literacy has been extended to include the set of “skills, motivation and confidence to apply such knowledge and understanding in order to make effective decisions […]� (OECD 2012), increasingly referred to as “financial capability.� As discussed in chapter 1, there are two main conceptual approaches for deter- mining which skills, attitudes, and behaviors should be considered as part of financial capability. One option is the “cognitive-based� approach, which assumes that the decision-making process is guided by knowledge: if the individual is sufficiently knowl- edgeable, then she will take the necessary steps to translate knowledge into positive outcomes. This approach is also defined “normative� because the types of skills, atti- 1  For example, see Mandell (1997); Chen and Volpe (1998); and Lusardi and Mitchell (2006). 2  See Kotlikoff and Bernheim (2001); Hogarth, Beverly and Hilgert (2003); Lusardi and Mitchell (2011); Cole, Sampson, and Zia (2011). See Duflo and Saez (2003); Karlan and Valdivia (2011); Bertrand and Morse (2010); and Cole, 3  Sampson and Zia (2011). 32 CHAPTER 2. Measuring financial capability in low- and middle-income countries tudes, and behaviors that are considered capable are identified in advance through economic theory or previous evidence. For example, the necessary skills will include numeracy and general literacy, and capable behaviors will include saving, diversifying investments, choosing the most convenient financial products for one’s needs, and so on. By adopting a normative approach, the researcher designing a survey of finan- cial capability will know in advance what topics should be covered and can proceed directly to identifying the best questions to measure the selected concepts. As discussed in section 2.6, the next methodological question is how to synthesize the information collected with such instrument into a measure of financial capability. An alternative approach, developed by the United Kingdom’s Financial Services Authority (FSA), recognizes that financial capability is a broad concept that may include knowledge as well as skills, attitudes, and behaviors but it does not make any assumptions about what these should be, or about the causal relationships among them. This conceptual framework highlights the wide range of possible financial capability interventions that can be adopted. Examples include one-to-one guidance, peer-learning, development of decision-enhancing tools (decision trees, computer algorithms), improved consumer protection, shaping attitudes and motivations through social marketing, and the deployment of nudge techniques (such as auto-en- rollment in pension schemes). This approach is called “agnostic� as it makes no assumption about how the outcome can be achieved, but determines this through research. The guiding principle is that the operational definition of financial capability should be relevant for the specific settings of interest, and therefore the people who live in those settings should determine what makes one capable or incapable, based on what is conducive to good outcomes. When adopting a positive (empirical) approach, the researcher designing the questionnaire needs to conduct preliminary qualitative research to obtain an operational definition of financial capability. The RTF opted for this positive/agnostic approach as it seems to be particularly helpful for assessing financial capability in LICs and MICs where this type of assess- ment has not been done before. In particular, it was not known whether the same concepts, behaviors, skills, and attitudes would be relevant for financial capability in LICs and MICs as in HICs. In addition, the normative conceptual framework assumes that financial products are readily available for individuals to use for consumption smoothing, which may not be the case in developing countries with limited levels of access to finance. Research has shown that even low-income people in developing countries have sophisticated financial lives that do not necessarily require interac- tion with formal financial services (Collins et al. 2009). Measuring financial capability by simply applying the definition adopted in HICs may not lead to an understanding of what individuals in developing countries really need to achieve better financial outcomes. 33 Financial capability in low- and middle-income countries: measurement and evaluation Increasing evidence from behavioral economics research shows how a number of social, psychological, and emotional factors such as cognitive biases, hyperbolic discounting, procrastination, self-control, etc., can dramatically affect behavior but are not considered in traditional economic theory. In the presence of these phenomena, the causal link from knowledge to behavior does not work anymore; for example, lack of self-control may cause an individual to not stick to her budget plan, or procrastination may lead people to postpone decisions about retirement planning even if they know they have to act on them. Not knowing in advance which of these mechanisms may be at play in the area of financial capability in a specific context, a positive approach based on research on people living in that context can help to identify the relevant behavioral issues. The main drawback of this approach is the challenge for measurement. A single concept (and particularly one such as knowledge, where there can be clear right and wrong answers) is much easier to measure, to compare across countries, and to interpret in economic analysis. Measuring capability is particularly chal- lenging because of the behavioral issues discussed above, which can prevent an individual from manifesting his or her own skills, but also because there are some types of capabilities that are very difficult to observe independently from the effect of external factors. For example, keeping records of one’s own expenses mostly depends on one’s ability and willingness to do so, while the ability to save also depends on the amount of available resources and on obligations towards family or community members. This constitutes a big challenge for measurement, and while the survey presented in this chapter was designed to take into account the effect of the enabling environment in which individuals operate, in some cases it was not possible to completely separate these intertwined aspects. Using a composite measure of financial capability raises a number of questions: Is it possible to construct a single measure of financial capability, or does it need to be a multidimensional measure (a set of indicators)? Which are the most important dimensions and are these comparable across countries? The rest of this chapter discusses how these questions can be answered by implementing the U.K. approach in LICs and MICs. 2.3 The FSA approach and lessons learned from HICs This section draws from the FSA report (FSA 2005) to briefly review the U.K. study, which was the first to develop and adopt a “positive� approach to measuring finan- cial capability, and which was then replicated in other HICs. These studies provided many lessons for developing both a conceptual framework and a methodology for 34 CHAPTER 2. Measuring financial capability in low- and middle-income countries questionnaire design. The objective of the FSA study was to conduct a baseline survey of financial capability in the United Kingdom by designing a questionnaire that could be used to construct a measure of capability and that took into account specific individual circumstances (FSA 2005). The research plan included a literature review to define a conceptual framework, a qualitative research phase to test the conceptual model and to identify potential ways of measuring capability, and several waves of cognitive interviews to help design, test, and refine the questions. The conceptual model was mostly based on a framework developed by the FSA and the Basic Skills Agency (2004). Behavior was considered as “evidence of financial capability,� and it was assumed to be the result of the application of knowledge and skills, with indirect effects from external circumstances and personality. The rele- vance of this conceptual framework was tested through qualitative research with the general public using focus groups. A different conceptual model emerged from these discussions, because people perceived financial capability in much more behavioral terms than anticipated. The focus groups identified a range of manifestations of financial capability that subsequent analysis found could be grouped into four broad domains: managing money, planning ahead, making choices, and getting help. The results of the focus groups were used to restructure the conceptual framework. Very interesting for the perspective of developing countries, the U.K. focus groups conducted with people living on low incomes focused mostly on managing money, as their ability to plan ahead was limited by income, and they only used a limited range of financial products and sources of advice and information. The focus group results clearly indicated the need to develop the questionnaire around the measurement of behaviors. For each key area of capability, the focus groups indicated the type of things financially capable people do: within “managing money,� they would make a budget, resist the temptation to borrow or to overspend, and keep debt under control; within “planning ahead,� they would save, plan for the long term, and plan for the unexpected; within “making choices,� they would shop around and read the small print; within “getting help,� they would gather information from TV and newspapers, know where to go for advice, and be aware of their rights. Personality and experience were perceived as elements determining the extent of a person’s financial capability. Other key lessons learned from the qualitative research included the need to develop questions that were income- and culture-neutral to account for different levels of income (acknowledging the fact that poor people can be capable and rich people can be incapable) and for cross-country and generational differences in perceptions about the use of certain financial products. The U.K. study also developed a methodology for constructing a measure of financial capability. Building on evidence from the qualitative research, a multidimensional 35 Financial capability in low- and middle-income countries: measurement and evaluation approach for constructing this measure was considered appropriate for the concep- tual framework: the fact that the focus groups had identified four different areas of financial capability and that they had recognized the possibility that some individuals could be good in one area and weak in another suggested that separate indicators (or scores) should be constructed for each area. Several methodological options were considered: (1) a simple arithmetic sum of scores assigned to answers to each question; (2) regression analysis to predict the probability of achieving the desired outcome based on behavioral indicators measured by the survey; and (3) factor anal- ysis to obtain a score for each domain from a linear combination of the questions relevant for that domain (see appendix B for further details of factor analysis). Factor analysis was identified as the most appropriate option because: (1) it was consid- ered difficult to assign a score to each question (there would not always be a right or wrong answer); and (2) identifying a clear desired outcome for each aspect of financial capability was not always possible (for example, in the area of information seeking). The objectives of the factor analysis conducted on the U.K. data were to: produce scores for each of the four domains of financial capability and describe the charac- teristics of people who were least and most capable in each domain. Factor analysis was conducted to construct a score for each domain, and cluster analysis was used to group individuals with similar scores. The results of the factor analysis suggested that two separate scores should be calculated within the first domain, as two distinguish- able factors emerged from the analysis: making ends meet and keeping track of money. In addition, the U.K. study segmented the population based on scores in these domains using cluster analysis. As described in more detail in appendix B, cluster analysis calculates a measure of similarity between individuals based on the scores, and groups together individuals who are similar to each other based on this measure. By looking at the average characteristics of the individuals assigned to each group (or cluster), it is possible to describe each group in terms of sociodemographic vari- ables (for example, average age, income, years in education, percentage of women) and in terms of their financial capability scores (average group scores in each of the domains). The analysis identified 11 clusters, ranging from a group of people with high average scores across all domains that were, on average, older-age couples with high income and using many financial products to a group scoring very low in all domains and composed, on average, of young parents living on low incomes. The main lesson learned from the U.K. study was that financial capability is mostly about behavior, although taking attitudes into account helps to mitigate the constraining effect of low income on behavior and therefore on the measurement of capability. In addition, the study found that financial capability spans across different dimensions (or domains) that cannot be assessed with a single measure, and there- fore the level of an individual’s capability should be measured across the various 36 CHAPTER 2. Measuring financial capability in low- and middle-income countries domains. Indeed, it was rather common to find that people were very capable in one domain but not in others. Many other countries, including, for example, Canada (Statistics Canada 2009), Ireland (Financial Regulator 2009), Italy (Ambrosetti 2008), the Netherlands (CentiQ 2008), and the United States (FINRA 2009) have adopted this “financial capability� approach, although most of them have not developed their questionnaires through the same long process entailing qualitative research before the actual survey. Most countries have simply used existing financial capability questionnaires and adapted them based on the judgment of experts and policy makers, instead of conducting focus groups with the public. They have also drawn on the U.K. approach to analyze the survey data. 2.4 Key issues in LICs and MICs Before describing the details of the process through which the RTF adapted the U.K. approach to the context of developing countries, this section revisits the key features of LICs and MICs already discussed in chapter 1 to highlight their implications for measurement of financial capability in these settings. 2.4.1 Access Typically, people living in LICs and MICs have lower levels of access to financial services. There is no conclusive evidence on the causal relationship between finan- cial access and financial capability. Do people use financial services because they are more capable than those who do not? Or does the use of financial services create an incentive for consumers to improve their understanding of the products and their ability to select and manage them? As research in developing countries has shown (Collins et al. 2009), people can be very capable even without using formal financial products. On the other hand, using financial products incorrectly (for example, an excessive use of credit cards or loans beyond the level that can be repaid) can be an indicator of limited capability. An important challenge for capability measurement is to first assess whether the use of financial services should be considered an element of financial capability; second, if one determines that access is a related but separate matter, then the chosen measure of financial capability must be as independent of the level of access as possible. In other words, if a low-income person living in a rural area is able to make ends meet, to save for unexpected events, and so on, it may not be appropriate to consider her less capable because she is not using a bank account. From a practical point of view, constructing a measure of capability that relies on the use of financial products (for example, based on how people selected a product, or on their punctu- ality in repaying their credit card debt) would have very limited applicability in devel- 37 Financial capability in low- and middle-income countries: measurement and evaluation oping countries because a large part of the population does not hold any products. The choice and use of financial products can be therefore considered a separate area for which a measure (if any) could be constructed for the subsample of people who have products. 2.4.2 Poverty The higher number of people living on low incomes in LICs and MICs compared to HICs has important consequences for the measurement of financial capability. Low income limits the ability to cope with unexpected shocks and to make ends meet. For example, if a person has such a low income that she cannot even pay for food, no matter how careful she is in spending money, she will run short of it. That does not mean she is not capable. It is therefore essential to try and separate aspects that are under the individual’s control from the external conditions that may affect the outcome regardless of the individual’s efforts. To limit the possibility that the level of income could bias the measure of capability, a partial solution is to take into account attitudes toward the key behaviors (whether the person tries to save as much as possible, whether the person is concerned with being able to cover unexpected expenses, etc.). Of course, it is not possible to design a questionnaire on financial capability that is entirely income-neutral, so results from the analysis should be inter- preted carefully. 2.4.3 Education The population living in developing countries typically has lower levels of literacy and educational achievements compared to that in HICs. A crucial challenge for financial capability measurement in these countries is therefore to design questions that can be easily understood by everyone, including people who cannot read or write, but that still apply and are relevant for higher-educated respondents. Extensive testing must be conducted to make sure that the questions measure a respondent’s level of financial capability rather than his ability to understand the question. 2.4.4 Location A large percentage of the population in developing countries lives in rural areas, with limited access to financial services. In some cases, rural areas are also denoted by a greater importance of the community in an individual’s social and economic life. For example, people support each other financially by forming informal networks for credit and risk management. This has implications both in terms of a higher incidence of informal financial services compared to formal services, and in terms of limitations to the individual’s ability to make financial decisions independently (for example, in some communities it is impossible to deny financial help to a member of the same commu- nity asking for it). 38 CHAPTER 2. Measuring financial capability in low- and middle-income countries When designing an instrument to measure financial capability, it is therefore important to obtain a good understanding of how financial decisions are made within the household and about what type of resources; for example, whether a person is only responsible for her own money or also for managing the household money; whether the person or household is regularly supporting other people outside the household; and what are the main sources of income in the household. The existence of informal networks also affects the decision to make provisions for the future in other ways. For example, if in a particular context there is a well-estab- lished tradition according to which adult children support their parents in their old age, it may make perfect sense for a person to use her savings to finance her chil- dren’s education rather than investing it in financial products. 2.4.5 Informality Many workers in LICs and MICs have informal jobs and therefore do not have access to old age pension, health insurance, and other benefits typically associated with jobs that comply with social security regulations. For these people, taking an active role in financial planning and risk management is particularly important because they will not benefit from government- or employer-provided support in old age or in case of illness or similar life events. In terms of measurement, to assess the level of capa- bility of people who are not actively planning for their future, it is important to know whether they are at least covered against risks through social security. For example, some people do not make any provisions for retirement because they are mandato- rily enrolled in old age/disability/survivor insurance provided through their job. Another aspect related with informality in LICs is the large incidence of self-em- ployment and family businesses. Many people who run small businesses in LICs do not separate the household budget from business finances. As a consequence, when designing a survey instrument to measure capability of managing money, it is important to clarify what money the respondent has in mind when answering the questions. 2.4.6 Risk management Individuals in LICs are exposed to a variety of risks that create higher uncertainty about day-to-day income sources, compared to those in HICs. This element should be taken into account when evaluating the type of insurance products and arrange- ments that people make in these settings. For example, if an individual living on a low income opts to buy a short-term insurance product (for example, weather-re- lated insurance to support agricultural activities) instead of saving the money for old age, this could be a sign of capability. 39 Financial capability in low- and middle-income countries: measurement and evaluation 2.5 Overview of selected approach The key objective of the RTF measurement program is to develop and test a new instrument that can be used to measure financial capability across different income levels, different cultures, and over time. The “positive� approach adopted by the RTF requires an extensive amount of preliminary research before the actual survey ques- tionnaire can be designed, and institutions managing surveys rarely have the time and resources available to go through such a process. The RTF supported efforts by a group of LICs and MICs that wanted to conduct a survey of financial capability by providing the expertise of a group of international experts on qualitative research methods and questionnaire design, by granting financial resources, and by coordi- nating the work of country teams to develop a common methodology that could be used across countries in a comparable way. The main assumption of the RTF approach is that financial capability, like other broad and abstract concepts such as intelligence or personality traits (Spearman 1904), cannot be measured directly, but that it is possible to measure a set of mani- festations of this underlying capability. Another assumption borrowed from the U.K. conceptual framework is that financial capability spans different domains, although it was not known whether the same domains would be relevant for developing coun- tries. It was also not known whether it would ultimately be possible to construct a single measure of capability or if it would be more sensible to evaluate capability in each specific domain. The selected approach posed two key methodological questions. The first was about the type of manifestations that should be considered, and therefore which questions to include in the questionnaire to measure them. The second key ques- tion was how the information about these manifestations could be used to produce a segmentation of the population according to its range of capabilities and to construct a measure (or measures) of capability. The RTF addressed these issues through a process that entailed many steps, from ascertaining the parameters of financial capability to data analysis: (1) development of an operational definition of financial capability through identification of its key manifestations; (2) development and testing of survey questions to measure the manifestations; (3) data collection; (4) identification of the key components and domains of financial capability through factor analysis and assignment of scores; and (5) identification of potential target groups for policy intervention through cluster analysis. The next section describes each step in detail by presenting the objectives, methodology, implementation process, and results. Following the “positive� approach, the project team identified the set of key mani- festations through the judgment of people from different walks of life in each of the 40 CHAPTER 2. Measuring financial capability in low- and middle-income countries Box 2.1  From basic concept to questionnaire The main idea of the RTF approach was to define the concept of financial capability through the views of the general public in the context of interest and to develop questions to measure levels of capability based on this definition. The key conceptual steps were as follows: ƒƒ Step 1: Use qualitative methods (focus groups) to identify characteristics or other elements associated by the general public with people who are financially capable or financially incapable. The output of this step consists of recorded transcripts of focus group discussions. ƒƒ Step 2: Identify from an analysis of the content of the focus group discussions a number of stylized concepts that represent the key manifestations of financial capability. The output of this step is a list of concepts that need to be measured in the survey. ƒƒ Step 3: Design questions to measure the key manifestations of capability. This was done by looking for relevant existing questions for each concept from previous capability surveys, and by designing new questions when necessary. The output of this step is a draft interview guide that contains multiple choice answer and open-ended questions about each manifestation. ƒƒ Step 4: Use qualitative interview methods (two waves of semi-structured cognitive interviews) to choose among alternative questions and to identify the most appropriate way of recording replies. The output of this step includes transcripts or detailed interviewer notes with feedback on each question. ƒƒ Step 5: Finalize the quantitative survey questionnaire. This was done by conducting pilot surveys to test the questions and the precodes selected in the qualitative phase. countries participating in the developmental work to help ensure that the developed measure of capability would be relevant for these settings. As discussed in more detail in the next section, these manifestations of financial capability were identified through focus groups discussions. Focus groups participants were asked to describe to the researcher both financially capable and financially incapable people. From the group discussion, a number of different aspects of capability emerged: behav- iors, attitudes, and motivations. The set of core manifestations of capability was then determined by identifying the concepts that were mentioned most frequently across all the groups in all the countries participating. The next step was to design appropriate questions to measure them. Available questions from existing capability surveys were reviewed, building on previous survey stocktaking done by the OECD (2009). The project team then designed new questions for concepts that were not adequately covered by existing surveys. Regardless of their source, questions were tested to make sure that they were well understood, that they provided mean- ingful information, and that they had the same interpretation across countries. The countries participating in the project tested the questionnaire with two rounds of cognitive (in-depth) interviews. These were very similar to the final survey interviews, 41 Financial capability in low- and middle-income countries: measurement and evaluation except that follow-up questions or prompts were used after many of the ques- tions to check the respondent’s understanding and reaction to the questions in the pretest. Detailed feedback was also gathered from the interviewers to detect any problems with specific questions, words, or expressions. The feedback was then compared across countries and necessary adjustments were made to the questionnaire (changes in wording, dropping difficult questions, choosing between alternative versions of the same question). The final draft ques- tionnaire was then administered in pilot surveys of 100–200 individuals in each country to test and refine the overall format, skip patterns, data entry procedures, etc. The last step in data collection was the implementation of national surveys in the participating countries. The data were analyzed both by the country teams and the project team, which constructed the measures of financial capability. Factor analysis was used to construct these measures (or scores) as in the U.K. study. This data reduction method uses correlation among the variables to find the smallest possible number of linear combinations of these variables that best synthesize the information contained in the data (see the next section for more details, and appendix B for a more technical reference). This technique has two steps: in the first step, information contained in the questions is aggregated into the components of financial capability (that are empir- ical counterparts of the manifestations of financial capability identified in the focus groups); in the second step, the relationship among the components is examined to identify the key domains of financial capability. By looking at which variables “load� onto a specific factor, it is possible to define and name the corresponding domain. For example, if one factor is mostly a combination of making a plan for spending money, keeping track of expenses, not overspending, and similar concepts, one could call the domain “day-to-day money management.� Ultimately, both the number and the nature of the domains were determined empirically through data analysis. After identifying the components, a score was calculated for each individual in each component as the linear combination of responses to the questions loading onto the relevant factor. The weight attributed to each variable was determined by the results of the factor analysis. This important feature allows for the weighting of each vari- able differently and does not require determination of the weights by making a priori assumptions. By comparing scores of different individuals in the various components, survey respondents were grouped according to similar levels of financial capability using cluster analysis. Cluster analysis calculates a measure of similarity between indi- viduals based on their scores and then proceeds to aggregate pairs of individuals until the number of groups formed achieves the optimal balance between: (1) the similarity of the individuals grouped together (that one wants to maximize); and (2) 42 CHAPTER 2. Measuring financial capability in low- and middle-income countries Box 2.2  From questionnaire replies to financial capability scores The questionnaire contains multiple questions for each concept measured. This is done to capture different nuances and to obtain more robust measures of each concept. To construct indicators of financial capability that can be used for further analysis (for example, for segmenting the population according to capability levels), the data need to be reduced. This is done through several steps using factor analysis (with principal component factoring). The following key conceptual steps are highlighted: ƒƒ Step 1: Generate variables from responses to questions. This step frequently requires aggregating two or more questions, for example, because the same original question was split into two to simplify response scales (e.g., “Agree/Disagree� followed by “To some extent/Strongly�). ƒƒ Step 2: Identify and exclude from the analysis variables with high nonresponse rates or ones that are too correlated with income if a more neutral alternative variable on the same concept can be constructed. ƒƒ Step 3: Aggregate variables measuring the same concept. This is done by conducting a factor analysis of all variables jointly. The result of the analysis is a number of components, each of which is a weighted combination of variables measuring the same concept. These components are the empirical counterpart of the concepts/manifestations resulting from the focus groups. The component scores are then compared across countries to assess the possibility of international comparisons at this level. ƒƒ Step 4: Identify key domains of financial capability. After a measure has been constructed for each component of financial capability, a factor analysis is conducted on the components to see if they can be further aggregated to create domain scores. If one single domain emerges, then a single score can be constructed by weighting all the components into the same scale. The domains are then compared across countries to test whether the same scoring method can be used. ƒƒ Step 5: Use the scores to segment the population. Depending on the results of the analysis, either domain scores or component scores can be used to identify subgroups of the population with similar capability levels. This is done through cluster analysis, which aggregates individuals with similar levels of the selected scores. the number of groups or clusters (that one wants to minimize). This method enables distinct groups to be identified in terms of their capability (for example, people who are doing well in all domains, or people who are very weak in one domain but are doing well in all others, etc.). By describing the average sociodemographic charac- teristics of people in a specific group, useful information can be provided to policy makers who want to target an intervention at a restricted group of individuals. For example, if the results suggest that there is a group doing well in every component except budgeting and that this group mostly contains young people, policy makers might want to study an intervention that improves young people’s ability to budget, while a completely different type of intervention might be advisable for older people. 43 Financial capability in low- and middle-income countries: measurement and evaluation 2.6 Implementing the approach in LICs and MICs 2.6.1 Overview of implementation The experimental nature of the work of the RTF on financial capability measurement required the involvement of counterparts in LICs and MICs to participate in the methodology development and testing. The overall work program was developed and managed by the World Bank RTF Secretariat, based in the Social Protection Unit of the Human Development Network, supported by a Technical Advisory Group composed of external experts and academics, and guided by a World Bank Steering Committee composed of representatives of different departments working on related topics. The Secretariat developed a selection procedure for allocating avail- able resources to relevant survey projects in LICs and MICs where the implementing agencies were interested in adopting a common methodology and willing to partici- pate in the development process according to guidelines provided by the RTF. In April 2010, a first call for proposals was issued to select the projects that would participate in the entire process, including the preliminary qualitative research phase and the final quantitative survey implementation. Proposals were submitted through a World Bank staff person working in the relevant regional department and acting as the Task Team Leader (TTL). Proposals were reviewed by the RTF Secretariat and by the experts. Eligibility criteria included: a focus on LICs or MICs and on low-in- come groups in the first phase; commitment to participate in the RTF workshops; and agreement to follow the RTF guidelines. In addition to eligibility criteria, projects were selected based on the quality of the proposal, the skills and commitment of the team involved, and the quality of support by national institutions, as well as on cofinancing requirements to ensure full commitment from local counterparts. Out of the 16 proposals submitted, the RTF selected six projects (covering eight countries: Colombia, Mexico, and Uruguay in the World Bank’s Latin American and Caribbean Region; Tanzania and a joint project for Namibia, Zambia, and Malawi in the Africa Region; and Papua New Guinea in the East Asia and Pacific Region). A new call for proposals was issued in March 2011 to select additional countries to join the program by implementing national surveys to test the questionnaire devel- oped through the preliminary qualitative research. The purpose of this second round of funding was twofold. One objective was to achieve coverage of regions that had not participated in the preliminary phase of the program, such as Eastern Europe and Central Asia, Middle East and North Africa, and South Asia. The second objective was to test and implement the common questionnaire in countries that had not partici- pated in the development stage. Following a very similar selection procedure, new grants were awarded to Armenia, Lebanon, and Turkey. From the group of initially 44 CHAPTER 2. Measuring financial capability in low- and middle-income countries selected countries, Colombia, Mexico, and Uruguay went on to implement a full survey, while Papua New Guinea only conducted a pilot, and the African countries did not participate in the final stage due to the lack of cofinancing or to time frames that were not compatible with the overall RTF program. In October 2012, Nigeria joined the country pilot with a project that implements the financial capability questionnaire as a module attached to a large panel household survey. With support and guidance from a senior World Bank staff from DEC acting as team leader for the overall measurement project and from the Technical Advisory Team, the RTF Secretariat developed the research program and guidelines for implementa- tion. The Secretariat also coordinated efforts of the local counterparts and organized three technical workshops where the country teams, the RTF Secretariat, and the Advisory Group met to discuss the implementation details and the results obtained in the various steps of the process. The complex development process followed by the RTF offered many opportunities to learn from the testing done by the country teams in each stage of the project. The complexity also presented several challenges: as a collaborative effort involving very different countries, close cooperation among the country teams was required, for example, through participation in the three workshops, and constant interaction with the RTF team supported by the experts. The methodological approach required that the core questions included in the instrument be determined by the outcome of the focus group discussions. This principle provided a helpful criterion to discriminate between questions that all countries should include in their survey and questions that could be considered optional. For example, the knowledge of financial concepts and products did not emerge as a key concept in the focus groups, and therefore was not considered to be part of the core questionnaire; however, the many coun- tries interested in including this were advised to adopt a common module for it based on the OECD international pilot (Atkinson and Messy 2012), separately devel- oped as a different project under the same RTF. The workshops with the country teams and experts proved to be very helpful for clarifying the objectives and methodological approach of the project, for providing country teams with a forum for discussions and sharing experience, and for building a collaborative team. Coordinated timing was essential to ensure that each revision of the interview guide or the questionnaire was based on the broadest possible feed- back, and it was certainly positively affected by the need to meet deadlines for the upcoming workshops. 2.6.2 Focus groups: identifying manifestations of capability Qualitative research methods to identify empirical indicators for instrument develop- ment have been used extensively in psychology and in the marketing research litera- 45 Financial capability in low- and middle-income countries: measurement and evaluation ture (Churchill 1979), where qualitative methods such as focus groups were originally introduced (Calder 1977). More recently, these techniques have been used in health care research (Pett, Lackey and Sullivan 2003) and indicated in official guidelines for the construction of patient-reported outcome measures (FDA 2006). Focus groups are a particular type of interview conducted by a facilitator with a small group of 8–10 people invited to discuss the topic of interest. The RTF focus group participants were asked to describe financially capable and incapable people, with minimal intervention by the facilitator, whose role was to get the conversation started and to make sure that every person actively participated in the discussion, without suggesting specific topics or expressing personal views. The topic guide for facilitators was developed by the team of experts and presented at a workshop with the country teams, where the teams were also provided with some training on the style of facilitation that should be used, the type of people to be invited as partic- ipants, etc. The original topic guide was slightly modified following discussions at the workshop, because some of the country teams (mostly in Africa and Papua New Guinea) were concerned about using concepts that were too abstract and felt that the discussions needed some minimal framing. The final guide included the following sections: „„ Short warm-up (introductions by participants); „„ Discussion of general money/resource issues, including the key questions “Tell me about someone who is [not/very] financially capable, what kind of person are they?� or the alternative, more concrete “What marks out someone who handles these things well from somebody who doesn’t?� (In both cases, the probes were “What sorts of things do they do/not do?,� “What skills and knowledge do they have/not have?,� “What motivations or attitudes do they have?,� “Anything else?,� “Which of these is most important?�); „„ Checking the relevance of specific areas (money management, planning for the future, selecting and using financial products, getting information and advice) if not previously mentioned, by describing somebody who is good/bad at it; and „„ Summing up the discussion (making sure that no important concepts were omitted, and noting the most and least important things discussed). The guidelines provided to facilitators about the style of the focus groups recom- mended: using the standard probes included in the guide; keeping any additional probe neutral (for example, by asking “Tell me more about that,� “What do the rest of you think?,� etc.); and intervening if multiple conversations started at the same time, bringing the conversation back to points that were mentioned but not further explored; and asking quiet people to express their views. 46 CHAPTER 2. Measuring financial capability in low- and middle-income countries In the implementation process, 6–13 focus groups were conducted in each of the eight participating countries (for a total of 70 focus groups). All the focus group discussions were recorded and transcribed to facilitate their analysis. For each country, the first focus group transcript was reviewed by the lead expert, who noted the main concepts that emerged from the discussion onto a thematic grid. This early review helped to detect any problems in the way the focus groups were conducted and to provide additional guidance to the country teams. The local teams then completed the grids with results from the other focus groups. For each focus group, the grid contained the concepts cited about: (1) people who are financially incapable; (2) people who are very capable; (3) day-to-day money management; (4) planning for the future; (5) choosing/using products, getting information; (6) personality; and (7) other topics. The focus group transcripts were then given to the experts who reviewed them against the thematic grids sent by the country teams, and a master grid was produced by evaluating the results across countries. The master grid listed the key manifestations of capability mentioned by the focus groups and their frequency. The results were presented at a second workshop with the country teams, where initial suggestions for questions around the proposed topics were also presented by the Technical Advisory Team. From a technical perspective, the focus groups provided surprising results; many similarities were found among the participating countries and among people with lower incomes in HICs like the United Kingdom and Ireland. The most notable differ- ence from HICs was that there was very limited mention of use or knowledge of financial products and their characteristics as an aspect indicating financial capa- bility—although even in the United Kingdom this was much less important for people with lower incomes. It also emerged that psychological characteristics such as impulsivity or action orientation were frequently used by focus group participants when describing financially capable (or incapable) people. The key manifestations of financial capability that emerged from the focus groups focused on the areas of day-to-day money management (budgeting, keeping track of expenses, not over- spending, prioritizing expenses, borrowing within affordable limits, etc.) and planning for the future (trying to save, planning for the children’s future, having strategies to cover expenses in old age, etc.). The full list that emerged from the focus groups is presented in the discussion of key questionnaire features. 2.6.3 In-depth interviews: designing and testing the questions The next step was identification of existing questions or design of new questions for the key concepts that emerged from the focus groups. The Technical Advisory Team consulted a review of 26 existing national and international surveys compiled for the 47 Financial capability in low- and middle-income countries: measurement and evaluation OECD under the same RTF, to maximize use of the number of questions already tried and tested. The Technical Advisory Team designed the remaining questions to be tested. The main criteria for questions were that they should: „„ Capture a key concept identified in the focus groups; „„ Work across countries and apply to the whole population (i.e., be culture- and income-neutral); „„ Be unambiguous; „„ Discriminate between more capable and less capable people; „„ Avoid scales based on value judgments; and „„ Allow the use of different statistical tools for data analysis, including factor analysis. In addition, the questions should be as objective as possible: for example, asking about the frequency (or similar details) of a certain behavior identified as capable by the focus groups was preferred to asking respondents hypothetical questions or asking them to rate themselves on a capability scale.4 In some cases, the expert team identified more than one possible question from existing surveys that could be used to capture a particular manifestation of financial capability. In these instances, all possible questions were tested in the interviews to identify those that best captured variations in financial capability and met the above criteria. The first draft of the instrument was a semi-structured interview guide to be tested in two waves of cognitive interviews, gradually refining the questionnaire. Inter- viewers were asked to record for each question whether it was: not understood and why; difficult to answer and why; or not appropriate for the respondent’s circum- stances and why. Interviewers were also asked to record any inconsistencies in the replies given by respondents across questions and to identify questions that did not accurately capture the respondent’s level of financial capability as indicated by the generality of questions. This was also a useful tool for ensuring that relevant topics that arose during the interview were adequately covered in the questionnaire, and that appropriate precoded answers were provided. Several questions were initially kept open-ended to identify the key answers to be included in the precodes. For example, after the question about planning how to spend the money, respondents All surveys on any topic are potentially subject to measurement error and to bias determined 4  by respondents overstating their own qualities or skills. To limit this effect as much as possible, the questions specifically avoided any wording that could sound judgmental. 48 CHAPTER 2. Measuring financial capability in low- and middle-income countries who did not plan were asked, “Why don’t you plan how you will use your income?�; after a question about setting priorities, people were asked, “What are your main priorities?� and probed with “What else?� Other common questions used were “Why do you say that?,� “Do you feel that the questions you have been asked provide an accurate picture of how you [manage your money day-to-day/plan for the future/ etc.]?,� and “Do you feel that we have missed out anything important about how you [manage your money day-to-day/plan for the future/etc.]?� Respondents were also encouraged to provide comments about the questions they were asked. In addition to cognitive testing, the purpose of the in-depth interview stage was to reduce the number of questions by removing duplication of coverage and focusing on a core set of questions that worked well across countries and all sections of the populations. About 15 in-depth interviews were conducted per country in each round, for a total of 117 interviews in the first round and 111 in the second round. Interviewers were requested to provide written notes or transcripts of each interview in addition to their feedback on questions that were not understood, produced inconsistent replies, were not relevant, or did not capture the respondent’s level of financial capability. This feedback, combined with debriefing sessions organized between the RTF Secre- tariat, the experts, and the country teams, informed the subsequent revisions of the questionnaire and the drafting of its pilot version. After each round, the project team and experts reviewed the interview transcripts and the feedback provided by the country teams to identify the questions, wording, and precodes that worked best. The in-depth interview stage was very helpful in identifying problems with the struc- ture and wording of some questions, particularly those that were taken from existing surveys carried out in developed countries. Relatively abstract concepts such as “managing� and “budgeting� could not be used as they were not well understood by lower educated respondents; for this reason, in some cases, multiple questions had to be used instead of a single question to ensure that every practical aspect related with the abstract concept was explored (in the case of “managing,� for example, this included planning, making decisions, being responsible for decisions, etc.). It was found that respondents with lower education levels had difficulty with the following: „„ Understanding long questions or questions containing negative forms; „„ Remembering some of the precodes that were read out after a question; and „„ Using response scales that were not binary (for example, a five-point agree/ disagree scale). 49 Financial capability in low- and middle-income countries: measurement and evaluation To the extent possible, the questions were revised to be short, with a simple construct, and to require a yes/no answer. Many questions split into two then had to be combined in the analysis (for example: Q1: “Do you agree with…?� A1: Yes/ No, followed by Q2: “And do you agree/disagree strongly or only to some extent?� A2: Strongly/To some extent). Results showed that, in some instances, asking two questions instead of one took less time because the questions were more readily understood by the respondent without the need for further clarification from the interviewer. The cognitive testing also stressed the need to customize the wording of some ques- tions to the role played by the respondent in managing the household’s finances, since the object of study was how people deal with the resources for which they are responsible. For example, people who are only responsible for their own expenses were asked whether they themselves had money left over after they had paid for necessary items, while respondents managing their household’s resources were asked whether their household had any money left over. The original questionnaire was revised extensively as a result of this process, demon- strating the importance of this step. 2.6.4 Pilot testing After the two rounds of in-depth interviews, the survey questionnaire was final- ized and sent to the country teams for testing in pilot surveys before the full-scale data collection. In each country, between 100 and 200 people were interviewed for the pilot phase. The collected data were then analyzed by the project team. A few changes were implemented in the final questionnaire as a result of the analysis, mostly aimed at improving the selection of the proper version of the questionnaire (depending on the resources for which the individual was responsible) especially for young respondents, adding checks on key demographic information about the respondent, clarifying/adding precodes, and simplifying the sequence of questions about household income. 2.6.5 Key questionnaire features The full survey instrument was composed of three parts: (1) the main questionnaire, to be completed by the respondent randomly selected within the household; (2) a questionnaire to be completed by one knowledgeable person for each enumeration area (the “location questionnaire�); and (3) a questionnaire completed by each inter- viewer (the “interviewer questionnaire�). The main questionnaire was divided into several sections with various purposes, as shown in table 2.1. 50 CHAPTER 2. Measuring financial capability in low- and middle-income countries Table 2.1  Content of the main questionnaire Section Purpose Section R: ƒƒProvide information on the type of household the respondent lives in to provide data that will allow various Household typologies based on demographic composition, education levels, activity levels, and decision making. This Roster introductory section is answered by a responsible adult available in the household. After information is col- lected about all household members, the respondent for the rest of the interview is randomly selected from the eligible members. This selection process is necessary whenever the sampling frame is based on dwell- ings/households instead of individuals. ƒƒProvide the criteria for determining which household members are eligible to be respondents for the survey; the number of eligible people is then used in the Kish table to generate a random selection of the respondent. All members aged 18 or older are considered eligible and proceed to further screening in Section A. Section A: ƒƒCollect information directly from the respondent about her role in managing money and making financial Role in decisions. managing ƒƒIdentify individuals who really play no role in financial or spending decisions: people who are not responsible money for planning the household’s expenses, for ensuring bills are paid, or for making any type of financial decision, and who are also not responsible for their own spending were not interviewed as there are no means to measure their financial capability. It should be noted that this further eligibility requirement was very diffi- cult to implement in practice in a rigorous way, and it generated some serious sampling issues (discussed in Kempson, Perotti, and Scott 2013a). The manual accompanying the questionnaire suggests that future sur- veys of this kind should simply aim to achieve a sample that is representative of the adult population, among which there will be a fraction of people for whom most of the financial capability questions would not apply (Kempson, Perotti, and Scott 2013b). ƒƒPeople who have some role in managing the household’s money or in making financial decisions for the household are directed to one version of the questionnaire. People who do not have any role in the house- hold’s financial decisions but who are responsible for their own spending are directed to an alternative version of the questionnaire. The two versions have very minor differences, mostly in wording and in the examples, to make sure that the questions are relevant for the person’s particular circumstances. Section B: ƒƒUnderstand how people manage their day-to-day money. This includes planning spending, spending on food Day-to-day and other necessary items, keeping track of spending, borrowing, and generally managing money. money management Section C: ƒƒUnderstand whether and how people plan for future expenditures, including for: known expenditures, unex- Planning pected expenditures or emergencies, old age, and/or for their children. Section D: ƒƒUnderstand how people choose financial products: whether they check the features, terms, and conditions Financial before buying financial products; whether they look for information before buying products; and whether they products seek advice or information before making financial decisions. ƒƒObtain a broad indication of the level of financial inclusion of the individual by asking which financial products the respondent holds. Section E: ƒƒCapture underlying motivations that influence the way people behave. This includes questions on attitude Motivations toward the future, impulsivity, and action orientation. Section F: ƒƒObtain information on the variations in income that the individual respondent and her household face. To do Sources of this, information is collected for all sources of income and then on how total income varies throughout the income year. Section G: ƒƒUnderstand whether the respondent seeks information or advice before making important financial decisions. General ƒƒUnderstand if the respondent would like to have more information about general aspects of money manage- questions ment discussed during the interview, and if so, what type of information he or she would like to have. 51 Financial capability in low- and middle-income countries: measurement and evaluation Table 2.2 presents the mapping of the concepts that emerged from the focus groups, the number of groups that mentioned them, and the identification numbers of related questions included in the final instrument. In addition to measuring concepts mentioned in the focus groups, the questionnaire collected information on other aspects considered relevant for understanding finan- cial capability; for example, the type of financial products held, the role the individual has in making financial decisions, and the sources and variability of individual and household income. As previously noted, some sections (B, C, and D) had two different versions: one for people asked about the money they manage both personally and for the household (printed on white pages), and one for people asked only about the management of their own personal money (printed on green pages). Finally, optional modules were available for interested countries to add to the main questionnaire. These included questions on: financial literacy (knowledge), banking, financial inclusion, credit cards, remittances, financial intermediaries, and consumer protection. These sections were taken from existing surveys and were not subject to the same rigorous testing as the core questions on financial capability or the explan- atory variable questions. The questionnaire developed by the RTF was designed to be used in face-to-face, paper-and-pencil interviews. When possible, however, computer-assisted personal interviews (CAPI) are very helpful to improve data quality, because quality checks can be embedded in the CAPI software to prompt the interviewer when there are incon- sistencies in the responses to different questions, or to ensure that answers are in the appropriate range. Due to the cost of the hardware and in some cases to safety considerations, CAPIs were conducted only in Mexico and Uruguay. The questionnaire used to interview members of the public was supplemented by two further questionnaires to provide context information. A “location questionnaire� was completed at the enumeration area level to provide additional explanatory vari- ables about living standards and available infrastructure in the area. The nature of the information collected through this section was inevitably somewhat subjective, but supplemented nationally collected statistical data on the survey respondents’ localities, which are often not available at such a fine level of detail. An “interviewer questionnaire� collected information about basic sociodemographic characteristics of the interviewers, their professional experience, and their degree of comfort with administering the survey. These data, appropriately matched to the main data set, were useful in analyzing and controlling for nonresponse and other data quality issues. 52 CHAPTER 2. Measuring financial capability in low- and middle-income countries Table 2.2 Mapping of capability concepts with specific questions No. of groups Focus Group concept mentioning Question(s) Day-to-day money management Plans spending against income and sticks to it 54/58 B1–B5 Prioritizes spending on essentials 53/58 B27 Self-disciplined/doesn’t “waste� money versus spends impulsively 51/58 B23–B24 (and B26–B27) or to impress others B28 E7–E11 Lives within means versus runs short of money or has to borrow 45/58 B6–B7 for essentials B9–B10 B13–B16 Tries to save when can/puts money aside at end of budgeting 31/58 C25–C30 cycle Keeps track of money and spending/knows how much money has 21/58 B17–B20 Economizes/knows how to make the most of their money 26/58 Unable to design a suitable question Puts others needs before own 20/58 E31–E34 Maximizing income 10/58 Not included in core because not income-neutral and only applies to those on low or inadequate incomes Planning for the future Thinks and plans ahead versus living for today 50/58 E1–E6 Saves/plans for unexpected expenses or events 42/58 C6–C9 Saves/plans for expected or known expenditure in the future 41/58 C2–C5 Enterprising/focuses on self-improvement 41/58 E10–E12 Saves whenever can 40/58 C25–C30 Puts money aside for regular commitments 25/58 Decided to drop after wave 1 because some people don’t have regular bills to pay Plans/makes provision for children’s future 35/58 C22–C24 Plans/makes provision for old age 21/58 C11–C21 Invests in business 13/58 Not included as only applies to those who are self-employed Selecting and using financial products Doesn’t borrow more than can afford 30/58 B21–B22 Seeks out information before deciding on products will buy/use 28/58 D5–D7 Checks product features before selecting/buying 28/58 D8–D9 Keeps money in an account for safe-keeping 16/58 Other Seeks information/advice before making financial decisions 30/58 G1–G2 Able to distinguish between reliable and unreliable information 16/58 Not possible to design a Q that would work across countries Learns from/listens to others 10/58 B17 Note: The table reports information collected from the focus groups received by the established deadline (58 out of 70). Transcripts received after the deadline were also reviewed to ensure that no important differences were missed. 53 Financial capability in low- and middle-income countries: measurement and evaluation 2.6.6 National surveys After the pilot surveys and following new debriefing sessions with the country teams and preliminary data analysis conducted by the project team and Technical Advisory Team, minor adjustments were made to the questionnaire to further simplify the wording and improve the question flow. The research teams of the participating countries developed their own sampling strategies, in line with the following broad recommendations provided by the RTF: 1. The survey is administered to individuals. 2. The sample should be representative of the national adult population (aged 18+). To be eligible for the interview, the respondent must participate in the household’s financial or spending decisions and/or be at least partly respon- sible for his own spending.5 3. Probability sampling must be used and related standard techniques should be used to determine sample size. 4. Both urban areas and rural areas should be domains of study. 5. It is generally assumed that the sampling frame is a list of dwellings (a sampling frame of individuals was not available in any of the participating countries) and that individuals will need to be selected for interview. 6. The individual to be interviewed must be randomly selected within the house- hold that is contacted using a Kish table, which provides a method by which each eligible person in a household has an equal probability of selection into the survey sample. The initial questions of the survey were designed to deter- mine whether the respondent has any role in managing money and is therefore eligible for the interview. 7. The definition of household is country-specific, in line with that used by national statistical offices. Details on sampling and other aspects of survey implementation are described in the project report (Kempson, Perotti, and Scott 2013a; also available on the RTF website www.finlitedu.org). 5  As noted earlier, this further selection step for determining eligibility is not recommended. It is instead suggested that the people who do not have any role in managing either the household’s or their own money be asked the questions in Sections E and F. 54 CHAPTER 2. Measuring financial capability in low- and middle-income countries 2.6.7 Factor analysis: constructing components Questions were designed to measure these manifestations of financial capability and the collected data were then reduced to construct measures for the key components of financial capability that broadly corresponded to the manifestations (or groups of similar manifestations) identified in the focus groups using factor analysis. The key advantages of this method are that questions measuring the same underlying compo- nents are identified empirically and that the relative importance of the questions in determining a score for the component are not determined in advance, but are calcu- lated through the empirical analysis. This is explained in greater detail in appendix B. A number of key questions about the relationship between these components and the underlying, unobserved abilities remain to be answered through empirical anal- ysis, including: „„ How are the components of capability related to one another? „„ Is it possible to trace these components back to a few key underlying concepts (the unobserved abilities or domains of financial capability)? How many? „„ Ultimately, is it possible to trace these components back to a single underlying ability and to construct a single score for financial capability? A similar process, using factor analysis, can be used to determine whether the identified components can themselves be combined in a smaller number of under- lying domains and, where they can, to calculate scores for that domain. This is also described in greater detail in appendix B. Figure 2.1 illustrates these two key steps. The factor analysis was conducted by the RTF Secretariat supported by the Technical Advisory Team. The data were analyzed separately both by country and by pooling countries together to compare results and identify differences and similarities. It proved possible to identify 12 robust components that corresponded to the manifestations identified in the focus groups. Ten of these applied to the whole population: budgeting (planning spending and keeping to it); not overspending (encompassing prioritizing spending on essentials and not wasting money); living within one’s means and not borrowing for essentials; keeping track of money; saving money when possible; covering unexpected expenses; gathering informa- tion (including learning from others and gathering information before making major financial decisions) and three more general motivations—not being impulsive, not focusing on the present, and being achievement-oriented. A further two applied to only part of the population: planning for one’s old age (only people aged under 60); and choosing appropriate financial products, which encompasses shopping around and checking the features of the product bought, and only applied to people who were active product purchasers. 55 Financial capability in low- and middle-income countries: measurement and evaluation Figure 2.1  Factor analysis associates variables with components and domains Variables Components Domains Planning: frequency Planning: accuracy Budgeting Planning: keeping to the plan Controlled budgeting Buying unnecessary items: frequency Not overspending Buying unaffordable items: frequency Moreover, each of these 12 components could be scored from 0 (least capable) to 100 (most capable) so that it was possible to identify areas where financial capa- bility was lowest and highest. This scoring found that across all countries, people are better at living within their means and not overspending than they are at planning their spending, keeping track of their finances, or saving. They also tended to have short time horizons, being more focused on the present rather than the future. Both the components themselves and the scores derived for them proved to be robust across countries. This meant that scores could be directly compared across countries in a meaningful way. On the whole, the lower the scores were in the overall analysis, the greater the variability across countries. So, for example, in budgeting (where average scores tended to be low), the population of Colombia had the highest average scores (79.54) while the population of Lebanon had the lowest (39.51). For living within one’s means (where average scores were high), the highest average scores were found in Lebanon (81.56) and Uruguay (81.28) while the lowest ones were in Turkey (67.85) and Armenia (67.59). Full details of this analysis are in appendix B. Along these estimated components scores the countries can be compared and assessed and a first impression on the distribution of the capability received. Figure 2.2 presents such a comparison, and orders individuals by component scores in a cumulative distribution function: the more upward concentrated the distribution, the more capable the population in this component (and vice versa). The results 56 CHAPTER 2. Measuring financial capability in low- and middle-income countries show quite some differences across components and between countries but no dominance of one country in all components; the comparative positions of countries shift between the components. With repeated country surveys, such a comparison can also be done for one country over time to identify changes related to progress in policy interventions and areas where still more needs to be done. After calculating and comparing scores across countries, it was possible to identify the types of people most strongly associated with high scores for each component. This was done first across all countries and then for each country in turn, using regression analysis to examine the extent to which respondents’ individual charac- teristics related to the component being studied, holding constant the influence of all other characteristics included in the analysis. For example, in the pooled data for all the participating countries, higher scores for budgeting were associated with: being a woman; having a high level of education; living with a partner and having depen- dent children; having a low income; and being retired. Full details of this analysis are contained in appendix B. The underlying regressions of the component scores allow also an identification of key drivers of the scores (country-specific scores’ regressions are presented in appendix B). The pooled regression results across all countries are presented in table 2.3 and offer first insights about a possible interpretation. While more time and analysis will need to be spent on this, the results suggest the following: „„ The components in table 2.3 show main differences with regard to overall fit as well as the number of significant parameters. Two sets of components have a better overall fit as measured by the adjusted R2 and the number of signifi- cant parameters: –– Set (a): The components budgeting, not overspending, and achievement orientation, with R2 between 0.128 to 0.181; and –– Set (b): The components saving, attitude toward the future, and living within means, with R2 between 0.131 and 0.171. „„ Related to both sets are further components with lower overall fit and less significant parameters—monitoring expenses, covering unexpected expenses, not-impulsive, choosing products, and using information have a much lower fit (with R2 between 0.05 and 0.09) and fewer significant parameters. „„ The country dummies (with Mexico as the benchmark) are mostly highly signif- icant and exhibit major size differences between countries in both directions. „„ As regards the explanatory variables, a number of interesting results emerge that give rise to ad hoc hypotheses to be elaborated and tested in future studies. The most striking ones are: 57 Financial capability in low- and middle-income countries: measurement and evaluation Figure 2.2 cumulative distribution of component scores across pilot countries (continued) 58 CHAPTER 2. Measuring financial capability in low- and middle-income countries Figure 2.2 cumulative distribution of component scores across pilot countries (continued) 59 Financial capability in low- and middle-income countries: measurement and evaluation Table 2.3 Results of pooled regression of component scores across pilot countries Living Covering Within Monitoring Using Infor- Not Over- Unexpected Variable Budgeting Means expenses mation spending Expenses Female 6.263*** −0.527 0.95 2.402** 1.009 0.642 Age 18−30 −4.143** −0.014 −2.256 −2.374* −7.837*** −0.647 Age 31−40 −1.837 −0.252 −0.039 0.036 −2.758** 0.323 Age 41−50 (baseline) Age 51−60 1.741 1.156 2.624 −1.113 2.004 1.108 Age 60+ −2.722 4.095*** 1.764 −2.414* 4.15** 1.481 Primary education at most −1.888 −0.966 −3.601** 1.067 4.365*** −1.869 Secondary education (baseline) Tertiary education 4.13*** 4.187*** 0.401 −1.427 −3.041 0.317 # household members 18+ −0.043 −1.314*** −2.551*** −0.012 0.309 −0.692 Living with a partner 2.155* 0.42 −1.405 3.966*** 1.142 0.589 Has dependent children 3.188** −4.064*** 2.306 0.214 2.405** −2.58* Income group 1 2.471* −4.658*** 0.163 0.67 5.095*** 0.087 Income group 2 (baseline) Income group 3 0.017 4.179*** 3.375* −0.149 −1.704 5.889*** Income group 4 −3.138 7.496*** 4.839** −2.196 −4.712** 11.956*** Has financial products 3.687*** −9.6*** 4.225*** 4.124*** −2.494** 0.803 E1: formal employee (baseline) E2: informal employee −3.36 −4.3*** −1.731 −2.699* 1.714 −3.992* E3: self-employed −4.486* −3.1** −5.177** −0.742 −0.941 2.071 E4+5: unemployed −4.316 −6.393*** −3.726 −0.678 0.924 −0.289 E6: student −5.91 3.926* 2.729 2.231 −2.99 −0.778 E7: retired 4.7* −3.503** 1.904 0.926 4.241** −2.32 E8: sick 0.357 −7.729** −1.983 −1.736 8.667** −7.636 E9: housework −2.447 −0.63 −1.383 0.28 3.242** −0.291 E10: other −3.532* −2.807** −1.786 −0.68 −0.333 2.465 Responsible for day to day 2.713 −0.259 5.492*** 1.001 −0.028 0.048 Responsible for planning 3.029* 0.151 1.744 5.285*** 2.776* 1.931 Armenia 17.654*** −7.315*** 21.491*** −1.061 13.907*** 2.063 Colombia 24.501*** −2.004** −3.849** 7.613*** 7.63*** −5.012*** Lebanon −14.282*** 0.969 2.633 −1.128 2.085 7.152*** Mexico (baseline) Turkey 8.016*** −9.077*** 10.253*** −4.173** 1.58 1.021 Uruguay 12.31*** 4.598*** 1.869 1.782 13.45*** −10.529*** Constant 43.046*** 90.141*** 39.097*** 61.789*** 63.892*** 62.725*** Adjusted R 2 0.181 0.171 0.091 0.067 0.159 0.053 N 9,971 9,893 9,966 9,741 9,995 9,873 (continued) 60 CHAPTER 2. Measuring financial capability in low- and middle-income countries Table 2.3 Results of pooled regression of component scores across pilot countries (continued) Attitude Toward the Not being Choosing Achievement Variable Saving Future impulsive product Orientation Female 2.988** −0.875 0.626 −0.380 −0.32 Age 18−30 3.675** 1.06 −5.126*** −3.479 3.273*** Age 31−40 0.521 2.083 −1.558 0.998 1.865* Age 41−50 (baseline) Age 51−60 −2.581 −1.287 −0.198 −4.640 −2.577** Age 60+ −1.78 −3.444* 3.096* −7.118* −8.47*** Primary education at most −4.578*** −6.177*** −3.1** −2.928 0.482 Secondary education (baseline) Tertiary education 0.546 7.732*** 3.303** 4.234* 0.619 # household members 18+ −1.29** −0.832* −0.761* 0.951 0.807*** Living with a partner 4.743*** 3.346*** 3.391*** 3.486* 1.229 Has dependent children 0.448 −1.089 0.686 −3.211 2.618*** Income group 1 −5.322*** −3.5*** −1.021 −3.018 −0.107 Income group 2 (baseline) Income group 3 4.825*** 3.912** 2.72* 4.779* −0.153 Income group 4 10.112*** 2.736 0.651 12.569*** −0.343 Has financial products 6.162*** 0.628 0.982 3.77*** E1: formal employee (baseline) E2: informal employee −7.035*** −5.085** −5.993*** −7.703** −1.853* E3: self-employed 1.892 −0.18 −1.506 2.106 0.541 E4+5: unemployed −7.299*** −2.426 −1.665 −7.139* −1.803 E6: student −7.863* −2.483 −4.778 0.341 2.505 E7: retired −3.148 0.937 −1.078 −2.592 −5.523*** E8: sick −12.642** −5.815 −5.627 −18.633* −14.411*** E9: housework −4.986** −2.636* −0.696 −2.119 −5.943*** E10: other −1.366 −0.453 −1.576 −1.798 −0.248 Responsible for day to day −1.725 2.222 2.69* 2.559 1.617 Responsible for planning 1.912 0.032 0.703 3.807 1.237 Armenia −8.288*** −7.2*** −2.445 1.262 2.093* Colombia −10.06*** 2.539* −0.823 −2.669 6.692*** Lebanon −18.996*** 16.76*** 12.313*** −3.628 −1.215 Mexico (baseline) Turkey −30.522*** 10.357*** 4.716** −5.816* −4.861*** Uruguay −16.117*** 2.366 −0.655 −7.318** 0.467 Constant 54.875*** 36.889*** 59.191*** 51.061*** 78.241*** Adjusted R2 0.148 0.131 0.063 0.055 0.128 N 9,929 9,822 9,880 5,367 9,890 61 Financial capability in low- and middle-income countries: measurement and evaluation –– Gender: Compared to the benchmark (male), there are three components with a positive and significant “female� parameter: Budgeting, Saving and Using Information. This is consistent with the general view that women are often in charge of budgeting and saving decisions. –– Age: There is quite some heterogeneity across the age groups and compo- nents. Interesting is the difference in parameter sign between sets (a) and (b) for age 60+: it is positive for set (a) and negative for set (b). This is consistent with the main preoccupation of this age group; these people stay within their resource constraint as they have few alternatives, and saving for the future is not a priority anymore. –– Education: The heterogeneity of results does not lend itself to an easy interpretation. That tertiary (compared to benchmark of secondary) educa- tion has a positive and significant parameter seems intuitive and emerges for Budgeting and Living within Means. Why the parameter is negative for Not-Overspending (with a similar but positive parameter for primary education) requires an explanation. One possibility is that higher education is associated with higher individual income (which is not included in the model specification), and higher income could lead to higher overspending (in fact, results here show that higher household income is associated with more overspending). The negative parameter for primary education on Saving and Attitude Toward the Future seems again intuitive and mirrors the positive parameter for tertiary education on Attitude Toward the Future. –– Income groups: The heterogeneity of the parameters across income groups and components has a number of intuitive interpretations, but not for all groups. The positive parameter for Budgeting for the lowest income group suggests that those with little means need to control their budget (and cannot overspend), while the negative parameter for the highest income group suggests that the reverse seems to be the case. Compared to that, the parameters for income groups have the opposite signs for Living within Means: they are negative for the lowest income group and positive for the highest. This can be interpreted that while the lowest income group attempts to budget and does not overspend (that is, they do not spend on non-essentials and they do not buy things they cannot afford), they do not succeed in living within their means, while this is possible for the highest income group. The interpretation of the Saving and Attitude parameter is similar: it is negative for the lowest income group and positive for the highest. –– Employment status: The signs of the coefficients for employment statuses other than “formal employee� make overall sense. The coeffi- 62 CHAPTER 2. Measuring financial capability in low- and middle-income countries cients are negative and at times significant for the unemployed, as they are for informal employees and those self-employed, interestingly. The similarity between these groups is not surprising, however, as the differ- ence between an informal employee and one self-employed in LICs and MICs is not great. „„ Not included in the pooled results is the relationship between financial capa- bility scores and financial literacy (as measured by ability to perform a simple division, knowledge of inflation, interest, and compounded interest rate). As not all countries asked the same questions, the results are only available at the country level (see appendix B). These results suggest that the level of financial literacy: –– Has little to do with capability components in set (a) (budgeting, not over- spending, and achievement orientation) and this is reflected in the erratic results across countries; and –– Is closely—and positively—linked with capability components in set (b) (saving, attitude toward the future, and living within one’s means) and hence almost consistent results are observed across countries. Overall, these results are broadly consistent with an emerging conjecture: daily money management and intertemporal decisions are seemingly two different animals. 2.6.8 Factor analysis: identifying domains After identifying and exploring the individual components of financial capability, a second round of factor analysis was conducted to see if these components could be combined into a smaller number of domains (or even a single overall domain). This was done first for the pooled data set for all six countries for which the same components were available (leaving Nigeria out of the analysis)and then for each country in turn. (Appendix B contains more details on both the methods and findings of this analysis.) The analysis found that it was not possible to construct a single domain (and there- fore a single score) combining the 10 components that applied across the whole population. It was, however, possible to identify two underlying domains: „„ Controlled budgeting, which comprises planning spending, monitoring one’s finances, using information, not overspending, and (very weakly) being achievement-oriented „„ Making provisions for the future, which includes living within one’s means, saving whenever possible, being able to cover unexpected expenses, being future-oriented, and not being impulsive 63 Financial capability in low- and middle-income countries: measurement and evaluation (Note that making provisions for one’s old age and choosing appropriate financial products were not included in this analysis as they only applied to a subset of the population.) In other words, financial capability was found to be a composite of skills, not a single skill that could be measured with a single score. Similar analysis conducted individually for each of the participating countries found two broadly similar domains, but there were subtle differences across countries. This indicated that domain scores across countries could not be calculated and compared in a statistically robust and meaningful way. While this is a disappointing outcome, it is not altogether surprising given the diversity of the countries involved. It does, however, leave open the possibility of conducting such analysis at a country level and developing bespoke scores. The number of domains required to capture all the components of financial capability in a meaningful way will differ across countries, from two to four. 2.6.9 Cluster analysis: identifying vulnerable groups (intervention targets) Of particular interest for policy makers is identification of specific subgroups of the population that show weaknesses in one or more particular areas of financial capability. If the problem turns out to be concentrated in one specific group of the population (for example, people under 30, or young parents on low incomes, etc.), it will be easier for policy makers to design a more tailored intervention to improve their capability. Cluster analysis was used by the project team to segment the population into groups with comparable levels of capability. This iterative procedure aggregates into groups individuals that have shared attributes that distinguish them from others in the population (in this case, similar financial capability strengths and weaknesses). This approach was used, for example, in the United Kingdom and the Netherlands, and is described in more detail in appendix B. Since it was not possible to identify domains that could be applied across all coun- tries, the cluster analysis was conducted using the 10 components that applied to the whole population. The analysis identified five broad groups (or clusters) in the population across each of the seven countries; in each case, these ranged from a group with low scores across most (if not all) of the components to a group with high scores across most components. In between, in each country there were three further groups, each with some areas of weakness and strength. Having done this, it was then possible to identify the average personal characteristics of the individuals within each group using regression analysis. Results for Mexico are presented as an example. Results for the other six countries are included in 64 CHAPTER 2. Measuring financial capability in low- and middle-income countries appendix B. Table 2.4 shows the average characteristics of the five clusters identified in Mexico. Around one in five in the Mexican population can best be described as unsophisti- cated money managers. They were the group with the lowest average scores and had below average scores on all but two components, but they were not inclined to overspending and they lived within their means. Their scores for planning how their money would be spent (budgeting) and monitoring their finances were very low. So, too, were their scores for saving and for choosing products (among those that had them) and they also had the lowest levels of financial inclusion. Compared with the rest of the population, this group was more likely to have a low income (83 percent of them were in the two lowest income groups). Their incomes were also variable and a quarter of them were working in the informal economy (19 percent) or were self-employed (5 percent). They had low levels of education, with almost half (47 percent) having only received primary education, and they had the lowest financial literacy scores of the five groups. Despite including above average numbers of people aged over 60 (18 percent), they had the lowest proportion of people describing themselves as retired (3 percent), suggesting that many were continuing to work in old age due to lack of financial provision. They included the second highest proportion of people living in a rural area, but even so almost 6 in 10 of them (56 percent) lived in towns and cities. This group would almost certainly be a focus of any strategy to raise levels of financial capability. However, given their low incomes, it may be that many of them were living day-to-day with little scope for saving and making provision for the future. Similarly, since they were living within their means and not overspending, it could be that their very low incomes meant that they could keep control over their finances without planning or monitoring expenditure. Their low scores on choosing financial products coupled with a high level of financial exclusion similarly suggest that they largely managed their money in cash and would be vulnerable consumers of financial products and services. With so many areas of weakness, almost certainly more than one intervention would be required and prior- ities would need to be set for the most pressing areas to be tackled. Moreover, given the incidence of informal and self-employment, it is almost certain that their needs would span both their “business� and personal finances and that, in all likelihood, these would not be separated. Their low scores for seeking information and low levels of levels of education suggest that it is unlikely that they would be reached by classroom-based financial education initiatives. The short-term money managers (cluster 2) were particularly strong at aspects of day-to-day money management, and budgeting in particular; but they were poor at saving and making provision for the future, particularly for unexpected expenses. They also had the second lowest score for choosing financial products, as only about a half of them (49 percent) were financially included. As a group, these people had 65 Financial capability in low- and middle-income countries: measurement and evaluation Table 2.4 Mexico: average characteristics of the five clusters Variable CLUSTER 1 CLUSTER 2 CLUSTER 3 CLUSTER 4 CLUSTER 5 Total Budgeting 1.01 73.62 74.89 2.63 75.59 51.86 Living within means 76.41 77.63 67.28 85.56 82.93 78.05 Monitoring expenses 24.88 34.23 62.29 51.23 40.14 38.78 Using information 60.81 70.50 71.92 76.37 79.23 71.25 Not overspending 71.62 72.32 47.26 69.53 75.89 69.57 Covering unexpected expenses 49.04 38.13 87.12 87.44 86.92 63.53 Saving 33.95 44.99 56.14 78.42 77.20 55.05 Attitude toward the future 30.92 33.09 30.68 36.39 40.26 34.39 Not being impulsive 52.23 61.15 40.42 70.18 69.25 59.32 Achievement orientation 78.11 84.69 81.64 84.30 90.99 84.38 Choosing products 30.34 53.37 68.08 72.02 72.65 58.73 Covering old-age expenses (under 60) 54.83 62.13 73.86 54.66 74.09 64.76 Female 0.49 0.57 0.53 0.53 0.52 0.53 Age 18–30 0.32 0.28 0.37 0.26 0.26 0.29 Age 31−40 0.22 0.20 0.31 0.29 0.30 0.25 Age 41−50 (baseline) 0.16 0.22 0.17 0.18 0.21 0.19 Age 51−60 0.12 0.15 0.10 0.08 0.11 0.12 Age 60+ 0.18 0.16 0.05 0.18 0.11 0.14 Primary education at most 0.47 0.38 0.25 0.36 0.30 0.36 Secondary education (baseline) 0.49 0.56 0.66 0.59 0.54 0.55 Tertiary education 0.04 0.06 0.09 0.05 0.16 0.09 # of household members 18+ 2.87 2.91 2.68 2.66 2.81 2.82 Living with a partner 0.65 0.68 0.68 0.70 0.73 0.69 Has dependent children 0.49 0.57 0.54 0.52 0.57 0.55 Rural area 0.44 0.35 0.31 0.47 0.36 0.38 Has financial products 0.37 0.49 0.59 0.61 0.57 0.51 E1: formal employee (baseline) 0.12 0.22 0.40 0.29 0.29 0.24 E2: informal employee 0.19 0.16 0.13 0.11 0.13 0.15 E3: self-employed 0.05 0.01 0.03 0.04 0.03 0.03 E4: unemployed 0.07 0.06 0.04 0.02 0.03 0.05 E5: waiting for busy season 0.02 0.02 0.02 0.02 0.01 0.02 E6: student 0.05 0.04 0.02 0.04 0.03 0.04 E7: retired 0.03 0.05 0.02 0.04 0.05 0.04 E8: sick/disabled 0.01 0.02 0.00 0.03 0.01 0.01 E9: housework 0.30 0.29 0.22 0.27 0.26 0.27 E10: other 0.16 0.13 0.12 0.15 0.16 0.15 Financial literacy score 2.61 2.83 2.62 2.71 3.00 2.79 Responsible for day to day 0.64 0.76 0.75 0.77 0.79 0.74 Responsible for planning 0.67 0.74 0.75 0.75 0.82 0.75 Responsible for choosing financial product 0.50 0.64 0.79 0.64 0.75 0.66 Income group 1 0.47 0.45 0.29 0.32 0.24 0.37 Income group 2 (baseline) 0.34 0.35 0.32 0.28 0.36 0.34 Income group 3 0.11 0.11 0.26 0.11 0.26 0.17 Income group 4 0.08 0.10 0.13 0.29 0.14 0.13 Income seasonality: no income 0.13 0.12 0.06 0.05 0.09 0.10 Income seasonality: variable income 0.58 0.61 0.67 0.63 0.55 0.60 Income seasonality: stable income 0.29 0.27 0.27 0.32 0.36 0.30 Number of observations 441 606 279 189 507 2,022 66 CHAPTER 2. Measuring financial capability in low- and middle-income countries the second lowest incomes (80 percent were in the two lowest income quartiles). Compared with others, they were more likely to be middle-aged (between the ages of 41 and 60) and to have dependent children. On the whole, they were urban dwellers; only 3 in 10 lived in a rural area. They accounted for almost a third of the population and formed the largest of the three groups. This group would almost certainly be targeted with interventions to encourage them to plan for their finan- cial needs, but their low levels of financial inclusion and vulnerability with regard to choosing and using financial products and services would also need to be addressed. The third group was the young overspenders (cluster 3), whose main areas of weakness were a strong tendency to impulsivity and overspending; perhaps as a consequence, they also had the lowest scores for living within their means. On the other hand, they were strong in money management (both budgeting and monitoring their finances) and financial planning (both for unexpected expenses and for their old age). They formed one of the two smallest groups, and accounted for just over 1 in 10 of the population. Compared with the rest of the Mexican population, they were relatively young and reasonably well-educated. They had the highest level of formal employment and incomes that were slightly higher than average. This group would almost certainly be targeted with interventions designed to help them curb their tendency to overspend and rely on credit to make ends meet. They would be particu- larly vulnerable to a change in their circumstances leading to a reduction of income. A fourth group can, perhaps, best be described as affluent but disorganized (cluster 4). They were very poor budgeters; indeed most of them made no attempt to plan how their money would be used. They also had the lowest score for planning for their old age. That said, they did live with their means and were particularly strong at some aspects of planning for the future, including saving when they could and making provision for unexpected expenses. They were also good at choosing finan- cial products and had the highest level of financial inclusion. This was the smallest group of all, accounting for less than 1 in 10 of the Mexican population. They had high incomes compared with the rest of the population—which almost certainly explains why they were able to live within their means without budgeting and plan- ning their expenditure. They were the group that was most likely to live in rural areas (47 percent) and to have been educated to secondary level. It is unlikely that they would be a high priority for interventions to improve their budgeting skills. On the other hand, given their high incomes, it is likely that they would be a target group for interventions designed to increase saving toward old age. Finally, there was a group of people (cluster 5), representing a quarter of the popu- lation, who were careful money managers and planners. They had the highest average capability scores and had above average scores for everything except moni- toring their finances. That said, in common with the rest of the Mexican population, 67 Financial capability in low- and middle-income countries: measurement and evaluation they had low scores for both monitoring their finances and having a short-term time horizon with regard to money. Compared with the rest of the population, this group was more likely to have steady incomes in the middle of the income distribution, was well educated (16 percent were educated to tertiary level), and had the highest scores for financial literacy. It is unlikely that there would be a need to target this group with interventions to raise their financial capability. The fact that they did not monitor their spending (yet managed to live within their means) suggests that it was not necessary for them to do so. Although this segmentation was based on five broad groups, cluster analysis would allow these groups to be subdivided if a more fine-tuned breakdown were required for policy purposes. 2.7 Main outputs, lessons learned, and next steps The generosity of the RTF allowed for application of very rigorous sets of procedures to both develop a method of assessing levels of financial capability in a population through a national survey and to analyze the data collected. Rarely is such rigor possible. Through this process, a range of outputs have been developed that are freely available for others to use, a number of important lessons have been learned, key analytical insights derived, and possible next steps developed. More technical details on output, the many detailed lessons and the resulting methodological suggestions are provided in Kempson, Perotti, and Scott (2013a). The key outputs produced by the RTF are freely available and include: „„ An extensively tested questionnaire that has been shown to be relevant across a wide range of LICs and MICs; „„ Supplementary questionnaires that collect contextual information about the localities in which survey respondents live and about the interviewers; „„ Guidance on conducting the survey and a set of interviewer instructions to accompany the associated questionnaire (Kempson, Perotti, and Scott 2013b); „„ Documentation on the qualitative research phase (a focus group topic guide, in-depth interview guides, and feedback forms and instructions provided to research teams and interviewers); „„ Survey data sets from seven countries and related documentation; and „„ A full report on both the qualitative and quantitative research conducted to develop the survey methodology, process of implementation, and survey results (Kempson, Perotti, and Scott 2013a). 68 CHAPTER 2. Measuring financial capability in low- and middle-income countries The final version of all these materials is available on RTF’s website (www.finlitedu.org). The main lessons learned include: „„ Perhaps most important, it is possible to identify, through a vox populi approach, a range of manifestations of financial capability that apply across very diverse LICs and MICs, from Papua New Guinea to Mexico. Moreover, these manifestations resonate with the findings among low-income groups in HICs such as the United Kingdom. These manifestations are primarily related to behavior, with motivations being important in ensuring income neutrality. „„ It is possible to design a questionnaire that works across different income groups and quite different cultures to capture their manifestations pretty accurately without introducing a distorting income or culture bias. „„ It is possible to create scores for individual components of financial capa- bility (mirroring the manifestations identified) that are robust and meaningful across different countries. Along these components’ scores, the countries can be compared and assessed and a first impression on the distribution of the capability received. The results indicate differences across components and countries, with some components having high scores for a large share of the population (such as living within means) while the scores of others compo- nents are more concentrated in the population (such as saving). With repeated future country surveys, such comparison may also be done over time to identify changes related to the effectiveness of interventions and strategies. Furthermore, the score distribution indicates that each country has areas of weakness and strength, and no country has dominating scores across all components. „„ It is not statistically meaningful to collapse these components into a single score for the “overall level of financial capability.� It is possible to identify two broad domains using the pooled data, designated “controlled budgeting� and “making provision for the future.� Cross-country comparisons at this level are not statistically robust, as there are subtle differences across countries in the number and composition of these domains. Such analysis can, however, be conducted at the individual country level, although the number of domains used to capture all the components of financial capability will differ, from two to four. Despite some robustness reservations with regard to country comparability for the broad domains “controlled budgeting� and “making provision,� such a comparison is undertaken in figures 2.3 and 2.4 to compare the cumulative distributions for coun- tries. These results confirm that a larger share of the population has high scores for 69 Financial capability in low- and middle-income countries: measurement and evaluation Figure 2.3  Cumulative distribution of domain “controlled budget� Figure 2.4  Cumulative distribution of domain “making provisions for the future� 70 CHAPTER 2. Measuring financial capability in low- and middle-income countries budget-related components than for provision-related components across all coun- tries. As regards the relative position of countries, the change is between the broad domains. For example, Lebanon emerges as a laggard for “controlled budgeting� and as a leader for “making provision.� Populations of individual countries can be segmented into groups with varying levels of capability across all 10 components. The strengths and weaknesses of these groups can be determined—as can their characteristics. These groups can be as fine- tuned as is required to inform approaches for increasing levels of financial capability. Some of the components identified can be tackled through education; for example, learning how to plan spending or to monitor finances. Others will require other types of interventions, as outlined in chapter 3. The results from the survey show that across all countries, people are better at living within their means and not overspending than they are at planning their spending, keeping track of their finances, or saving. They also tend to have short time horizons, being more focused on the present than the future. On the whole, the lower the scores were in the overall analysis, the greater the variability across countries. A few of the many analytical insights emerging from the project with regard to defining and measuring financial capability in LIC and MICs and that will require further deepening by using the current data and that of new country pilots include: 1. Financial capability was not perceived by focus groups participants as related with income or education. However, income clearly affected behavior and therefore attitude and motivations had an important role in distinguishing who is capable and who is not among lower income groups. 2. As the effect of income could not be completely excluded from the measures of capability; members of higher-income households were found to be more capable in saving, planning for unexpected expenses, and choosing products. 3. Even with the limitations implied by some sampling issues, the survey results suggest that a very large majority of adults contribute to household financial decisions or are at least responsible for their own expenses. 4. The proportion of adults involved in the household’s financial decisions appeared to be smaller in three-generation households compared to nuclear households, and in households where the head was older or a male. 5. Members of larger households, with dependent children particularly, seem to have less financial control and have lower scores for living within their means and covering unexpected expenses. 71 Financial capability in low- and middle-income countries: measurement and evaluation 6. In terms of differences across demographic groups, women were found to have higher levels of capability than men in budgeting and saving (and using information, when analyzing all countries together); young individuals were more achievement-oriented but more inclined to overspend and be impulsive, and had lower budgeting skills. Higher education was in general found to be associated with higher capability, except for overspending. 7. The link between financial inclusion and financial capability was not easy to interpret, but, for example, it seemed to be rather weak in areas of capability like budgeting and monitoring expenses. 8. Financial knowledge (literacy) did not have a clear link with the financial capa- bility scores across countries: while in most countries there was a positive correlation between financial literacy and the scores for saving, using informa- tion, and choosing financial products, most countries did not show any signifi- cant relationship in the other areas such as budgeting. The results of the comprehensive RTF project have advanced the measurement agenda of financial capability within a short time span by an enormous distance. Much has been achieved but as the lessons and analytical insights highlight, much more needs to be accomplished. This section ends with a few suggestions for priority steps in the measurement agenda, such as: „„ Disseminating the current results to countries and the research community worldwide and making them aware of the available survey instruments to be applied and the existing survey data to be exploited. „„ Finding ways and means to encourage other LICs and MICs to apply the instru- ments in a scientific (i.e., controlled and recorded) manner to establish data that are comparable, and to supply these future pilot countries with the exper- tise to do so. „„ Encouraging the research community to review the approach, data, and results of the project to identify gaps and ways of improvement, and to help address the many new issues that have been raised. A more detailed discus- sion of key open issues is offered in Kempson, Perotti, and Scott (2013a). References Ambrosetti. 2008. “L’Educazione Finanziaria in Italia: La prima misurazione del livello di cultura finanziaria degli italiani.� Consorzio PattiChiari-European House-Ambrosetti. Executive summary available at http://www.ambrosetti.eu/it/download/ricerche-e- presentazioni/2008/icf-pattichiari-leducazione-finanziaria-in-italia-la-prima-misurazione- del-livello-di-cultura-finanziaria-degli-italiani. 72 CHAPTER 2. Measuring financial capability in low- and middle-income countries Atkinson, Adele, and Flore-Anne Messy. 2012. “Measuring Financial Literacy: Results of the OECD/International Network on Financial Education (INFE) Pilot Study.� OECD Working Papers on Finance, Insurance and Private Pensions, No. 15, OECD Publishing. Basic Skills Agency. 2004. Adult Financial Capability Framework. London, Basic Skills Agency. Bertrand, M., and A. Morse. 2010. “Information disclosure, cognitive biases and payday borrowing.� University of Chicago Booth School of Business Working Paper. Chicago. Calder, B.J. 1977. “Focus Groups and the Nature of Qualitative Marketing Research.� Journal of Marketing Research 14: 353–64. CentiQ. 2008. “Financial Insight among the Dutch.� Summary available at http://www. wijzeringeldzaken.nl/media/13191/summary_financial_insight_amoung_the_dutch.pdf. Chen, H, and Volpe, R.P. 1998. “An analysis of personal financial literacy among college students.� Financial Services Review 7:107–28. Churchill, G.A. 1979. “A Paradigm for Developing Better Measures of Marketing Constructs.� Journal of Marketing Research 16:64–73. Cole, S., T. Sampson, and B. Zia. 2011. “Prices or Knowledge? What Drives Demand for Financial Services in Emerging Markets?� Journal of Finance 66:1933–67. Collins, D., J. Morduch, S. Rutherford, and O. Ruthven. 2009. Portfolios of the Poor: How the World’s Poor Live on $2 a Day.� Princeton: Princeton University Press. Duflo, E., and E. Saez. 2003. “The role of information and social interactions in retirement plan decisions: Evidence from a randomized experiment.� Quarterly Journal of Economics 118: 815–42. Food and Drug Administration (FDA). 2006. “Guidance for Industry: Patient-Reported Outcome Measures.� Washington, DC: U.S. Department for Health and Human Services. Financial Regulator. 2009. “Financial Capability in Ireland Methodology Report. Report.� available at http://www.centralbank.ie/publications/Documents/Financial%20Capability%20 Methodology.pdf. FINRA. 2009. “Financial Capability in the United States.� Available at http://www.finrafoundation. org/web/groups/foundation/@foundation/documents/foundation/p120536.pdf. FSA. 2005. Measuring Financial Capability: An Exploratory Study. London: Financial Services Authority. Hogarth, J.M., S. G. Beverly, and M. Hilgert. 2003. “Household financial management: the connection between knowledge and behavior.� Federal Reserve Bulletin 89: 309–22. Karlan, D., and M. Valdivia. 2011. “Teaching entrepreneurship: Impact of business training on Microfinance clients and institutions.� Review of Economics and Statistics 93: 10–527. Kempson, Elaine, Valeria Perotti, and Kinnon Scott. 2013a. Measuring Financial Capability: A New Instrument and Results from Low- and Middle-Income Countries. Washington, DC: World Bank. —. 2013b. Measuring Financial Capability: Questionnaires and Implementation Guidance for Low- and Middle-Income Countries. Washington, DC: World Bank. Kotlikoff, L. J., and B. D. Bernheim. 2001. “Household Financial Planning and Financial Literacy.� In Essays on Saving, Bequests, Altruism, and Life-cycle Planning. Cambridge, MA: MIT Press. 73 Financial capability in low- and middle-income countries: measurement and evaluation Lusardi, A., and O. Mitchell. 2011a. “Financial Literacy and Planning: Implications for Retirement Well-being.� In O. Mitchell and A. Lusardi, eds., Financial Literacy: Implications for Retirement Security and the Financial Marketplace, 17–39. New York: Oxford University Press. —. 2011b. “Financial literacy around the world: An overview.� Journal of Pension Economics and Finance 10: 497–508. —. 2013. “The Economic Importance of Financial Literacy: Theory and Evidence.� NBER Working Paper 18952. Cambridge, MA. Mandell, L. 1997. “Personal Financial Survey of High School Seniors.� Jump Start Coalition for Personal Financial Literacy, March/April. Washington, DC. OECD. 2009. “Framework for the Development of Financial Literacy Baseline Surveys: A First International Comparative Analysis.� OECD Working Papers on Finance, Insurance and Private Pensions, No. 1, OECD Publishing. —. PISA 2012 Financial Literacy Framework. Paris, OECD. http://www.pisa.oecd.org/ dataoecd/8/43/46962580.pdf. OECD/INFE. 2011. “Measuring Financial Literacy: Core Questionnaire and Guidance Notes for Conducting an Internationally Comparable Survey of Financial Literacy.� Paris: OECD. Pett, M., N.R. Lackey, and J. J. Sullivan. 2003. Making Sense of Factor Analysis: The Use of Factor Analysis for Instrument Development in Health Care Research. Sage Publications. Spearman, C. 1904. “‘General Intelligence’ Objectively Determined and Measured.� American Journal of Psychology 16: 201–93. Statistics Canada. 2009. Canadian Financial Capability Survey (sponsored by Human Resources and Skills Development Canada, Finance Canada and the Financial Consumer Agency of Canada), documentation available at: http://www23.statcan.gc.ca/imdb/p2SV. pl?Function=getSurvey&SDDS=5159&lang=en&db=imdb&adm=8&dis=2#a4. 74 M CHAPTER 3 easuring the effectiveness of financial capability-enhancing interventions 3.1 Motivation and overview of the evaluation program Worldwide interest in financial literacy and broader concepts that are expected to improve financial outcomes has increased dramatically over recent decades, with the interest and efforts moving from developed to developing countries in the last years (as discussed and documented in the previous chapters).1 Yet despite this interest in the topic, the empirical knowledge and understanding of what may actu- ally drive the outputs of interest—financial literacy or more broadly financial capa- bilities—remain very limited. The likely main reasons for these limitations are briefly identified and enumerated: „„ The vast majority of interventions so far have been in the area of financial education with the objective of increasing knowledge and skills, and at times changing attitudes and behavior. If this cognitive approach to financial capa- bility has limitations (as discussed in chapters 1 and 2), they will also apply to the interventions designed to improve financial capability. „„ Very few financial education interventions have undergone rigorous M&E. A realistic figure would put the share of rigorously monitored and evaluated interventions in the developed world below 1 percent, and in the developing world, close to zero. „„ Alternative interventions that use the noncognitive route and non-education methods are only gradually increasing, albeit they seem to be subjected more often to rigorous M&E. But for a number of these alternative interventions, the traditional M&E approach faces methodological challenges that are not yet fully addressed. This chapter has benefited greatly from the support provided by Mattias Lundberg, Senior 1  Economist with the Human Development Network at the World Bank, and Elaine Kempson, Senior Consultant with the World Bank. 77 Financial capability in low- and middle-income countries: measurement and evaluation The limited rigorous evaluation of financial literacy/capability-oriented interventions begs the question of the reasons for such resistance, as they need to be identified to inform a strategy and an action plan to overcome them. The perceived main reasons that eventually guided the development of the World Bank’s evaluation work program under the RTF include: „„ A lack of understanding among program providers of the importance of rigorous M&E for the design, implementation, and improvement of interven- tions; „„ A lack of a toolkit for M&E that goes beyond generic instructions and addresses the concept and issues within a financial capability framework; „„ The high cost of rigorous M&E for new interventions, which often exceeds $100,000 in addition to costs related to the design and implementation of the intervention itself; „„ The public good character of rigorous M&E, which produces knowledge at no cost for worldwide public consumption, but whose cost is born by the provider of the individual intervention; and last but not least, „„ Resistance from program providers (often NGOs) to an evaluation, as it may reveal unfavorable results and/or limited impact, consequently risking funding from financiers (often financial institutions, foundations, or public bodies). Against this background, the work program of the RTF was developed to address the lack of rigorous M&E in a variety of approaches. The program also entailed the use of the results of the completed and existing evaluations in the field to better under- stand which interventions work, which don’t, and under what circumstances. The key items of the program include: „„ The development of an M&E toolkit geared toward financial capability inter- ventions, which include financial education but also any other financial capa- bility-enhancing mechanisms. The development of this toolkit was informed by a number of RTF-financed field evaluations. „„ The financing and provision of expert advice for field impact evaluations of competitively selected interventions across regions and income levels. The interventions themselves were not financed. A comprehensive review of the existing impact evaluations was initiated to identify gaps and guide the selec- tion process for interventions. „„ The financing of evaluations for a set of interventions that conducted an inte- grated and comparative testing of effectiveness, which is typically not done; 78 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions i.e., they compared the effectiveness of a specific intervention across coun- tries or alternative interventions under comparable settings in one country. „„ A set of related activities to strengthen the results, such as: advocating the importance of rigorous M&E in client countries and through the OECD Inter- national Network on Financial Education; using the World Bank research committee for review and validation of research methods for the RTF-financed field evaluations; organizing regional workshops for field projects to share experiences and lessons, including collaborations with the Sub-Saharan Africa projects financed through DIFD’s Financial Education Fund (FEF). The remainder of this chapter is structured as follows. Section 3.2 discusses the conceptual issues and options and the development of objectives that guided the RTF evaluation program, followed by a more detailed description of the chosen approach and the associated knowledge products and field evaluations in section 3.3. Section 3.4 discusses lessons learned from the process and results. Section 3.5 explores policy implications and suggests direction for future research. 3.2 Issues and options: from financial education to broader financial capability interventions In spite of growing attention by governments and policy makers on the urgency of addressing the financial capability gap among populations, there has been little concrete guidance on the design of interventions. For example, while many coun- tries around the world have developed national strategies for financial literacy and education, other than defining goals to improve literacy, they generally fall short of providing concrete instructions on the types of interventions needed to achieve results. This is primarily a consequence of inconclusive evidence in the existing liter- ature for identifying successful approaches. There could be many reasons for the paucity of evidence needed to formulate policy; however, three elements are easily identified as having contributed to this outcome. First, and as mentioned earlier, the debate around this topic in recent years has evolved conceptually as a result of limited evidence linking financial literacy to behavior change desired for good outcomes. It has shifted from an initial focus on financial literacy defined as knowledge and numeracy to a broader definition that includes skills, attitudes, and behavior (the conceptual development is discussed in more detail below). This progress has expanded the range of interventions beyond a mere focus on traditional financial education to include a variety of alternative methods that may follow different designs and theories of change. 79 Financial capability in low- and middle-income countries: measurement and evaluation Second, until more scientific research among the academic community began in recent years, the field was generally characterized by the inadequacy of ex ante and rigorous evaluations. Good evaluations need to be integrated into program design to determine causality and properly attribute outcomes. Most of the evaluations completed have been ex post, limiting the extraction of lessons. Third, even when conducted, evaluations have tended to target financial education programs and have not necessarily explored alternative methods. These factors and their implications are discussed in more detail below. 3.2.1 Conceptual development As documented by a number of studies, the earlier policy responses to help indi- viduals make better financial decisions predominantly focused on financial literacy, defined as knowledge and simple numeracy (Mandell 1997; Lusardi and Mitchell 2006), and therefore on education programs to improve literacy. This was primarily motivated by survey evidence documenting the correlation between financial literacy and household well-being. For example, lower levels of finical literacy were found to be negatively related with engagement in saving, credit, and investment (Hilgert, Hogarth and Beverly 2003), with planning for retirement (Lusardi and Mitchell 2007), borrowing at high interest rates (Stango and Zinman 2009), and with the use of informal sources of borrowing (Klapper, Lusardi, and Panos 2011). However, while some financial education programs may have been effective in improving literacy levels, their impact on altering actual behavior has been incon- clusive, if not exclusively bleak (Braunstein and Welch 2002; Cole and Shastry 2008; Gale, Harris, and Levine 2012; Mandell and Klein 2009). To investigate this phenom- enon and determine whether the relationship between financial education, literacy, and desired outcomes is in fact causal, a number of scientific experiments have been undertaken in recent years. The results of these studies have been mixed (Bertrand and Morse 2010; Cole, Sampson, and Zia 2011). For example, Duflo and Saez (2003) conducted a randomized study to measure the impact of a benefit fair on retirement plan enrollment, and found small effects on take-up. Focusing on developing coun- tries, Cole, Sampson, and Zia (2009) conducted a randomized study in Indonesia to measure the impact of financial education on savings, and also found no substantial impact. Similar findings reporting on the limitations of financial education on behav- ioral outcomes were found by Karlan and Valdivia (2011) in Peru and by McKenzie and Weber (2009) in Uganda. Moreover, recent efforts by behavioral economists in drawing lessons from psychology have been insightful in helping understand some of the barriers that may prevent people from moving from knowledge and intentions to actions. This litera- ture argues that due to a number of cognitive, emotional, and social factors, individ- 80 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions uals often fail to make optimal decisions. To list a few examples: there is evidence that people: are loss-averse—that is, they attribute greater value to losses than gains (Kahneman and Tversky 1979); have status quo preference—that is, in the presence of many options, they tend to avoid changing their course of action (Samuelson and Zeckhauser 1988); discount the long term in comparison to the present (Loewenstein and Prelec 1992); and are highly influenced by emotions, peers, and common sense (Kahneman, Slovic and Tversky 1982; Andrade and Ariely 2009). Exploring the decision-making process specifically in the context of poverty, the latest experimental work in this area finds that many of these biases are even more pronounced among the poor (Banerjee 2000; Mullainathan and Thaler 2000; Duflo 2006). One key observation from this work is that poverty creates myopia, which refers to a focus on solving short-term problems without the ability (or mental space) to make decisions that take into account long-term implications. As a consequence, many decisions reached under these conditions are detrimental for the future (for example, high interest rate loans). In retrospect, what this literature suggests is that there is more to changing behavior than the traditional microeconomic model would predict. Simply providing individuals with financial education and access to instru- ments is insufficient; additional or alternative behavioral treatment mechanisms might be necessary to improve outcomes. These results, from both field experiments and behavioral research, have led to a shift away from financial literacy toward a wider concept that includes “a combi- nation of financial awareness, knowledge, skill, attitude and behavior necessary to make sound financial decisions and ultimately achieve financial wellbeing� (OECD/ INFE 2011).2 The expansion of the definition in broad terms represents a shift in conceptualization and measurement from a cognitive-based to an outcome-driven approach. According to the outcome-driven approach, if an individual makes a good financial decision, he or she is considered financially capable regardless of whether the decision followed a cognitive or a noncognitive route. This approach provided the basis for the measurement instrument developed by the RTF. Yet even with this broader definition, problems in identifying effective interventions persist. In fact, a key challenge in adopting a more holistic definition is that the interventions become more diverse in their design and may follow different theories of change. As the range of interventions expands, careful thinking is required to develop a standard results framework that allows for accurate comparison of findings across projects and countries. 2  See chapter 2 for more details on the conceptual development. 81 Financial capability in low- and middle-income countries: measurement and evaluation 3.2.2 Implications for program design and evaluation The discussion on the conceptual development has direct implications for program design and evaluation. When examining and discussing the merits of different types of interventions to improve consumer financial decision making, the debates usually revolve around two main categories: financial literacy programs and broader financial capability interventions. As explained earlier, by definition, financial literacy programs are designed to achieve outcomes through formal education, assuming a causal chain reaction from knowl- edge to skills and behavior. While these interventions may also include other delivery mechanisms to transfer education, the basic premise is that improving knowledge leads to desired outcomes. Financial capability programs, on the other hand, do not assume a particular relationship between the source factors, and hence may either follow a noncognitive approach or combine financial education with other mecha- nisms. From this category, in addition to programs that employ standard classroom and workshop models, five other types of interventions more widely encountered in recent studies include edutainment, social marketing, personal counseling, consumer protection, and behavioral treatments. Edutainment (educational entertainment) is a form of entertainment designed to educate the audience. It usually involves delivery of educational content through amusing and emotional tales in TV films, soap operas, theater, computer-based games, and others. Unique features of this intervention are that it can reach a wide audience, appeal to emotions, and disseminate messages in a way that stick to memory. The field of public health has extensively and successfully used edutain- ment to influence behavioral change in a number of areas, including nutrition, tobacco and alcohol use, safe sex, physical exercise, and immunization. However, in consumer finance, the application of this mechanism is still in the early stages of experimentation. To the authors’ knowledge, before the RTF efforts began, the only edutainment program for personal finance was Makutano Junction in Kenya, a TV soap opera funded by DFID.3 The program, however, has never been evaluated. More recent examples that involve evaluation include RTF-financed experiments in South Africa, Nigeria, and Kenya, where content-specific messages are integrated in a soap opera, a feature film, and comics, respectively (discussed in section 3.3). Social marketing refers to the systematic application of standard commercial marketing to alter preferences and influence a behavioral change. Though similar to edutainment, social marketing is less focused on education and more on persuasion 3  For more information see http://www.makutanojunction.org.uk/. 82 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions through short messages. These include anything from print handouts and billboards, to public service announcements linked to TV shows and films, to celebrity endorse- ments, street theater, and formal in-school and in-the-workplace presentations. Examples are many, such as in health (anti-tobacco, safe sex), road safety (buckle up, wear bike helmets), environment (recycling, energy saving), and politics (voter partic- ipation in elections). In promoting responsible financial management, this mechanism is also at its early stages of application. In a paper by Lee and Miller (2012) prepared for the Consulta- tive Group to Assist the Poor (CGAP), the authors reviewed 100 case studies where social marketing has influenced financial behavior; however, none of these programs have been rigorously evaluated. To the authors’ knowledge, the only intervention to date that employs social marketing coupled with a rigorous evaluation in the field of personal finance is the RTF-financed experiment in Kenya, which involves a national marketing campaign to promote savings (discussed in section 3.3). Personal counseling involves individualized and one-on-one training or assistance at a particular point in time. This method is best known from credit counseling, designed to either avoid debt or to help individuals develop debt management and repayment plans. It differs from traditional financial education aimed at teaching more general concepts. Personal counseling is customized and targeted to a specific individual and topic, and is usually applied at the time when it is relevant. It may include interventions where an individual is directed to budget properly and develop expense plans, save for a particular goal, or establish and improve credit history, or to guide product selection. It is mostly useful in light of major life events, such as loss of job, unexpected reduction in income, domestic violence, etc. For example, following the results of a baseline survey of financial capability, the United Kingdom’s Financial Services Authority (FSA) developed a pathfinder service offering individual information and guidance on a wide range of financial matters (such as budgeting, retirement planning, and choosing appropriate financial prod- ucts). This was delivered face-to-face, by telephone, and through an interactive website, and was found to be effective (Kempson and Collard 2010).4 A more recent example includes a RTF-financed experiment in India that involves a combination of standard classroom training with reminders sent through mobile phones and personal counseling through home visits (discussed in section 3.3). Consumer protection refers to institutionalized laws and policies mandated by governments and organizations to ensure rights of consumers and the free and appropriate disclosure of information. They are designed to protect consumers More information on the evaluation may be found at: http://www.bristol.ac.uk/geography/ 4  research/pfrc/themes/advice/pfrc1002.pdf. 83 Financial capability in low- and middle-income countries: measurement and evaluation against businesses and financial service providers. Interventions that involve consumer protection are considered a form of financial capability because they often involve mechanisms to direct individuals toward “better� decisions or to guide them to information to influence behavior. Another example in the United Kingdom developed in response to a baseline survey that revealed low levels of capability with regard to choosing products entailed development of a range of tools based on “decision trees� designed to automati- cally take consumers through the optimum pathway for selecting products.5 A more recent example is a RTF-financed experiment in Mexico that tests the impact of a program that modified product-specific disclosure statements to help consumers better understand their choices (discussed in section 3.3). Behavioral treatments are motivated and guided by the literature from psychology and behavioral economics. They refer to interventions that either sidestep or harness behavioral biases. The most well-known intervention in this category includes “nudging,� which refers to programs designed to push people toward better deci- sions by altering the choices they face (Thaler and Sunstein 2008). This may take the form of a choice architecture, sending reminders, providing incentives, or a combina- tion of these. Well-known examples are default options for retirement or savings plan enrollments. Two other types of behavioral interventions that have received attention lately include programs that aim at influencing actions through “learning-by-doing,� peer pressure, and social networks. The premise behind learning-by-doing is that once individuals become users of formal financial systems, their interactions with banking may provide the basis for learning and skills development. The assumption behind programs that explore social networks is that both learning and decisions may be spread from certain participants who receive a treatment to others who do not but who are in contact or proximity with the treated individuals. More recent examples include RTF-financed experiments in Kenya, Uganda, and Malawi, where researchers are investigating the impact of: learning-by-doing through participation in automated savings accounts; social networks on transferring practices; and activating mental accounting to effect savings (discussed in section 3.3). In general, financial capability programs may come in different forms and follow different theories of change. Understanding the differences between interventions is important for proper evaluation and for accurate interpretation of results. For eval- uation, having a clear understanding of the program’s objectives and its conceptual framework is critical for developing a proper hypothesis, mapping variables along More information on this initiative may be found at: http://www.pensionsnetwork.com/ 5  pensions_calculator/pdfs/trees.pdf. 84 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions the casual chain, and identifying indicators from inputs to outputs and outcomes. In extracting general lessons, whether the intervention involves financial education, nudges, edutainment aspects, or a combination of them is an important distinction because the corresponding theoretical designs have direct consequences in moving from research to policy formulation. 3.2.3 Paucity of the existing evidence base Rigorous empirical research designed to identify the impact of financial capability interventions is very recent, with most studies financed by the RTF. Earlier and completed evaluations are generally characterized by a lack of rigor. To the authors’ knowledge, the first comprehensive review of past evaluations of financial capability initiatives was commissioned by the FSA in 2008. The review concluded that “not only has there been relatively little work in the past on financial capability in the United Kingdom or other countries, but also that rigorous, credible policy evalua- tion showing the incremental impact of financial capability work is difficult to find� (Atkinson 2008). This was later confirmed by another stocktaking exercise conducted by the OECD in 2009, the conclusions of which were presented at the 4th session of the November 14, 2009, OECD/INFE Conference in Brazil. In 2010, the World Bank and the RAND Corporation undertook another stocktaking exercise, built on the previous reviews by the FSA and OECD and aimed at capturing the more recent studies. The review also aimed to expand the scope of the reviewed evaluations to account for alternative and broader financial capability interventions (see the next section and appendix C—available online at www.finlitedu.org—for a more detailed discussion of the review). The results confirmed the paucity of evalua- tions around financial capability programs in general and revealed the following: „„ Most of the evaluations lacked rigor and did not address the key principles of good evaluations related to internal and external validity, such as confounding factors, selection bias, spillover effects, contamination, and heterogeneity. „„ From the ones rigorously conducted, the majority applied experimental methods, which are effective in determining cause-and-effect but may fall short in explaining how and why impacts were (or were not) observed and may have uncertain external validity. „„ The majority of the evaluations lacked information about the intervention. They also lacked details about the impact measures (most reported that there was no change in financial literacy) or even the use of inappropriate measures given the nature of the interventions. „„ Linked to the above points, the majority reached sweeping conclusions, such as that financial education has no impact on financial literacy/capability. A 85 Financial capability in low- and middle-income countries: measurement and evaluation closer look at these studies, however, reveals that their results apply to that particular intervention, delivered in a specific way, to a particular audience, at a specific time, and by a particular agency; they do not necessarily generate lessons that apply to other interventions and circumstances. „„ There was an overemphasis on measuring changes in knowledge, as opposed to other outcomes, such as changes in attitudes, behavior, etc. „„ Most of the programs evaluated were workshop-based. The lack of rigorous evaluation is not necessarily specific to the domain of financial capability. Generally, evaluation is undersupplied in large part because it is consid- ered a public good and faces the classical free-rider problem. It creates knowledge that can benefit the public, but the costs are usually incurred by the provider. Even when mandated, however, evaluation is not welcomed because it is costly and tech- nically challenging and can pose reputational risks. Costs for the design and imple- mentation range anywhere from $50,000 to $500,000 and more. Considering that most programs, especially in developing countries, face budget constraints, diverting resources to research is a difficult decision for managers. A well-designed evaluation is also challenging to implement. To be able to answer both the “what� and the “why,� evaluation often needs to employ a combination of quantitative, process, and theory-based approaches, and follow a well-defined results framework. Also, when using randomized experiments, planning of the research design is required before the program begins. Furthermore, evaluation may be politically controversial, especially if it reveals that a program hasn’t achieved impact. Lastly, as with any field in its infancy, there are a number of narrowly defined research questions that need to be addressed. These questions often determine program characteristics that are required to permit proper testing. Therefore, in addition to evaluating existing programs, it is often important to work with program developers to design innovative interventions in the understudied areas. 3.3 Selected approach: field evaluations and a toolkit The discussion in the previous sections highlights that the field of financial capa- bility is relatively new and characterized by limited empirical evidence. Established to tackle essential gaps in research, the RTF evaluation program developed three parallel initiatives: „„ Funding of a number of individual and cross-country field evaluations that allow experimentation with different program designs and research tech- niques; 86 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions „„ Development of a methodological and operational toolkit for M&E of financial capability programs in LICs and MICs; and „„ Capacity building for design and implementation of rigorous evaluations in client countries through a series of clinical evaluation workshops. The overall evaluation program followed a similar management structure to that of the measurement cluster explained in chapter 2. The development of a strategic plan and its daily operations were handled by the Trust Fund’s Secretariat, consisting of World Bank staff based in the Social Protection Unit of the Human Development Network. The Secretariat was supported by a Technical Advisory Group consisting of external senior experts and academics who offered guidance in defining research priorities and designing evaluation methodologies. A World Bank Steering Committee, composed of staff representing different departments working on related topics, offered a broader institutional perspective to guide the direction of work. The next sections describe in more detail the development and implementation process of the individual knowledge products and associated activities. They also describe the coordination efforts that were required to orchestrate a multidimen- sional and highly interconnected work program. 3.3.1 State of knowledge and identification of research gaps To better understand the state of knowledge in terms of which programs work and which don’t, and to identify gaps in literature to guide the funding of evaluations, the RTF undertook a stocktaking exercise in the early stages of program development (briefly mentioned earlier in the chapter). This effort was led in collaboration with senior experts and the RAND Corporation. While other reviews of financial capability programs had been conducted in the past, they were limited in terms of the informa- tion provided about the reviewed evaluations. The project team was not interested in simply knowing what programs were evaluated and what the results were, but was more specifically focused on gathering information related to program characteris- tics and the rigor of the evaluations. In this effort, a set of evaluations related to the topic of financial education and broader financial capability both completed and in progress was systematically collected. Among other items, the information gathered about these evaluations consisted of: „„ Intervention design „„ Topics covered within the broader domain of financial capability „„ Delivery (mediums through which the interventions were delivered) „„ Targeted audience „„ Potential learning and results 87 Financial capability in low- and middle-income countries: measurement and evaluation „„ Methodological approaches: quantitative, qualitative, or mixed „„ Whether programs included process evaluation „„ Sample size „„ Whether the evaluation entailed a pre- or post-design „„ Whether the evaluation entailed a comparison group „„ Key outcome measures „„ Whether evaluations entailed measurement of changing effects over time The review also covered the 15 evaluations financed by DFID’s FEF, which primarily focused on African countries (appendix D—available online at www.finlitedu.org­— provides a summary of FEF’s evaluations).6 Broadly speaking, evaluations were sourced in four categories: a standard academic literature search for published studies related to financial capability, financial education, and financial training; a literature search for working papers and unpublished studies; a review of the World Bank’s internal Impact Evaluation database; and extended queries of funders and research organizations to locate studies in progress. The review excluded from consideration certain studies based on content and quality, such as: „„ Studies that did not focus on a specific reform or intervention; e.g., work examining relationships between financial capability measures and outcomes in a population, reviews of other literature, opinion pieces, etc. „„ Studies of interventions aimed at increasing access alone through expansion of financial supply; e.g., opening bank branches, providing free bank accounts, etc. „„ Interventions aimed solely at increasing the quality of regulatory disclosures in the developed financial systems. „„ Studies that did not meet the most basic evaluation quality criteria; e.g., work that contained no discussion of comparison groups or selection into the program. The review included certain studies for which information was not complete, such as: „„ Studies recently awarded and reported to be in progress by the funding sources. „„ Studies reported to be in progress by reliable sources (World Bank, published articles, etc.). For more information on DFID’s FEF, see http://www.dfid.gov.uk/work-with-us/funding-oppor- 6  tunities/countries-and-regions/fef/. 88 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions „„ Studies with incomplete methodology but validated by third party sources; e.g., the World Bank’s Impact Evaluation database. „„ Studies in two related fields that appeared in search results: choice architec- ture and health/agricultural/other education. Based on a carefully selected sample of 129 evaluations, a number of underresearched areas were identified. First, in terms of thematic coverage, budgeting and saving were found to be covered most widely. Other topics that are highly relevant were found to be underrepresented, such as financial capability related to credit, loans, investments, and the usage of insurance products. Second, the majority of the evaluations were found to focus on the delivery of education through workshops and seminars, followed by classroom and counseling. Alternative means of delivery, such as the use of media, marketing, behavioral treatments, or even mixed interventions, were found to be understudied. Third, the majority of evaluations either used process or randomized controlled trials. Very few employed mixed methods or examined spillover effects. Furthermore, the review revealed a gap in conducting comparison studies that measure the impact of interventions across countries, contexts, and populations. Lastly, a large number of choice architecture interventions appeared to be under way, particularly in the field of commitment devices. However, studies appeared to treat behavioral economics methods as a distinct alternative to financial capability training, rather than adopting behavioral economics insights to improving financial capability. Appendix C provides a more detailed discussion of the stocktaking and the main findings. This exercise helped establish five priority areas to guide invitations of proposals and selection of evaluations. Table 3.1 outlines these priorities divided into five broad categories. 3.3.2 Filling the gaps through field evaluations The 17 RTF-financed field evaluations represent a highly coordinated approach across themes and countries and involve a large number of clients and technical experts. These include: (1) country-specific evaluations; (2) cross-country evaluations; and (3) programmatic activities to integrate the learning generated from the evalua- tions into knowledge products. Guided by the priorities identified in table 3.1, the RTF awarded grants for evaluations through five tranches, distributed to internal World Bank staff and external parties.7 7  See appendix E (available online at www.finlitedu.org) for a full list of World Bank RTF-financed field evaluation projects as well as for more detailed descriptions of individual evaluation designs, methodological approaches, and respective policy implications. 89 Financial capability in low- and middle-income countries: measurement and evaluation Table 3.1 Priority areas guiding the selection of RTF field evaluations Thematic focus ƒƒBudgeting and savings ƒƒPlanning for known and unexpected events (savings for old age, partic- ipation in pension schemes, precautionary savings, formal and informal insurance) ƒƒPlanning and making provisions for known needs and changes in cir- cumstances (investments, loans and debt, education planning, changes in finances due to job loss, sickness, disability, or seasonal variations in income or crops) Program ƒƒBroadcast media or new media, such as radio, television, print material, delivery SMS-mobile technology, internet-based communication, and computer games ƒƒPrograms that link financial education to government cash transfers Special ƒƒComparing the relative impact of different methods of delivering research financial education, including relative cost-effectiveness. For example, questions evaluation of the impact of school-based models or workshops versus nontraditional learning, such as radio, games, theater, TV, films, print media. ƒƒExamination of the importance of the intensity and duration of expo- sure to education or information. For example, measuring how much information is optimal and how often it needs to be reinforced to impact behavior. Evaluation ƒƒPreference for mixed methods and underrepresented approaches. methodology These may include, for example, regression discontinuity design, matching estimators, and difference-in-difference techniques. Regional ƒƒPreference for underrepresented regions: Eastern Europe, Central Asia, coverage and the Middle East and North Africa Source: Project team. Projects were identified through calls for proposals. In addition to financial support, grant recipients received technical assistance in the design of evaluations. Following selection, project managers were matched with evaluation experts to develop research plans and validate the rigor of methodologies. One novelty of this arrange- ment was that it offered expertise to applicants who were able to identify programs but did not possess the technical capacity to develop research proposals. This was especially helpful for program sponsors, as it enabled them to compete for funding in parallel with researchers. The first call for proposals The first call for proposals was issued in April 2010, competitively open to both World Bank staff and outside parties. Proposals were invited to focus on two main topics: personal finance (day-to-day financial management) and planning for the long term and the unexpected (budgeting, saving, investments, retirement, and insurance). Invi- 90 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions tations also emphasized a preference for research that incorporated a combination of experimental and qualitative methods. Four projects were selected for evaluation in Brazil, India, South Africa, and Uganda: „„ Brazil: a randomized experiment to measure the impact of financial education incorporated in standard high school curriculum. „„ India: a randomized experiment combined with process evaluation to measure the impact of financial education delivered through doorstep banking. „„ South Africa: an experimental encouragement design to measure the impact of financial education delivered through TV soap operas. „„ Uganda: a study designed to develop qualitative methods for evaluation based on Financial Diaries. The second call for proposals The second call for proposals, also open to World Bank staff and outside entities, was issued in March 2011. The priority areas were similar to those in the first call for proposals, with slightly stricter requirements to target topics other than budgeting and savings and interventions that utilize nontraditional methods of delivering educa- tion (i.e., other than in schools and workshops). The invitations emphasized a pref- erence for comparative examinations to measure the relative impacts of delivering the same program through different channels (for example, schools versus media, or a combination) and of testing the same intervention across different settings and target groups. Through this process, three projects were selected for evaluation in India, Mexico, and Nigeria: „„ India: a randomized experiment to measure the impact of a mixed interven- tion, involving classroom education, personal counseling, and reminders sent through mobile messaging. „„ Nigeria: a randomized experiment to measure the impact of financial capabili- ty-related messages delivered through a feature film. „„ Mexico: a randomized experiment combined with process evaluation to measure the impact of product-specific information disclosure. The third call for proposals The third call for proposals, issued in August 2011, was open to external parties to identify research opportunities outside the World Bank. Parties were invited to 91 Financial capability in low- and middle-income countries: measurement and evaluation submit proposals based on the same priority areas and selection criteria as those in the second call for proposals. From this process, two projects were selected to conduct work in Kenya and Nigeria: „„ Kenya: a randomized experiment to measure the impact of an intervention that combines high school classroom education with comic books and radio. „„ Nigeria: a difference-in-difference evaluation to measure the impact of a national marketing campaign to promote savings. The fourth call for proposals A final call for proposals for country-specific evaluations was issued in November 2011. It was open internally to World Bank staff and attempted to identify programs that had not been covered through previous solicitations. Four projects selected through this process included evaluations in Malawi, the Dominican Republic, Uganda, and Brazil: „„ Malawi: a randomized experiment to measure the impact of financial educa- tion training combined with mental accounting to facilitate savings. „„ Dominican Republic: a randomized experiment to measure the impact of financial education incorporated in a conditional cash transfer program. „„ Uganda: a randomized experiment to measure the impact of network effects on transferring knowledge and practices. „„ Brazil: a randomized experiment to measure the impact of financial education related to investments delivered in combination with an online stock market simulator. Cross-country comparative evaluations In addition to financing country-specific evaluations, the RTF funded a complemen- tary program of four projects in early 2010 within the World Bank’s Research Depart- ment to provide a comparative analysis of programs across different settings and countries. These projects assessed whether context had an impact on outcomes. The World Bank–wide Research Committee, with representation from regions and sectors, managed the proposal review process with external and internal reviewers. The proposals included studies in South Africa, Mexico, Australia and New Zealand, and Kenya: „„ South Africa: a randomized experiment to measure the impact of a financial education seminar on remittance behavior and product selection; comparison with a similar intervention in India. 92 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions „„ Mexico: a randomized experiment to measure the impact of financial literacy training on credit card usage; comparison with similar interventions in other countries (comparing countries to be identified). „„ Australia and New Zealand: a randomized experiment to measure the impact of financial education on remitting behavior; results compared between temporary migrants from Pacific islands in Australia and New Zealand. „„ Kenya: a randomized experiment to measure the impact of financial educa- tion on weather insurance take-up; results compared between Kenya. 3.3.3 Individual project designs The RTF evaluations are aimed at addressing critical gaps in research by contributing both methodologically and conceptually. But what are the program features and the research methods of these evaluations? And how do they fill the gaps and add to the overall state of knowledge in the field? This subsection provides a brief description of the individual designs and their methodological characteristics. It discusses them in three broad categories: traditional financial education through schools and work- shops; nontraditional education through entertainment media and marketing; and financial education through mixed interventions. On methodology, while the RTF takes an open view on research design, all of the studies funded aim to determine both causality and attribution. In most cases, this is satisfied through randomization, but other methods for identifying counterfac- tuals have also been adopted, especially in cases of media and social marketing interventions where treatment is not easily excludable. In addition, some studies have supplemented the methodologies with qualitative and process evaluation to explain how and why change occurs. Conceptually, the evaluations intend to answer a diverse set of questions, and while individually they are designed to test specific theories, collectively they intend to contribute to the same broader policy questions. Traditional financial education: schools, workshops, and training seminars Does formal financial education delivered through school curricula, workshops, and trainings work in improving knowledge and desired outcomes? The six evaluations described in box 3.1 explore this question. Throughout this section, the reported results on impact are statistically significant with at least 90 percent confidence. As with most efforts to study the impact of policies, the main challenge in estimating the effect of financial education on individuals’ knowledge and behavior is establishing counterfactuals; that is, determining what would have happened to those individuals 93 Financial capability in low- and middle-income countries: measurement and evaluation in the absence of the intervention. Almost all design discussions are concerned with this question, and in general, the more attention that can be dedicated to the design of the project ex ante, the simpler and more robust is the estimation of results ex post. All six studies discussed above achieve this through random assignment of beneficiaries to treatment and control groups. Data are then collected to measure outcomes, in some cases combined with qualitative and process evaluations. Random assignment requires that the evaluation objectives and methodological development for data collection and analysis are in place before the program is rolled out, and this requires collaboration with program providers early in the process. Close collaboration among the learning and program staff is especially important to address ethical issues, ensure compliance and tracking among both treatment and control groups, and minimize attrition and spillovers. The projects discussed above adopt innovative methods to address these concerns. Appendix E (available online at www.finlitedu.org) provides more detailed descriptions of their approaches. Furthermore, considering the absence of more conclusive evidence on the impact of traditional financial education, these projects are designed to provide additional insight on whether different aspects of program design, such as content, delivery mechanism, context, and duration, affect results. For example, the novelty of the Brazil project is that it evaluates financial education incorporated across a number of subjects in the standard school curriculum and delivered during three consecu- tive academic semesters. Although financial education in schools has been studied before, this is the first time it has been integrated so extensively in the school program of study. The two India studies explore the impact of classroom education when combined and reinforced over time through reminders and personal coun- seling. This, too, is different from evaluating single and one-time interventions usually delivered in much shorter periods of time. Nontraditional financial education: use of mass media and social marketing As discussed earlier, lessons from psychology suggest that behavioral treatments can be effective in achieving outcomes. The five RTF evaluations described in box 3.2 represent the first experiments that incorporate lessons from behavioral psychology into financial capability programs. All of the studies discussed above experiment with different ways of incorporating financial education into alluring stories or short messages transmitted via mass media and social marketing campaigns or innovative technologies. Conceptually, the studies are designed to shed light on whether media and marketing tools can improve indi- viduals’ financial capabilities and improve decision making, or whether school-based 94 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions Box 3.1 RTF case studies using traditional financial education Financial Education and Behavior Formation: Large-Scale Experimental Evidence from Brazil ƒƒ Pathways to change: Classroom financial education ƒƒ Thematic focus: Budgeting, savings, and general financial management ƒƒ Target groups: High school students ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Self-administered questionnaires ƒƒ Results: Impact on improving knowledge, attitudes, and behavior The study assesses the impact of high school financial education in Brazil. It includes nearly 900 schools and 26,000 students. Administration of the program through schools allowed for a broad coverage of content in the curriculum. To control for quality of content, the educational material was developed by experts. Separate training was provided to a group of parents of the students to examine whether inside-the-household interactions influenced behavior. Results found that the program increased student financial knowledge by a quarter of a standard deviation, which led to a 1.4 percentage point increase in savings—a relatively large and economically relevant effect. A complementary workshop for parents induced children to save even more. Both current attitudes and forward-looking intentions to save improved. The Impact of Financial Literacy Training for Migrants: Evidence from Australia and New Zealand ƒƒ Pathways to change: Group-based financial literacy seminar ƒƒ Thematic focus: Remittances, credit and financial product selection ƒƒ Target groups: Migrant workers ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Face-to-face surveys ƒƒ Results: Impact on increasing knowledge and information-seeking behavior and reducing the risk of switching to costlier remittance products. No impact on changing frequency or levels of remittances. The study examined the impact of financial education targeting migrant workers and their remitting behavior in Australia and New Zealand. Training consisted of a two-hour session on reasons to remit, strategies for comparing costs, and information about different remittance products. Results show that the training led to increases in financial knowledge; migrants were more likely to know it is cheaper to send one large transfer than individual smaller ones, and more likely to know cheaper methods of remitting. The study also found that migrants changed behavior in response to knowledge gained; however, training was not found to change frequency of remitting, amount remitted, or the take-up of products. (continued) 95 Financial capability in low- and middle-income countries: measurement and evaluation Box 3.1 RTF case studies using traditional financial education (continued) Does Financial Education Affect Savings Behavior? Evidence from a Randomized Experiment among Low-Income Clients of Branchless Banking in India ƒƒ Pathways to change: Classroom training seminar ƒƒ Thematic focus: Budgeting, savings, and general financial management ƒƒ Target groups: Low-income households ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Face-to-face surveys ƒƒ Results: Impact on savings and attitudes related to financial management. Financial literacy levels did not improve. The study measured the impact of a classroom financial literacy training on the uptake of branchless banking and on savings behavior. The intervention consisted of a two-day training that covered the role of formal banking in people’s lives; and responsible borrowing, spending, saving, and cash management. The experiment was conducted on a random sample of 3,000 clients of branchless banking across two adjacent districts in the state of Uttar Pradesh. The results reveal that the intervention had impact on savings and that attitudes related to financial management improved, but overall financial literacy did not. This suggests that a causal chain reaction from knowledge to behavior might not necessarily be required in such order to achieve desired outcomes. Measuring the Impact of Financial Literacy on Savings and Responsible Credit Card Use: Evidence from a Randomized Experiment in Mexico ƒƒ Pathways to change: Classroom training seminar ƒƒ Thematic focus: Saving, retirement, use of credit ƒƒ Target groups: Bank credit card customers ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Face-to-face surveys ƒƒ Results: Impact on improving knowledge and savings rates, but no impact on credit card usage The study tested the impact of financial literacy training on savings and borrowing behavior and credit card usage patterns of credit card customers in Mexico City. It involved approximately 40,000 bank consumers. The training course lasted for about four hours and consisted of four modules on savings, retirement, credit cards, and responsible use of credit. Results show a 9 percentage point increase in financial knowledge, and a 9 percentage point increase in saving outcomes, but no impact on credit card behavior, retirement savings, or borrowing. Moreover, administrative data suggest that the savings impact is relatively short-lived. The results point to the limits of using general-purpose workshops to improve financial literacy and decision-making patterns for the general population. (continued) 96 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions Box 3.1 RTF case studies using traditional financial education (continued) The Impact of Financial Education on Financial Knowledge, Behavior, and Outcomes: Evidence from a Randomized Experiment in South Africa ƒƒ Pathways to change: Group-based interactive financial literacy seminar ƒƒ Thematic focus: Remittances, credit, and financial product selection ƒƒ Target groups: Members of burial societies ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Face-to-face surveys and self-administered questionnaires ƒƒ Results: The study is still ongoing. Based on early observations, the intervention was found to improve budgeting and savings, reduce gambling, and decrease risk aversion. The study examines the impact of one-day financial education training on savings, financial planning, budgeting, and debt management. The purpose is to encourage individuals to seek more efficient financial vehicles, as well as to save and use credit wisely. The target population consists of members of burial societies and women’s development groups in the Eastern Cape area of South Africa. It uses a randomized experiment involving approximately 1,300 individuals. Outcome measures are obtained from a variety of sources, including individual surveys and administrative data. While the study is still ongoing, preliminary results show that the intervention had impact on budgeting skills and savings behavior, as well as on reducing gambling and decreasing risk aversion. The Role of Financial Access, Knowledge, and Service Delivery in Savings Behavior: Evidence from a Randomized Experiment in India ƒƒ Pathways to change: Classroom seminar, counseling, and reminders through phones ƒƒ Thematic focus: Savings, budgeting, and selection of financial products ƒƒ Target groups: Low-income households ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Face-to-face surveys ƒƒ Results: No results reported at this stage. The intervention is ongoing. The study measures the impact of a mixed and multilayered intervention on savings behavior. The program consists of three treatments: a classroom financial education training, followed by reminders through mobile SMS and voice messages over a period of several months, and personal counseling through physical visits to participants’ homes. It aims to measure the overall impact but also to disentangle the separate effects from the individual treatments. The novelty of this study is that it experiments with combining traditional financial education with behavioral treatments and explores a multilayered intervention over a longer period of time as opposed to a single and one-time treatment. 97 Financial capability in low- and middle-income countries: measurement and evaluation Box 3.2 RTF case studies using nontraditional financial education Harnessing Emotional Connections to Improve Financial Decisions: Evaluating the Impact of Financial Education in Mainstream Media in South Africa ƒƒ Pathways to change: Entertainment media (television soap opera) ƒƒ Thematic focus: Debt management ƒƒ Target groups: Low-income households with and without existing consumer debt ƒƒ Evaluation method: Randomized encouragement design ƒƒ Data collection: Phone and face-to-face surveys and qualitative focus groups ƒƒ Results: Improvement of knowledge and borrowing behavior The study investigates whether debt management may be improved through TV soap operas. To control for content quality, the project develops a soap opera storyline through focus groups. It involves around 1,000 randomly selected individuals divided into treatment and control groups. The treatment watches a soap opera with financial literacy messages, called Scandal!; the control watches a different show aired at the same time but with no financial literacy messages. Financial incentives are provided to ensure compliance. Results report that individuals assigned to watch Scandal! had higher financial knowledge on issues highlighted in the storyline. Scandal! viewers were more likely to borrow from formal sources, less likely to engage in gambling, and less prone to enter hire purchase agreements. Learning by Doing? Using Savings Lotteries and Social Marketing to Promote Financial Inclusion: Evidence from an Experiment in Nigeria ƒƒ Pathways to change: National marketing campaign and savings lottery ƒƒ Thematic focus: Savings and use of banking services ƒƒ Target groups: Existing and new bank users ƒƒ Evaluation method: Regression; discontinuity design ƒƒ Data collection: Microdata collected from the banks on daily account balances ƒƒ Results: Increased savings and use of additional financial products within a week of intervention. However, no evidence on persistent changes after incentives removed. The study measures the impact of a national marketing campaign launched by one of the largest banks in Nigeria to encourage savings. It entails lottery prizes for individuals who open a savings account and maintain a threshold amount in the account for 90 days during a period of three months. The lottery is publicized through advertisements with celebrity endorsements and media releases. The research assesses the extent to which the different components of the campaign affect the take-up rate. It also measures the impact of “learning-by-doing�—the extent to which interactions with banking motivate people to continue to save. Results report that during the intervention there was improvement on savings behavior and on the usage of the bank’s other financial products. However, there was no evidence that the incentive program led to persistent changes after explicit incentives were removed. (continued) 98 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions Box 3.2 RTF case studies using nontraditional financial education (continued) The Impact of Cartoons and Comics on the Effectiveness of Financial Education: Evidence from a Randomized Experiment in Kenya ƒƒ Pathways to change: Comic books ƒƒ Thematic focus: Financial management and saving ƒƒ Target groups: Schoolchildren ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Face-to-face surveys ƒƒ Results: No impact on literacy levels and on savings behavior. However, it found impact on the likelihood that students want to start a business in the future. The study tests the absolute and relative impact of different program delivery mechanisms on the financial capability and behavior of Kenyan youth. It compares delivering education though classroom with that through comic books and radio shows. It uses a sample of 220 high schools, randomly assigned to two main treatment groups, a placebo group and a control. One novelty of this study is that in addition to completing both baseline and endline surveys, students are also asked to make financial decisions using real resources. This allows recording how actions differ from stated intentions, and how both stated intentions and actions change over time. Results show little evidence that the interventions improved financial literacy. Similarly they find no effect on stated and actual savings behavior. However, the study finds impacts on the likelihood that students want to start a business in the future. The Impact of Financial Literacy through Feature Films: Evidence from a Randomized Experiment in Nigeria ƒƒ Pathways to change: Entertainment media (feature film) ƒƒ Thematic focus: Savings and credit ƒƒ Target groups: Low-income households and small business owners ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Surveys and self-administered questionnaires ƒƒ Results: The study is still ongoing. Early observations report on the impact on perceptions and trust in microfinance institutions and increasing the take-up of savings accounts in the short run. Limited evidence of an impact on longer-term behavioral change. The study involves a sample of 3,000 individuals in Nigeria to assess the extent to which a feature film can promote responsible borrowing and improve savings. A basic premise on which the study is developed is that emotions have an influence on actions, and while the emotional state might be transient and short-lived, the decisions reached under the emotional state could potentially provide the basis for future actions. To capture immediate actions, the experiment includes the presence of microfinance institutions at the movie screening venues. Furthermore, to control for spillover effects—that is, the extent to which individuals in the treatment and control groups might talk to each other and share information—the experiment adopts a mix of individual and cluster randomization. (continued) 99 Financial capability in low- and middle-income countries: measurement and evaluation Box 3.2 RTF case studies using nontraditional financial education (continued) The Impact of Financial Education and Learning-by-Doing on Household Investment Behavior: Evidence from Brazil ƒƒ Pathways to change: Online stock market simulator ƒƒ Thematic focus: Stock market participation and investments ƒƒ Target groups: Stock market simulator users and participants ƒƒ Evaluation method: Randomized controlled trial design and regression methods ƒƒ Data collection: Stock market simulator data and individual stock market data ƒƒ Results: No results reported at this stage. The intervention is ongoing. The study measures the impact of financial education on stock market participation and investment in Brazil. It involves an online stock market simulator that serves as a platform through which financial education is passed to participants and as a mechanism through which participants develop practical experience. A combination of education, warning, and reminders about good practices and learning-by-doing is expected to improve decisions over time. The study uses data of approximately 600,000 investors. Understanding why people make the investments they do can help identify interventions to improve consumer protection and support development of capital markets. financial education material is more effective when presented in a more entertaining way. Also, they provide valuable insight into the extent to which these interventions, by appealing to emotions and sticking to memory, lead to more effective decision making in the future. Conversely, “learning-by-doing� experiments test the reverse hypothesis; that by “nudging� and otherwise affecting behavior directly (for example, enrolling in automatic savings), individuals will become more interested in interacting with financial institutions and a broader range of financial products. Mass media campaigns present a unique evaluation challenge and require innova- tive evaluation methods. First, because television is not excludable, it is difficult to distinguish between treatment and control groups and to minimize spillovers. Second, because these interventions tend to be multidimensional and contain several levels of treatment, it is difficult to identify the specific program characteristics responsible for impact. The teams have developed novel mechanisms to maximize the separation between treatment and control, to motivate participation, and to design placebo treat- ments as falsification tests. These methods are explained in more detail in appendix E. Financial education though combined interventions Five RTF evaluations explore the impact of mixed or combined interventions on knowledge and behavioral outcomes, as described in box 3.3. In addition to adopting rigorous methods to measure impact on behavior and outcomes and addressing methodological concerns discussed throughout this 100 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions Box 3.3 RTF case studies using combined interventions Social Networks, Financial Literacy, and Index Insurance: Evidence from a Randomized Experiment in Kenya ƒƒ Pathways to change: Comic books ƒƒ Thematic focus: Long-term planning; index-based weather insurance ƒƒ Target groups: Rural, small-scale farmers ƒƒ Evaluation method: Randomized controlled trial design and regression methods ƒƒ Data collection: Face-to-face and phone surveys ƒƒ Results: Impact on encouraging the take-up of index-based drought insurance The study presents a randomized field experiment measuring the direct impact and social network spillovers of providing financial literacy and discount vouchers on farmers’ decision to purchase index-based drought insurance in Kenya. The experiment covers around 14 villages and uses comic books as a delivery mechanism of financial education; the comic details the index-insurance product and how it can help families protect themselves from the risk of drought. The study finds social network spillovers to the provision of financial literacy materials, but no spillovers to the provision of discount vouchers on farmers’ decision to purchase insurance. It further finds that financial materials have spillover effects on farmers’ attitudes toward insurance but limited effects on understanding as narrowly measured in the survey. These results provide suggestive evidence that financial literacy materials are efficacious in encouraging take-up when farmers’ social contacts similarly receive access to financial literacy materials. Understanding Financial Capability through Financial Diaries and In-Depth Interviews in Uganda ƒƒ Pathways to change: Classroom training sessions ƒƒ Thematic focus: General financial education topics ƒƒ Target groups: Low-income households ƒƒ Evaluation method: Qualitative (financial diaries) ƒƒ Data collection: Face-to-face surveys ƒƒ Results: Qualitative findings suggest changes in knowledge, skills, and attitudes; however, they also indicate that they do not always translate directly into behavior change, at least not within a short time frame. The study uses financial diaries in combination with in-depth interviews in Uganda to understand and measure the financial capabilities of low-income households. It compares changes in knowledge, skills, attitudes, and behaviors of respondents in the treatment and comparison groups, highlighting situations where the former underwent a change that might be the result of the impact of the financial education. Results suggest that financial education affects knowledge, skills, and attitudes. Nevertheless, they also indicate that they do not always translate into behavioral change and affect decision making, at least not within a short time frame. However, there is some suggestion of change in savings behavior in terms of saving at home. (continued) 101 Financial capability in low- and middle-income countries: measurement and evaluation Box 3.3 RTF case studies using combined interventions (continued) Evaluating the Effectiveness of Loan Disclosure Reforms on Consumer Understanding and Financial Decision Making: Evidence from Mexico ƒƒ Pathways to change: Product disclosure formats, and mobile SMS and telephone counseling ƒƒ Thematic focus: Savings and credit ƒƒ Target groups: Low-income consumers of credit ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Surveys and credit reports ƒƒ Results: Preliminary results suggest that disclosure and transparency improve the ability of consumers to select the best product out of several options and to identify the least expensive of several credit products offered. The study measured the impact of product-specific information disclosure on financial decisions. It assumes that the more transparent and relevant the information, the better consumer decisions will be with regard to product selection. It involved development and testing of a series of alternate product-specific disclosure formats, which were then used by low-income consumers in Mexico to choose between a series of credit or savings products. The testing of formats was complemented with financial education information delivered prior to the exercise to some participants, either via SMS messages or by phone consultation. Preliminary results suggest that disclosure and transparency improve the ability of consumers to select the best product out of several options and the ability to identify the least expensive of several credit products offered. These findings point to potential benefits of focusing on product-specific information disclosure and consumer education. Increasing the Impact of Conditional Cash Transfer Programs through Financial Literacy in the Dominican Republic ƒƒ Pathways to change: Professional and peer trainings ƒƒ Thematic focus: Household and business financial management ƒƒ Target groups: Conditional cash transfer beneficiaries ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Surveys ƒƒ Results: No results reported at this stage. The intervention is ongoing. The study assesses whether conditional cash transfer (CCT) programs can be leveraged to deliver financial education and affect both knowledge and behavior. Evaluations of CCT programs have shown that they can be successful in increasing usage of health care and education services. This project explores the extent to which CCTs can improve financial capabilities. Working with around 60 beneficiaries of the Solidaridad cash transfer program in the Dominican Republic, the study randomly selects one group of beneficiaries to participate in the training and another group that does not to serve as a control. The experiment is further divided into subtreatments to test whether training delivered by professionals versus peers has different effects. It also measures the relative impact of business training versus soft job skills training on decreasing unemployment among beneficiaries. (continued) 102 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions Box 3.3 RTF case studies using combined interventions (continued) Financial Development and the Psychology of Savings: Evidence from a Randomized Study in Malawi ƒƒ Pathways to change: One-on-one training; labeled banking accounts ƒƒ Thematic focus: Household and business financial management ƒƒ Target groups: Low-income agricultural wage earners and smallholder farmers ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Surveys ƒƒ Results: No results reported at this stage. The intervention is ongoing. The study investigates innovative ways to address low levels of formal savings by leveraging psychological mechanisms. The target population consists of low-income agricultural wage earners and smallholder farmers in Malawi. It examines whether direct deposit of wages as opposed to cash payments can help individuals match desired savings and expenditure patterns with actual behavior. It also tests whether labeling particular bank savings accounts with particular expenditures and labels (for example, college fund, car purchase, etc.) reinforces commitments to save. The Impact and Network Effects of Financial Management and Vocational Training in Informal Industrial Clusters: Evidence from Uganda ƒƒ Pathways to change: Practice-based vocational training and classroom-based business training ƒƒ Thematic focus: Vocational and business training, and business network effects ƒƒ Target groups: Small-scale industries and business owners ƒƒ Evaluation method: Randomized controlled trial design ƒƒ Data collection: Face-to-face surveys ƒƒ Results: Very preliminary results suggest some short-term effects of training on financial literacy and technical knowledge, optimism, and adherence to technical standards, but not on core business outcomes. The intervention is ongoing. The study explores the extent to which personal financial choices are affected by peers. It consists of a randomized evaluation in Uganda to identify the impact of a comprehensive financial management and vocational training program for small-scale industries, focusing on network effects. The study identifies business networks and examines whether the enhanced knowledge received through the training program spreads to other businesses and across networks, influencing certain behavior among the untreated population. Potential positive spillovers would constitute efficient ways to scale the impact of trainings and provide a natural source of leverage for these programs. chapter, the five studies described in this section all explore a number of questions that have been peripheral to the field until recently. These topics include providing product-specific information disclosure, leveraging CCTs, and experimenting with mental accounting and network and peer effects on knowledge and practice transfer. Among other things, these studies expand the range of programs that can be used to improve financial capabilities. The evaluation approaches adopted encompass a range of methods, predominantly through random assignment, closed-form data 103 Financial capability in low- and middle-income countries: measurement and evaluation collection, with some studies combining these methods with qualitative and process techniques. These methods are presented in more detail in appendix E. 3.3.4 A toolkit for M&E A second main initiative under the RTF evaluation program entailed the development of a toolkit for M&E of financial capability programs. Lack of robust evaluation is one reason why evidence on effective interventions for financial capability improve- ment is sparse across LICs and MICs. In LICs in particular, there are many obstacles to conducting rigorous evaluations. For example, there are problems with physical infrastructure, with securing financial and human resources, and with maintaining randomization, among many others. To this end, the RTF set out to develop a toolkit specifically targeted to these circumstances and informed by ongoing field experi- ences. Its development was led by RAND Corporation, contracted by the World Bank through competitive procurement.8 Through this effort, the RTF aimed to make evaluation expertise available to stake- holders operating in resource-scarce environments and to illustrate technical aspects of evaluation in simple terms and through case studies. While there are many handbooks that address different aspects of M&E, the RTF toolkit is different in a number of ways: „„ It provides an overview of all evaluation and research methods, including impact and process evaluation, and quantitative and qualitative techniques; „„ It is designed in a context of financial capability, including the challenges of evaluating different types of programs, especially programs that utilize media mechanisms and social marketing (that usually require special attention from an evaluation point of view); „„ It entails development of outcome measures specific to financial capability; and „„ It discusses the results by including details of the interventions not just the evaluations. The first part of the toolkit introduces financial capability as a concept and discusses how it translates into interventions. It outlines examples of programs and their theo- retical approaches, including traditional classroom models, broadcast media and social marketing, and initiatives linked to CCTs and government to person (G2P) inter- See the World Bank Russia Financial Literacy and Education Trust Fund website (www.finlitedu. 8  org/evaluation/wb/toolkit/) for more information on the toolkit development and the final product. 104 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions actions, as well as new novelties that draw on behavioral economics. The second part discusses the importance of developing a sound program theory, including goals and objectives, and defining a logical argument of how a series of inputs, activ- ities, and outputs are expected to accomplish specific goals (known as a theory of change). It provides a sample results framework to help readers understand how the different variables are usually mapped in project planning. The third part focuses on the three main categories of M&E, namely monitoring, process evaluation, and impact evaluation. It explains the difference between these approaches, defines the questions they intend to address, and illustrates cases of their application. As part of impact evaluation, the toolkit details the different research designs and their varying strengths and limitations. These include randomized controlled trials, regression discontinuity design, instrumental variables estimation, propensity score matching, and the difference-in-difference design. The reader is provided with the technical underpinnings behind these methods, their application through case studies, and a discussion around how to choose appropriate methods. It also explains the importance of qualitative methods, such as interviews, focus groups, site visits and observations, and case studies. It argues that qualitative data help capture the processes behind observed results that are not fully identified through quantitative methods. However, it also recognizes that they are limited in their ability to conclusively attribute changes to the program evaluated. In conclusion, it suggests that when combining different types of evaluations as well as different methods for data collection, the research design can better answer both the “what� aspects of the intervention responsible for change and the “why.� A major contribution of the toolkit is that it discusses both theory and methods through real-world examples. It takes the reader through concrete steps, from conceptual formulation of the evaluation, to choosing methods, to conducting data collection and analysis. It combines the latest evaluation technology with challenges and best practices realized among the diverse range of programmatic approaches, countries, and methods covered by the RTF field evaluations. In this way, it aims to lower the barriers to high-quality evaluation in LICs and MICs and help advance the evidence on the effectiveness of financial capability interventions. Appendix F (avail- able online at www.finlitedu.org) provides a more detailed description of the toolkit and its individual chapters. 3.4 Lessons and policy implications What are some of the lessons that may be drawn from the results and the prelimi- nary observations from the RTF program activities and evaluations reviewed above? This section outlines some general inferences and suggestions. 105 Financial capability in low- and middle-income countries: measurement and evaluation 3.4.1 Process of Implementation Close collaboration with the measurement component of the RTF activities strengthened the orientation and impact of the learning. The RTF consisted of two main components—the first to define and measure what it means to be financially capable (through survey development), and the second to learn how one can achieve financial capability (through evaluations). The learning half benefitted enormously from the measurement half: the goals of programs to enhance financial decision making have been shaped by a better understanding of what financial capa- bility really entails. This experience suggests that future activities in this field ought to be built on both measurement and evaluation simultaneously. Measurement work defines the universe; it identifies the levels of financial capability in a population and the different target groups, but evaluation work identifies the right interventions needed to improve the financial capability gaps. Policy makers are interested in both the identification of targets and policy measures to address them. Rigorous evaluation of existing programs is important; however, the knowl- edge generated by them may be limited in addressing critical research gaps. To be able to test new and refined theories, evaluations must sometime entail development of interventions. The objective of the RTF was to exclusively finance the impact evaluation efforts, not the interventions. It was assumed that programs existed and that through systematic solicitation, interventions that permitted different types of hypothesis-testing would be identified. On the contrary, it was a challenge to identify programs that responded to specific research constraints. The theory to be tested often determines the design of the program. For example, to compare whether a program is more effective when delivered through comics than through traditional classroom text books, an existing program would have to be identified that entailed the delivery of the same financial education content through these two mechanisms while targeting the same audience. Such programs are often absent in practice. Therefore, after gaps in literature are identified, it is often neces- sary to partner with researchers and program providers to either design new or modify existing interventions to respond to the necessary research characteristics. It is critical to clearly identify knowledge gaps to target financing in priority areas. A systematic review of the existing and ongoing literature, including both survey-based and experimental work, helps identify the existing knowledge and the gaps and describe how the proposed work is related to or builds on prior efforts. While this may sound obvious, when the RTF joined the international efforts, there had been no systemic review of evaluations in the field—at least not to an extent that provided detailed information related to research scopes, characteristics of program design, or methodological approaches. To address this void, the RTF under- took a stocktaking effort based on which a number of gaps emerged, both on the 106 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions methodological and conceptual fronts. Building on this exercise and maintaining a database of these studies is a valuable resource to ensure that duplication of effort doesn’t occur, that resources are invested in priority areas, that the results from different studies can be combined to inform the broader policy questions, and that future work builds on the existing studies to explore their implications fully. Collaboration among policy makers, program staff, service providers, and those responsible for learning helps to enhance validity. The design and imple- mentation of innovative, appropriate, and sustainable interventions to advance financial capability require coordination among the various important stakeholders in the process. This is especially true of the private sector, whose participation is vital, and who will not be encouraged to participate in activities which are inimical to their interests. This requires that programs are designed to be “incentive-compatible�; that is, the interests and objectives of all stakeholders, including consumers, are taken into account when designing programs. Programs must provide or enhance the services that consumers believe best suit their needs, as well as encourage consumer take-up. However, this is often a challenge for researchers, especially when conducting studies specifically designed to test hypotheses and generate find- ings. Strong collaboration with stakeholders during project development is crucial for appropriate research design, proper implementation, and adequate inferences made from the data collected. 3.4.2 Research methodology When feasible, one should adopt mixed methods for evaluation to answer both whether X causes Y, and also why and how. In choosing impact evaluation methods, it is important to think about cause and effect as well as why and how a program works or doesn’t work. Otherwise, even after a highly rigorous approach is adopted, results could be inadequate, or worse, could misguide policy. Take, for example, a randomized assessment to measure the impact on savings of a financial education seminar delivered to employees of a company that finds no significant impact. Such a result could be interpreted to suggest that the program doesn’t warrant replication. In contrast, it could be that minor aspects of the intervention affected the outcome, such as the timing of the day when the seminar was delivered, the motivation of the particular teachers, or a number of other unobserved factors. In other words, knowing whether a program works or doesn’t is only half of the battle. Understanding why it doesn’t work helps improve program design. In this view, employing mixed method approaches, such as relying on randomized controlled trials when feasible but also supplementing that with qualitative and process tech- niques, helps tell a more complete story. 107 Financial capability in low- and middle-income countries: measurement and evaluation In evaluating multidimensional interventions, it is important to isolate the specific aspects of programs that are responsible for change. This is especially relevant for media-based and marketing interventions that often entail a number of treatments administered at the same time. For example, one of the key aspects of these interventions is relying on alluring stories to appeal to people’s emotions. In this case, it is difficult to precisely estimate whether the outcome of interest is driven from the actual quality of the content delivered, the channel of delivery, the extent to which the audience favors the actors involved, or other factors. One solution is to include placebo groups, “sham treatments� designed to have no effect; however, the more complex interventions are, the more difficult and costly it becomes to develop multiple and layered treatments. The evaluation of comics as a delivery mechanism in the Kenya example discussed earlier employs a similar approach. For these inter- ventions, combining qualitative and quantitative methods can be particularly valuable for identification of outcomes. When feasible, it is useful to include a “Latin-square�- type design, in which different combinations of interventions or components can be tested. These evaluations require larger samples, however, because they aim to answer many questions simultaneously. One should conduct comparative examinations to measure the relative impact of programs and projects delivered in different settings or across different target groups. Results suggest that an understudied area of research is the compar- ative examination of the impact of programs; that is, evaluating not only whether a particular program works, but whether it works better or worse compared to other programs. It is especially important to experiment with nontraditional modes of delivering financial education and information, including utilization of technology, marketing, and behavioral treatments. Of equal importance is comparing delivery methods or alternative versions of the same delivery method against one another, and testing one delivery method in different contexts. This will help understand whether a program works across settings, and whether it works better or worse when disseminated through different delivery channels. The relevance of quality and intensity or dosage of the service (e.g., educa- tion and information) delivered should be explored. Another area that warrants further attention is the quality of the financial education and information delivered. It is unclear to date the extent to which the outcomes are driven by content as opposed to other aspects of the program structure. Most studies, though not all, do not include content development or content testing as part of the overall program development and evaluation. Having relevant and engaging financial education mate- rial significantly explains differences in financial literacy (Mandell and Klein 2007). As such, the quality and relevance of the content to the target audience should be an integral part of the research design. Related is the examination of the intensity and duration of exposure to education and information. For example, how much 108 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions education and information is optimal and how often should it be reinforced to impact behavior? Studies should be developed to look at time effects; that is, at what point do marginal benefits from exposure to education and information start to decrease and how long do effects last? The insights from psychology and behavioral economics into intervention design and implementation should be incorporated and tested. A behavioral perspective presents explanations as to why individuals are prone to behavioral biases when confronted with financial decisions. Psychological factors such as over- confidence, loss-aversion, status quo preference, and hyperbolic discounting, among others, have been found to impact financial decision making. Kahneman and Tver- sky’s Prospect Theory addresses some of these aspects and how they account for deviation from the classical full-information utility maximizing model. The relevance of this literature in consumer financial decision making is widely documented and is largely undisputable. However, aside from a couple of studies funded by the RTF, not much has been done elsewhere in incorporating these insights in developing and testing new theories and models. It is therefore of crucial importance to further study this area and examine the types of behavioral biases that impact decisions, to develop a theoretical framework to collect data around the psychology of financial decision making, and to experiment with designing interventions that help overcome such biases. The cost-effectiveness and feasibility of scaling up interventions should be investigated. It is one thing to experiment with interventions and determine whether a program works, and another to determine the extent to which that partic- ular intervention is scalable, especially from a budget point of view. Cost-effective- ness analysis relates the costs of a program to its outcomes and can greatly help policy makers in developing countries allocate scarce resources; as such, it should be integral in rigorous impact evaluation. This isn’t to suggest that the analysis should take a programmatic approach where the cost-effectiveness of all interven- tions in the field are compared. That would be a challenging, extremely costly, and time-consuming undertaking. Rather, interventions can either be compared with a single intervention or a small set of similar interventions or with an agreed upon benchmark representing the assumed willingness of policy makers and program providers to invest. 3.4.3 Identification of impact Evidence suggests that financial capability has an impact on consumer finan- cial decision making, but results are somewhat mixed, pointing to certain areas that require further investigation. Of the 17 RTF projects, 11 are complete and have reported results. The remaining six are still undergoing implementation or 109 Financial capability in low- and middle-income countries: measurement and evaluation end-line analysis, though they have reported on preliminary findings and observa- tions.9 Overall, the results from this set of projects suggest that financial capability enhancement, whether achieved through traditional classroom models or through innovative ways that utilize media, mobile technology, entertainment, or behavioral treatments, can be effective in both improving knowledge and awareness, and also in changing behavior. Yet generalizing from each individual impact study to other settings and populations is a challenge, as is the case with impact studies in general. Like with any randomized experiment, the more rigorous the experiment, the more likely the study is to lose external validity. However, the results from the completed studies combined with preliminary observations from the ongoing studies seem to suggest that, in general, financial capability enhancement programs work better when: „„ Financial education content is targeted and relevant. „„ It addresses the consumers at “teachable moments.� „„ It is entertaining and appeals to emotions. „„ The exposure to information is longer term. Evidence seems to strongly suggest that one-time interventions (such as short courses or workshops) can have an impact in the short term, but effects tend to fade over time. This has important policy implications in the field of financial capability because programs are generally aimed at altering long-term behavior. 3.5 Conclusion and proposals for next steps The M&E research program of the RTF implemented two main and complemen- tary initiatives in parallel: the goal of the first was to incentivize the application of rigorous M&E for interventions designed to improve financial capabilities of popula- tions in LICs and MICs; the goal of the second was to advance the knowledge agenda and the level of empirical findings on what works and what doesn’t in this area. This chapter provided an overview of this research program, its conceptual and method- ological development, the main lessons learned, and the potential policy implications. This final section offers a concluding assessment on whether the results of this work 9  The six RTF evaluations still in implementation or in the end-line analytical phase are expected to be completed by October 2013. Project completion reports and technical papers from these studies will be posted on the RTF website at www.finlitedu.org. The authors of these projects have reported preliminary findings and early observations with regard to impact; these have been incorporated in the overall lessons and recommendations made in this chapter. Comple- tion of these studies is not expected to affect the overall RTF program conclusions provided in this report. 110 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions Table 3.2 Summary of impact from the RTF evaluations Mixed Evidence Evidence of Study Evidence of of Lack of Incomplete Intervention Effectiveness Effectiveness Effectiveness (as of April 2013) School-based/seminars and workshops Brazil (schools) X Australia and New Zealand (seminar) X India (seminar) X Mexico (seminar) X South Africa (Seminar) X India (seminar, counseling, SMS reminders) X Entertainment Media and Marketing South Africa (soap operas) X Nigeria (feature films) X Kenya (comics) X Nigeria (marketing) X Brazil (online stock market simulator) X Other and mixed interventions Kenya (comics) X Uganda (financial diaries) X Mexico (information disclosure) X Dominican Republic (CCTs and seminars) X Malawi (labeled banking accounts) X program are in line with promises and expectations and makes a few proposals for possible next steps. The M&E incentivizing element consisted of three distinctive and complementary objectives: (1) to develop a toolkit on M&E to facilitate its rigorous application in LICs and MICs; (2) to (co-) finance the M&E component of the interventions to overcome financial constraints and disincentives generally associated with evaluations; and (3) to provide technical support for development of research plans and their proper implementation. The value of the M&E toolkit is in its elaboration and application of highly rigorous technical research methods in the context of financial capability. It is designed to educate the relevant stakeholders of financial capability enhancement interventions about proper M&E and to provide illustrations of the real-world application of tech- 111 Financial capability in low- and middle-income countries: measurement and evaluation nical tools through case studies in LIC and MIC settings. Its focus on resource-scarce environments is especially important to help practitioners ensure rigor of research in these countries. The financing of impact evaluations for a diverse set of interventions through the RTF program has validated the various conjectures about financial constraints and the public good issues. But it has also clarified that for impact evaluations to generate useful lessons and help improve programs in the longer term, the rigor of research designs needs to be ensured. This requires an engagement of technical experts at the early stages of project development, including at the level of hypothesis creation and intervention design. For many of the RTF-financed evaluations, provision of tech- nical expertise was the most valuable aspect of the program. The knowledge creation element also had three distinctive features: (1) to assess the effectiveness of interventions designed to improve financial capability and to ultimately yield better financial decisions; (2) to move beyond traditional finan- cial education and explore whether other alternative mechanisms are effective in achieving desired outcomes; and (3) to address specific gaps in literature identified by researchers as crucial for generating a more conclusive body of knowledge to guide policy. For all these dimensions, the research targeted poor segments of popu- lations in resource-scarce environments. The program made main progress in conducting a systematic review of existing and technically sound evaluations to identify research gaps both theoretically and meth- odologically. This stocktaking exercise represents one of the first efforts in the field to generate a database of evaluations that reports on both intervention and research characteristics. Furthermore, the program was successful in promoting evaluations of new and innovative interventions beyond traditional financial education. This was triggered by an outcome-based and agnostic approach that conceptually did not hold predispositions on the pathways needed to achieve good outcomes but instead explored different theories of change. This helped expand the research scope to include a variety of innovative programs that had not been addressed by previous studies, such as the utilization of media, marketing, and behavioral treatments in program design. Final results and preliminary findings from the 11 completed and 6 ongoing RTF-com- missioned research projects provide valuable insights and leads that need to be further explored by the research community to be able to offer more conclusive evidence. The following lessons are especially important: „„ Financial capability may be achieved through different interventions; programs that utilize mass media and social marketing tools promise to 112 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions be especially effective. Results suggest that financial capability programs, whether delivered through schools and workshops or through more inno- vative methods such as mass media and marketing, can be effective in changing both knowledge and behavior. Alternative interventions, such as using television soap operas, films, and promotion campaigns, prove to be especially effective. For these interventions, two features are presumed to affect the outcome. First, they can be more entertaining and have the capacity to transmit messages through appealing stories that stick to memory; and second, they can serve as mechanisms through which messages can be repeated and reinforced over time, keeping the audience engaged in treat- ment for longer than traditional financial education mechanisms do (example, soap operas can last for months or years). „„ The quality of the content delivered affects outcomes. Results tend to suggest that financial capability programs work better when the content is relevant, targeted at the right audience, and delivered at teachable moments. While this might come across as an obvious point, most financial capability programs, developed by both program providers and researchers, do not tend to focus much on developing the financial education content. The stocktaking exercise conducted by the OECD and the World Bank RTF finds that most of the programs in the field either use similar financial education material or develop generic curriculum that encompasses key concepts on financial management without tailoring the subject to the targeted audience or the specific setting. „„ Financial behavior improves even in the absence of improvements in financial literacy. Especially important, results find that programs can be effective in changing individual decision making even in cases when the finan- cial literacy of the treated individuals does not improve. This speaks to the importance of paying more attention to the mechanisms that drive behavior, be that knowledge or behavioral treatments. If the policy objective of a program is to improve savings rates, for example, providing financial education might not be necessary, and in certain cases might not even be cost-effective. Interventions that employ media and marketing tools could improve savings rates without following a cognitive route. This does not suggest that financial education should not be promoted, but it rather suggests that interventions other than traditional financial education are available to policy makers to help people make better financial choices. „„ Impacts from one-time interventions fade over time. Repetition of treat- ment is required for behavioral change maintenance. Results suggest that improvement of financial capability among the poor is extremely difficult, 113 Financial capability in low- and middle-income countries: measurement and evaluation especially when the improvement in question refers to longer-term behavior change and habit formation. The results from RTF and other studies suggest that while interventions may be successful in influencing immediate post-in- tervention behavior, the effects tend to fade over time. This observation is in line with research in other areas that focuses on mechanisms to change deci- sion making, for example, in health (promoting a healthy lifestyle) or energy (changing consumption patterns). This is not a demonstration that the financial capability interventions are not effective, but rather that treatments might need repeating over time to maintain their effect. The RTF supported a wide range of interventions in many different contexts that have yielded information of great value to policy makers and program staff alike with respect to the orientation, content, and effective delivery of financial capability information across the developing world. Nowhere before have the research and policy communities joined forces in such a coordinated manner to understand these lessons so comprehensively and rigorously. But even this comprehensive effort leaves many questions unanswered. In moving forward, the focus areas for the next generation of research need to be identified. This report recommends investing in the following priority research areas: 1. Explore interventions other than financial education. The limitation of traditional financial education in changing consumer financial behavior is well documented by research and is confirmed by the results from RTF projects. And even if they work, they typically have high treatment costs per individual. In contrast to traditional and classroom-based financial education, interven- tions that utilize mass media (television programs, radio, commercials, etc.), social marketing techniques (promotion campaigns, etc.), and behavioral treatments (reminders, choice framing, peer pressure, etc.) have often exhib- ited promise to be effective in influencing consumer choices, have mostly low treatment costs per individual, and can also be taken rapidly to national scale. A main condition is high technology penetration that is nowadays met in most LICs and MICs. This suggests that future research needs to further explore the scope and limitations of these interventions to better understand what specific mechanisms work better and in which settings. 2. Explore the difference between financial literacy and financial capa- bility and conduct comparative research to measure the relative impact on consumer decision making. These two terms, while to an extent used interchangeably in literature in recent years, mean different things and may have different objectives and outcomes. Achieving financial literacy means improving knowledge that may improve decisions for some financial outcomes, for example, budgeting or specific investments. Achieving financial capability 114 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions means improving financial decisions for the same or, perhaps, other outcomes, for example, increasing the savings rate or asking for financial advice. This conjecture emerges strongly from both the financial measurement work as well as from impact pilots, but the differences and overlap with regard to policy interventions (say classroom intervention versus mobile phone reminders) and their effectiveness are little known. 3. Validate the results of the Brazil pilot that financial education in school, if very well done, works both to increase financial literacy as well as financial outcomes (saving) and has (hopefully) long-term effects. To achieve such a result, it is suggested to require: (1) a relevant high-quality material/textbook developed by experts; (2) a well-trained and highly motivated staff (through incentives); (3) multiple treatments over a longer period (three consecutive academic semesters); and (4) last but not least, the involvement of parents. Yet even then it is not yet clear if the created effects are sustained. In consequence, it is important to replicate and document the approach and results in different countries to gain external validity, or at least to understand under what conditions a replication can be achieved (or not). 4. Explore the importance of intensity or dosage of the service delivered. There is little evidence on how much education and information is optimal and how often should it be reinforced to achieve desired outcomes. Studies should be developed to look at optimal dosages for different interventions and examine timing and repetition effects; that is, whether impacts diminish as the time spent in classes or exposure to messages declines and how long effects last. 5. Focus on maintenance of long-term behavior change. Related to the point above, the results from the RTF suggest that most interventions that find impact usually refer to short-term impact. Individuals generally tend to regress to former behavior (pre-intervention). It is not enough for researchers to iden- tify interventions that succeed in controlled clinical trials; they must also show that the impact detected can continue over time, otherwise an intervention cannot be deemed successful and suggested for replication. Future research should focus on the maintenance stage; i.e., ways (or additional treatments needed over time) to help individuals maintain behavior in the longer term. 6. Apply behavioral treatments to address limitations and biases of individ- uals to ensure effective policy design. The recent revolution in behavioral economics research has provided a range of empirical evidence that contra- dicts the rational agent model, suggesting that people often behave in contrast to their best interest and that they are very sensitive to the way choices are framed. This is especially relevant in the field of decision making and personal 115 Financial capability in low- and middle-income countries: measurement and evaluation finance. Future research should therefore explore this area further and examine how behavioral models can inform personal financial decision making; specifically, it should examine what types of behavioral biases impact choices, in what contexts, and how to collect data around behavioral characteristics. Furthermore, it should examine the interaction between individual behav- ioral characteristics and external environmental factors that might influence behavior. 7. Integrate cost-effectiveness analysis in research design to inform scale. Scientific research must report on both the impact of an intervention on desired outcomes and on the cost of developing and delivering the inter- vention. This is especially important for LICs and MICs, because if a program is deemed effective but expensive, countries might not have the means to deliver at scale. It is understandable that in the early stages of research, when “successful interventions� have yet to be identified, it might not be necessary to invest in cost-effectiveness analysis (especially if no impact is detected). However, as the second-generation research in this area advances, it is crucial that studies that find impact report on both the benefits and costs. They must present cost-effectiveness ratios in terms of treatment cost of desired outcome per individual (or similar measures). Studies are also encouraged to focus on cost comparisons between alternative interventions, such as the cost of traditional financial education programs versus alternative methods. References Andrade, Eduardo B., and Dan Ariely. 2009. “The Enduring Impact of Transient Emotions on Decision Making.� Available at: http://duke.edu/~dandan/Papers/transientEmotions.pdf. Atkinson, Adele. 2008. “Evidence of Impact: An Overview of Financial Education Evaluations.� Consumer Research No. 68. FSA: London. Banerjee, Abhijit. 2000. “The Two Poverties.� Nordic Journal of Political Economy 26 (3): 129–41. Bertrand, Marianne, and Adair Morse. 2010. “Information disclosure, cognitive biases and payday borrowing.� Working paper, University of Chicago Booth School of Business: Chicago. Braunstein, Sandra, and Carolyn Welch. 2002. “Financial Literacy: An Overview of Practice, Research, and Policy.� Federal Reserve Bulletin 88 (11): 445–57. Cole, Shawn, Thomas Sampson, and Bilal Zia. 2009. “Financial Literacy, Financial Decisions, and the Demand for Financial Services: Evidence from India and Indonesia.� Harvard Business School Working Paper 09-117. Cambridge, MA. —. 2011. “ Prices or Knowledge? What Drives Demand for Financial Services in Emerging Markets?� Journal of Finance 66(6): 1933–1967. 116 CHAPTER 3. Measuring the effectiveness of financial capability-enhancing interventions Cole, Shawn, and Gauri Kartini Shastry. 2007. “If You Are So Smart, Why Aren’t You Rich? The Effects of Education, Financial Literacy and Cognitive Ability on Financial Market Participation.� Mimeo, Harvard Business School. Cambridge, MA. Duflo, Esther, and Emmanuel Saez. 2003. “The Role of Information and Social Interactions in Retirement Plan Decisions: Evidence from a Randomized Experiment.� Quarterly Journal of Economic. 118(3): 815–42. Duflo, E. 2006. “Poor but Rational?� in Understanding Poverty, eds. Abhijt Banerjee, Dilip Mookherjee, and Roland Benabou. New York: Oxford University Press, 367–78. Gale, William, Benjamin Harris, and Ruth Levine. 2012. “Raising Household Saving: Does Financial Education Work?� Social Security Bulletin 72 (2). www.ssa.gov/policy/docs/ssb/v72n2/ v72n2p39.html. Gibson, John, David McKenzie, and Bilal Zia. 2012. “The Impact of Financial Literacy Training for Migrants.� World Bank Development Research Group. Policy Research Working Paper 6073. Impact Evaluation Series No. 57. World Bank, Washington, DC. Hilgert, Marianne, Jeanne M. Hogarth, and Sandra Beverly. 2003. “Household Financial Management: The Connection Between Knowledge and Behavior.� Federal Reserve Bulletin 89: 309–22. Kahneman, D., and Amos Tversky. 1979. “Prospect Theory: An Analysis of Decisions Under Risk.� Econometrica 47(2): 263–91. Kahneman, D., P. Slovic, and Amos Tversky. 1982. “Judgment Under Uncertainty: Heuristics and Biases.� New York: Cambridge University Press. Karlan, D., and M. Valdivia. 2011. “Teaching entrepreneurship: Impact of business training on Microfinance clients and institutions.� Review of Economics and Statistics 93: 10–527. Kempson, Elaine, and Sharon Collard. 2010. “Money Guidance Pathfinder: A Report to the Financial Services Authority.� Personal Finance Research Centre University of Bristol. Available at: http://www.bristol.ac.uk/geography/research/pfrc/themes/advice/pfrc1002. pdf. Klapper, Leora F., Lusardi, Annamaria and Georgios A. Panos. 2011. “Financial Literacy and the Financial Crisis: Evidence from Russia.� Available at SSRN: http://ssrn.com/ abstract=1786826 or http://dx.doi.org/10.2139/ssrn.1786826. Lee, Nancy R., and Margaret Miller. 2012. “Influencing positive financial behaviors: the social marketing solution.� Journal of Social Marketing 2 (1): 70–86. Loewenstein, G., and Drazen Prelec. 1992. “Anomalies in intertemporal choice: Evidence and an interpretation.� Quarterly Journal of Economics May: 573–97. Lusardi, A., and O. Mitchell. 2006. “Financial literacy and planning: Implications for retirement wellbeing.� Pension Research Council Working Paper No. 1. Pennsylvania: Wharton School. —. 2007. “Financial Literacy and Retirement Preparedness: Evidence and Implications for Financial Education.� Business Economics 42 (1): 35–44. Mandell, L. 1997. “Personal Financial Survey of High School Seniors.� Jump Start Coalition for Personal Financial Literacy, March/April. Washington, DC. Mandell, L., and L. S. Klein. 2007. “Motivation and Financial Literacy.� Financial Services Review 16: 106–16. 117 Financial capability in low- and middle-income countries: measurement and evaluation —. 2009. “The impact of financial literacy education on subsequent financial behavior.� Journal of Financial Counseling and Planning Education 20 (1): 15–24. McKenzie, David, and Michaela Weber. 2009. “The Results of a Pilot Financial Literacy and Business Planning Training Program for Women in Uganda.� Finance and Private Sector Development (PSD) Impact Note 8. World Bank, Washington, DC. Mullainathan, S., and R. Thaler. 2000. “Behavioral Economics.� National Bureau of Economic Research Working Paper 7948. Boston: NBER. OECD/INFE. 2011. “Measuring financial literacy: Core questionnaire and guidance notes for conducting an internationally comparable survey of financial literacy.� Paris: OECD. Samuelson, W., and R. Zeckhauser. 1988. “Status Quo Bias in Decision Making.� Journal of Risk and Uncertainty 1: 7–59. Stango, V., and J. Zinman. 2009. “Exponential Growth Bias and Household Finance.� Journal of Finance 64 (6): 2807–2849. Thaler, Richard, and Cass Sunstein. 2008. Nudge: Improving Decisions about Health, Wealth and Happiness. New Haven: Yale University Press. 118 C CHAPTER 4 onclusions: from the main results to the next steps T his final chapter presents the main results from the World Bank executed RTF work program, puts the results into the context of international developments in this area, highlights open issues at the conceptual and empirical levels, and proposes priorities for the next steps. 4.1 Main results The results are presented against the ambitions and objectives of the RTF work program as outlined in chapter 1. As a reminder, the RTF was established to generate or deepen knowledge to assist LICs and MICs with implementing national strate- gies (NSs) to improve financial capability and outcomes at large, and particularly for the poor. Pursuing these global objectives, the results of the RTF should help in the implementation of Item 6 of the Declaration of the 2006 G8 Summit: “We acknowledge the importance of better financial education and literacy for improving the ability of people to use financial services and to make effective decisions with respect to their present and future welfare.� Handing this trust fund to the World Bank as fiduciary and implementer, the government of the Russian Federation strengthened its new role as a main financier of development aid and knowledge creation, but it also acknowledged the Bank’s role as a leading nonacademic research institution and think-tank on development. The OECD was a beneficiary under a separate RTF work program with a focus on stocktaking, policy analysis, and the development of principles and guidelines. This arrangement exploited the two organizations’ comparative advantages to their mutual benefit and that of the RTF. The World Bank’s work program focused on two measurement-related topics: (1) how best to measure financial capability over time and across different settings and individuals of different income levels; and (2) how to measure the effectiveness of interventions to improve financial capability that include financial education but go well beyond. This choice reflected a number of considerations that led to the specific design and implementation of the work program, such as: 121 Financial capability in low- and middle-income countries: measurement and evaluation „„ Effective NSs require a measurable concept of what they seek to improve that needs to be grounded in analysis that can be reliably undertaken in LIC and MIC settings with a special focus on poor people. „„ A workable concept needs to go beyond financial knowledge and skills (i.e., the traditional financial literacy concept) to include attitudes, behavior, and outcomes. This is embodied in the emerging concept of financial capability. „„ The concept of financial capability is positive; i.e., it is empirical and based on measurable outcomes and behavior that can be defined using the vox populi as a means to distinguish between capable and noncapable behavior; and it is a priori agnostic about which type of interventions can be effective in improving financial capability. „„ Identifying effective interventions to improve financial outcomes requires rigorous and comprehensive monitoring and evaluation (M&E) that is rare throughout the world and especially sparse in LICs and MICs. Rigorous here refers to the use of the best counterfactual available, while comprehensive refers to the use of both quantitative and qualitative methods: the first to establish causality, the second to shed light on the underlying change mecha- nism—the why and how of program impact. To establish a method for the measurement of financial capability levels in LICs and MICs, an extensive research effort was undertaken in eight countries. This began with focus groups to identify concepts and behavior that define a financially capable person and ended with a fully coded household survey questionnaire used to identify common topics (i.e., domains under one single score) for each participating country and across countries. The questionnaire was initially piloted in seven of the eight countries and later in four additional ones. The resulting findings and guidance on the use of the instrument and analysis of data, effectively comprising a financial capa- bility measurement toolkit, are now available for public use. To promote rigorous and comprehensive program impact evaluation in LICs and MICs and to garner knowledge about the effectiveness of specific financial capability interventions, the RTF program had three elements: (1) development of a toolkit providing methodological guidance on the design and implementation of evaluations of financial capability programs; (2) a stocktaking exercise to identify knowledge gaps to guide the selection of interventions to be assessed; and (3) a process for soliciting and financially supporting M&E for 17 competitively selected interventions in LICs and MICs. The following summarizes the key results from RTF work program. 122 CHAPTER 4.  Conclusions: from the main results to the next steps The results from all of these components confirmed the importance of an integrated approach to the measurement of financial capability and the asso- ciated interventions for its enhancement. The work on measurement helped to sharpen the understanding and definition of financial capability and how to best measure and apply it. The positive/agnostic approach to financial capability measure- ment helped open the possibilities for consideration of a much larger range of inter- ventions. The issues around the measurement of their effectiveness, in turn, helped in the development of the conceptual thinking on capability measurement. The work on the financial capability measurement toolkit and the analyses of the results of the questionnaires in seven countries (out of 12 participating overall) offer a rich set of information that leads to a number of conclusions, such as the following: „„ Perhaps most importantly, through the application of the positive/agnostic approach and use of the vox populi method, it is possible to identify a range of common attributes of financial capability that apply across very diverse low- and middle-income settings. The development work was undertaken in countries ranging from Papua New Guinea, Nigeria, and Armenia to Colombia, Lebanon, and Mexico. Most but not all of the identified topics resonate with the findings from similar efforts in HICs, including the United Kingdom where the approach was originally developed, Ireland, and other countries, where this method has been tested. „„ For LICs and MICs and specifically for lower-income groups, the relevant elements of financial capability are oriented to managing resources on a day-to-day basis and planning for the future. These have also been identified in HICs. However, two other topics found to be most relevant in HICs—the capability to choose among alternative financial products and finding and assessing information, help, and advice—found only limited resonance in the other settings. „„ The results are based on the capability of designing a questionnaire that both works across different income groups and quite different cultures and captures relevant manifestations of financial capability accurately without introducing an income or cultural bias. Clearly, this is a challenge that needs further testing and elaboration. „„ From the coded replies to the questionnaire, it is possible to create scores for individual components of financial capability (mirroring the manifestations identified above) that are robust and meaningful across different countries. The statistical process using the pooled data suggests the identification of two domains of capability that are relevant—“controlled budgeting� and “making 123 Financial capability in low- and middle-income countries: measurement and evaluation provisions for the future.� However, it is not statistically meaningful to collapse these into a single score for a uniquely comparable level of financial capability across different settings. While the two domains emerge in all investigated countries so far, cross-country comparisons need to be taken with a grain of salt as they may not be statistically robust given the subtle differences in the composition of these two domains across countries. While such analyses can be conducted at the individual country level, the number of domains needed to capture all the components of financial capability may differ from two to four. „„ Comparing countries’ scores for “controlled budgeting� and “making provi- sion� offers nevertheless interesting insights. The tentative results confirm higher scores for aggregate budget-related components and lower scores for “making provision�-oriented components across all countries. As regards the relative position of countries, the change is between the broad domains. For example, Lebanon emerges as a laggard for “controlled budgeting� and as a leader for “making provision.� „„ The results from all countries strongly suggest that people are better at living within their means and not overspending than they are at planning their spending, keeping track of their finances, or saving. They also tend to have short time horizons, being more focused on the present than the future. On the whole, the lower the scores were in the overall analysis, the greater the variability across countries. „„ Critical for guiding and focusing national strategies and financial capability interventions, the results demonstrate that the populations of individual coun- tries can be segmented into groups with varying levels of capability across all 12 of the identified components of financial capability. The strengths and weaknesses of these groups for each component can be determined—as can their demographic, social, and economic characteristics. Thus, it is possible not only to identify vulnerable groups that show low scores of financial capa- bility, but also to offer, through the identified components, first indications of the most appropriate intervention: financial education in some cases, behav- ior-oriented interventions in others. Whether such suggested interventions are truly effective, however, needs to be settled through rigorous evaluation of the program. The work on the impact evaluation toolkit, the preparation and implemen- tation of the 17 M&E pilots, and their preliminary results offer a wealth of information that will be further enriched as longer-term evaluations are under- taken. However, some early conclusions can already be drawn. The strongest and most valuable relative to the RTF objectives include: 124 CHAPTER 4.  Conclusions: from the main results to the next steps „„ Supporting rigorous and comprehensive M&E of financial capability inter- ventions works very well and makes a lot of sense from a public policy perspective: it provides firsthand knowledge on promising interventions; the interventions themselves can be solicited or at least influenced; the results allow an early understanding and correction of what works and what does not; the results save valuable financial resources over the medium term; and the approach is highly incentive-oriented, or can be structured to this end. However, this will require some innovative thinking and management efforts, as presented in the report. „„ Getting the most out of M&E, and at times making it worthwhile at all, requires upstream and downstream efforts. The lessons from the M&E support under the RTF work suggest that one should: (1) design the evaluation in conjunction with the program design upstream to be very clear about the objectives and hypothesis to be tested; (2) clearly identify knowledge gaps among interven- tions to target financing for priority areas; (3) wherever feasible, adopt mixed methods for evaluation to answer not only if there is a measurable outcome but also why and how; (4) attempt to isolate the specific factors responsible for the change—an approach particularly needed for outcomes and financial behavior subject to many influences; (5) conduct comparative examinations to measure the relative impact of programs and projects delivered in different settings or across different target groups; (6) explore variations in the quality and intensity or level of the service (e.g., education and information) delivered; and last but not least, (7) incorporate and test the insights from psychology and behavioral economics into intervention design and implementation. „„ The knowledge creation under the program did very well in reviewing existing, methodologically acceptable evaluations to identify gaps in interventions. This exercise was the first of its kind and was extremely helpful in guiding the solicitation and selection process for the evaluation studies that were supported. The program also did very well in promoting and evaluating new and promising interventions beyond financial education. „„ The 17 impact evaluations offer important lessons that governments, program providers, and researchers should consider in future interventions to improve financial capability to affect financial outcomes. In particular: –– Financial capability may be achieved through different interventions; programs that utilize mass media and social marketing tools promise to be especially effective and cost-efficient (but this needs further testing and evaluation). 125 Financial capability in low- and middle-income countries: measurement and evaluation –– The quality of content and delivery significantly affects outcomes for all interventions—both traditional financial education and more modern programs. –– One financial education program in schools in Brazil had a significant (short-term) impact on knowledge and outcomes (saving) that may be due to getting the relevant things right: it used high quality material; had well- trained and motivated teachers; used multiple interventions over a long period; and included parents. –– There are indications that, although knowledge and financial capabiity are correlated, in many instances, financial behavior improves even in the absence of improvements in financial literacy. This suggests that alter- native interventions and lessons from behavioral economics and finance should be explored. –– There are also strong indications that the impacts from one-time interven- tions fade over time. This calls for repeated treatment and other measures to achieve sustained effects. „„ Finally, regarding the aspiration to substantially move the knowledge agenda of what works or does not in traditional areas of interventions, the results to date are somewhat both sobering and encouraging. The limited evidence suggests that enhancing financial capability among poor households is extremely difficult to achieve, particularly with regard to planning for the future. Some pilots indicate that financial education may work for both knowledge and savings outcomes using a high-quality intervention that “gets everything right,� but this promising result needs to be confirmed under more varied circumstances. One general observation, however, seems to be that the more rigorous the evaluation, the less likely the program is to demonstrate a positive impact. This highlights the importance of rigorous evaluation rather than providing an encouragement to evaluate less rigorously. 4.2 The RTF results in an international context To better appreciate the RTF work program results, this section puts them in the context of the international discussion about issues, gaps, and priorities and high- lights their contribution. The most critical contributions are that: „„ The measurement of the improvement in financial literacy or capability and the rigorous evaluation of related interventions remain top priority topics. They were identified and selected by the RTF as critical items in the FL&E area 126 CHAPTER 4.  Conclusions: from the main results to the next steps in 2006/08, and they remain priority topics in 2013 for formal financial insti- tutions and NGOs alike. This is documented in the recommendations by the 2012 Citi Foundation commissioned “Report on Bridging the Gap� (Deb and Kubzansky 2012) and in the conclusions of the 2011 MasterCard Foundation commissioned report on “Taking Stock: Financial Education Initiatives for the Poor� (MasterCard Foundation 2011). „„ The RTF results are based on the first conceptually consistent and empirically rigorous work on the measurement of financial capability and of the effec- tiveness of the related interventions. The work is promising and ready for full replication in other countries across the world. Two fully fledged toolkits—the questionnaire toolkit and the M&E toolkit—can be downloaded, applied, and used to compare new results with those of the RTF pilots. The processes underlying the development and production of the toolkits are fully docu- mented and available on the RTF website (and summarized in this report and its appendixes). „„ The financial capability measurement results under the RTF are open for full comparison with other national measurement attempts by various countries (and reported in the stocktaking exercise by the OECD), as are the OECD/INFE trial project for a modular-type financial literacy/financial capability survey, the financial knowledge measurement questions by Lusardi and Mitchell (2006), and other attempts. This allows for a comparison at the conceptual level, but also for testing whether results are similar both within and across countries. „„ Along the same lines, the approach and results of the RTF M&E pilots can be compared with the few other attempts in this direction. For example, the impact evaluation sponsoring approach was used by DFID’s FEF, which sponsored 15 evaluation projects in Africa,1 and the RTF maintained a close relationship with its work (appendix D—available online at www.finlitedu. org—summarizes those results [FEF 2012]). Lastly, the Poverty Action Lab and the Innovation for Poverty Action are focusing on rigorous impact evaluation and are now extending the evaluated interventions to financial education writ large, with studies forthcoming.2 1  See www.financialeducationfund.org. 2  See www.povertyactionlab.org and www.poverty-action.org. 127 Financial capability in low- and middle-income countries: measurement and evaluation 4.3 Key open issues The World Bank–led RTF work program significantly pushed the knowledge agenda and frontier on measurement issues. Despite the progress made, many questions remain unanswered and new issues have been raised. This section very briefly summarizes the key open questions associated with conceptual issues, empir- ical issues, costing and financing of interventions, and networking and knowledge sharing. All of these issues are being increasingly raised by policy makers and formal and informal financial sector institutions. 4.3.1 Conceptual issues While considerable progress has been made on conceptual issues, a lot of heavy lifting still remains. This requires more country experiences with the now developed and alternative measures of financial capability and many more lessons from impact evaluations. Nevertheless, the to-do list includes: „„ Developing a broadly shared vision, definition, indicators, and measurement (i.e., conceptual framework) for financial capability. „„ Developing a conceptual framework regarding what kind and form of inter- ventions work best for each financial domain and situation (country, individual characteristics, etc.). For example, teaching individuals to draw up a budget may well be amenable to financial education and learning, while teaching them to plan for retirement may require some education, some nudging, and some advocacy. „„ Defining the role of financial sector actors in financial education and clearly delineating financial education from product promotion. 4.3.2 Empirical issues Progress on the conceptual side requires more empirical results to be better able to distinguish what matters for good outcomes and what does not, and to determine which interventions do work—when, why, and how—and which do not. The priority list includes the following issues: „„ Establishing empirically what matters for good financial outcomes. What is the importance of financial knowledge and cognitive skills (i.e., the original financial literacy idea) and for what kind of outcomes? What is the impor- tance of financial capability as defined by the vox populi and for what kind of outcomes? 128 CHAPTER 4.  Conclusions: from the main results to the next steps „„ Merging the measurement of financial capability and effectiveness of interven- tions into one conceptual and empirical framework. The indicators to measure the effectiveness of an intervention would also be part of the financial literacy/ financial capability indicator set. „„ Exploring further the measurement of the effectiveness of noncognitive interventions (such as advocacy, edutainment, etc.) on financial behavior. The outcome will have a major impact on the content and process of any national financial strategy. 4.3.3 Costing and financing of interventions Cost and finance issues have been largely ignored so far, or if acknowledged, have not yet provided good answers. This list of priority issues includes: „„ Identifying the costs of intervention, data for which are very limited in most countries. Addressing and solving the measurement and evaluation issues helps to identify effective interventions but does not yet identify cost-efficient ones. „„ Closing capability gaps where they exist. Such gaps are claimed to have widened in recent years due to the increase in access to financial products and services in LICs and HICs, while financial education is lagging well behind. This creates a nonsustainable and potentially very dangerous situation for the poor and for development outcomes, and is costly to eliminate. „„ Acknowledging the cost burden and externalities of interventions. The ques- tions of who profits from the interventions, who should pay for them, and who ultimately pays the burden have been little raised, and no good guidance is available. Ignoring externalities and fixed costs, the costs should be borne by those who profit from the intervention. Acknowledging the initial costs and externalities of interventions on development and economic growth provides space for the role of public subsidies, mandated interventions, and structured cost-sharing between the government, financial sector providers, and individ- uals. 4.3.4 International networking and knowledge sharing Tightly linked with the knowledge gaps around financial literacy/financial capability are the gaps in networking and knowledge access. The following highlights some priority areas revealed during the RTF work program: „„ Integrating private sector financial services providers (such as formal financial institution, MFIs, or NGOs offering financial services, financial education, or 129 Financial capability in low- and middle-income countries: measurement and evaluation both) into existing international networks to facilitate knowledge exchange on these issues. „„ Including private sector providers, who claim a shared (conceptual) framework and a knowledge infrastructure, to avoid redundancy and wheel-reinvention.3 „„ Providing public sector agents, at least in LICs and MICs, with detailed, rigorous, and constantly updated toolkits and knowledge banks in the area of FL&E. These may supplant previous sources such as Livings Standard Measurement Study (LSMS) surveys, Poverty Reduction Strategy Paper support, or more recently, CCT programs. 4.4 Proposed next priority steps Against this background of progress in knowledge and the large list of remaining knowledge gaps, the proposed next priority steps are as follows. „„ Strengthening the knowledge platform for all actors by: –– Encouraging countries to apply the financial capability survey tool (enhanced by modules on financial knowledge or other topics as they deem useful), undertake the analyses on domains and scores, repeat the survey periodically, and share their findings internationally. –– Motivating instigators of financial capability interventions to engage with M&E teams early on and to prepare for the discussion, preparation, and implementation by reading the toolkit for evaluation of financial capability programs. –– Exploring cost-effective ways to more systematically collect the results and lessons of impact evaluations across the world, particularly in LICs and MICs. „„ Using national financial (education) strategies more strategically by: –– Establishing the priority areas where more knowledge about the effective- ness of traditional interventions is needed and promising innovating inter- ventions are expected. –– Motivating the application of rigorous and comprehensive M&E by cofinancing the evaluation part and offering technical support from the very beginning of a project. See the recommendations in Initiative 2 of the 2012 “Report on Bridging the Gap� by Deb and 3  Kubzansky (2012). 130 CHAPTER 4.  Conclusions: from the main results to the next steps –– Encouraging peer learning among the providers of financial capability inter- ventions through facilitation, technical support, and knowledge dissemina- tion. „„ Reviewing and expanding financial (education) strategies by: –– Finding financial and administrative ways and means to expand impact evaluations from short- to medium- and long-term effects to gain a much- needed understanding about the time profile of their effectiveness—for financial education and other interventions. –– Exploring promising and highly cost-effective interventions such as edutainment and social marketing to improve financial behavior on a large scale. –– Making the search for cost-effectiveness of financial capability interven- tions a guiding principle of strategy and knowledge generation. References Deb, A., and M. Kubzansky. 2012. “Bridging the Gap: The Business Case for Financial Capability.� A report commissioned and funded by the Citi Foundation. Cambridge, Mass: Monitor, March. Financial Education Fund (FEF). 2012. “Lessons Learnt: An Overview of Lessons Learnt from Projects funded by the Financial Education Fund in Africa.� http://www. financialeducationfund.com/storage/files/Evaluation_Lessons_Learnt_from_FEF_ Final_2012.pdf. Lusardi, A., and O. Mitchell. 2006. “Financial Literacy and Planning: Implications for Retirement Wellbeing.� MRRC Working Paper No. 2006-144. MasterCard Foundation. 2011. “Taking Stock: Financial Education Initiatives for the Poor.� A report. Toronto: MasterCard Foundation. 131 The Russia Financial Literacy and Education Trust Fund was established in 2008 at the World Bank with funding provided by the Ministry of Finance of the Russian Federation. The work supported by the Trust Fund is jointly managed by the World Bank and the Organisation for Economic Co‑operation and Development (OECD) and is directed toward improving public policies and programs to enhance financial knowledge and capabilities in low- and middle- income countries. This effort has focused on the review of national strategies for financial education, the development of methods for the measurement of financial knowledge and capabilities, methods for evaluating the impact and outcome of programs, and research applying these methods to programs in developing countries. The products of this program of work can be found at the Trust Fund website at: www.finlitedu.org MINISTRY OF FINANCE OF THE RUSSIAN FEDERATION