T H E W O R L D B A N K O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T Adult Literacy A Review of Implementation Experience Helen Abadzi Director-General, Operations Evaluation: Gregory K. Ingram Acting Director: Ajay Chhibber 2003 Manager: Alain Barbu The World Bank Task Manager: Helen Abadzi Washington, D.C. ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank's Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. OED Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Contact: Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) e-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/oed Abbreviations and Acronyms ADB Asian Development Bank AIDS Acquired Immune Deficiency Syndrome BELOISYA Basic Education and Livelihood Opportunities for Illiterate and Semiliterate Young Adults CAAS Computer-based Academic Assessment System CIDA Canadian International Development Agency DABE Directorate of Adult Basic Education (Indonesia) DIKMAS Directorate of Adult Basic Education (Indonesia) DNFE Directorate of Non-Formal Education (Bangladesh) EFA Education for All EWLP Experimental World Literacy Programme fMRI Functional magnetic resonance imaging IDA International Development Association INA Institut National d'Alphabétisation IEC Information, éducation, communication LIL Learning and Innovation Loan msec milliseconds NFED Non-Formal Education Division (Ghana) NORAD Norwegian Agency for Development Cooperation NGO Nongovernmental Organization OPEC Organization of Petroleum Producing Countries OSEO Organisation Suisse d'Entraide Ouvrière PRA Participatory Rural Appraisal PET Positron Emission Tomography PRSP Poverty Reduction Strategy Paper REFLECT Regenerated Freirean Literacy Through Empowering Community Techniques SDC Swiss Agency for Development Cooperation SIDA Swedish International Development Cooperation Agency UNESCO United Nations Educational, Scientific and Cultural Organization UNDP United Nations Development Programme UNICEF United Nations Children's Fund USAID United States Agency for International Development i Acknowledgements Thanks are due to the Swiss Development Cooperation, whose grant through the World Bank Consultant Trust Funds Department inspired this report and supported literacy research in Burkina Faso. Thanks are also due to Gregory Ingram, Director General, Operations Evaluation Department, and Alain Barbu, Manager, OEDST, who supported the efforts to undertake this review. Furthermore, I thank my World Bank colleagues Martha Ainsworth, David Klaus, and John Oxenham, as well as Profs. Richard Venezky (University of Delaware), Irving Biederman (University of Southern California­Los Angeles), Alexandra Reis (University of Algarve, Portugal), John Comings (Harvard University), and James M. Royer (University of Massachusetts) for the diligent reviews and helpful comments. Thanks are also due to Keiko Sugishita and Kerri Henager for their help as well as to Maria Pilar Barquero. Finally, I would like to thank the National Literacy Institute of Burkina Faso, Organisation Suisse d' Entraide Ouvrière (OSEO), the Alpha Program of the Swiss Cooperation, the Promotion Femmes Développement Solidarité, and the Association Manegdbzanga for their help in carrying out literacy research in Burkina Faso. Helen Abadzi Sepember 15, 2003 iii Contents Acknowledgements ................................................................................................................i Executive Summary..............................................................................................................v Objectives and Scope of Review............................................................................................1 The Long History of Adult Literacy......................................................................................1 Bank Lending and Strategy for Adult Literacy in the 1990s............................................5 Outcomes of the 1990s Projects...................................................................................8 How Effective Are Literacy Programs Financed by Other Donors?....................................10 Limitation: Program Data Are Sparse and Hard to Interpret.......................................11 Dropout and Completion or Graduation Rates...........................................................13 Reading Achievement................................................................................................14 Income Generation as a Means to Acquire Literacy....................................................16 Social Benefits of Adult Literacy Programs ................................................................17 The Cost of Adult Literacy Programs.........................................................................18 NGO Implementation Involvement and Expertise........................................................22 Issues and Conclusions........................................................................................................23 Projects Reach and Retain More Participants............................................................23 ...but Do Participants Acquire Stable Literacy Skills?.................................................23 Instruction and Information Processing--The Forgotten Variables..............................25 Lessons Learned from Completed Bank- and Donor-Financed Projects.......................27 Appendix: Supplemental Tables.........................................................................................27 Annex A..............................................................................................................................33 Annex B ..............................................................................................................................39 References...........................................................................................................................41 Tables Table 1. Bank Lending Trends in Adult Literacy..................................................................... 4 Table 2. Ratings of Completed Adult Literacy Projects since 1990......................................... 8 Table 3. Efficiency Indicators Reported in Some Basic Adult Education Program Evaluations ......................................................................................................................................... 12 Table 4. Private Rates of Return to Education in Bangladesh, 1995/96................................. 19 Table 5. Cost Breakdown of the Senegal Women's Pilot Literacy Project ............................ 20 iv Supplemental Tables Table A. 1. Objectives and Components of Bank-Financed Literacy Projects, 1990­2002 .. 27 Table A. 2. Literacy Projects Financed by the World Bank in 1990-2002............................. 28 Table A. 3. Bank Projects with Literacy Components in 1990-2002..................................... 29 Table A. 4. Adult Education Lending, 1990-2002.................................................................. 31 Table A. 5. Unit Costs in Constant 1996 US Dollars of Bank-Financed Projects.................. 32 Table A. 6. Costs of Adult Literacy Programs and Four Years of Primary Schooling .......... 32 EXECUTIVE SUMMARY The Education for All goals include "achieving a 50 percent improvement in levels of adult literacy by 2015, especially for women, and equitable access to basic and continuing education for all adults." Adult literacy is also essential in fulfilling the Millennium Development Goals. This OED study assesses the extent to which adult literacy projects financed by the World Bank help towards achieving this goal. Many countries carried out literacy campaigns in the 1960s-80s through government-led, top- down and brief courses without follow-up. A minority of eligible participants enrolled, and of those about 50 percent dropped out. Of those who stayed on, about 50 percent passed literacy tests, and of those about 50 percent were estimated to have dropped back into illiteracy. Overall, many of the 1970s campaigns had efficiency rates of about 12.5 percent, with few participants acquiring stable literacy skills. The World Bank included literacy components in 45 of the 304 education projects financed between 1963 and 1985 in countries such as Yemen, Somalia, Afghanistan, Ethiopia, and Malawi. However, lending for literacy was only about 1­3 percent of education lending, and Bank- financed components for any type of nonformal education often accounted for less than 10 percent of the project costs. Given cost-recovery concerns of that era, there was almost no support for reading materials, library services, or audiovisual aids and no attempts to strengthen implementation capacity. Predictably, objectives could not be achieved with such limited investments. Among donors and governments, interest in adult literacy activities peaked in 1975­79 and diminished thereafter. The Bank financed almost no adult literacy in the 1980s and continued a cautious approach in the 1990s. In 1990-2002, just seven literacy-only projects were approved (in Indonesia, Ghana, Bangladesh, and Senegal), of which four were completed by the time of this review. Fourteen other projects had literacy as a component, usually a small one. Contrary to earlier times, the stand-alone adult literacy projects of the 1990s were large and complex operations. They focused on management issues and strengthened government support services. They relied extensively on nongovernmental organizations (NGOs) for implementation and instructional expertise, empowering them to teach literacy as they knew best (an approach called "faire faire"). The literacy-only projects of the 1990s were largely implemented as planned, and their outcomes were rated satisfactory by OED. Altogether, they reached about 10.8 million illiterates including many women and showed that it is possible to bring large numbers of people to classes. (Nevertheless, the projects that included literacy as a minor component rarely met their numerical or learning targets.) Overall, borrowers showed commitment and their performance was satisfactory, as was Bank performance. However, the extent to which participants learned to read and remained literate is uncertain. Monitoring and evaluation proved too cumbersome, and it was impossible to verify learning outcomes and to monitor closely the extent to which classes actually took place or how time was used. Thus, sustainability of the interventions is uncertain. Three Bank sector documents published since 2000 have summarized outcomes of adult literacy programs financed by governments and other donors. Data on student attendance and performance are sparse, hard to interpret, and have large variance. Of 32 literacy programs for which statistics were available, the median completion rate was 78 percent, median attendance (five programs only) was 62 percent, and pass rate of a final test was 56 percent. Relapse into illiteracy is rarely reported; some neoliterates can read years after instruction (for example the mainly previously schooled participants in Uganda), but only 12-60 percent of graduates sampled from other programs met literacy criteria later on (e.g. in Ajmer-India, Bangladesh, Kenya). Even when graduates remember the mechanics of reading, evidence suggests that they may understand little of what they read. Overall, course dropout and completion rates seem to have improved, but pass rates are about what they were around 1976, and long-term outcomes are uncertain. Some literacy documents emphasize social outcomes, such as self-confidence and empowerment. However, these benefits are typically assessed only once and through self-reports, so long-term effects cannot be ascertained. Better documented is the consistently cited willingness by participants to send their own children to school. Unit costs reported in sector documents also show much variance (US$12-1,246 annually), depending on whether teachers are paid or whether Ministry-levelstaff salaries are included. Volunteer teachers are hard to retain and replace, so the cheaper programs may be less efficient; however, the limited data available show no clear relationship between cost and acquisition of stable literacy skills. Overall, costs per participant are seen as low and are lower than primary-education costs, but costs per successful graduate are about three times higher. Despite low unit costs, adult literacy programs make up 1­5 percent of government or donor budgets, and they remain severely underfunded in comparison to primary education and number of potential beneficiaries. The modest efficiency of the interventions may be one reason why governments and donors are still ambivalent about financing adult literacy. Donors and governments financing literacy expect that once taught, people will remain literate. If a substantial number of neoliterates cannot read a few months after training, literacy instruction is seen as inefficient. A vicious circle of low financing and low efficiency may be created. So, the contribution of adult literacy programs in increasing literacy rates in various countries thus far has been marginal. Given its current scope and efficiency, this educational vehicle probably will not contribute significantly to a 50 percent reduction in adult illiteracy rates by 2015. Some features of human memory affect reading performance. Brain imaging studies show three regions activated in the brain during reading: two slower analytical neural pathways that are used by beginning readers and an express, instant word recognition pathway used by skilled readers (Annex A.) Fast reading is needed because our short-term memory is exceedingly brief; to understand written messages, a person must read a sentence within about 12 seconds. However, the express pathway may not be activated in adults as easily as it is in children. The slower analytical pathways used by novice readers challenge the limits of short-term memory. The buffer is filled with individual letters and overflows; thus people may forget by the end of a sentence what they read in the beginning. Novice readers, who must make conscious decisions about letters, can only read small amounts of text and may have to read a message repeatedly. To read a sentence within the 12-second deadline, learners at the end of a literacy course should be able to read at least a word per 1-1.5 second with about 95 percent accuracy. However, few attain this performance level; spotty attendance and wasteful use of class time are partly to blame. Research in Burkina Faso found that new literates took 2.2 seconds to read a word and were correct only 80-87 percent of the time (Annex B). Literacy skills are more likely to become permanent when read people fast and effortlessly. Skills that are not automatized tend to be forgotten rapidly after training. Furthermore, cognitive obstacles may add to the burdens of poverty. Cognitive research points towards a minimum standard for literacy acquisition: By the end of a literacy course, learners should read a word in about 1-1.5 second with about 95 percent accuracy. At this rate, readers decipher many script features automatically though they still struggle with comprehension. Adult literacy instruction should focus on increasing speed and accuracy, objectives that are usually not central in literacy courses. Literacy tests should be timed. Policy implications. Large-scale adult literacy dissemination is complex task that poses considerable managerial, technical, and financial challenges. But eliminating such investments may not be an option. In countries with very low quality of education, the next generation of adult illiterates is forming. Dropouts may later resort to literacy centers, and this instructional medium may become ever more important in meeting Education for All goals. Adult literacy instruction must become much more efficient than it currently is. Agencies and donors financing adult literacy should experiment with methods based on scientific research and select those that can be viably disseminated to teachers. Such methods (e.g. teaching phonological awareness and increasingly faster reading) are more complex and will probably require larger expenditures for teacher training and materials. Better qualified and better paid teachers may be necessary to impart the necessary quality and intensity of instruction. However, the benefits may be worth the costs in the long term. Though per-participant costs are likely to rise, per-graduate costs may remain the same or even drop as program efficiency rises. Hopefully implementing agencies can make the transition to more effective literacy programs. Key lessons from literacy projects and research discussed in this document are: · Donor attention to organizational and financing issues is crucial, but not sufficient to help learners acquire stable skills. Attention to instructional variables and scientific research is also necessary. The challenge is how to locate, abstract, operationalize and disseminate effective instructional methods to large numbers of implementers worldwide, many of whom have limited education. · Stand-alone adult literacy projects are more likely to be implemented as planned than isolated literacy components in projects of various sectors (e.g., rural development). If literacy is not the main objective of a project or at least not a significant component, it typically receives little financing, attention, or expertise in implementation and yields limited outcomes. The poor implementation record of isolated literacy components suggests that without resources for supervision, outcomes may continue to be unsatisfactory. · Though literacy instruction is considered to have low costs per participant, the costs per graduate made permanently literate are higher, and management costs are sizeable. The lowest-cost programs depend on a volunteer teacher corps, which may be unstable. A vicious circle may be created, whereby cheap but ineffective programs disappoint financiers and preclude more expenditure that might make them more efficient. Countries that decide to engage in adult literacy should consider their long-term commitment and should determine the extent to which they are willing to fund more effective but also more expensive programs. · Intensive government training and supervision of NGOs is important. Though many NGOs can carry out quality literacy programs, others need considerable support and monitoring. NGOs may reach beneficiaries in specific local areas effectively, but they do not always possess expertise in instructional delivery. · The monitoring designs conceived thus far in adult literacy projects are not sustainable. The research and evaluation capacity of organizations must be strengthened, but evaluation designs must also be simplified so that they can actually be carried out. 1 OBJECTIVES AND SCOPEOF REVIEW 1. Adult literacy is important in fulfilling the Millennium Development Goals and the Education for All goals. To what extent are Bank-financed literacy projects making progress towards achieving these goals. This OED study provides some answers by focusing on the following evaluation criteria: · Relevance: In what respects is adult literacy strategy fulfilling various human development needs? · Efficacy: To what extent have adult literacy projects of the 1990 met their stated goals? · Efficiency: How do the Bank's adult literacy project costs and benefits compare with the costs and benefits of projects financed by other donors? How well are participants trained in literacy, given costs and effort? · Sustainability: Will adult literacy programs continue to deliver efficient training as long as necessary? Will the reading skills of neoliterates become stable and useful? · Institutional development: What institutional support is needed for successful adult literacy programs and to what extent have Bank projects developed adequate structures? 2. The outcomes of Bank projects from the 1990s are juxtaposed with evidence from programs funded by other donors as well as with research related to adult literacy. 1 Annex A contains a summarized review of research on how adults learn to read fluently and how the process can be speeded up. Annex B contains the summary of a research study on improving literacy outcomes through cognitively based methods in Burkina Faso. THE LONG HISTORYOF ADULT LITERACY 3. Worldwide, nearly a billion adults, at least 600 million of them women, are illiterate.2 Over 70 percent of them live in nine large countries: Bangladesh, Brazil, China, Egypt, India, Indonesia, Mexico, Nigeria, and Pakistan.3 Most have been unschooled, but others are illiterate because of early dropout or inefficient schooling. The number of adult illiterates worldwide is larger than the number of primary-school students who amount to roughly 700 million. 4. Adult literacy is highly relevant to poverty alleviation efforts worldwide, because in the 21st century much of the information needed to make decisions and improve one's economic, personal, family, or political conditions is presented in written form. People must be able to decipher a script code quickly, understand the contents of the documents, and decide upon options transmitted in them. For these reasons, reduction of adult illiteracy is an important component of the Education for All (EFA) initiative, a global effort to achieve universal completion of primary education by 2015 and 1. To obtain answers regarding Bank lending in adult literacy, published Bank documents as well as internal reports were studied (that is, staff appraisal reports (SARs), project and implementation completion reports (PCRs and ICRs), project appraisal documents (PADs), and project performance assessment reports (PPARs) of projects that had had any literacy components). Requests for evidence of literacy effectiveness were sent to 17 task managers, 6 of whom responded. The project-related information in this report was updated to June 2002. 2. UNESCO statistics indicate that there were an estimated 962 million illiterates in the world in 1990, 885 million in 1995, and an estimated 887 million in 2000, constituting 27% of the adult population in the developing countries (Education for All Monitoring Report, 2002, http://www.unesco.org/education/efa/efa_2000_assess/studies/literacy_summary.shtml). 3. Aftab 1994, citing UNESCO report. 2 eliminate gender disparities in education by 2005. Since the 1990 World Conference on Education for All, the World Bank has acknowledged and committed itself to expansion of quality education, including goals to: · Reduce the 1990-levelof adult illiteracy (23.8 percent male, 39.5 percent female) by half by the year 2015; and · Expand the provision of quality basic education and training in other essential skills required by youth and adults.4 5. After a decade of effort, however, progress remains uneven and inadequate. Clearly, the EFA goals will not be met in at least 28 countries unless the pace sharply accelerates.5 But the gap in information availability between the rich and the poor of the world may actually be widening.6 Worldwide, over 100 million children are still out of school, and an undetermined number of others attend school for only 1 or 2 years. Instructional time in the schools of the poor tends to be limited, and dropouts are often illiterate. Unschooled or under-schooled children are the next generation of illiterate adults. So, as they slip through the schooling efforts, nonformal programs should be in place to capture them and efficient teach them their country's literacy code and related basic skills.7 Thus, effective and efficient instruction in basic adult literacy is needed more than ever. 6. Most developing countries acknowledge the need for continuing large programs in adult literacy and commit themselves to special efforts to eradicate literacy in the coming decade.8 (For example, the government of Senegal would like to make one million adults literate per year.) However, the delivery of adult basic education has a long and disappointing history. In the 1960s- 1970s, conventional wisdom regarded nonformal education as a cheap and rapid alternative to educate an entire population.9 Many countries carried out literacy campaigns in that period, a time of considerable thinking into literacy theory and practice.10 But these early programs offered government-led, top-down and brief courses without follow-up and made few people literate. A minority of eligible participants enrolled, and of those about 50 percent dropped out. Of those who stayed on, about 50 percent passed literacy tests, and of those about 50 percent were estimated to have dropped back into illiteracy. Overall, many of the 1970s campaigns had efficiency rates of about 12.5 percent, with few participants acquiring stable literacy skills.11 The ineffectiveness of those literacy programs dampened the interest of development agencies in this educational vehicle. 4. Targets are country-specific. UNESCO 2001. 5. Education for All Global Monitoring Report, 2002; Education for Dynamic Economies 2001. 6. Comings, Smith, and Shrestha, undated. 7. UNESCO defines literacy as a person's ability "to read and write with understanding a short simple statement of his/her everyday life" (UNESCO Statistical Yearbook 2001). The terms "nonformal education" and "adult basic education" (ABE or ABEL) are often used interchangeably with "adult literacy." However, nonformal education includes all forms of deliberately organized education outside the mainstream education system. ABE may include bridges to the formal system or other skills. 8. Many country reports are shown on the UNESCO web page, www.unesco.org. 9. Jones 1990. 10. For example, Couvert 1979, Hamadache and Martin 1986, King 1978, Scribner and Cole 1981. 11. UNESCO//UNDP 1976. The literacy programs discussed in this document were not random, and results may have been presented in an effort to neutralize the effects of socialist literacy campaigns (Jones 1988). 3 7. World Bank experiences with adult literacy were also modest. The institution included literacy components in 45 of the 304 education projects financed between 1963 and 1985 in countries such as Yemen, Somalia, Afghanistan, Ethiopia, and Malawi (Table 1).12 However, the Bank often offered adult literacy in lieu of primary education because the latter was deemed too expensive. 13 Thus, the institution financed literacy centers for young adults in Burkina Faso, while discouraging primary schools in rural areas (First and Second Education Projects, Cr. 430-UV and Cr.956-BUR in FY73 and FY80). A similar vehicle was used in Senegal (Second Education Project, Cr. 530-SE, in FY75). Stakeholders did not support what they perceived as second-rate education and instead demanded primary education.14 8. Nonformal education components were small. Bank lending for literacy was only about 1­3 percent of education lending. In most cases, the components accounted for less than 10 percent of the project costs, and even those amounts were often deleted from country budgets.15 Perhaps because of Bank cost recovery policies at that time, there was almost no support for reading materials, library services, or audiovisual aids. There was also no strengthening of implementation capacity. Predictably, objectives could not be achieved with such limited investments. Gradually, nonformal education components were converted into formal education. Interest in adult literacy activities peaked in 1975­79 and diminished thereafter. 9. A review of nonformal education projects approved in FY63­85 documented the limited support given to literacy components16 and concluded that the results were due to neglect during appraisal and supervision. The authors recommended greater attention to the nonformal subsector, actual training of adults, more investment, and a focus on learner demand. However, the conclusions of the review were construed to mean that literacy and other subjects of nonformal education were ineffective investments, and almost no literacy projects were initiated for a decade.17 10. The difficulties of the early adult literacy programs prompted reflection in the donor community regarding the definition of literacy and whether adults really needed to be literate.18 One school of thought argued that people need basic skills for living rather than literacy per se. Also, an expectation developed that universal primary education would expand rapidly, and that literacy programs would be unnecessary, so many donors declined to finance them.19 In the 1980s, however, it became obvious that access to primary education itself was linked to parental literacy and that universal primary education was going to take much longer than expected. Donors restarted literacy projects in the 1990s, trying to improve project designs. However, the Bank remained reluctant to 12. Romain and Armstrong 1987, p. 46. Only two were freestanding nonformal education projects, Loan 1486-IND and Loan 2355-IND in Indonesia. 13. Haddad 1994. The policy of limiting primary education in Burkina Faso to urban areas only was first conceived by the French government in 1959, and other donors followed suit ( p. 179). 14. Only eight of the planned 20 centers were built in Senegal, and eventually they became primary schools. 15. Romain and Armstrong 1987. 16. Romain and Armstrong 1987. The review encompassed 92 nonformal education projects, of which 49 percent financed adult literacy, while 77 percent financed rural skills development and income generation. Only about 35 projects had significant nonformal education components. In the 1980s, the Operations Evaluation Department did not have detailed project ratings, so exact outcome ratings are not known. But the eight adult literacy projects considered successful had strong country commitment, institutional support, strong materials support, and availability of counterpart funds (Romain and Armstrong, p. 35). 17. Lauglo 2001, p. 13. 18. Bennett 1995; Eisemon, undated. 19. Iredale 1994. 4 finance literacy through the 1990s. Adult literacy was specifically not dealt with in the 1995 education sector strategy paper, though the authors acknowledged that adult illiteracy would remain a major problem.20 This stance prompted some criticism that the institution was neglecting adult needs due to orthodox economic ideology, devotion to a narrow human capital theory, and need to appear a commercially viable institution.21 Table 1. Bank Lending Trends in Adult Literacy Period Activities Rationale/Policies Financing Design Features 1963­ 45 projects with adult literacy - Skills training, particularly - Civil works - Government agencies 1985 components of young farmers - Technical assistance teaching (2 stand-alone projects) - Limiting expansion of - Fellowships literacy formal education - Limited funding for - Supply-driven, top-down - Confused focus, unclear materials - Limited Bank supervision whether primary or literacy - Limited needs assessment - Instruction in official - No provision for recurrent languages expenditures - Insufficient counterpart funding - Under-use of technology (correspondence, broadcasting) - Few evaluations 1985­ - Almost no new lending Literacy activities considered 1989 for adult literacy ineffective or unnecessary - Second Indonesian project 1990­ 7 stand-alone projects; - Poverty alleviation - Recurrent expenditures - Community demand 2002 14 projects with adult literacy - Targeting women - NGO operating - NGOs teaching literacy components - Adult productivity expenditures under government - Maternal competence - Teacher salaries supervision - Complementarity with - Teaching materials - Post-literacy universal primary education - Limited civil works - Intensive supervision for efforts literacy-only projects - Instruction in local - Limited supervision for languages literacy components - Evaluation efforts, though of limited success 11. Since 1990, the Bank has financed only seven free-standing adult literacy projects (four of which were completed at the time of this report; Appendix Tables A1 and A2), with total commitments amounting to US$200.2 million. As of January 2002, adult literacy was a component in 14 other education, women in development, and rural development projects, of which six have closed. In all cases, the literacy component has been minor; it accounts on the average for about 5 percent of costs in projects where the literacy cost could be estimated (that is, about US$11.7 million out of US$229.66 million; Appendix Figure 1). However, lending is on an increase. Six of the projects that include literacy as a component became effective since 1999, and more Bank lending is in the pipeline. For example, Morocco is implement a Learning and Innovation Loan (LIL for US$4.1 million) in adult literacy, and the Poverty Reduction Strategy Papers (PRSPs) of countries such as Burkina Faso, Mauritania, and Uganda, mention adult literacy activities as a means of poverty alleviation and attempt to link the formal and nonformal aspects of education in a single strategy. About 15 countries in Africa alone have preparatory activities for nonformal education. In addition, the Bank provided a grant (through the Development Grant Facility) to an NGO that teaches literacy 20. Priorities and Strategies for Education. 1995. p. ix, p. 15. 21. Jones 1997, pp. 370-372. 5 through innovative means (Box 1). The Human Development Network education anchor (HDNED) has staff with an exclusive responsibility for nonformal education. The anchor has supported regional capacity building, a distance learning course in managing literacy programs, a website (in English, French and Portuguese), and several literature reviews. Bank Lending and Strategy for Adult Literacy in the 1990s 12. In contrast to the earlier financing of supply-driven, top-down government instruction implemented by weak public institutions, the projects of the 1990s focused on generating user demand and financed the activities of NGOs, which often ran programs that retained participants and appeared to be more efficient. Government institutions were to be used mainly for policy and monitoring. Thus, the Bank's strategy for the 1990s generation of projects was to: · Strengthen government adult literacy agencies to prepare materials, administer funds, monitor and supervise various types of community and nongovernmental associations (NGOs); · Shift responsibility for instructional delivery of adult literacy from governments to NGOs; · Develop eligibility criteria for screening NGOs, finance programs of qualified organizations; · Give NGOs support and textbooks and empower them to teach adult literacy as they know best (an approach that came to be called in French "faire faire" ­ let them do it); · Teach in local languages when possible and keep course length at about nine months (followed by post-literacy when possible); · Focus on women and out-of-school adolescents; · Focus on groups of workers, such as cooperatives; and · Where possible, link literacy with microcredit, income generation, and rural development. 13. All seven of the Bank-financed adult literacy projects state their goals and components in terms of actions for increasing access to adult basic education, developing institutions capable of overseeing nonformal education, and specifying numerical targets to be reached (Appendix Table A1). The Senegal Female Pilot Project expected to lower the national illiteracy rate. All featured significant monitoring and evaluation activities. Briefly, the completed projects had the achievements and challenges described below. (Information culled from project implementation completion reports (ICRs); also see Appendix Table A2.) 14. The Indonesia Third Nonformal Education Project(FY92­FY00) was the latest in a series that started in Indonesia in the 1970s to educate the 23 million illiterates in the country and provide income-generation opportunities for many of them. Initially, the government provided most training, but in the last three years of the project, more latitude was given to community organizations. The project lasted nearly seven years and reached about 6.2 million people, exceeding by a large amount its initial target of 1.8 million. Literacy was taught only in Bahasa Indonesia. Instructional quality was variable. The basic, three-year centralized and book-based literacy program failed, as many participants abandoned it, on average after 1.6 years. (Often the later courses were unavailable.) Classes were large, books were often not distributed, and supervision was minimal. Teachers were essentially volunteers; they received only five days of training, used class time inefficiently, and had 6 high turnover. Income-generation programs did not meet their targets, and small village libraries did not function as expected. However, a later, learner-centered approach that relied on community organizations, retained more participants. The project supported a large administrative structure (the Directorate of Adult Basic Education-DIKMAS), that paralleled the formal education system and absorbed 31 percent of project cost. Proceeds helped improve a national support system for teacher training, book development, and community outreach. However, an effective monitoring, supervision, and evaluation system could not be established, partly due to understaffing. The Bank supervised the project infrequently. Project outcome was rated marginally satisfactory. 15. The Ghana Literacy and Functional Skills Project (FY93­FY97) enrolled 1.3 million participants, of whom 726,714 had completed a nine-month training course by the end of the project. Dropout was about 22 percent, but participant achievement was not tested in the earlier years and remained unknown. Teachers were volunteers, but their retention rate for the next course cycle was 85 percent. The project's first four years were largely occupied with establishing a nonformal education administration, which absorbed 30 percent of project cost. Project performance was affected by the unsatisfactory leadership and large turnover of the nonformal education department as well as by limited counterpart funds. Other problems included staff training and technical assistance to participating NGOs. Literacy was taught in 15 languages, a strategy that posed logistical problems. A beneficiary assessment upon completion found that informally sampled graduates could read fluently out of the Bible and their textbooks, but had low reading comprehension. Demand for newspapers was much lower than what had been expected.22 The health messages transmitted in classes apparently did not change many participants' behaviors. Monitoring and evaluation functions were partly implemented, and class records were very weak, sometimes falsified; complete statistics on participants existed even when basic class records had not been kept, partly because teachers found the forms hard to complete. Many supervisors failed to visit regularly the 15 classes assigned to them. The project was rated satisfactory, and a follow-on operation was initiated (Cr. 3251-GH; Appendix Table A1), which built upon the lessons of the previous project and is emphasizing quality of learning, class supervision, learning assessments, and monitoring. 16. The Bangladesh Nonformal Education Project (FY96­FY02) was led by the Asian Development Bank with cofinancing by the World Bank. The project supported capacity development of the Directorate of Non-formal Education (DNFE) to reach large numbers of people effectively and make about 2.5 million participants literate through contracting with qualified NGOs, providing textbooks, and training. The nonformal education directorate simultaneously implemented three other donor-financed projects and was overextended. The evaluation and monitoring capacity could not be developed. Literacy was offered in voluntary nine-month classes, as well as through total literacy movements, where all able-bodied community residents were asked to study. The project reached 2.9 million people, of whom 64 percent were women. Dropout was estimated by the government at 10 percent; women became more aware of the services available to them and were more likely to complete the course. About 92 percent of the participants who took final tests passed them, but many neoliterates preferred not to take the tests. A 1997 validation study retested completing participants in 307 of the 6,075 centers and found that only 32.3 percent of the participants passed the same test. Another study found that 2.5 years after completing the program, only 34.5 percent of sampled graduates passed a reading test, 32.4 percent passed a writing test, and 26.9 percent passed an arithmetic test. By comparison, 87 percent of graduates reported that they wrote from time to time and 97 percent reported 22. Korboe 1997; p. 9, 19. 7 that they still read two years after program completion.23 The project was rated satisfactory, and a follow-on post-literacy project became effective in March 2001 (Cr. 3251-BAN; Appendix Table A1). 17. The Pilot Female Literacy Project of Senegal (FY97­FY02) supported the first phase of the government's 10-year program, which aimed to reduce the illiteracy rate of people aged 10­39 years from an estimated 55 percent in 1995 to 25 percent by the year 2005; female illiteracy would be reduced from 66 percent to 30 percent. The multistage project was to lower the illiteracy rate to about 40 percent overall, and to 47 percent for women, by reaching about 300,000 beneficiaries, 75 percent women during the first phase. The project finance private providers of literacy, some of which were pre-existing NGOs. Village associations played an important role in organizing classes and selecting instructors, who taught in local languages. However, pressures to politicize the process delayed implementation, and NGO training capacity proved lower than expected. An undetermined number of the participants had attended primary schools. In 1997, 50.7 percent of the participants passed reading tests (37.8 percent writing tests), and in 1998, 54.2 percent passed reading tests (47.7 percent writing). According to supervision mission observations, arithmetic problem solving performance was quite poor. The midterm review concluded that performance in certain classes was weak and that the objective of teaching sustainable literacy was not being achieved, despite progress. Ultimately, 191,557 people (87 percent women) were reached. The completion report indicated that in 2001, 75 percent met criteria for reading proficiency, 63 percent for writing, and 41 percent for problem solving. Though numerical targets were not met, there was considerable institutional development, and the outsourcing of adult literacy classes to private persons proved an effective way to deliver large-scale literacy instruction. 18. OED carries out project performance assessment reviews (PPARs) on samples of projects and rates outcomes on the basis of information gathered during on-site visits and interviews with beneficiaries. As of spring 2003, none of these projects had been assessed. Ratings are based on information reported in the completion reports (Table 2). 23. Ahmed and Lohani 2001, p. 8. These results, produced in 1997, were not reported in the ICR, which rated project outcomes satisfactory. 8 Table 2. Ratings of Completed Adult Literacy Projects since 1990 Credit amount Project Outcome Institutional Bank Borrower Development Sustainability Performance Performance and Cancellations Indonesia: Third Moderately Modest Uncertain Satisfactory Satisfactory US$69.5 million Nonformal Satisfactory Education Total US$4.8 Cr. 3431 million cancelled Ghana: Literacy Satisfactory Modest Uncertain Satisfactory Satisfactory US$17.4 million and Functional Skills (Cr. 2349- US$1.07 million GH) cancelled Senegal: Pilot Satisfactory Substantial Likely Satisfactory Satisfactory US12.6 million Female Literacy (not No cancellation Cr. 2873 reviewed by OED) Bangladesh: Satisfactory Substantial Likely Satisfactory Satisfactory US$10.5 million Nonformal US$1.7 million Education cancelled Cr. 2822 Note: Outcome, possible ratings: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Institutional Development Impact, possible ratings: High, Substantial, Modest, Negligible. Sustainability, possible ratings: Highly Likely, Likely, Unlikely, Highly Unlikely, Not Evaluable. Bank and Borrower performance, possible ratings: Highly Satisfactory, Satisfactory, Unsatisfactory, Highly Unsatisfactory. 19. Projects including adult literacy only as a component. Since 1990, 14 projects had a literacy component or subcomponent, mainly in the agriculture or rural development sector (Appendix Table A3). Descriptions in project documents were often cursory, and often no clear costs were assigned to literacy activities. Typically the activities targeted a few hundred people, and there was little evidence of institutional capacity to carry out the task. The performance of these components resembles that of components in the projects of the 1970s and 1980s. The Women in Development Project of Côte d'Ivoire spent US$60,000 on literacy activities and reached 550 women in 12 villages out of a target of 600 women; no achievements were stated in the ICR. The Gambia Women in Development Project targeted 10,000 women in 175 literacy centers but reached only 7,000; dropout was 75 percent, and only 30 percent of those who continued learned to read and write simple sentences in their own language. The Third Education Project of Gambia had vague targets and no indicators of achievement. Task manager responses indicate that supervision of these small components has been minimal, as staff in sectors other than education do not feel qualified to do it. Rural development projects in countries that are developing a reputation for effective literacy instruction (such as Burkina Faso) have received more attention, partly because communities demand literacy. Projects in Nepal and the Philippines reportedly reached larger numbers of illiterate women, but documents provided little detail. One operation where the literacy component received much attention has been the Morocco Social Priorities Project. It was expected to reach about 295,000 people by 2000 through 39 NGOs. Outcomes of the 1990s Projects 20. Numerical targets were met... The stand-alone adult literacy projects of the 1990s were complex and ambitious operations. Altogether they aimed to reach about 11.8 million illiterates over the period 1990­2001 (10.8 million through literacy-only projects and 1 million through components of other projects; see Appendix Tables A1 and A2). The projects that targeted women, reached large numbers of them, as expected. However, the number of illiterates reached in each country is modest in relation to the illiterate population. (For example, the 2.9 million participants in Bangladesh represent about 7 percent of the adult illiterate population.). Nevertheless, success in meeting targets is an important foundation for future expansion of programs. 9 21. ...but the acquisition of literacy skills is not well documented. Unfortunately, the outcomes and impact of the projects are uncertain. It is unknown how many participants attended until the end of classes and how many of the graduates acquired sustained literacy skills. Competencies were rarely tested. A number of participants had already been to school (for example, in Senegal) and may already be counted in official censuses as literate.24 Where literacy was a minor project component (as in Benin, Côte d'Ivoire, Gambia, and Lesotho) programs were less effective, and few people were made literate through the projects. The outcome measures reported above from the four completed projects are similar to those reported by other donor-financed projects (see below). 22. The main reason for these uncertainties is that the monitoring systems did not function well in any project, and it was impossible to collect the quantity or quality of data needed. Though all four stand-alone literacy projects had monitoring components, logistics for monitoring have proved complex and impractical. The number of staff needed to supervise the thousands of classes scattered in villages around a country may rival the number of teachers in the system. Countries have balked at these top- heavy and expensive arrangements, and better alternatives have not yet emerged. 23. Efficacy and efficiency were modest... Overall, adult literacy projects have not been large enough or efficient enough to improve significantly the borrowers' capacity to use their human, organizational, and financial resources efficiently. The modest literacy completion and pass rates suggest that many of the participants may not acquire stable literacy skills, or become reliably empowered.25 24. ...and sustainability is not evaluable. The sustainability of two completed adult literacy projects has been rated as uncertain. The extent to which literacy programs will continue to produce net benefits as long as intended and even longer is unclear. To reach large numbers of participants, strong institutions with a clear mandate to expand literacy are necessary, and the four evaluated literacy projects of the 1990s achieved that. However, the large expenses for institutional development may not be sustainable in the long run, particularly if donor assistance is limited. Also, reliance on NGOs has uncovered significant implementation weaknesses (see below), which create doubts regarding the extent to which the system can expand further through their leadership. 25. In the literacy-only projects, Bank performance was satisfactory. (For literacy components of other projects, Bank performance was moderately unsatisfactory.) The Bank paid much attention to management and administration--project documents are replete with discussions regarding how to set up a national literacy fund in Côte d'Ivoire or how to measure the economic impact of literacy through household surveys. The issues discussed in the documents reflect expertise in management and economic analyses. For example, in 2001 the Bank conducted a training program through videoconferencing with the Ministry of Literacy and National Languages of Senegal to strengthen staff capacity to deal with NGOs and disseminate knowledge to other West African countries. On the other hand, the Bank paid almost no attention to instructional effectiveness. None of the documents examined discussed methodology, classroom management, learner motivation, or how to improve learning outcomes. Very little attention was also given to teacher training structures and curricula. One reason might be that task managers were often economists or other specialists, such as architects, while technical consultants hired for appraisals did not maintain continuity for project 24. Reaching large numbers of illiterates is unlikely to make a difference in literacy statistics, because UNESCO, which publishes data, extrapolates literacy rates from the number of students reaching grade 4 rather than reporting survey or census data. 25. Empowerment was not a specific objective of any of the projects discussed but is often considered an important benefit of literacy instruction. 10 supervision. The Bank's strength in management was not complemented with technical expertise to ensure adequate information processing and skills acquisition in classrooms. 26. Borrower performance was satisfactory. Countries such as Bangladesh, Ghana, and Senegal showed much political will and ownership. However, borrowers were overwhelmed by the magnitude of the task. The significant budget requirements for counterpart funds resulted in large- scale staff recruitment, but also understaffing and turnover. Competent staff had to oversee the expansion of the programs, and their positions had to have continuity. Also, the local and community authorities had to be committed to the programs if success was to be ensured. Where only components of literacy were implemented, they were often not a priority, and they tended to be neglected.26 Also, where Bank-supported programs merely put literacy courses in place, the effect was limited. 27. Regardless of problems, relevance was high. The literacy projects addressed a high-priority area and indicate considerable implementation progress in this sector since the 1980s. HOW EFFECTIVE ARE LITERACY PROGRAMS FINANCEDBY OTHER DONORS? 28. Bank Sector Studies. Like the Bank, bilateral and multilateral donors have attempted to improve project designs and effectiveness in the 1990s. To understand the broader issues and lessons, three Bank reviews were carried out on the performance of various adult literacy programs in 2000- 02. This OED review reports data from these studies: · BELOISYAreport.27 A capacity-building and evaluation initiative for adult literacy was put in place in the Africa region, financed by Norwegian trust funds. The BELOISYA project examined the international evidence regarding the functioning of adult literacy programs in 1989­98 and conducted a series of workshops in 1999. Issued by the Human Development Network Education Department in 2001, the publication synthesized 27 schematic analyses of 33 evaluations of adult basic education programs financed by donors with a core of literacy. · "Engaging with Adults." This is a review of data that support investing in adult basic education in Sub-Saharan Africa.28 · "Including the 900+ Million."29 This is a synthesis of the BELOISYA and other reports that focuses on evidence for outcomes and effective implementation modes in 18 countries. 29. The following sections synthesize the findings of these studies as well as relevant information collected by the World Bank's Human Development network on programs that donors have implemented. 26. Eisemon, undated; Barnett 1995. 27. Basic Education and Livelihood Opportunities for Illiterate and Semiliterate Young Adults (Diagne 1999). 28. Lauglo 2001. 29. Oxenham and Aoki 2002, draft. The report covered activities in Bangladesh, Burkina Faso, Ecuador, El Salvador, Ghana, India, Indonesia, Kenya, Mali, Mexico, Morocco, Namibia, Nepal, Nicaragua, Sudan, Tanzania, Mozambique, and Uganda. 11 Limitation: Program Data Are Sparse and Hard to Interpret 30. The number of literacy programs worldwide is large; many are financed by governments and NGOs without external aid and some may target just a few hundred people. Main donors include UNESCO, UNICEF, bilateral agencies, such as Canada, Norway, Netherlands, United Kingdom, and large international NGOs, such as Africare, Plan International, Aide et Action, and Oxfam. In addition, a multitude of small local NGOs and communities are involved. Occasionally, the Asian and African Development Banks finance adult literacy. There is no estimate of the amount donors and governments spend on this subsector per year. 31. Adult literacy programs financed by other donors, like those of the Bank, have important data weaknesses. For example, only 13 of the 27 evaluations examined by the BELOISYA project reported on mastery of skills, and they used different methods to assess participant learning.30 (Table 3 presents the data that could be located.) Due to differing testing criteria, these statistics are not really comparable, and they are probably not representative of all studies. However, this is the available evidence. 32. Overall, effectiveness indicators are sparse, and it is unclear how many participants entered a program and how many emerged literate. Variance is very large, with tested pass rates ranging from 5 percent to 88 percent. It is unclear whether completion rates include all participants persevering in a class or just those who agreed to take a final test. Very few programs report attendance. Implementing literacy agencies and NGOs tend to have weak and possibly unreliable record-keeping, and often lack clear records regarding their participants and their progress. (Sometimes data may be inaccurate or falsified, as in the Bank-financed Ghana project). The nonformal nature of literacy is partly responsible. Participants may attend sporadically, and class composition may change substantially by the end of a course. Teachers, who are responsible for keeping the statistics, may have very low education. Many beneficiaries fail to take end-of-course tests when administered, and it is possible that the ones most likely to pass the tests take them. Thus outcome statistics may be inflated. Some programs (such as REFLECT) do not give final tests, and neoliterates are merely asked whether they have become literate. Participants are rarely retested after a lapse of time. 33. Clearly, data are difficult to obtain. Researchers report crossing rivers to go to remote villages and test graduates, who on that day may be away.31 And research in Burkina Faso found that correlation between self-reports on literacy acquisition and actual achievement was near zero.32 Therefore, the sparse and sketchy data leave questions about the effectiveness of adult literacy unresolved. Even when statistics are reported, there are usually no research designs to establish cause-effect relationships. Studies publish few instructional details or information about frequency and duration of classes, how instruction is carried out, or how efficiently class time is used. Documents do not report correlations between various non-literacy benefits (such as self-confidence) and reading performance, previous schooling, age, or number of hours of actual instruction. No research was found that controlled for the effects of prior knowledge or instructional quality. No studies were found that attempted to guard against the selection and attrition threats to the internal validity of the research or try to estimate them. Little long-term research exists, such as panel studies to measure performance of graduates and literacy's long-term impact on their welfare. 30. These were evaluations on Bangladesh, Burkina Faso, El Salvador, Ghana, India, Indonesia, Kenya, Mexico, Namibia, Nepal, Nicaragua, Tanzania, and Uganda. Several other studies also had limited data. 31. Cawthera 2001, on the difficulties of retesting learners in Bangladesh. 32. Abadzi 2003b. 12 Table 3. Efficiency Indicators Reported in Some Basic Adult Education Program Evaluations Project Attendance Ratesa Completion Ratesb Tested Pass Ratesc PRE-1975 GOVERNMENT PROGRAMS - UNESCO/WOALP Irand1972 -- 81.4% 64% - Turkey Government with primary school teachers, 1971­72d -- 67.6% 56.4% - Thailand Government 1971d -- 87.4% -- POST-1975 GOVERNMENT PROGRAMS - Kenya (1980­85)e 49.3% - 5% - program of 3096 participants 48% 29% reading, 21% writing - Namibia (1992­99)f -- - 92/99 Stage 1 -- -- 73% - 93/99 Stage 2 -- -- 79% - 93/99 Stage 3 -- -- 71% - 1996 evaluation - 70% 55% - Nepalg -- 50% 40% - Mozambique--multidonor project - 43.4% - POST-1975 GOVERNMENT PROGRAMS SUPPORTED BY THE WORLD BANK - Indonesiah (1977­97) -- 78.7% -- Level 1 -- 87.8% -- Level 2 -- 90.8% -- Level 3 -- 88.1% -- Level 4 -- 80% 17-27% - Ghanai (1992­97) 75%+ 89.9% - - Senegal (1996-2002) 66% reading 92% after course - Bangladesh--multidonor project) 10% (estimated) - 33% (later, 2 samples) NONGOVERNMENT PROGRAMS - ActionAid REFLECT (1996) Bangladesh 45.4% 77.9% 60.0% El Salvador -- 75.0% 67% Uganda -- 69.0% -- - ActionAid Control -- -- 88% Bangladesh -- 55.1% 25.4% El Salvador -- 40.0% -- Uganda -- 75% 56% - ActionAid REFLECT (1999) 70 - 100% Bangladesh 44 - 92% -- - World University Service REFLECT 48 - 100% -- Sudan (1999) -- - Nijera Shikhik (Bangladesh) tested 75% after one year - 56% - Save the Children/ USA, Kolondieba.l Overall 97.5%, (Mali) October 1998--May 1999, 11 groups with between 4 and 8 47-100% --- months instruction Range 86-100% - Gambia m 20% Median values of reported statistics 62.15% 78% 56% a. The general unreliability of attendance records has prevented most programs from including class attendance rates in their reports. A review of 27 programs (Diagne, World Bank, 1999) found that only three mentioned attendance rates. b. Completion rates indicate the proportions of enrolled participants reported to have formally completed the prescribed course, but do not indicate whether or not they achieved satisfactory levels of attainment. c. Tested pass rates indicate the proportions of enrolled participants reported to have passed whatever tests of achievement the program set. Many learners do not take them, and results may not be indicative of achievements. d. Oxenham 1974, p. 66. Oxenham's figures for Turkey were taken from the instructors' registers and numbers of tests submitted. e. Carron et al. 1989, p. 37-38, Tables II.3 and II.4. and p. 94, Table IV.21. A similar report on the literacy program in Tanzania found the reported statistics so unreliable that it declined to use them. f. Lind, A., 1996, p. 70-71; Government of Namibia, 1999, Ministry of Basic Education and Culture, Directorate of Adult Basic Education, The National Literacy Program in Namibia: Statistical Information. g. Comings et al. 1992, p. 212­226. h. Directorate of Community Education, 1998, Impact Evaluation of Nonformal Education Program in Indonesia. i. Implementation completion report on Ghana Literacy and Functional Skills Project, 1998. i2. Senegal : Pilot Female Literacy Project. Implementation Completion Report , June 2002. j. Archer and Cottingham 1996. Saldanha et al. 1999. Mace 1999. k. Cawthera 1997. l. Sanogo et al. 1999. m. BELOISYA (Diagne, p.11) Source: Table adapted from "Including the 900+ Million" 2001 draft. 13 34. Self-confidence and empowerment are frequently hailed as a benefit of adult literacy programs, but they are often measured through one-time self-reports. Self-reports may be the only practically available means to assess program outcomes, and verification may be time-consuming and expensive. But they are prone to cognitive illusions33 and social desirability of responses, particularly when no baseline data are collected. Yet, assessment instruments usually are not designed to deal with these biases. So, studies may report increased self-confidence or use of reading, but they do not offer more specific illustrations of these benefits, performance before training, or verify information independently. It is hard to generalize from unverified ethnographic self-reports about the social benefits of literacy,34 especially if beneficiaries later lapse back into illiteracy. Only one study of the 1990s (in Uganda) measured graduate performance in relationship to several variables but it did not assess what proportion of the initial cohort passed a literacy test.35 35. Perhaps the biggest concern about generalizing outcomes is self-selection. Participants decide when and if to go to literacy classes. It may not be possible or desirable to prevent self-selection, but the variables that determine it have not been researched, either. Which people are more likely to join and stay in classes? Motivation (however measured), age, slightly higher financial means, family support for women, higher or lower community status, group membership, age (for example, teenagers who do not yet work full-time), general intelligence, prior schooling are all possibilities, but their relative importance is not understood. Those possessing certain characteristics may be the easiest to teach, and it may be imprudent to generalize findings from self-selected participants to the general population. 36. The Bank's sector studies argue that there is sufficient economic justification for investing in literacy, but lack of good data makes it difficult to evaluate adult literacy projects in relation to alternative investments. More systematic donor-financed evaluations are needed to find out in greater detail what participants are and are not learning, as well as program impact in individual economic and social well-being. To find out more, the Bank started tracer studies in Ghana and Bangladesh in 2002. Dropout and Completion or Graduation Rates 37. Adult literacy programs tend to be heavily subscribed when they open, but often experience relatively low and irregular attendance rates and sometimes disappointing completion rates. This phenomenon suggests that demand exists, but the content does not meet the needs of some students or other factors intervene to limit attendance.36 The median dropout rate for 32 programs reporting results was 22 percent (Table 3).37 This is an improvement from the rough 50 percent rate of the 1970s, but dropout may be disguised. The median daily attendance rate for the five programs that reported this statistic is 62 percent (Table 3), which suggests that many participants often do not attend classes regularly or may effectively drop out but remain in the registries from which completion rates are reported. It is hard to become literate with desultory attendance. Observations 33. Bjork 1994, 1999. 34. For example, an ethnographic Philippine study showed that adults can learn to read and write under their own steam, albeit with much difficulty; and stressed the importance of promoting transition from literacy to literacy practices for sustainable literacy. 35. Okech et al. 1999. 36. BELOISYA (Diagne p. 8). In an ethnographic study, Kenyan participants reported desultory attendance to women's heavy workloads, negative attitudes toward adult education in their communities, and "laziness." (Fujisawa 2001). 37. For example, compared to UNDP 1976; Oxenham and Aoki 2002; Lauglo 2001, p. 78. Literacy classes usually last about nine months. 14 suggest that participants often repeat classes, but none of the above programs has reported statistics on repeaters.38 38. Data indicative of performance show great variability. The median completion rate of the programs reporting it was 78 percent (Table 3). Programs evaluated by the BELOISYA project reported relatively high rates; three of them reported completion rates of 90 percent or better, two reported 70­75 percent, and two others 30­50 percent. However, many programs do not collect this information. Some are still unsuccessful; a multidonor program in Mozambique reported low coverage and a dropout rate of 56.6 percent.39 Some programs report only those who pass a final test, while others report the percentages of participants who simply stayed in a course through the end. 39. Few of the evaluation documents reported data on the socioeconomic, motivational, and affective aspects of literacy programs, such as whether participants like to go to class.40 Such variables may be important because participation in classes may partly depend on the classroom environment.41 Fun, socially satisfying classes and no punishment for lack of participation may produce high retention rates. Good teachers may be able to retain high numbers of students, while bad teachers may lose many students. Obtaining well-designed qualitative data may help the adult literacy community design better programs. Reading Achievement 40. Of the programs reviewed in the "900+ Million" report, 20 reported completion rates and 14 reported pass rates at an end-of-course literacy test. (Since content and grading criteria differ, rates are not comparable.) The programs show pass rates of 5­89 percent, with a median of 56 percent of those initially enrolled (Table 3). Some of these figures may only indicate the percentage of those who passed among those who took the test rather than among those who completed a course. 41. Whatever the definition of pass rates, achievements are modest. For example, a 1997 evaluation of the Bank-financed Indonesian three-year nonformal education program42 reported pass rates of 17­27 percent. In terms of practical achievement, those graduates who reported they could read agricultural pamphlets were 6 percent before the program and 40 percent afterwards; the graduates who could read medicine instructions increased from 44 percent before the program to 75 percent, while those who could read newspapers increased from 12 percent before the program to 17 percent.43 Reading progress was reported as limited after the first year of study, though those who enrolled in subsequent years reported progress in writing and arithmetic. Almost no relapse into illiteracy was noted after completing the course. However, 62 percent of Indonesian participants had been to school (for an average of 2.4 years), so many of them may have been partly literate. 42. Reports from other countries have shown mastery by about half the participants or fewer. In Tanzania in 1990, 60 percent of a sample of 269 participants taking a reading proficiency test could read a short story of more than one paragraph, while another 11 percent could read syllables and simple words. Of the completers, 75 percent could do four arithmetic operations with three-figure 38. Kinda, Jules. June 2002. « Amerlioration de la qualité de la lecture ». Ouagadougou, OSEO. Unpublished report. 39. Patel et al. 2000. 40. In Nepal, the lowest-income women tended to drop out of the programs (Burchfield et al. 2002). 41. John Oxenham, personal communication, 1999. 42. Impact Evaluation of the Nonformal Education Program in Batch I and II Intensive Kecamatan; 1997. 43. Moulton 1997. 15 numbers, but only half of the completers could read, understand, and solve a real life problem involving such operations.44 In Kenya, 29 percent of 291 sampled graduates demonstrated that they were able to read and understand a short text related to their daily life, while 51 percent could read and understand simple sentences, and 20 percent performed below that level; 73 percent could do arithmetic operations.45 The Mexican National Adult Literacy Institute caters to about 1.1 million adults per year, of whom only 28.63 percent reportedly learned to read successfully through the courses provided.46 According to an extensive study of 967 Nepalese women in a three-year program, participants scored only 36 percent of items correct at the end of year one, 41 percent at the end of year two, and they had practically no writing skills; even the most involved group of women scored only 63 percent correct.47 Such performance levels raise doubts about the utility of skills acquired. 43. The most extensive and best-designed evaluation study to assess outcomes was carried out in Uganda in 1999. About 793 graduates were chosen to participate in a series of tests and interviews regarding lifestyle changes and were compared to 170 randomly selected nonliterates who had not taken an adult literacy course; 73 percent of the sampled graduates had been to school (for an average of 4.5 years), as had 35 percent of the nonliterates (average 3.5 years). In 12-18 months of study, graduates reached the achievement level attained in grade 4 of the "better" public primary schools. (Schools teach in English, whereas literacy centers teach in national languages.) Nearly every one of the graduates could read and understand very simple questions about their name, date, or name of the President, but few could answer more complex comprehension questions. The average comprehension score was just over 60 percent, and scores for writing tasks had a mean of just below 40 percent. Thus, the literacy achievement was rather limited.48 However, a two-year lapse since graduation led to virtually no erosion of skills in reading and calculation. The skills of some Ugandan graduates were shown to improve over time. 44. The tendency for primary education dropouts to attend literacy centers must become better understood. Perhaps under certain conditions literacy courses serve as "finishing school" for motivated school dropouts. This may be important in countries with high dropout rates, such as many African countries. It reinforces the hypothesis that instruction in foreign languages in primary school has an adverse impact in poor rural areas.49 Since literacy is typically delivered in local languages, participants do not have to struggle to understand basic information in a foreign language, as children do in primary schools. Given the limited instructional time and poor quality of many primary schools, perhaps students who did not manage to learn usable skills through official languages try to learn them through literacy courses. However, the research reported in Uganda and Indonesia did not relate language to literacy attendance, and data are not available to study this relationship. 44. Carr-Hill et al. 1991. 45. Carron et al. 1989, p. 189. 46. Instituto Nacional de Educación para los Adultos, 1998. Cited in Ardila et al. 2000. 47. Burchfield et al. 2002. 48. Okech et al. 1999; p. 19. Since many illiterates do not comprehend extensive spoken material well, the study might have assessed listening comprehension of readers. Literacy evaluations could benefit from baseline listening comprehension measures. 49. Some studies suggest that learning in a foreign language may create instructional difficulties for the poor and may cause dropout (e.g., Thomas and Collier, 1997). Also, students learning to read in a foreign language, particularly one with irregular spelling, like French and English, do not have a well-developed phonological processor in these languages to decode words from the context. 16 45. The various studies rarely report on relapse into illiteracy, which depends on class quality and amount of teaching.50 Given the usually deficient monitoring, it hard to trace graduates, but it is uncertain how much instruction actually takes place in various locales. Samples of learners retested in Bangladesh months or years after course completion showed that only 32­34 percent were able to pass. In the Bank-financed Ghana program, 40 percent were found to have lapsed into illiteracy in 1994, while in a government-financed Total Literacy Campaign in Ajmer, India, only 15 percent retained literacy.51 Another study measured relapse in the Nijera Sikhi program of Bangladesh. Three project sites were visited in 1997 and again in 2000, and the available and willing graduates were tested. Though this program reportedly has thousands of participants, only 19 could be retested over time. This number attests to the difficulty of locating beneficiaries and obtaining repeated measurements. In two sites, 73 percent of the tested participants were found to have maintained literacy skills, whereas at a third site 41 percent did. The low numbers and high selectivity of tested participants preclude any generalizations. An informal assessment in Burkina Faso in 2002 showed that many sampled rural participants who had passed a literacy test in 2001 had forgotten most letters a year later.52 46. It is unclear what standards should be used in judging literacy performance. One sector study considers 60 percent completion or pass a "respectable" outcome.53 But even if one accepts this standard, the self-selection bias complicates issues. If programs that have a 60 percent completion or pass rate are expanded to a less selected population, it may be hard to maintain even this completion level. To bolster the acceptability of the current literacy outcomes, it has been argued that they compare favorably with the high dropout and low attainments of primary education.54 However, schools for the poor may catch a population of children self-selected on different criteria and often spend very little time effectively teaching them. Failing to learn in a low-quality primary school is not a sufficient justification for accepting modest efficiency rates in adult literacy. Income Generation as a Means to Acquire Literacy 47. To improve literacy outcomes, livelihood skills and microcredit have been included in various projects. Often, participant groups are first taught skills that help their work, and they may progress to literacy only when they decide that they need it (an approach called "literacy second").55 A number of participants in these projects already know the rudiments of literacy. Income-generation programs are attractive to participants and may help keep them in classes. Some classes actually have literacy requirements, and those that do not are longer programs. However, the evidence of economic outcomes for these programs is not very positive;56 participants report some income improvements when asked, but these outcomes are not verifiable. There is little empirical research to prove that adult literacy programs enable unemployed learners to obtain new jobs or to make major career changes. One problem is that literacy instructors are usually not qualified to teach livelihood skills, and hiring special instructions often proves difficult and expensive. Overall, evidence from 50. Comings 1995. 51. Karlekar 2000. 52. Jules Kinda, informal report, June 2002. 53. Lauglo 2001, p. 37. 54. Lauglo 2001. 55. Oxenham and Aoki 2002, draft. 56. Oxenham et al. 2002. Female literacy participants reported increases in an index of social and economic development (Burchfield et al. 2002), but it was unclear what behaviors were associated with this development. 17 developing countries indicates that adult literacy programs rarely lead to actual economic improvements in participants' daily lives.57 Social Benefits of Adult Literacy Programs 48. The Bank sector documents "Engaging with Adults" and "Including the 900+ Million" discuss the "externalities" of literacy programs, benefits additional to the acquisition of literacy. The benefit best documented is increased willingness of participants to send their own children to school. A comparison of illiterates and neoliterates in Uganda showed that the latter were more than twice as likely as the former to report that they discussed schoolwork with children. Similarly, Nepalese neoliterate women became more engaged in their children's studies, and children of neoliterates in Bangladesh are more likely to go to school.58 Women who had participated in the extensive literacy crusade of Nicaragua in the 1980s tended to have fewer children, to have lost fewer children, to send their children to school, and to keep them there.59 Beneficiaries of the Bank- financed project in Bangladesh reported that they send children to school.60 It is unclear from the reports whether this benefit was a result of learning to read or of messages about children's schooling transmitted in classes. 49. Other benefits mentioned in conjunction to literacy programs are more effective oral (and sometimes written) communication, decontextualized language use, improved family health, more productive livelihoods.61 The effects of literacy on participants' quality of life, health, fertility, and productivity are well known when these have been acquired in primary schools, but they are less well documented for adult literacy classes. The beneficiary assessments of Ghana, Uganda, Kenya, and Tanzania suggest that relatively few graduates change attitudes or behave according to health messages transmitted during literacy courses. 50. Other benefits attributed to literacy classes are empowerment, self-confidence, and autonomy. Empowerment is central to the Bank's poverty alleviation strategy, not only because of poverty reduction potential, but because empowerment is connected to stronger governance and a participatory political structure. Literacy is considered crucial to empowering the poor (particularly the women). Most evaluations and reviews referenced in this document mention the self-confidence and attitude changes that participants receive from literacy classes. For example, Kenyan female learners in an ethnographic study talked about the desire to live well, to open eyes, to be enlightened.62 The individual empowerment effects are reported as strong and well documented, and adult basic education is considered a means by which women can take more charge of their lives.63 Evaluations and documents of the widely used REFLECT methodology (Box 1) extensively discuss the empowerment benefits of the program.64 57. Wagner 1995, p. 342-343; Oxenham 2002. 58. Okech et al. 1999; Burfield et al. 199, Laug1o 2001, p. 16-17. 59. Sandiford et al. 1995. 60. Bangladesh: Education Sector Review, 2001, vol. 2, p. 43. 61. Nepalese women's knowledge of family planning, HIV/AIDS, and awareness of sexually transmitted diseases increased for participants in a literacy class but not for a control group (Burchfield et al. 2002). 62. Fujisawa 2001. Statement of a 33-year-old female Kenyan learner, who had never been to school. 63. Lind 1997. 64. Archer 1996; Riddell 2001. 18 Box 1. REFLECT: Beneficiaries May Feel Empowered but not Be Literate The World Bank's Development Grant Facility was used in 1997-2000 to support a program whose methodology appeared highly effective in teaching literacy. Project REFLECT obtained US$980,000 for evaluation surveys, materials development, preparation of training videos, and an external evaluation. The Regenerated Freirean Literacy Through Empowerment Community Techniques (REFLECT) helps groups identify local issues and problems of shared concern through visual materials that are generated in each community: maps, matrices, calendars, or diagrams. Participants are encouraged to communicate about and act upon the problems. Also, they learn to recognize the words of the problems they identify through a whole-word method. Reportedly the approach interweaves literacy with empowerment and creates motivation, confidence, and self-esteem, particularly for women.65 Participants do not typically use textbooks; they generate their own according to their particular needs. Also, their literacy performance is not tested; at the end of the course, they are asked to estimate their reading ability on a rating scale. REFLECT has spread rapidly from small-scale programs in 25 countries to more than 350 organizations in 60 countries that include governments and NGOs. There are training and coordination units in countries such as Bangladesh, Brazil, El Salvador, Ghana, India, Mali, Mozambique, Nepal, Pakistan, Peru, South Africa, and Uganda; 250,000 participants were in the program in August 2000. In its earlier stages, the project had high rates of participant retention (69 percent) and reported rates of achieved literacy of over 60 percent (Salvador 86 percent; Bangladesh: 77.5 percent; Uganda: 99 percent).66 However, subsequent evaluations report mixed results.67 In Bangladesh, 50 percent of the participants enrolled in the course "circles" could read a paragraph and make basic calculations up to 100 after attending for a year. But for three sites of India, at least 50 percent of the participating women were unable or unwilling to learn literacy. Evaluators estimated that only about 25 percent of the Indian participants were becoming literate. Those who did were generally the younger persons in the circles. The others could identify and write words that had arisen in the course of the construction of the graphics, but they were unable to decipher most of the letters. People often dropped out when the program started teaching individual letters.68 Although this program was initially billed as a means to teach literacy, this activity is now just one option. Participants decide what to work on, and less than a quarter of them take up reading.69 Perhaps the method confers significant empowerment and development benefits to its participants, as self-reports indicate. However, without control groups, baseline data, random assignments, and measurements on important variables, it is impossible to know. The impact of the Bank's investment in this institution is uncertain. 51. Group communication and action have long been known to be effective in finding solutions that may alleviate participants' poverty. Participation in community activities may help improve literacy outcomes; for example, one study found that level of community involvement was the most important variable influencing literacy performance.70 However, it is unclear how literacy instruction per se creates empowerment and self-confidence. The learners who become fluent readers may become confident and empowered. Do the non-readers also feel the same? Do literate graduates stay self-confident if they lapse back into illiteracy? No answers have been found in the literature, and no analyses have been found that linked reading achievement with self-confidence. As mentioned earlier, empowerment reports may be due to social desirability of responses and cognitive illusions. Also, 65. Archer and Cottingham 1996. 66. Archer and Cottingham 1996. 67. Saldanha 2000; Riddell 2001. 68. Riddell 2001. 69. ActionAid International 2001. 70. Burchfield et al. 2002, p. 136. 19 participants may feel empowered because they are stronger as a group, or class attendance may give them an improved status in society. But merely being counted among the schooled may have a "placebo" effect on empowerment, and its development value may be dubious. 52. Many reports describe empowerment benefits,71 so it is possible that these exist independently of measurement artifacts or memory tricks. It is difficult at this stage to establish a cause-effect link between literacy courses and social benefits, including self-confidence and empowerment. Clearly, more behaviorally oriented research is needed to sift through the possible explanations and verify the extent and sustainability of social benefits to literacy instruction. The Cost of Adult Literacy Programs 53. Returns on Investment. None of the 1990s projects has yielded reliable rates of return for adult literacy.72 However, the1995/96 Bangladesh Household and Expenditure Survey was used to estimate a table on the private rates of return to adult education in Bangladesh (Table 4). The data suggest that successful completion of an adult literacy course can be valued as more than `No education' and equated with the effects of `Less than full primary schooling'. If so, then the private rate of return for adult literacy would be substantial. Table 4. Private Rates of Return to Education in Bangladesh, 1995/96 Level of Education % Less than primary over no education 37 Primary over less than primary 45 Junior secondary over primary 15 Senior Secondary/Higher School Certificate 22 Higher over secondary 24 Source: Bangladesh Post -Literacy and Continuing Education for Human Development Project, project appraisal documents, 2001p. 49 54. Costs. Only 9 of the 27 evaluations studied through the BELOISYA project offered cost information. Unit costs per participant ranged between US$12 (just instructional expenses) and US$1,246 (including development costs, international consultants and travel).73 Unit costs per completer ranged between US$61 (instruction only) and $984 (including development costs). Other data indicate that the unit costs of adult basic education programs vary from US$1.93 per participant in strictly voluntary programs up to US$37, when various levels of paid staff are included.74 For example, the REFLECT program, which enrolled 140,000 students in 26 out of 45 districts in Uganda, reported unit costs of US$4­5 per year, when teachers were volunteers, but US$13­14 when they were paid.75 The Indonesia program cost Rh 18,000 per participant annually (about US$90 in 1991), with volunteer teachers and organizers. A review of Bank-supported projects in five countries in 1997­2002 found that the unit cost per enrolled participant has ranged between US$5.46 and 71. Discussions on the meaning of empowerment: e.g., Hashemi, Shuler, and Riley 1996; Jejeebhoy 1996. 72. One effort to calculate returns was made at the end of the First Indonesia Nonformal Education Project, when individual rates of return at time (rate of growth of individual income compared to rate of growth of the cost of training) yielded a rate of about 22 percent. That compared favorably to a 22 percent rate of return to investment in primary education in Indonesia (Oxenham 2002). 73. BELOISYA 1999 (Diagne p. 14). 74. Oxenham and Aoki 2002. 75. Lauglo 2001, p. 33. 20 US$57.67, with a median of US$12.85. The cost per person completing a course has ranged between US$11.73 and US$73.65.76 Total cost per successful learner has ranged between US$36.43 and US$174.60, so it is about three times higher (Appendix Table A4). Thus, costs appear focusing on participants rather than on successful graduates, a method that underestimates the cost of literacy provision. 55. The low per-participant costs are presented as an argument in favor of expanding nonformal education,77 but they rarely represent all costs associated with literacy provision. Development costs are significant. The large monitoring and administrative institutions behind them have recurrent expenditures, which may not be included in the calculations. For example, the funds devoted to building up the Indonesia and Ghana administrative structures took up 30 percent of the total project costs, whereas those of the Senegal project were lower (Table 5). There is a need for better understanding of how various expenditures contribute to program effectiveness. Table 5. Cost Breakdown of the Senegal Women's Pilot Literacy Project - Remuneration of personnel 40% - Instructional materials 25% - Operational expenses 15% - Training personnel 10% - Institutional support 10% - Total 100% Source: Oxenham 2002 (draft) 56. These administrative costs are important to consider, because making large numbers of people literate through adult education may require a long-term investment and some "recycling" of participants. Tanzania and Cape Verde devoted about 10 percent of their national education budgets to nonformal education; yet it took them about 25 years (1961­86 and 1975­99, respectively) to raise adult literacy rates to nearly 90 percent in conjunction with primary education.78 Governments may take a short-term view and plan campaigns to make large numbers of people literate once and for all. However, the limited instructional effectiveness of the programs and the requirements of adult life imply that few people may be served at one time, and even those may be served inefficiently. Participants may take classes more than once.79 Where reading opportunities are limited, participants get little daily-life practice. So, it may not possible to make large numbers of people permanently literate over a short period of time. However, the experiences of Tanzania and Cape Verde suggest that with more investment and a long-term horizon, nonformal education may help the literacy level rise, perhaps partly due to increased demand for children's primary education. 57. Efforts have been made to compare nonformal and formal education costs. Oxenham and Aoki (2002) compared cost data available from eight programs and concluded that the unit costs of a program range from a high of half those of a country's primary school annual unit costs to a low of about one-seventh. In Nepal, a graduate of a 9-month adult literacy program can master the skills of a 5th or 6th grade primary school student.80 Data from Ghana, Bangladesh, and Senegal indicate that 76. Oxenham 2002. Due to currency devaluations, costs at completion were often lower than appraisal costs. 77. Lauglo 2001, p. 25-26. 78. BELOISYA 1999 (Diagne p. 18). 79. Abadzi 2003b. 80. Comings et al. 1998. In principle, some development effects of 4 to 6 years of primary schooling could be attained by respectable proportions of adolescent and adult learners through less than one year of adult/youth basic education (10-12 hours per week) at between one-sixth and one-fortieth the cost of a primary school graduate. The skills and information acquired could be applied immediately to improve quality of life (Oxenham and Aoki, 2002). However, schools in low- income provide limited skills in this time span. 21 adult literacy courses cost only 13-33 percent of primary school costs (Appendix Table A5), but it this is only under the assumption that adults will become and stay literate through courses of 300-400 hours compared to children's courses that last in principle about 3000-4000 hours. Ironically, adult literacy costs compare well with primary education because the latter often involves wastage, low quality, and high dropout. Yet, many of the participants have been to school and may have acquired some skills. The cost to make them literate should include the money already spent on them. 58. Literacy project costing creates dilemmas. One implication is that formal and nonformal education compete for the same funding, and perhaps expenditures on children should be preferred over expenditures to adults, who have a more limited lifespan. There is some internal Bank debate on this issue, and no clear consensus has emerged. One school of thought believes that children's education takes precedence, and adults should be allowed to remain illiterate. Others see literacy as a global public good that should be available to people of any age, particularly given the externalities that adult literacy may have for women and their improved ability to take care of their children's health and schooling. 59. Teacher pay. The success of the programs rests on the performance of literacy teachers. Because the literacy task is large, it is important to keep salary costs low. So, programs pay teachers very little or search for volunteers or part-volunteers, whom they pay with honorariums or gifts such as sewing machines, or bicycles. This means that a rather limited population is willing to teach literacy. These may be primary school teachers (as in Indonesia), adolescent students, and even literacy course graduates. They may have limited education and receive limited training, often for only five days. 60. Considerable literature exists regarding the pros and cons of paying teachers,81 and the conclusion is that they should be paid if programs are to be sustainable. Volunteerism is not an easy way to staff a program, especially a long-term program. For example, 170,000 volunteers were permanently needed in Indonesia; the turnover rate was high, and limited demands could be made of their time. In Bangladesh, it was also found that teachers were hard to sustain without sufficient remuneration. Ironically, the civil servants who staff the nonformal education departments receive regular salaries and training to oversee the programs, but those who actually deliver it receive very little. 61. The donor community now understands the importance of paying teachers sufficiently; the Bank's Human Development Network expects teacher salaries in formal education to be 3.5 times a country's per capita income.82 To sustain literacy programs, the same guidelines should apply to literacy teachers. If costs are low and volunteer teachers are used, programs appear inexpensive, but costly inefficiencies may be introduced. Money saved by not paying teachers may have to be spent on paying administrative staff to handle the turnover and on additional courses to teach those who have failed. If teachers are trained and paid sufficiently, the quality of the programs may improve, but their cost will also rise. 62. The effectiveness of cheaper versus more expensive programs has not been systematically evaluated, but experience shows that few have both low per-participant costs and stable, well- performing teachers. Program costs and effectiveness need to be considered with a long-term strategic perspective, in order to avoid a trap of low cost and low effectiveness. 81. For example, Rogers 1991. In the 1970s, efforts were made to pay teachers according to the numbers of people made literate (e.g., in Nepal), but it was found that literate people sometimes came to classes and passed tests. 82. Education for All (EFA) Fast Track Initiative ­ Draft Proposal. Human Development Network, May 9, 2002. 22 NGO Implementation Involvement and Expertise 63. Donors and governments in the past decade have empowered NGOs to become leaders in teaching literacy. However, it is unclear how much instructional or organizational expertise these organizations have. Availability of funding has encouraged some NGOs to emerge and vie for it or to expand into literacy beyond their areas of expertise. On some occasions (such as the Senegal Pilot Female Literacy Project), private persons may provide literacy instruction. Though this may be a legitimate means to teach large numbers of people, it is unclear which persons and NGOs are effective in teaching literacy. Under the worst conditions, they may be entities set up by relatives of various officials to take advantage of financing.83 The Bank-financed Indonesia and Ghana project reports indicate that the pedagogical support services to NGOs were not effective in improving outcomes. Some irregularities were reported in Bangladesh, such as NGO teachers pooling classes of multiple centers or appointing low-paid assistants in their stead. 64. A 2001 workshop in Dakar, brought about the following views about NGO delivery of nonformal education:84 · NGO strengths: there may be transparency, equity of access, increase in demand for literacy, systematization of the partnership approach, coherence of government action, reinforcement of capacities, greater expertise on literacy instruction. · NGO weaknesses: operators may have limited capacities, be insufficiently decentralized, not integrating literacy with other skills, having high staff and institutional turnover, not always focusing on quality. 65. The "900+ Million" review concluded that reliance on the work of NGOs and community- based organizations would not satisfy the targets to which the Bank is committed. Governments must at least provide an encouraging framework within which voluntary and community association can develop the capacity to reach program participants. Therefore, the Bank's projects should strengthen both government and nongovernment agencies. The authors contend that programs run by government agencies are not necessarily less efficient than those offered by voluntary organizations. However, the reliability of the statistics presented is unknown; both governments and NGOs are known to be generous in their estimates of their own achievements. Where voluntary and community- based programs are not feasible, governments may have to implement nonformal basic education directly.85 83. Eisemon, undated. 84. Diagne, Amadou Wade. 2001. Programme de formation sur la stratégie du `faire faire' en éducation des adultes. Dakar: Ministry of Literacy, May 7--July 5, 2001. 85. Oxenham and Aoki 2001. 23 CONCLUSIONS "Does Adult Education contribute to development? Yes but... no wonder-working."86 Projects Reach and Retain More Participants... 66. Overall, the Bank and the rest of the donor community implemented similar strategies toward adult literacy since the 1990s and obtained similar results. Compared to earlier projects, designs have become more elaborate, focused on organizational effectiveness and financial viability. Women are targeted and reached effectively. Literacy efforts have moved from vertical government-only campaigns to strong institutional development, culturally sensitive textbooks, NGO participation, faire faire, and reliance on community and group demand for income generation. Bank and other donor projects are able to reach millions and do so at a cost that appears low, though some areas of savings, such as low teacher pay, may create inefficiencies. These newer operations have lower dropout rates and may retain more participants in classes than in the past (for example, a reported 22 percent dropout and up to 78 percent retention compared to 50 percent dropout and completion rates reported by UNESCO/UNDP in 1976). Thus, contemporary projects help more participants complete literacy courses than the projects of the 1970s, which retained roughly 50 percent of the participants. Attention to management, financing, partnerships, and institutional development has paid off. ...but Do Participants Acquire Stable Literacy Skills? 67. Participant performance has apparently not changed much since the 1970s. The 56 percent median completion rate for reported projects of the past decade (Table 3) is near the estimate reported by UNESCO/UNDP in 1976.87 Lack of monitoring and testing standards brings even these modest results into question. Should the Bank and other donors finance projects that have a known 40 to 50 percent failure rate even with a self-selected clientele? It is arguable that the glass is half-full, and with the large numbers of people reached, this inefficiency may be expected. If such results were widely considered satisfactory and obtainable, several large projects would have been developed. However, governments and many Bank staff seem to prefer pilot projects or components in formal education operations that account for about 5 percent of project costs. This percentage is low, as was the 1­3 percent of education lending devoted to literacy among projects in 1963­85. Despite rhetoric regarding the desirability of adult literacy, there is continuing ambivalence toward financing such activities.88 Similarly, literacy budgets in many countries are just 1­2 percent of education budgets.89 68. The reluctance may be due to the modest outcomes. These may perpetuate a vicious circle, whereby cheap but ineffective programs disappoint financiers and preclude more expenditure that might make them more efficient. Governments and donors continue to spend very little on literacy, so the implicit choice is low-cost and low-efficiency programs. 86. Oxenham 2000, workshop presented at the World Bank. 87. The few available data on relapse into illiteracy include Uganda (almost no relapse) and samples of the Bangladesh program (a drop to 32-34 percent proficiency). They offer conflicting evidence, but relapse was roughly estimated to be about 50 percent in the 1970s. 88. Jones 1997. 89. The figure includes certain vocational programs for unschooled adults (Oxenham and Aoki 2002). No study was found that compared the costs of various programs with literacy outcomes, possibly because the costing methods are not directly comparable. 24 69. An example of disappointing outcomes is limited or forgotten skills. Forgetting is a normal operation of human memory, and literacy skills are not exempt. To justify an investment in literacy, however, donors and governments expect that once taught, people will remain literate. This can indeed happen, but only if reading functions become automatic. Given the desultory teacher and participant attendance, many or even most learners probably fail to become automatic readers. If participants read infrequently and must make conscious decisions about letters, they read only small amounts of text. Their skills will probably be lost rapidly after training, like other rarely practiced skills.90 Research in Burkina Faso indicated that most literacy graduates need 2.2 seconds to read a word and are correct only 80­87 percent of the time. The speed and error rate overwhelm the short- term memory.91 If program graduates read infrequently and must make conscious decisions about letters, they may forget the beginning of a sentence by the time they reach the end. Failure to understand and limited opportunities to practice may make them forget the sounds of many letters and the ability to decode words. Imperfect skills acquisition has spurred lending for post-literacy, but few adults can take the time needed for continuing education. If a substantial number of new literates become unable to read a few months after training, financing literacy with the current methods may lead to wastage. 70. In an effort to justify continuing investment in the face of modest results, Bank sector documents emphasize the social benefits of literacy. Indeed, evidence suggests that participants send their own children to schools, and even indirectly, adult literacy may promote progress in the Education for All goals. But the health and empowerment benefits are poorly documented and uncertain, and it is uncertain whether they arose from ability to read or from messages and group dynamics during class. Ultimately, the implication is that the development effects are those of consciousness-raising rather than actual literacy acquisition or behavior change; literacy should be financed not because it is economically justifiable, but because parents' and children's literacy are complementary, and children's schooling tends to improve with parental education.92 However, these arguments make a weak case for large-scale investment. Donor interest seems to go through cycles of pickup and abandonment. If results continue to be modest, the cycle may move downwards, and literacy may be abandoned again. 71. Implications of the findings. Adult literacy is implicated directly or indirectly in the Education for All goals (Table 6) and the second Millennium Development Goal, which is universal primary education. In 2003, it appears unlikely that goal no. 4, achieving a 50 percent reduction in adult illiteracy rates, will be met in 2015 through literacy programs. Due to modest efficiency and limited scope, the long-term effect of these programs on adults' ability to read may be marginal. 72. If primary education becomes universal, will adult literacy programs be needed less? The evidence is discouraging. Students' reading achievement is low in some countries that have greatly increased enrollments among the poor and the girls.93 An evaluation in Guinea indicated that little is learned in each grade, and only a small percentage of students achieve mastery of the material. The data suggest that the vast majority of Guinean students are functionally illiterate at the end of grade 6.94 Similar trends were reported in Ugandan tests.95 Where students spend little time in class and 90. Abadzi 2003a 91. Royer et al., 2003. 92. Sandiford 1995. 93. Moulton et al. 2001. 94. Barrier et al. 1998. The students scoring at or above 75% in French reading and math were : 15% and 11% in grade 2, 10% and 5% in grade 4, and less than 5% in grade 6. OED mission observations in 2003 confirmed that many students about to graduate from primary school could not read or explain what they read. 25 learn through official languages (that furthermore have spelling complexities), the next generation of adult illiterates may be forming. A number of school dropouts may later go to adult literacy centers, as has already happened in Uganda. Thus, adult literacy may have to shoulder the responsibility for Education for All in some low-income countries in the years to come. Its ability to make large numbers of people literate therefore remains a priority issue. Table 6. The Education for All Goals (Updated Version from the Dakar Framework for Action) 1. Expanding and improving comprehensive early childhood care and education, especially for the most vulnerable and disadvantaged children; 2. Ensuring that by 2015 all children, particularly girls, children in difficult circumstances, and those belonging to ethnic minorities have access to and complete free and compulsory primary education of good quality; 3. Ensuring that the learning needs of all young people and adults are met through equitable access; 4. Achieving a 50% improvement in levels of adult literacy by 2015, especially for women and equitable access to basic and continuing education for all adults; 5. Eliminating gender disparities in primary and secondary education by 2015, with a focus on ensuring girls full and equal access to basic education of good quality; 6. Improving all aspects of the quality of education so that recognized and measurable learning outcomes are achieved by all, especially in literacy, numeracy and essential life skills. ISSUES AND RECOMMENDATIONS Instruction and Information Processing--The Forgotten Variables 73. Can literacy outcomes improve and help meet the Education for All goals? And if so, will donors and governments be more willing to finance adult literacy? It may be worth exploring. 74. The peculiarities of human memory may be at the root of limited performance and government ambivalence. Cognitive science research indicates that literacy skills can become permanent if reading becomes quite fast and at least partly automatic (Annex A). This is because verbal short-term memory holds seven items for only about 12 seconds. To function as a reader, a person must read and understand a sentence within about 12 seconds. Therefore, at the end of a literacy course, learners should be able to read about a word per 1-1.5 second with about 95 percent accuracy. However, few attain this performance level, given the spotty attendance, wasteful use of class time, and various information-processing issues of illiterate adults. For example, donor- financed research in Burkina Faso found that new literates took 2.2 seconds to read a word and were correct only 80­87 percent of the time. Perhaps some adults do not become fast readers easily. If program graduates read infrequently and must make conscious decisions about letters, they may forget the beginning of a sentence by the time they reach the end. Failure to understand and limited opportunities to practice may make them forget the sounds of many letters and the ability to decode words. 75. Overall, the probability that learners will maintain skills similar to reading is determined by their prior knowledge, amount of time spent practicing, and individual ability.96 Donors and 95. National Assessment of Performance in Education, 1999. The percentages of students scoring `adequate' or better were in grade 3 18% for English and 39% for math, and in grade 6, 15% for English and 42% for math. Only 3% of rural student scores in reading were considered `adequate'. 96. Cognitive variables follow a normal curve distribution, and a small area at the extreme of the curve may learn and retain material no matter how weak programs are. How much of the cognitive ability spectrum should donor-financed programs 26 governments cannot do much about prior knowledge or individual ability, but they can do a lot to see that illiterates are taught effectively. However, donors are best equipped to deal with management aspects, so efforts are directed toward planning and management, organizing the delivery of instructional inputs. The instructional variables, quality and effectiveness issues have received surprisingly little attention. Some donors consider purported NGO expertise a sufficient safeguard. Thus, none of the Bank or donor studies has dealt with classroom issues or described how classes are delivered and how learners process information. Failure of monitoring systems has contributed to a limited understanding of how much instruction actually takes place and how well class time is used. Ultimately, the disregard of learning variables may prove expensive. 76. The donor community has not probed much the applications of cognitive science on poverty alleviation. Practically no research has been done on how to teach illiterate adults more efficiently or help them remember reading. Yet, scientific research conducted in the past 10 years has direct implications for improving the outcomes of adult literacy programs. Neuropsychologists (mainly in Europe) have been working to understand how literacy affects various parts of the brain; others scan the brains of dyslexics in hopes of finding out how to help them read better. Science has been focusing on methods to make memory coding more durable under difficult instructional circumstances. Concepts such as automaticity, phonological awareness, word superiority effect, and perceptual learning are staple concepts of reading research but are virtually unheard of in the literacy and donor community. Donor staff and researchers clearly do not communicate. Connecting researchers with the illiterate populations that are mainly in Asia and Africa and with funding could be the next frontier in providing sustained literacy skills. As the literacy research in Burkina Faso shows (Annex B), this is quite feasible. 77. One obstacle is the limited understanding of these issues among government, donor, and NGO staff. Adult literacy experts worldwide typically lack training in cognitive science and may not appreciate the importance of working memory limitations or perceptual learning intricacies.97 Research conducted by many literacy departments and agencies worldwide focuses on social and motivational issues, such as empowerment or community literacy. These are certainly needed, but efficient learning and rapid recall of reading information seems to be the most important issue. Effort is needed to familiarize government and NGO officials with cognitive concepts and to help them focus on methods that are most effective. 78. Cognitively based methods have been tried only on a few occasions, and their large-scale applicability is unknown. (Annex A offers some examples.) They are more complex and will have higher dissemination costs, including longer teacher training requirements. Computer displays may speed up reading automaticity, but few countries with large illiterate populations (India, South Africa, Brazil) can make them available on a large scale. Thus, more attention to instruction will cost more in terms of supervision, materials, teacher training, and probably teacher pay. Increased sophistication will probably increase unit costs per participant. However, it may reduce unit costs per graduate or keep them at the same level. The multidonor Education for All financing initiative could help countries experiment with the more feasible cognitively based literacy instruction methods and determine which ones (for example, phonological awareness, rapid decoding techniques) can be implemented in classrooms with limited means and little-educated teachers. Experiments may be aim to teach? If programs effectively reach people whose abilities are one or more standard deviations above the mean, this is not satisfactory performance if the Education for All goals are to reach all people This issue has not been discussed for literacy or for primary education, which often has high failure rates. 97. Working memory actually consists of multiple parts, including an executive decisionmaker and short-term storage buffers that hold auditory and visual material (Reisberg 2001, p. 25). For the purpose of this document, working and short- term memory are used interchangeably. 27 carried out to find out which methods and give the best results, given their cost. Also, the cost of specific components (teacher pay, giving participants financial aid) may be varied to find out how outcomes increase per extra dollar spent. 79. Better targeting for improved outcomes. Most of donor-financed literacy courses tend to be one-size-fits-all. They assume that adults are illiterate and teach the rudiments of literacy, with language of instruction as the main attention to individual needs (where needed). Pretests and classes for more advanced people or for the improvement of reading comprehension are rare. However, the low-quality primary education that many adults have had implies a more targeted approach. Also, different dropout rates and outcomes would be expected. For example, the mainly schooled illiterates of Uganda would be expected to have different outcomes from the mainly unschooled participants of Bangladesh, and the large differences in reading skills two years after a course are suggestive of participants' differences. If complete illiterates are put in a class with persons who can decode but have low comprehension, both types may feel unserved and drop out. However, little in project documents or design is focused on increasing the probability that participants will achieve their goals. The lack of emphasis on systematic individual needs may be due to project size, difficulties, and the emphasis on management that donor programs tend to have. However, means must be found to teach people what they need. 80. Even with the best of efforts, it is hard to capture illiterates once they have become adults and have acquired their own lives and priorities. It is critical to capture and keep participants when they are primary education students. However, many children drop out of school before becoming literate, not just out of poverty but because schools do not spend enough time teaching them basic skills. To prevent the next generation of adult literacy participants, primary education teaching must become more efficient. 81. Implications for future lending instruments. The Bank has been shifting to programmatic lending, whereby multiple sectors will receive budgetary support, and task managers will mainly focus on high-level coordination. There may not be many more self-standing literacy projects. Given the unsatisfactory performance of literacy components in larger projects of various sectors, there is a concern that the performance of literacy lending may deteriorate rather than improve. There are increasingly fewer staff resources available to pay attention to the important detail-level instructional issues brought forth in this document. Staff with appropriate knowledge is needed to give sustained attention to project supervision, and using ever-changing consultants may not provide needed continuity. Given the changing lending circumstances, an important issue requiring high-level decision is not only how to make literacy classes more effective, but also how to ensure knowledgeable and effective Bank supervision and interface with effective borrower institutions. Linking formal and nonformal education delivery modes in country strategies may succeed in giving adult literacy higher priority. 82. Task manager training in adult literacy issues. Task managers in education usually have studied other disciplines and may have no special training in education. The Human Development Network is developing curricula to disseminate sectoral strategy and lending issues. State-of-the-art knowledge about the policy and instructional issues facing adult education could be inserted in the curriculum. Lessons Learned from Completed Bank- and Donor-Financed Projects 83. Key lessons from literacy projects discussed in this document are: 28 · Significant dropout rates and limited instructional efficiency undermine the achievement of literacy skills by participants. Unless learners can largely read automatically by the end of a literacy course, they may forget how to read. Given the inefficient instructional conditions, it is uncertain how many become automatic readers. · Donor attention to organizational and financing issues is crucial, but not sufficient to help learners acquire stable skills. Attention to instructional variables and scientific research is also necessary. The challenge is how to locate, abstract, operationalize and disseminate effective instructional methods to large numbers of implementers worldwide, many of whom have limited education. · Stand-alone adult literacy projects are more likely to be implemented as planned than isolated literacy components in projects of various sectors (e.g., rural development). If literacy is not the main objective of a project or at least not a significant component, it typically receives little financing, attention, or expertise in implementation and yields limited outcomes. The poor implementation record of isolated literacy components suggests that without resources for supervision, outcomes may continue to be unsatisfactory. · Though literacy instruction is considered to have low costs per participant, the costs per graduate made permanently literate are higher, and management costs are sizeable. The lowest-cost programs depend on a volunteer teacher corps, which may be unstable. A vicious circle may be created, whereby cheap but ineffective programs disappoint financiers and preclude more expenditure that might make them more efficient. Countries that decide to engage in adult literacy should consider their long-term commitment and should determine the extent to which they are willing to fund more effective but also more expensive programs. · Intensive government training and supervision of NGOs is important. Though many NGOs can carry out quality literacy programs, others need considerable support and monitoring. NGOs may reach beneficiaries in specific local areas effectively, but they do not always know how to teach adults most effectively. · The monitoring designs conceived thus far in adult literacy projects are not sustainable. The research and evaluation capacity of organizations must be strengthened, but evaluation designs must also be simplified so that they can actually be carried out. · The local and community authorities must be committed to literacy programs to ensure their success. Where Bank-supported programs merely put literacy classes in place, benefits to the poor were limited. 27 Appendix: Supplemental Tables Table A. 1. Objectives and Components of Bank-Financed Literacy Projects, 1990­2002 Project Overall goal or Objective, if Literacy instruction Other Skills or Components Institutional Development Evaluation Any Indonesia: Continue the provision of basic Implement an intensive program, Strengthen the Directorate of Third Nonformal literacy to illiterates and primary special strategy targeted to low- Community Education (DIKMAS) Education Project school dropouts; income beneficiaries; upgrade through the national support services Cr.3431 private vocational training courses, component; offer technical assistance offer scholarships, apprenticeships, to income-generating groups. FY92-98 small business management skills Bangladesh: Nonformal Expand and improv e NFE Institutionalize a Directorate of NFE Education Project programs to reach about 2.5 staffed by qualified personnel able to Cr.2822 million participants of the targeted function in related tasks such as age group through (NGOs) and planning, training, monitoring, and FY96-01 the local district administrations evaluation; Ghana: Literacy and Improve the quality of life of Improve the existing literacy Give new literates access to a Strengthen the institutional capacity Evaluate all program Functional Skills Project the poorest Ghanaians, program so that a large range of reading materials in of the Non-Formal Education Division aspects frequently, do Cr. 2349 especially in rural areas. proportion of the 840,000 adults Ghanaian languages; (NFED) to manage effectively the research, provide feedback, who will be allowed to participate Expand radio coverage for entire literacy program; FY93-98 become fully literate; f unctional literacy an educational programming in rural areas Senegal Lower the illiteracy rate to Teach sustainable literacy and Strengthen the capacity of the private Pilot Female Literacy about 40 percent overall, and post-literacy through 250-300 sector to deliver effective programs, Project to 47 percent for women hours of instruction; appropriate and the capacity of the public sector Cr. 2873 content and materials for adults; to monitor, evaluate and coordinate knowledgeable teachers, public investments in literacy FY96-02 supported by training; beneficiary programs. participation in design and implementation Côte d'Ivoire Pilot Test functional literacy programs; Develop and test essential Provide technical support to Literacy Project Funding the selection of sub- strategies for post-literacy to ensure nongovernmental organizations to Cr.3275 projects proposed by the scaling up and sustainability of build staff capacity in all required provider/beneficiary groups the program. areas of expertise while creating an FY00-03 operational network of resource institutions. Ghana Develop flexible systems for Increase the number of Ghanaian Support the second phase of the Second Adult Literacy technical support and adults (15 years or older), National Literacy and Functional and Functional Skills financing delivery to providers particularly of women, who Skills Program of the Government of Program Project beneficiaries for country -wide acquire literacy and functional Ghana Cr. 3251 skills and cease to be illiterates. Serve about 1 million participants FY99-05 in 5 years Bangladesh Increase the functional Provide post-literacy and Finance a post-literacy program. Strengthen institutional capacity of Build monitoring and Post-Literacy and application of literacy skills continuing education to neo- Improve the quality and the Directorate of Non-Formal evaluation into current Continuing Education for literates through a more accessibility of continuing education Education by improving its program operations, and Human Development comprehensive and enhanced services, help organize village-level administrative, financial, information, develops the capacity of the Project system of non-formal education. continuing education centers, help planning, implementation, technical technical staff. Cr. 3467 service providers develop and offer service, and monitoring systems, courses, and develop linkages strengthens its management FY01-06 between participants and services. information system 28 Table A. 2. Literacy Projects Financed by the World Bank in 1990-2002 Country/ Project Name Project Total Cost Donor Financing Implementing Beneficiary Targets Beneficiaries Served Comments Dates (US$ (US$ million) Agency for Literacy million) Indonesia: Nonformal C3431 US$99.3 IBRD ­ US$69.5 Directorate of Basic literacy 900.000 Basic literacy 5.3 million Exceeded enrollment targets. Total 8.2 m Education Project P003966 Govt. ­ US$ 29.8 Community Education Intensive literacy 600.000 Intensive literacy 855,567 participants reached (DIKMAS) Income generation 300.000 Effective: Income Generation 118,000 04/30/92 Low quality, poor teachers, evaluation Closed: showed dropout in middle of 3-year course. 03/31/99 No comprehensive testing of participants. Ghana: Literacy and C2349 US$57.6 IDA ­ US$17.4 Nonformal Education 840,000 illiterates Enrollment: 1,373,571 Drop-out rate: 40% relapse in illiteracy, as tested in 1994 Functional Skills Project P000917 Norway US$2.9 Division, Ministry of 22% 80% graduates performed adequately in Education Effective: UNICEF ­ US$0.6 reading-writing; 76% numeracy, 73% 07/21/92 Selected NGOs Govt. ­ US$10.0 attitude change; 69% unschooled, 74% multiple literacy yearsa Closed: 12/31/97 Bangladesh: Nonformal C28220 US$24.3 ABD ­ US$26.7 Directorate of Reach 2.5 m. illiterates Reached 2.9 m, 57% female; in 64% of participants were women; reported Education Project P009560 IDA ­ US$10.5 Nonformal Education 29,503 NGO-run centers; 183,090 success 60% participants in total literacy Effective: SDC ­ US$3.8 Selected NGOs 92% passed tests after class completion; movements. 02/02/96 Govt. ­ US$10.0 two samples showed 32- 34% long-term Closed: retention of literacy (Ahmed and Lohani 06/30/01 2001) Senegal: Pilot Female C2873 US$14.0 IDA ­ US$12.6 Ministry of Basic 300.000 (75% women); About 191,577 (87% women) In 1997, 50.7% met reading criterion Literacy Project P035621 Govt. ­ US$0.5 Education & National Reduce illiteracy from 55% completed courses; 180,000 books (37.8% writing criterion) Languages procured; outsourcing method Effective: Beneficiaries & NGOs ­ to 36.5% 1995­2005 In 1998, 54.2% met reading criterion proved effective 9/06/96 US$0.9 Selected NGOs. (180,000 persons/year) (47.7% writing). 42% read with hesitation. ICR reported 75% met criterion in reading, Closed: 63% writing, 41% problem-solving 2/28/2002 Côte d'Ivoire: Pilot Literacy C32750 US$5.7 IDA ­ US$5.0 Ministry of Education 15,000 per year, of whom Project P055073 Govt ­US$ 0.7 75% women; 70% participants should become Effective: literate 05/12/00 Closing: 12/31/2004 Ghana: National Functional C3251 US$46.0 IDA ­ US$32.0 Nonformal Education 1,000,000 illiterates Basic Literacy & Development Activity for Program P000974 Govt. ­ US$13.5 Division/Ministry of (60% women) 200, 000 Education Effective: Districts/Communities English Pilot Expanded. 06/17/1999 US$0.5 Selected NGOs Closing: 12/31/2004 Bangladesh: Post-literacy P050752 US$71.57 IDA ­ US$53..3 Directorate of 1,600,000 neo-literates Implementing through 230 field offices and Continuing Education Effective SDC ­ US$7.0 Nonformal Education (50% women) 6,900 village-level Continuing Education for Human Development 06/11/01 (DNFE) Centers. Project govt. ­ US$11.24 Closing: Selected NGOs 12/31/2005 a. Reported in Burnett, 1995 p. 13 29 Table A. 3. Bank Projects with Literacy Components in 1990-2002 Country/Project Credit No. Total Cost Approximate Implementing Literacy Activities Target Achievement of OED Project Comments Name ID (million) Amount for Agency Population Targets Outcome Approval Literacy Rating Date Gambia: Women In C2141 US$15.1 m US$1.5 m for all Ministry of Education, Training in numeracy & 10,000 women 7000 women Satisfactory ActionAid NGO involved Development Project P000822 training Nonformal Education literacy 175 centers reached, 75 % in many programs. Effective Dep. dropout, 30% pass Literacy unsustainable, 12/27/90 high dropout, no Closed supervision 12/31/97 Côte d'Ivoire: Women in C3251 US$1.24 m US$0.06 m Ministry for the Literacy & numeracy in 12 Adult female Only 550 women Unsatisfactory Development Pilot P001192 Advancement of villages, 600 women. illiteracy rate: 69%. reached in Center- Support Project Effective Women Training in literacy, East, No. other info. 2/22/91 hygiene & home Closed economics. 12/31/97 Nepal: Basic and C2357 US$53.8 m US$4.7 m Ministry of Education Nonformal classes for No. specific targets Women 265,362 Satisfactory Primary Education P010395 & Culture women, children, other Adults 45,659 Project Effective adults Youth 6,625 07/08/92 Closed 12/31/99 Philippines: Second C3244 US$410.0m US$7.6 m for all Dept. of Education Literacy sub-component of 6,000 classes, Reached 493,915 Satisfactory Elementary Education P004565 training Drop-out Intervention 130,000 students women through Project Effective Program per year 29,908 classes 8/30/1990 Closed 12/30/96 Bolivia: Second Social C2532 US$70.0 m Unknown Social Investment Literacy as targeting No specific targets Number unknown, Satisfactory Despite earlier plans, very Investment Fund Project P006200 Fund women, focus on bilingual considered few literacy activities took Effective literacy health, fertility, insignificant place 06/08/93 nutrition & gender relations Closed 12/31/98 Burkina Faso: C2229 US$17 m Amount for literacy Ministry of Land Subcomponent of Targets Number of Satisfactory Literacy centers built Environmental P000290 unknown Management community support unspecified beneficiaries Management Project Effective services unknown 2/11/92 Closed 12/31/98 Benin: Borgou Region C3098 US$5.0 m Part of component Ministry of Rural Adult & functional literacy 250 target Program reached ICR rated overall Successful literacy Pilot Rural Support P057345 costed at US$1.2 Development for women. Post-literacy, communities; 2878 men and 2760 project outcome component of community Project Effective m in community based 450,000 to be women highly satisfactory committees, existing 05/07/98 capacity building served (performance schools, good Closed unknown) management 06/31/02 Gambia: Third C3128 US$51.3 m US$0.64 m Ministry of Education: Adult & functional literacy Age group 15-39, N/A N/A Education Sector P035643 Department of State program piloted by the Target 4,000 Project Effective for Education (DoSE) private sector supported adults/year 08/07/98 by UNDP Closing 04/30/04 30 Country/Project Credit No. Total Cost Approximate Implementing Literacy Activities Target Achievement of OED Project Comments Name ID (million) Amount for Agency Population Targets Outcome Approval Literacy Rating Date Lesotho:Education C3192 US$46.2 m US$0.4 m Ministry of Education Basic literacy component Literacy rate raised Targets under N/A Sector Development P056416 (MOE) to expand distance teacher to 80% by 2011 planning Project Effective training, carry out literacy 03/25/99 campaign Closing 12/31/03 Mali: Improving C3318 US$5.52 m Part of a Ministry of Basic Functional Literacy for No specific targets 1491 persons served N/A Bilingual education: 6 Learning in Primary P056416 component costed Education community participation & until midterm national languages (in Schools Project Effective at US$1.36 m awareness 700 schools) and French 01/21/00 Selected NGOs Closing 06/30/03 Morocco: Social C4024 US$97.7 m Was initially Min. of Employment & Pilot literacy program Target population Possibly 295,000 N/A 80% complete classes; Priorities Program: P005501 US$3.7 m. Social Affairs for linked with employment 2.1 million reached through learning Basic Education Effective reduced to US$2.5 literacy promotion program 2000. Innovations Loan for 05/09/96 m In 1998-99 literacy in preparation Closing Selected NGOs 59,541 people 12/31/03 reached through 39 NGOs Guinea: Village C3174 US$38.7 m Unknown Ministry of Plan and Functional literacy as sub- No. of N/A Communities Support P05732 Cooperation and micro projects under beneficiaries no Program Effective community based rural available 01/25/99 development Closing 06/30/04 Senegal: Quality C3333 US$926 m Unknown Ministry of Education Adult literacy program in 6 Target 140,000 per N/A Use of national languages Education for All Project P047319 (Ministry of Ed. languages year for transition toFrench Effective Budget) Functional & post-literacy 03/20/00 IDA credit with faire faireapproach & Closing US$50 m participatory methods 12/30/03 Brazil: Ceará Basic C4591 US150.0 m Expanding Access Ceará Basic Subcomponent of the Access to 90,000 N/A Education Quality P059566 cost. US$22.1 m Education Secretariat quality improvement of youngsters & Improvement Project Effective Amount for literacy (SEDUC) education adults (15­39) 11/29/00 unknown Closing 06/30/06 Burkina Faso: C3536 US$41.0 m Communities Ministry of Land Subcomponent of Targets Project continues Community Based Rural P03567 support Management community support unspecified successful experiences in Development Effective US$13.5 m; services Burkina Faso. FY2001 Literacy unknown Closing 06/30/06 31 Figure 1. Adult Education Lending, 1990-2002 Adult Education Lending as Percentage of Total Education Lending, FY90 to 02 12% 10% 8% 6% 4% Percentage 2% 0% 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Fiscal Year Source: Enhancing the Contribution of Adult and Nonformal Education To Achieving Education For All and Millennium Development Goals. Draft Concept Paper, World Bank, Human Development Network, November 2002. 32 Table A. 4. Unit Costs in Constant 1996 US Dollars of Bank-Financed Projects Category ­ Completed Bank Projects Ghana-1 Ghana-II Bangladesh I Senegal (1992-97) (2000-2004) (1996-2000) (1996-2000) Total cost per successful participant $27.59 $37.07 $20.40 $57.67 (100% success) Total cost per successful learner (70% $49.27 $51.49 $36.43 $174.60 success) Differences as percentage of Bangladesh 135.2% 141.34% 100.00% 479.27% costs Source: Oxenham 2002 (draft). Appraisal estimate for Senegal was US$ 97.78 per learner. Table A. 5. Costs of Adult Literacy Programs and Four Years of Primary Schooling Category Ghana-I Ghana-II Bangladesh Senegal (1992-97) (2000-2004) (1996-2000) (1996-2000) Total cost per participant $ 27.59 $ 37.07 $20.40 $ 97.78 Total base cost per participant for four $208.00 $208.00 $61.24 $650.00 years of primary schooling Cost per participant as percentage of cost 13.3% 17.8% 33.3% 15.0% per primary school participant Source: Oxenham 2002 (draft) Note:The comparison implicitly assumes that a course of about 300-400 hours will enable an average adult participant t o develop knowledge and skills in reading, writing and calculating equal to those developed by a primary school pupil in four years -officially some 3,500-4,000 hours- of class-work. No empirical evidence is available to substantiate the assumption. 33 ANNEX A How to Improve the Performance of Adult Literacy Classes? Advice from Cognitive Research98 The modest literacy results may be due to inefficient instruction but also to certain features of human memory, which have important implications for adult literacy acquisition. The need to learn the rapid recognition of complex patterns poses problems that are not apparent to people who became expert readers in their childhood. Brain imaging studies show three regions activated in the brain during reading:99 - Two slower pathways involved in word articulation and analysis (in the left parieto-temporal area and in the inferior frontal gyrus near Broca's speech center). These pathways are used by beginning readers to link letters to sounds and decode words. Involvement of the Figure 2: Brain areas involved in reading speech area creates the tendency to sound out subvocally a text in order to decode it. - An express, instant word recognition pathway (in the left occipito-temporal region) is used by skilled readers. This region responds very rapidly (in less that 150 milliseconds) and sees the whole word as a pattern. This pathway is activated after a reader has analyzed and correctly read a word several times. An exact neural model of that specific word is formed, reflecting its Source: Shaywitz 2003, p. 78-82. spelling, pronunciation, and meaning, and is permanently stored in the occipito-temporal system. Human short-term memory is exceedingly brief. The short-term memory (or working memory) buffer needed to store the deciphered material lasts about 12 seconds and holds about 7 items in educated people; in illiterates it may even last less. If the information in the short-term memory is not rehearsed or transferred to long-term memory, it gets wiped out. The slower analytical pathways used by novice readers challenge the limits of short-term memory. The buffer is filled with individual letters and is overwhelmed. Novice readers, who must make conscious decisions about letters, can only read small amounts of text and may have to read it repeatedly. To understand a sentence within about 12 seconds, readers must read at least a word in about 1-1.5 seconds. They should also identify accurately at least 95 percent of the words. This 98. This section summarizes the literature review presented in Abadzi 2003a. 99Shaywitz 2003, p. 76-87. 34 rate is pretty fast and is not often attained in literacy classes. In Burkina Faso graduates needed 2.2 seconds to read a word and were correct only 80­87 percent of the time. At this accuracy rate, people may understand less than half of what they read. The slow speed implies that most literacy graduates do not sufficiently activate their fast word recognition pathway. Unless their environment enables them to get sufficient practice, they must rely on the slower pathways while they read. The necessity to read within the limits of working memory determines who is a functional literate in real life and points towards a minimum standard for literacy acquisition: By the end of a literacy course, learners should read a word in about 1-1.5 second with about 95 percent accuracy. At this rate, readers decipher many script features automatically though they still struggle with comprehension. When the rapid recognition pathway is activated sufficiently, skills become permanent. Automatic readers do not normally lapse back into illiteracy. For reasons that are unclear, adult neoliterates may remain slow readers who read letter by letter, whereas children tend to increase reading speed and become automatic readers. It is possible that the instant word recognition pathway is not activated in adults as easily as it is in children. Thus, adult literacy instruction should focus on increasing speed and accuracy, objectives that are usually not central in literacy courses. Literacy tests should be timed. Research on literacy is often carried out by adult education specialists who typically lack training in cognition and neuropsychology. Thus, the many literacy departments and agencies worldwide focus on social, motivational, and methodological issues. The importance of fast and accurate reading is not well understood. The instructional delivery of adult literacy could be reformulated based on state-of-the-art cognitive findings. However, more research is needed to find out which of these methods can result in sustained reading improvement and how feasible large-scale applications are. Also more dissemination of technical information is needed among high-level decisionmakers. Summarized below are some principles and methods to make literacy instruction more effective that are derived from cognitive and neuropsychological research. Improving the Cognitive Functions of Adult Illiterates People's level of education influences their ability to solve abstract problems, use readily presented data in decisions, recognize and name pictures of objects, and understand radio broadcasts. Most important, the unschooled perform less well in most memory tasks: recalling a series of digits backward and forward, remembering lists of words, reproducing a short story, reproducing complex figures that were presented, recalling common objects, remembering sequences. The limited memory and cognitive resources probably also reduce performance in literacy classes. To remedy areas of low cognitive performance, literacy could be combined with exercises to lengthen working memory, such as repeating increasingly longer series of digits. A successful such trial, called the Neuroalfa method, took place in Mexico (see Abadzi 2003a). Illustrative materials used in participatory rural appraisals to help the poor articulate their community knowledge (maps, calendars, diagrams, matrices) may be useful tools in improving the visuospatial skills of illiterates, which are particularly poor among women. Teaching Decoding of a Script 35 Phonological awareness tasks should be introduced early, such as deleting the initial or final vowels or consonants, counting the number of syllables (as appropriate in various languages), or rhyming. Rather than random words, Freirean "generative" words could be used. Instruction must help learners efficiently and quickly understand words and offer an analytical strategy for learning new letter combinations. Decoding techniques are often language- or script-specific, but many exercises have been devised, such as (a) phonographix, a system to show extensions of reading rules to similar patterns (for example, mat ­ cat ­ hat ­ ham ­ him), (b) reading pseudowords that look like real words, and (c) speed-reading letters together, though they constitute nonsense syllables. To get the benefit of the word superiority effect, textbooks and teachers might reduce the presentation of unconnected letters and include early on the letters in words. New letters could be presented before and after words that include them (for example, paper, e, p, a, r). Words seen more frequently are recognized faster. New texts could consist of words that have already been presented or have well-practiced letter patterns. Mnemonics have been used to help learners connect letters and sounds (such as "a is for apple"). The benefits and costs are unclear. They may cause learners to make irrelevant connections rather than the instant and effortless connections that are needed between letters and sounds. On the other hand, reminders of how letters look may make the connections memorable early in a course or help readers if they start to forget after literacy training. Mnemonics may create more stable connections if learners themselves make them up according to their existing knowledge. Fast Reading and Automatic Recognition of Text Readers must be helped to read ever faster through various technologies such as flash cards or computerized tutorials, if possible. Chanting, that is, quick turn taking with a teacher or peer establishes a rhythm and may force students to deal with larger amounts of print than they otherwise would. Also important is repeated reading of the same text and building texts from words that are already known or have well-practiced letter sound patterns. To bring about automatic recognition of words, learners may overlearn and automatize the recognition of small units. This can be accomplished by reading a pack of cards containing small units as fast as possible over several days and recording the time required to do this. (Sheets may also be used, but learners may learn the order of the words.) This may be done once per session, and the time may be recorded every day. Over several days, the time required for the task decreases. When learners show no further improvement, they get a new list of longer words. Along with the new words, interspersed practice is given on the ones mastered earlier, to maintain old skills and build new ones. Learners can efficiently practice linking sounds and letter groups by computer. Computerized presentations may also engage learners and motivate them to stay and work at the tasks longer than they would otherwise do in a class. Some computerized applications may work on equipment that is cheaper than computers, such as an electronic reader and speaker by the U.S.-based Leapfrog company or the Indian-made Simputer. This and other technologically based methods may improve the reading skills of the poor if they become widely available in poorer countries. For example, same-language subtitling and singing of words that appear before the readers have been used in India to improve reading fluency. 36 If learners master the basics of reading within a literacy course, then vocabulary and reading comprehension must be built. This level of skills frequently is left for post-literacy courses. Forgetting How to Read Unless learners become automatic readers by the end of a course, they may forget how to read. Information can be forgotten rapidly, but the amount of forgetting is a function of prior exposure (in primary school, for example), practice during the literacy course, environmental opportunities to practice literacy, and innate ability. Teachers must prepare learners for this eventuality and stress the importance of practice, even if it is laborious and discouraging. Often, neoliterates remember the shape of letters but forget the associated sounds. Suitable mnemonics may help them keep letters and sounds connected for long-term retention. Writing may be remembered after letter-to-sound linkages have become inaccessible and facilitate recovery of the skill. Tactile exercises might increase the likelihood of remembering letter sounds traced on a palm. Perceptual Learning, Discrimination of Varying Letter Patterns Textbooks could present letters in sequences that would help detection of critical features and discrimination of minute details such as the number of dots on Arabic letters. To help focus on the correct dimensions, letters should initially be of the same type font. Also, letter or word patterns must be consistently linked to specific sounds or concepts. But in later lessons, various type fonts could be introduced, including handwriting. Discrimination training could progress from easy to hard, initially avoiding teaching together the letters that look similar, but eventually juxtaposing them, after learners have connected them frequently to sounds and seen them inside words. Letters shaped differently from others (such as Latin X, Arabic k, or Hindi ksh) stand out and their sounds may be rapidly identified. Such salient letters can be introduced in textbooks and instruction and interspersed in such a way as to facilitate reading of other letters. People who still read serially can use such salient letters to search for words in texts or dictionaries that they otherwise do not recognize automatically. Given the limited time of literacy instruction, writing often has a lower priority. It may be more effective to focus on building fluency in reading and then move to writing. Yet, writing may facilitate perceptual learning; significant motor and visuospatial abilities are reinforced through copying during basic reading instruction. Writing may also prevent forgetting. For this reason, tactile exercises may be given when letters are taught: learners may use their fingers to trace large models of letters, then they look at the letters and trace them with a pencil on their free palm while naming them. They may also trace letters by finger on a rough surface while repeating the sound. Some reading problems are due to poor eyesight. Teachers could ask learners whether they can see certain designs on their books or on the blackboard and seat those who cannot see well nearer the blackboard. 37 Utility of Literacy and Numeracy as Motivators Group formation and a concrete or common goal facilitate staying in class and persevering. Also, livelihood training helps under rather limited circumstances to keep learners in class. To keep learners focused on a concrete goal, it should be shown early on how learners can use even rudimentary skills to improve their lives (such as looking for information in a newspaper, reading labels of drug bottles, and scrutinizing children's report cards). People are more likely to remember skills or material if they know what it will be used for. Number recognition and calculations must also be automatized. To achieve this, the units that are the building blocks of complex calculations, such as multiplication tables, could be memorized, but their use must be specifically demonstrated with local currency and transactions. Rather than focus on the abstract, which the unschooled may not handle well, calculations should focus on local currency and transactions. These should also help neoliterates avoid being cheated. Improving Use of Class Time ­ Reducing Absenteeism To attain the rate of 1-1.5 second per word, class time must be used much more efficiently. The frequently used method where one person reads and the others repeat does not offer sufficient practice in decoding skills. More effective use of time would be reading in small groups (reciprocal teaching). This means that teachers must learn grouping techniques and monitor learner groups for errors. Also, teacher and student absenteeism must be reduced. Teacher training is typically very short and must improve substantially, in terms of content as well as social issues. Training should include the basics of the cognitive principles needed to develop automaticity, phonological awareness, and working memory. Teacher selection, training, assignments, pay, classroom duration, and time on task are critical variables that determine course outcomes. Because of complexity and budget constraints, these issues may receive short shrift in planning and supervision, or they may not be carried out as expected. The low education of teachers must be taken into account for training. One viable training means is to use videotaped role modeling, which would show examples of effective and ineffective instruction. Assessing Empowerment and Other Social Benefits The social benefits of literacy have been hard to assess, and the one-time self-reports are unreliable measures of progress or sustainability. Program organizers must be quite clear about what empowerment benefits they should reasonably expect and how to bring them about. If a program has empowerment objectives, they should be worked into the curriculum timeframe, and teachers should be explicitly instructed on activities to be undertaken. To minimize memory bias of researchers and participants, baseline responses should be obtained to show change, including control groups if possible. Preparatory Skills to Facilitate Reading Acquisition Perhaps all courses should much time in the first two weeks to preparatory tasks. These could be: · Phonological awareness exercises for about 20 minutes daily. 38 · Learning to count through local money and transactions, discussing how to avoid being cheated. · Digit span and other exercises (integrated within counting and phonological awareness) to help lengthen the working memory; understanding pictures in the textbook, using data in syllogisms. · Simple visual tests to determine which learners might need to sit nearer the blackboard. (These could be included in one page of a textbook.) · Creating a utility for literacy, for example, asking learners to bring in materials they would like to read. It is important to find out which of the various paradigms discussed in this document are more effective in increasing reading speed and accuracy. Means may be found to apply those that are, at least in environments that can sustain them. At any rate, resources are increasingly becoming more affordable. Computer prices keep dropping, and solar energy makes their operation possible in remote rural areas. Given the urgency to achieve Education for All by the year 2015, it may be deemed desirable to invest more in applied literacy research and more efficient operation of literacy classes. 39 Annex B Improving Reading Performance in Adult Literacy Classes of Burkina Faso Summary The research carried out in Burkina Faso is a preliminary example of how a cognitive methods approach, supported by previous scientific work, helps improve literacy class outcomes on those crucial skills of learning how to read and write. (See Abadzi 2003b for details.) The research took place from May 2000 to August 2001. First, baseline measurements were taken and compared with U.S. norms. The comparisons indicated that student scores in basic arithmetic additions and subtractions approximated U.S. scores of the appropriate grades. However, the Burkinabé adults completing one phase of adult literacy training were generally performing at a level lower than that attained by 2nd grade students in both Burkina Faso and the United States. Learners in the process of completing two years of nonformal adolescent and adult classes read too slowly (about 2.2 seconds per word) and inefficiently (80­87 percent correct); given the limitations of human memory, they were probably not able to use reading extensively. Graduates of past years who became literacy teachers have become more efficient readers with time and approximated the speed and accuracy of formally educated literacy teachers and of secondary rural school students. Some of the tested literacy teachers, however, had scores only slightly above those of their learners. Subsequently, the research team tested the alternative hypotheses that (a) speeded reading of increasingly larger word units would improve learners' reading scores over those of a control group and (b) phonological awareness training would improve reading scores over those of a control group. Two cognitively oriented instructional methods were implemented in 15 adult literacy classes of Burkina Faso comprising 245 learners: phonological awareness and rapid reading of increasingly difficult words. The literacy centers were operated by a Swiss nongovernmental organization (Organisation Suisse d'Entraide Ouvrière - OSEO) and the governmental National Institute of Literacy (INA). Literacy centers were assigned at random to receive phonological awareness, rapid reading, a combination of the two, and no special treatment (control group). Learners were given reading tests and a computerized reaction time test. Although the literacy courses were shortened by the arrival of rains and government delays, the piloted methods helped adults read better than those in the standard, "control" classes. Learners enrolled in the experimental classes performed better on the outcome tests than did learners enrolled in control classes. Ninety percent of the possible comparisons between treatment classes and control classes favored classes receiving treatments, and 72 percent of the treatment class advantages were statistically significant. The evidence suggests that phonological awareness training is particularly effective in situations where the training period was short, and that rapid reading was more advantageous in longer training situations. 40 The findings indicate that even with the short literacy class duration and the various test administration problems in the field, effects could be discerned for both cognitive methods that were tried out. However, due to the short duration of the classes (3-4 months) learners apparently did not receive sufficient practice to consolidate skills. A year after instruction was completed, informal observations indicated that a number of these rural neoliterates had forgotten the sounds associated with some consonants. More exposure to print and better use of class time may be necessary for stable skills acquisition. 41 References Abadzi, Helen. 2003a. Improving Adult Literacy Outcomes: Lessons from Cognitive Research for Developing Countries. Operations Evaluation Department. Washington, D.C.: World Bank. ------. 2003b. "Teaching Adults to Read Better and Faster: Results from an Experiment in Burkina Faso". Policy Research Working Paper Series, No. WPS 3057. Washington, D.C.: World Bank (http://econ.worldbank.org) ------. 1996. "Does Age Diminish the Ability to Learn Fluent Reading?" Educational Psychology Review, 8: 373­396. ActionAid International. 2001. Final Documentation and Evaluation Material from the Comparative Action Research and Scale Up of REFLECT. Final Report. July 25. Development Grants Facility. Washington, DC.: World Bank. Aftab, Tahera. 1994. Fighting Illiteracy: What Works and What Doesn't: A Case Study of Female Literacy in Pakistan. Convergence, 27: 25­34. Ahmed, Manzoor and Shiv Lohani. 2001. NFE in Bangladesh: Synthesis of Experience and Future Directions. Dhaka: Directorate of Nonformal Education. Archer, David and Sarah Cottingham. 1996. "Consolidation Research Report on REFLECT: Regenerated Freirean Literacy Through Empowering Community Techniques. The Experiences of Three REFLECT Pilot Projects in Uganda, Bangladesh, and El Salvador." ODA Education Research Serial No. 17. ------. 1996. Action Research Report on REFLECT. London: Overseas Development Administration. Ardila, A; V. Ostrosky-Solis, and F. Mendonza, V.U. 2000. Learning to Read is Much More Than Learning to Read: A Neuropsychologically Based Reading Program. Journal of the International Neuropsychological Society. 6, 789-801. Barrier Emilie, Sekou Fernández, Jeannot Saa Tinguiano, Gononan Traore. 1998. Evaluation du Systeme Educatif Guinéen. Centre International d' Etudes Pédagogiques ­ Sèvre et Cellule Nationale de Coordination de l'Evaluation du Systeme Educatif ­ Conakry. Bennett, Nicholas. 1995. "Draft Preliminary Analysis of the World Bank and Adult Literacy." World Bank (mimeo). 42 Bjork, R.A. 1994 Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp.185-205). Cambridge, MA: MIT Press. Bjork, R. A. 1999. Assessing our own competence: Heuristics and illusions. In D. Gopher and A. Koriat (Eds.), Attention and Peformance XVII. Cognitive regulation of performance: Interaction of theory and application (pp. 435-459). Cambridge, MA: MIT Press. Burchfield, S. et al. "A Longitudinal Study of the Effect of Integrated Literacy And Basic Education Programs On Women's Participation In Social And Economic Development In Nepal." USAID. Girls' and Women's Education Policy Research Activity. December 2002. Canieso Doronila, Maria Luisa 1996. "Landscapes of Literacy: an Ethnographic Study of Functional Literacy in Marginal Philippine Communities". UNESCO Institute of Education. Carr-Hill, Roy, Aikel Kweka, Mary Rusimbi, and Rustica Chendgelele. 1991. The Functioning and Effects of the Tanzanian Literacy Programme. UNESCO, IIEP Research Report no. 93. Paris: International Institute of Educational Planning. Carron, Gabriel, Kilemi Mwiria and Gabriel Righa. 1989. The Functioning and Effects of the Kenya Literacy Programme. Paris: International Institute of Educational Planning, UNESCO. IIEP Research Report No. 76. Cawthera, Andrew. 1999. "Let's Teach Ourselves": A People's Literacy Movement in Bangladesh. Manchester Monographs, Centre for Adult and Higher Education. England: University of Manchester. Cawthera, Andrew. 2001. "Nijera Shikhi, and Adult Literacy. Impact on Learners After Five Years. Effectiveness When Operating as an NGO." February. Unpublished Report. Comings, John. 1995. "Literacy Skill Retention in Adult Students in Developing Countries". International Journal of Educational Development 15(1): 37­45. Comings, John, C.K. Shrestha and Christine Smith. 1992. " A Secondary Analysis of a Nepalese national literacy program." Comparative Education Review, 36: 212­226. Comings, J. P., C. A. Smith, S. LeVine, A. J. Dowd, and B. Garner. 1998. A Comparison of Impact From Schooling and Participation in Adult Literacy Programs Among Women in Nepal. Boston: World Education. Comings, J.P., C.A. Smith, C.K. Shrestha. (Undated). "Adult Literacy Programs: Design, Implementation and Evaluation." Boston: World Education, Inc. Couvert, Roger. 1979. Evaluation of Literacy Programmes. Paris: UNESCO Diagne, Amadou Wade. 2001. Programme de Formation Sur La Stratégie du `Faire Faire' en Education des Adultes. Dakar: Ministry of Literacy, May 7­July 5. Diagne, Mactar. 1999. "Basic Education and Livelihood Opportunities for Illiterate and Semiliterate Young Adults--BELOISYA--Especially Young Women in Countries with 43 Low Rates of Enrollment in Primary Schools." Synthesis of Project Evaluation Reviews. Human Development, Africa Region. Paper Prepared for the Review and Design Workshop. March 15­19. N'Djamena, Chad: World Bank.. Education for All: Is the World on Track? 2002. UNESCO Publishing. Eisemon, Thomas Owen; Kari Marble and Michael Crawford. 1995. Investing in Adult Literacy: Lessons and Implications. Human Development Department. Washington, D.C.: World Bank. ------. 1999. "Investing in Adult Literacy: Lessons and Implications." In Literacy. An International Handbook. Boulder Daniel; A. Wagner; Richard L. Venezky, and Brian V. Street (1999, Eds.): Westview Press, p. 360­366. Eisemon, Thomas. (Undated). "Strengthening Investments in Adult Literacy." World Bank (mimeo). External Evaluation Team, 1993. "The Final Evaluation of the Total Literacy Campaign of Ajmer District." A report conducted by the external evaluation team (mimeo). Fujisawa, Miki. 2001. "Rural Women's Perspectives on Adult Literacy Education and Development in Kenya." Presented at the Annual Conference of the Comparative and International Education Society. March 14-17. Washington D.C. Haddad, Wadi. 1994. The Dynamics of Education Policymaking. World Bank: Economic Development Institute. Analytical Case Studies No. 10. Hamadache, Ali and Daniel Martin. 1986. Theory and Practice of Literacy Work. Paris: Unesco. Hashemi, Syed, Sidney Schuler and Ann Riley. 1996. "Rural Credit Programs and Women's Empowerment in Bangladesh". World Development, 24 (4): 635-653. Indonesia Ministry of Education and Culture. 998. "Impact Evaluation of Nonformal Education Program in Batch I and II Intensive Kecamatan, Final Report." Jakarta: Directorate of Community Education, p. 2­12. Iredale, R. 1994. "Why do Donors Find Literacy Difficult?" Journalof Practice in Education for Development, 1: 31­33. Jejeebhoy, S. 1996. Women's Education, Autonomy, and Reproductive Behavior: Experience from Developing Countries. Oxford: Clarendon Press. Jones, Phillip. W. 1988. International Policies for Third World Education: UNESCO: Literacy and Development (Routhledge). ------. 1990. "Literacy and Basic Education for Adults and Young People: Review of Experience" A Special Study for the World Conference on Education for All. ------. 1997. "The World Bank and the Literacy Question: Orthodoxy, Heresy, and Ideology." International Review of Education, 43: 367-375. 44 Karlekar, Malavika. 2000. Reading the World: Understanding the Literacy Campaign in India. Mumbai: Asian South Pacific Bureau of Adult Education. King, Kenneth. 1978. Literacy In Developing Countries. Ottawa: International Development Research Center. Korboe, David. 1997. Ghana: Literacy and Functional Skills Project (Cr. 2349-GH). Beneficiary Impact Assessment. Accra: Ministry of Education and IDA. Lauglo, John. 2001. Engaging with Adults: The Case for Increased Support to Adult Basic Education in Sub-Saharan Africa. Africa Region. February. Washington, D.C.: World Bank. LeVine R., S. LeVine, and B. Schnell. 2001. "Improve the Women:" Mass Schooling, Female Literacy and Worldwide Social Change. Harvard Educational. Harvard Educational Review Vol.71, no. 1, pp. 1­50. Levinger, B. 1992. Nutrition, Health, and Learning. School Nutrition and Health Network Monograph Series 1. Newton, Mass.: Education Development Center. Limage, Leslie. 1999. "Comparative Perspectives on Language and Literacy." Selected papers from the work of the Language and Literacy Commission of the 10th World Progress of Comparative Education Societies in Cape Town (1998). Dakar: UNESCO Regional Office. Lind, Agneta and Anton Johnston 1990. Adult Literacy in the Third World. A Review of Objectives and Strategies. Stockholm, SIDA. Lind, Agneta. 1996. Free to Speak Up: Overall Evaluation of the National Literacy Programme In Namibia. Directorate of Adult Basic Education, Ministry of Basic Education and Culture, Windhoek. ------. 1997. "Adult Literacy in the Third World--A Review of Trends a Decade later." NORRAG NEWS, 21: 31­35. Mace, J. 1999. Women's Education Programme, Gedaref, Sudan. An Evaluation of a WUS (UK) Project, London, World University Service. Moulton, Jeanne et al. 2001. Paradigm Lost? The Implementation of Basic Education Reforms in Sub-Saharan Africa. Washington, DC: USAID Office of Sustainable Development Bureau for Africa. SD Publication Series, Technical Paper No. 109. Moulton, Jeanne. 1997. Non-formal Education and Empowered Behavior. Washington, DC: Academy for Educational Development. ------. 2001. Improving Education in Rural Areas: Guidance for Rural Development Specialists. Washington: D.C.: World Bank. National Assessment of Performance in Education. 1999. The Achievements of Primary School Pupils in Uganda in English and Mathematics. Uganda: National Examinations Board. 45 New Era. 1989. Evaluation Study of Literacy Campaign in Surkhet Valley. (mimeo). Kathmandu, Nepal. ------. 1990. Evaluation Study of Literacy Programme of SFDP and PCRW. (mimeo). Kathmandu, Nepal. Okech, Anthony; Roy Carr-Hill, and Ann Katahoire et al. 1999. Report of Evaluation of the Functional Adult Literacy Programme in Uganda. Kampala. Ministry of Gender, Labour and Social Development. Okech, A.; R. Carr-Hill; A. Katahoire; T. Kakooza; A.N. Ndidde; and J. Oxenham (Evaluation Team). 2001. Adult Literacy Programs in Uganda. Africa Region Human Development Series. Washington, D.C.: World Bank. Oxenham, John; Abdoul Hamid Diallo; Anne Ruhweza Katahoire; Anna Petkova-Mwang, and Oumar Sall. 2002. "Skills and Literacy Training for Better Livelihoods." Human Development Sector, Africa Region. Washington, DC.: World Bank. Oxenham, John and Aya Aoki. 2002. Including the 900 Million. Washington, DC: World Bank (draft). Oxenham, John and Maktar Diagne. 2001. "Synthesis of the Evaluations of 27 Programs in Adult Basic Education with Literacy and Numeracy." In John Oxenham (Ed.) Proceedings of a Workshop in Programs of Adult Basic Education and Training with Literacy and Numeracy. Washington D.C.: World Bank and International Literacy Institute. Oxenham, John. 2003. Review of World Bank Supported Projects in Adult Basic Education and Literacy, 1977-2002: Comparison of Costs. World Bank: Human Development Network (forthcoming). Oxenham, John., 1974. The Turkish Project in Functional Education for Family Life Planning. Harvard University Graduate School of Education, Cambridge, Mass. Patel; Balate; Tembe; Macabi, and Sánchez. 2000. Evaluation Report of Adult and Non-Formal Education Programs in Mozambique. Ministry of Education, Republic of Mozambique. Projet Alphabétisation Priorité Femmes. 1999. Revue à mon parcours. Rapport bilan. 1995­1999 (presented in Dakar on June 23­24. Mimeo). Riddell, Abby. 2001. "A Review of 13 Evaluations of Reflect." ActionAid International. June. Rogers, A. 1991. Partners in Literacy. Reading, U.K.: Education for Development. Romain, R. and L. Armstrong. 1987. Review of World Bank Operations in Non-formal Education and Training. Education and Training Series no. 63. Washington, DC.: World Bank. Roy, P. and J. M. Kapoor. 1975. The Retention of Literacy. Delhi: MacMillan Co. of India. Royer, J. 1997. "A Cognitive Perspective on the Assessment, Diagnosis, and Remediation of Reading Skills." In Phye, G. (Ed.) Handbook of Academic Learning. Academic Press. 46 Royer, J. M., Abadzi, H., & Kinda, J. 2003. "The impact of phonological awareness and rapid reading training on the reading skills of adolescent and adult neoliterates in Burkina Faso." International Review of Education (forthcoming). Saldanha, Denzil. 2000. The REFLECT Approach to Adult Education in India: Process and Policy Implications. DFID-India, New Delhi Tata Institute of Social Sciences. Mumbai. May. Saldanha, Denzil, K.S. Ahmed, A. Haq, S.D. Khan, R. Sengupta, S.R. Uddin. 1999. Evaluation Study of REFLECT in Bangladesh: A Participatory Collaborative Review, Dhaka, ActionAid-Bangladesh. Sandiford, P., J. Cassel, M. Montenegro, G. Sanchez. 1995. "The impact of women's literacy on child health and its interaction with access to health services". Population Studies, 49, No. 1. Sanogo, M., D. Kone, A. Konare. 1999. Rapport d'Evaluation à Mi-Parcours du Programme de Save the Children/USA, Kolondiéba, Kolondiéba, Mali, Save the Children. Scribner, S. and M. Cole. 1981. The Psychology of Literacy. Boston: Harvard University Press. Scriven, M. 1991. Evaluation Thesaurus (4th ed.). San Francisco, CA: Sage. Shaywitz, Sally. 2003. Overcoming Dyslexia. New York: Alfred Knopf. Thomas, Wayne P. and Virginia Collier. 1997. School Effectiveness for Language Minority Students (http://www.ncbe.gwu.edu) UNESCO: 1999­2000 Statistical Yearbook. ------. 2000. Education For All ­ The Dakar Framework for Action. Dakar, Senegal, April 26- 28. (http://unesdoc.unesco.org/images/0012/001211/121147e.pdf ) UNESCO/UNDP. 1976. The Experimental World Literacy Programme: A Critical Assessment. Paris: UNESCO Press. Wagner, Daniel. 1995. "Literacy and Development: Rationales, Myths, Innovations, and Future Directions." In International Journal of Educational Development. 15 (4): pp.341­362. Weiss, C. H. 1997. "Theory-Based Evaluation: Past, Present, and Future." In D. J. Rog & D. Fournier (Eds.) Progress and Future Directions in Evaluation: Perspectives on Theory, Practice, and Methods. (Special issue). New Directions for Evaluation 76: 41­56. World Bank. 1995. Priorities and Strategies for Education. Education Sector Human Development Network. World Bank. 2001. Bangladesh: Education Sector Review. World Bank. 2001. Education for Dynamic Economies: Accelerating Progress Towards Education for All. Human Development Network.