FEDERAL REPUBLIC OF NIGERIA WORLD BANK PROGRAM-FOR-RESULTS FINANCING BETTER EDUCATION SERVICE DELIVERY FOR ALL (BESDA) (P160430) TECHNICAL ASSESSMENT Prepared by the World Bank May 30, 2017 ABBREVIATIONS AND ACRONYMS ASC Annual School Census BESDA Better Education Service Delivery for All BPSR Bureau of Public Service Reforms CBN Central Bank of Nigeria CCT Conditional Cash Transfer DFID U.K. Department for International Development DLI Disbursement-linked Indicator DLR Disbursement-linked Result EMIS Education Management Information System FME Federal Ministry of Education HGSFP Home-grown School Feeding and Health Program IR Intermediate Result LGA Local Government Area LGEA Local Government Education Authority MDA Ministries, Departments, and Agencies M&E Monitoring and Evaluation MLA Monitoring Learning Achievement MTEF Medium-term Expenditure Framework NEDS Nigeria Education Data Survey NEI+ Northern Education Initiative NEMIS National Education Management Information System NLA National Learning Assessment NPC National Population Commission NPMT National Policy and Monitoring Team NPSC National Program Steering Committee PAP Program Action Plan PDO Program Development Objective SBMC School-based Management Committee SME State Ministry of Education SUBEB State Universal Basic Education Board TA Technical Assistance TDP Teacher Development Program TVET Technical and Vocational Education and Training UBE Universal Basic Education UBEC Universal Basic Education Commission UNICEF United Nations Children’s Fund USAID United States Agency for International Development Table of Contents I. Program Strategic Relevance and Technical Soundness ...................................................................... 1 A. Strategic Relevance ........................................................................................................................... 1 B. Technical Soundness ......................................................................................................................... 2 C. Institutional Arrangements ............................................................................................................. 15 II. Description and Assessment of Program Expenditure Framework .................................................... 17 III. Program Results Framework and M&E .......................................................................................... 19 A. Disbursement-Linked Indicators ..................................................................................................... 20 B. Data Sources for Disbursement-Linked Indicators......................................................................... 25 C. Managing Risks and Improving Capacity ....................................................................................... 26 IV. Program Economic Evaluation ....................................................................................................... 27 I. Program Strategic Relevance and Technical Soundness A. Strategic Relevance 1. This Program aims to support the Universal Basic Education (UBE) program’s goals of equitable and inclusive quality education by relaxing key binding constraints the UBE program has suffered from since its launch in 1999. The UBE program has taken different forms since its launch in 1999, and its current implementation relies on the UBE Intervention Fund allocation formula and the resulting categories of activities. Consequently, UBE program interventions in support of increasing access to basic education are overwhelmingly supply-driven—largely providing school infrastructure and furniture. This is even the case when the UBE program is aiming to increase enrollment of specific out-of-school population groups, such as almajiri students or girls. The UBE program interventions in support of improving quality of education consist of teacher in-service training and provision of textbooks and learning materials. While these are important inputs for basic education, they are arguably diffuse and do not target a fundamental shortcoming of the system, which is its inability to impart basic literacy and numeracy skills to students. The large majority of UBE program funds are currently allocated on an equal basis across states, regardless of state’s needs or performance. Fifty percent of the UBE Intervention Fund is so-called direct interventions where the Universal Basic Education Commission (UBEC) administers the funds and interventions that are carried out at the state level, and the great majority of funds are allocated equally across states. The other 50 percent of the UBE Intervention Fund is used for infrastructure matching grants, again allocated equally across states, but where different states have accessed the matching grant to different degrees because of the counterpart funding requirement and likely other state-specific considerations. Perhaps the greatest shortcoming of the present UBE program is lack of monitoring of access and quality results achieved through its interventions —partially through the unavailability of key system data. In addition, key federal and state actors are not held accountable for results and have little incentive to coordinate efforts and interventions. 2. Therefore, the strategic relevance of the Program lies in relaxing the UBE program’s interrelated constraints of lack of accountability for results; an approach to resource allocation to states that is untied to states’ needs and performance; and an overreliance on supply-side interventions to promote access to basic education (see table 1). The Program is geared toward rewarding those states with the highest out-of-school rates for demonstrating outcomes in terms of increased equitable access and improved literacy—by lifting demand- and supply-side constraints, as relevant. The Program also incentivizes data collection on key education system attributes and outcomes, use of these data in institutionalizing need- based state plans, and in coordinated strategic planning across federal and state levels. Accountability is strengthened at all levels, including reporting by UBEC and State Universal Basic Education Boards (SUBEBs), as well as validation of school data by school-based management committees (SBMCs). Implementation arrangements are structured to foster interagency coordination at federal and state levels between the Federal Ministry of Education (FME) and UBEC. 1 Table 1. UBE Program Binding Constraints and Operation Remedies Binding Constraint Operation Remedies 1. Lack of accountability for results for  Tracking of progress on education outputs and outcomes federal and state actors (disbursement-linked indicators [DLIs] 1, 2, 3, 4, and 5)  Mainstreaming of baseline information (DLI 4, Technical Assistance [TA] component)  Strengthening of interagency coordination at federal and state levels between line ministry and executive agency (under Program implementation arrangement)  Institutionalizing coordinated strategic planning at federal and state level (DLI 5)  Fostering interstate and intergovernmental policy dialogue and peer learning (through Program implementation arrangement)  UBE program progress report and evaluation (Program Action Plan [PAP])  Strengthening of school-level social accountability (DLI 4 on data validation by SBMCs) 2. Resource allocation to states untied to  Performance-based allocation under the DLI matrix need or performance  Institutionalizing need-based strategic planning at state level (DLI 5)  Targeting of states with largest out-of-school population (Program design) 3. Overreliance on supply-side  Incentivizing homegrown solutions to enrolling out-of-school children interventions to increase access to basic – whether demand- or supply-side interventions (DLI 1) education  Institutionalizing need-based strategic planning at state level (DLI 5) B. Technical Soundness Results Area 1: Increasing equitable access for out-of-school children in focus states 3. Using population projections from the National Bureau of Statistics and states’ out- of-schooling rates from the 2015 Nigeria Education Data Survey (NEDS), more than 15 million children ages 5–15 are expected to be out of school in 2022. This reflects Nigeria’s average population growth rate of 2.7 percent per year and the fact that the country’s fertility rate remains amongst one of the highest in the developing world. In other words, to decrease rates of out-of-schooling, states must not only absorb the existing stock of out-of-school children but also be able to provide schooling to the growing number of school-age children each year. 4. Using size of out-of-school population as the main criterion, 17 states are eligible for funds under Results Area 1 (and, therefore, also under Results Area 2). Six of these states are in the North East, seven in the North West, and one in each of the four other Nigerian zones. Table 2 provides the expected reduction in basic education out-of-school children in each of these states over the lifetime of the Program, assuming a consistent 10 percent reduction in the out-of-schooling rate in each state every year. This assumption is clearly tenuous as not all states will reduce their out-of-schooling rate at the same speed every year, and there may well be delayed or expedited implementation in some states versus others. Table 2 therefore serves mostly the purpose of providing an estimate of expected total reduction by state and by year during the Better Education Service Delivery for All (BESDA) Operation timeframe. It is important to note here that even with the reduction in numbers of out-of-school children under the present Program, the size of the out-of-school population in the 17 states alone is expected to 2 remain large—with around 6.9 million 5- to 15-year-olds in 2022. The issue of out-of-school children in Nigeria will, therefore, require concerted efforts over a longer period of time, and this Program can be considered a first phase for additional interventions, either targeting the same states or expanding the geographic scope or both. Table 2. Reduction in Number of Basic Education Out-of-school Children in Focus States (2018–2022) Reduction in Out-of-school Population Region State 2018 2019 2020 2021 2022 Total North East Adamawa 19,387 17,720 16,196 14,803 13,529 81,634 North East Bauchi 80,721 74,067 67,961 62,359 57,219 342,327 North East Borno 96,208 88,277 81,000 74,323 68,196 408,003 North East Gombe 35,940 32,925 30,164 27,634 25,317 151,980 North East Taraba 23,126 21,137 19,319 17,658 16,139 97,379 North East Yobe 51,194 47,010 43,169 39,641 36,402 217,417 North West Jigawa 61,876 56,554 51,690 47,244 43,180 260,544 North West Kaduna 58,511 53,520 48,955 44,779 40,960 246,726 North West Kano 102,992 94,429 86,577 79,378 72,778 436,153 North West Katsina 91,283 83,496 76,374 69,860 63,901 384,914 North West Kebbi 65,087 59,582 54,542 49,929 45,706 274,844 North West Sokoto 74,419 68,071 62,264 56,953 52,095 313,803 North West Zamafara 63,628 58,292 53,403 48,924 44,821 269,067 North Central Niger 40,199 36,885 33,844 31,054 28,495 170,477 South East Ebonyi 4,297 3,925 3,584 3,274 2,990 18,070 South South Rivers 8,659 7,945 7,290 6,689 6,138 36,722 South West Oyo 21,351 19,591 17,976 16,494 15,135 90,548 Total 898,878 823,426 754,310 690,996 632,999 3,800,609 5. Under Results Area 1, the Program aims to incentivize states to implement homegrown solutions to reducing the number of out-of-school children that span demand- as well as supply-side interventions. Several key out-of-school population groups are identified in annex 1—including girls, almajiri children, children of nomadic pastoralists, as well as internally displaced children. Rates of out-of-schooling are also generally higher among rural populations and the poor. There are some interventions currently under way in Nigeria that could be expanded or informed by international best practice to improve their implementation and results. Past experiences also provide valuable lessons and can be reintroduced through the availability of BESDA funds. The menu of possible interventions under this results area includes school feeding, conditional cash transfers (CCTs), integration of core basic education subjects into religious education, as well as expansion of schooling infrastructure. This proposed menu of interventions is indicative and not exhaustive and states will be guided under the Program in identifying their needs as well as possible solutions. 6. Some states, including Kaduna, have supported school feeding programs. The Home Grown School Feeding Program (HGSFP) is one of a series of social investment programs announced by the Government to begin in 2016 to tackle poverty and improve the health and education of children and other vulnerable groups. The HGSFP provides free school meals with food procured from local smallholder farmers and aims to (a) increase school enrollment and completion; (b) improve child nutrition and health; (c) strengthen local agricultural economies by providing a school feeding market in which farmers can sell their produce; and (d) create employment opportunities with jobs in catering, food processing, and so on. Federal, state, and local governments are to jointly fund and deliver the HGSFP, although to date, states have gone ahead and rolled out school feeding without having received federal support. 3 7. School feeding programs can help get children into school and keep them there, increasing enrollment and reducing absenteeism, and once the children are in the classroom, these programs can contribute to their learning, through avoiding hunger and enhancing cognitive abilities.1 Recent research shows that the most sustainable programs are those that are designed and implemented together by the education, health, and agriculture sectors. Countries are moving toward local sourcing and production of food, and greater recognition of the role of smallholder farmers in food production, 80 percent of whom are women globally. Similar to the Nigeria HGSFP, there are other national school feeding programs that are increasingly embedded in national policy on poverty elimination, social protection, education, and nutrition. Three common features of institutional arrangements appear to support program quality and effectiveness: (a) planning for adequate capacity at the national and subnational levels; (b) effective coordination that supports cross-sectoral linkages, especially given the increasing prominence of local agriculture as well as the importance of food nutritional quality in program design; and (c) creation of functioning mechanisms to ensure quality assurance and accountability. Regarding the latter, though mechanisms for monitoring are critical to program success, most countries that have implemented school feeding programs have lacked these mechanisms, with the exception of Chile and Ecuador, which have a comprehensive information management system, and Brazil, with its system of checks and balances that involves different stakeholder groups across Government institutions and civil society.2 However, despite the ubiquity of school feeding programs, there is a surprising lack of information on their impact, as few impact evaluations have been undertaken, and even fewer that could be described as having used a controlled or systematic trial design. In Nigeria itself, Osun State has a long-running school feeding program that is considered good practice, particularly with regard to the engagement in program monitoring from different stakeholders at many levels, which has provided a strong platform for improved transparency and accountability. 8. In addition to school feeding, several states are piloting CCT schemes in an effort to increase enrollment of girls in particular. A large body of rigorous evidence from CCT programs implemented around the world over the last 15 years demonstrates that financial incentives can play an important role in bringing girls to school and keeping them in school. For example, Bangladesh pioneered CCTs to make school affordable for girls and indirectly reduce instances of child marriage. In Nigeria, Kano State piloted a CCT program targeting approximately 12,000 enrolled or recently dropped out girls in grades 4, 5, and 6 from 300 randomly selected primary schools. This was in response to the 2009 Kano school census that showed that though there were 520,000 female students in primary school, only 93,000 female students continued on to junior secondary school. The CCT pilot in Kano incorporated an impact evaluation using a randomized-control design. The CCT program was found to lead to a 35 percent increase in the transition of girls to higher grades after just one payment. Though the program was temporarily suspended and no CCT payments were made for a year, the original cohort of beneficiary girls continued to show high transition rates—double the rates found in the control group. The CCT program was also found to have small positive impact on student attendance, and to lead to improvement in the average student performance in math and English. 1 Drake, Lesley, Alice Woolnough, Carmen Burbano, and Donald Bundy (editors). 2016. Global School Feeding Sourcebook: Lessons from 14 Countries. New Jersey: Imperial College Press. 2 Drake, Lesley, Alice Woolnough, Carmen Burbano, and Donald Bundy (editors). 2016. Global School Feeding Sourcebook: Lessons from 14 Countries, page xliv. New Jersey: Imperial College Press. 4 Despite these successes, the Kano CCT program was discontinued for lack of funding, and its achievements were hampered by the delays and uncertainties during implementation. 9. Globally, several other types of interventions have shown their effectiveness in increasing female enrollment, some of which are also currently under way to some degree in Nigeria. Hiring qualified female teachers is one such intervention that is already supported by World Bank-financed operations. The absence of female teachers has been shown to deter girls’ participation in education in other countries, for example, the Republic of Yemen. Studies also suggest that, in some countries, child marriage may account for 10–20 percent of dropouts among girls at the secondary level. Yet in 60 countries over the last 30 years, the share of girls marrying before the age of 18 only decreased from 51 to 4 percent. Some countries have taken legal action, for example, Malawi, banning child marriage through legislation that increases the legal age of marriage from 15 to 18. Experiencing violence in and outside of school also negatively affects girls’ enrollment. In Nigeria in particular, the Boko Haram insurgency in the North East that only recently subsided specifically targeted schools, and the 2014 abduction of the female secondary students in Chibok in Borno State caused worldwide consternation. Emergency education efforts in the North East of Nigeria have therefore taken into account the psychosocial needs of both teachers and students in their design. 10. The right learning environment is important for all students and for girls in particular, and the UBE Intervention Fund dedicates a substantial share of resources to infrastructure and equipment. Location and accessibility of schools can play a significant role in children’s ability to go to school. The distance between home and school is a greater problem for girls, especially in rural areas, where secondary schools are more likely to be distant from small villages. Parents are often reluctant to let girls walk long distances to school due to safety concerns. Building schools closer to villages or providing affordable, safe public transportation have been successful interventions to encourage girls’ attendance. In the province of Balochistan in Pakistan, the government constructed 200 community schools in areas without any girls’ schools within a 2 km radius. New school construction, which included boundary walls, positively affected female enrollment. Young girls may also forego education because schools lack girl-appropriate facilities such as separate latrines. In Kano State in Nigeria, girls’ access to junior secondary school was expanded through upgrading of single sex girls’ schools and providing separate toilet facilities for girls at coeducational schools. More generally in terms of the availability of appropriate school facilities and space for both boys and girls in Nigeria, table 3 provides data on key system attributes in eight states in Nigeria that completed the 2015/2016 Annual School Census (ASC). Table 3 shows that, on average, school size in these states is relatively small at around 300, with the exception of Katsina, where school size is closer to 700. Katsina also stands out in having a relatively high pupil/teacher ratio of 76, whereas the other states range from 30–50 pupils per teacher. It is important to note that despite the focus on infrastructure in the UBE Intervention Fund, the pupil/classroom ratio is high across all eight states, and again particularly high in Katsina at 114. There is, therefore, clearly still an infrastructure constraint in these eight states. 5 Table 3. Public Primary School Key Indicators, Select States Pupil/Teacher Pupil/Classroom Region State School Size Ratio Ratio North East Bauchi 299 49 81 North East Gombe 311 32 71 North West Katsina 662 76 114 North West Kebbi 274 35 72 North Central Niger 203 29 57 North West Sokoto 324 46 86 North East Taraba 252 34 67 North West Zamfara 302 51 85 Source: 2015/2016 ASC. 11. For almajiri children, a program was funded by the U.K. Department for International Development (DFID) in Kano, Kaduna, and Jigawa in the North West to integrate modern secular education with traditional Islamic education . This resulted in an integrated curriculum that was recognized and supported by the state government. DFID found that the biggest challenge to integrating secular subjects in the curriculum of Islamic schools was to overcome ideological resistance from the teachers and religious leaders. In northern Nigeria, modern schools have long been associated with colonial rule and an alien religion and thus viewed as tools of westernization. However, once the first barrier of ideological resistance was overcome, and schools became open to teaching of secular subjects, DFID found that the state simply does not have the resources to provide secular subject teachers for these schools. In addition, SUBEBs were unable to supervise all the Islamic schools because to address their unique needs, SUBEBs would have to fundamentally reorient their approach/model to engage with non-formal education institutions. SUBEBs argued that this would distract their attention from secular public schools and potentially clash with SUBEBs’ mandate as determined at the federal level. Building on this DFID experience, this Program-for-Results will incentivize states to perhaps revise their SUBEB’s approach to Islamic non-formal schools, as well as potentially offering training to existing Islamic school teachers in formal subjects, in particular language instruction. 12. In addition to the above efforts, a range of interventions are under way which target nomadic pastoralist and internally displaced children, though on a relatively small scale. For nomadic children, SUBEBs provide teachers to them if they spend a significant amount of time in one location, but the school infrastructure is often lacking. For the internally displaced, development partners have sought to provide non-formal literacy and numeracy programs to populations both inside camps as well as those living with host communities. There is now some collective experience in this regard that can be built on to further roll out such interventions. In addition, there will be a need for reconstruction and rehabilitation of school infrastructure once the internally displaced return to their homes. Other demand-side interventions, such as psychosocial training for teachers and students, are already foreseen under various development partner programs, including the World Bank, but might require additional funds for rollout. Results Area 2: Improving literacy in focus states 6 13. In fostering increased attendance in basic education, this Operation must additionally promote improved literacy so as to counteract overcrowded classrooms and schools that lead to no learning for students. Consequently, the BESDA Operation will incentivize and provide technical support to states that are eligible under Results Area 1 to additionally adopt intensive literacy improvement program under Results Area 2. In this regard, BESDA can build on existing initiatives supported by international development partners for literacy improvement. Specifically, currently Nigeria is implementing the following innovative literacy/reading improvement programs:  Funded by the United States Agency for International Development (USAID), the Northern Education Initiative (NEI+) five-year program is supporting participating states to improve children’s reading skills. NEI+ is currently implemented in Bauchi and Sokoto States. These states have been focusing the reading program in local government areas (LGAs) with highest number of out-of-school children, specifically girls. By incorporating early grade reading instruction concepts, techniques, practice, and materials into teacher training programs, teachers will be equipped with a full set of instructional skills that will have a direct and positive impact on their students’ reading proficiency.  Funded by DFID, the Teacher Development Program (TDP) aims to improve the skills—including language teaching—of 66,000 teachers in six states (Jigawa, Kaduna, Kano, Katsina, Niger, and Zamfara). The program started in 2013 and will last until 2019. Currently, TDP is operating in three states—Jigawa, Katsina, and Zamfara. The TDP’s scope covers key subjects such as English, mathematics, and science. Recently, the TDP also developed teaching and learning tools for students and teachers in Hausa. 14. Under this Operation, states will be encouraged to scale up existing literacy improvement programs in their state or adopt them. The Operation will provide guidance to states to enable them to select the most appropriate program. It is important to note that the choice of program should ensure the following essential characteristics of intensive literacy improvement interventions. The existing programs described above have these characteristics or can fill any gaps with additional resources:  Focus on the early grade: Research shows that the best time for students to learn to read is in the early grades. The ultimate goal of reading is comprehension, the ability to understand and get meaning from text. With limited fluency, the process of decoding text drains attention and insufficient attention is available for constructing the meaning of the text. Accuracy is the percentage of words that students can read correctly. Fluency and accuracy are proven to be valid measures of reading proficiency. Fast acquisition of reading proficiency is quintessential, as the ability to attain fluency diminishes with age. With constant practice, the visual word form area in the brain (fusiform gyrus) is sufficiently activated and multiple letters are recognized simultaneously, raising the reading speed significantly. The eyes take in 5 letters at once in about 250 milliseconds. The brain identifies words and each word or phrase becomes an item. At this stage, children reach automaticity and cannot help but read. They can then focus on the message and not the prints. Automaticity 7 once acquired will ensure children will not fall back to becoming illiterate. They will be transiting from learning to read to reading to learn. There is strong correlation between individuals’ capacity to read and their highest level of education attainment. International assessments show that learning deficits in early years accumulate and low performers will not be able to catch up later in life.  Ensure adequate availability of language textbooks: To learn to read letters, words, passages, and sentences, students need to have their eyes on the text. If three to four students share one textbook, they cannot see clearly. They often can only copy from teachers writing on the blackboard, in many instances copying wrongly. It is therefore of utmost importance that each student has one language textbook as sharing textbooks means that students will not have a chance to look at the text long enough to decode and understand it and do not have sufficient time to practice independent reading.  Provide appropriately designed textbooks: While spoken language learning is natural, reading starts with vision that helps decode sounds into letters/words/sentences. Textbooks for the ‘learning to read’ stage ought to facilitate efficient encoding/decoding of text. Before making sense of the text, letters must be attended to, located, and identified as shapes. Students must be able to map letters onto sounds. A big letter is needed in every page when a new letter is introduced so that students can see it clearly and can point to it with fingers. The new letter shall be blended with already known letters in systematic ways, preferably in meaningful combinations. Letter combinations and syllabaries are gradually introduced to form words, sentences, and a passage. The textbooks ought to classify materials in orderly fashion to enable students to locate information easily. They ought to use expressive and precise statements so that students remember them better. Pictures must be informative and demonstrate the topics of the text. Important items should be highlighted, and conclusions must be drawn. Text and questions should lead students to comprehend, analyze, synthesize and evaluate.  Provide appropriate teachers’ guides and training which follow both the textbooks and teachers’ guides: Teachers’ guides need to be structured according to the reading skill development in student textbooks. A training program for teachers needs to take into account the fact that teachers may be inadequately trained in the first place and face cognitive overload having to teach large classes of children from disadvantaged backgrounds. Copying on the blackboard or dictating verbatim from textbooks may be less mentally demanding for these teachers. They may also use the class time less efficiently (for example, teaching activities are disconnected from day to day; rarely asking questions or use local examples to explain new concepts; and so on). To increase the probability of teachers carrying out the expected activities, it is advisable to plan the training program centrally and prepare ‘scripted lessons’ for early grade language teachers. Teachers need to learn to perform the tasks fluently so that they can apply these tasks when they return to class. To increase student engagement, teachers need to be encouraged to (a) use scaffolding instruction “I do, we do, you do”; (b) call on students at random; (c) call student names first before giving the questions; (d) move away from the front-desk 8 position to attend to students at the back of the class; (e) monitor students’ seatwork instead of sitting unengaged; and (f) praise students for reading correctly. Teachers can be trained to use various types of questions to motivate and assess student reading comprehension, including (a) factual questions; (b) inferential questions (answers must be sought from multiple places in the text, need to be able to read between the lines); and (c) critical questions (answers not found in the text, need to be able to read beyond the lines). They also need to learn how to (a) prepare visual materials (alphabet and syllabaries) for classroom wall hanging; (b) use textbook/lesson plan efficiently; and (c) give appropriate homework.  Adequate time spent by teachers and students on reading/literacy improvement: To overcome barriers of a limited home learning environment in terms of availability of educational resources, students need to spend more quality time learning to read at schools to attain automaticity. Without explicit instruction, books, and practice, reading speed does not increase. Students who become fluent readers later may always read slowly and have limited comprehension. Reading comprehension comes from extensive practicing of sentence reading, passage reading, and listening (story read aloud). Schools participating in the program should assign significantly more time for teachers to teach reading to enable students to achieve proficiency.  Assessment of student progress is done regularly and learning gaps are identified and addressed on time: Reading goals for each grade need to be clearly set and teachers need to be equipped with skills to conduct the assessment. The assessment should measure letter sound recognition, decoding, fluency and accuracy. Assessments can be done at the classroom by the teachers (for example, asking student to read aloud letters, words, sentences, passages) and by the school (for example, reading and writing tests). The school test results need to be analyzed by all participating teachers to identify common areas of learning gaps and strategies to address them, including revisiting or re-teaching the lessons.  Coaching is regular, combining school-based and external support: Coaching is a professional development process supporting teachers to implement the intensive reading program in their classes by (a) modelling good teaching practices; (b) observing teachers’ lessons; and (c) providing feedback to help teachers master the new practices. Teachers teaching early grade reading need constant support through coaching. To provide good coaching, coaches must have mastered the teaching methods. Coaches must be well-trained to be able to model the teaching to teachers. They should visit schools and observe teaching regularly, minimum once every month. They should use standard observation tool and follow teachers with their expected timetable. They should help teachers set specific goals for improving their teaching practices. In addition to coaching, participating schools shall establish a teacher learning community that consists of all early grade reading teachers. At the cluster level, a cluster learning community shall be established to facilitate cluster- level exchange of good teaching practices between schools. 9  Ensuring a critical mass: Finally, for reading/literacy improvement programs to achieve their goals, a critical mass of students, teachers, and schools needs to participate for a sustained period. When a school adopts the program, it is important that all primary 1–3 students and language teachers take part, although the program can be introduced one grade every year. School-wide implementation helps increase the efficiency of implementation (reducing the need for cascade training for teacher; building a community of practice for sharing knowledge and experience; and increasing schools’ monitoring and professional development capacity). At the state level, at least 50 percent of primary schools shall adopt the intensive reading/literacy programs by the final year of BESDA. 15. The interventions to improve literacy can build on ongoing UBE program interventions in textbook provision and teacher training. It is critical that all students participating in the literacy program receive the grade language textbooks at the beginning of the school year. In the same manner, all teachers participating in the program shall receive the textbook, teachers’ guide, and teacher training manual before the school year starts. To ensure value for money, it may be efficient for states to identify the total number of books/teacher s’ guides needed well in advance and UBEC will procure the textbooks/teachers’ guides in bulk. Similarly, the training program shall ensure that participating teachers are able to implement the literacy program step by step for the school year. Training shall be provided to teachers before the school year starts and refresher training shall be provided during Term 2 and Term 3. The teacher training modality can follow UBEC’s current approach. Results Area 3: Strengthening accountability for results 16. Lack of accountability for results is one of the most critical governance challenges that hampers the implementation effectiveness of the UBE program. Existing accountability mechanisms and legal requirements are not operationalized and need strengthening, to increase the accountability of the Nigerian Government, at federal and state levels, and its implementing agencies, UBEC at federal and SUBEBs at state level, for results achieved under the UBE program in terms of increased access and quality of basic education. This strengthening will foster policy dialogue between stakeholders and coordination between the two tiers of government, as well as improve policy making. Accordingly, the Operation strengthens different types of accountability:  Managerial accountability at federal and state levels through the completion and disclosure of strategic plans and annual progress reports by UBEC and SUBEBs;  Political accountability at federal and state levels for resource allocation and progress on access and quality of education through the mainstreaming of data collection and use on inputs, outputs, and outcomes (learning achievements);  Fiduciary accountability through financial auditing and procurement review;  Social accountability through data validation by communities (through SBMCs) on school performance and the public disclosure of this information; and 10  Information for accountability by building the necessary evidence base (baseline information, tracking of progress) for policy making and dialogue. 17. On information for accountability, the objective of this results area is to improve the availability of basic education data on system attributes and outcomes and make use of these data in improving planning and reporting by state and federal levels toward improved effectiveness of UBE program implementation. To achieve this objective, the Program will focus on incentivizing (a) a system of National Learning Assessment (NLA) under the TA component; (b) implementation and publication of the ASC; (c) state reporting and planning for UBE program implementation; and (d) consolidated strategic planning by UBEC using progress report and midterm program evaluation findings. 18. Nigeria has a latent NLA system. The national Monitoring Learning Achievement (MLA) experience in Nigeria has been less than desirable, characterized by inconsistency in its conduct, inadequate stakeholder buy-in, ineffective dissemination, and ineffective utilization of results/outcomes to inform and drive improvements in education provision. Although the national MLA has been designated as a statutory responsibility of the Department for Educational Planning, Research, and Development of the FME, the exercise is far from being institutionalized. The four-year cycle for the conduct of MLAs has not been maintained. Two MLAs were conducted in 2003 and 2011 with long delays in report production and poor dissemination. During the same period, UBEC conducted separate learning assessments in 2003, 2006, and 2011. In addition, over the years, federal and state education ministries, departments, and agencies (MDAs) and international development partners have carried out variants of the MLA in the form of learning assessments specific to their programs to meet monitoring and evaluation (M&E) needs and other requirements. Most of these exercises do not relate to or build on each other. This uncoordinated and fragmented approach without a national benchmark of learning outcomes is a serious limitation to the credibility of program-specific variants of the MLA and their use. Their significance and contributions to systemic national tracking of learning outcomes, sector performance evaluation, and development of national strategies for improving education provision will continue to be very minimal. 19. It is critical that Nigeria develop a robust and integrated NLA system that can build the evidence base on system learning outcomes and inform the policy development for improving the country’s quality of education. An NLA policy framework was drafted in 2016 and it includes important parameters such as the rationale for the NLA system; NLA objectives; grades/subjects to be assessed; and frequency of assessments. However, the draft has not (a) specified the level of representativeness (sampling); (b) the framework for the policy analysis using the data collected; and (c) the reporting and dissemination protocol. Furthermore, for the NLA policy to be implemented successfully a clear budget line for the activity also needs to be specified. Finally, the draft proposes that the NLA will be housed at the FME, yet UBEC has the mandate for quality assurance in basic education. Hence, it is critical to delineate the NLA responsibilities in basic education to create synergy and coherence and to avoid overlap (for example, both the FME and UBEC conducted parallel learning assessments in primary education in 2003 and 2011). Consequently, under this results area the Operation will support the development and institutionalization of the Nigerian NLA system. This is a large-scale, low- stake, nationally representative survey of student achievement to measure the level of learning at a certain grade; identify factors (home, school, and policy) that associate with variation in 11 learning; and make policy recommendations for improving learning. The NLA can also help Nigeria measure in a comparable manner the country’s progress toward Sustainable Development Goal 4.1 that aims to improve the proportion of children and young people in (a) grades 2 and 3; (b) at the end of primary education; and (c) at the end of lower secondary education achieving minimum proficiency levels in mathematics and reading. 20. Assessing learning outcomes must go hand in hand with a clear understanding of the system’s key indicators related to infrastructure, human resources, and student body. While having a National Education Management Information System (NEMIS) policy document issued in 2007, Nigeria has not been able to publish key education statistics regularly and in a comprehensive manner. The ASC is intended to provide accurate, timely, and reliable data that will guide policy makers to ensure efficient and effective planning, decision making, and monitoring of the performance of the education sector. While it is run by the FME, states are expected to pay for its implementation. The ASC uses a unified questionnaire for data collection on every pre-primary, primary, junior secondary, and senior secondary school in the country. However, states that receive international development partner support have more resources to conduct the ASC using external enumerators, compile data into a school/LGA/state database, and publish the ASC report on key education indicators such as student population, number of teachers, pupil/classroom ratio, and school facilities. In 2014/2015, only 11 states collected the ASC and published the ASC report. In 2015/2016, 18 states collected the ASC and published the ASC report using the enumerator approach. The remaining states collected the data using reports provided by the Local Government Education Authorities (LGEAs). The data have not been consolidated into a national school database and no national education statistical abstracts have been published using these data. 21. The Operation will therefore incentivize states to undertake the ASC annually, conduct appropriate data verification, and report on the findings. States will collect the school data using the unified ASC questionnaire and compile the data into a computerized database. States will also analyze the data and publish the state ASC report, presenting key education indicators and their trends compared to previous years. Though currently ASC data are collected by enumerators deployed to schools, the United Nations Children’s Fund (UNICEF) is working with the FME and states to put in place school record keeping that would feed into the ASC as well as provide data to the school itself. When the ASC is collected through school reporting, it is important that data accuracy is assured. Training on how to fill in the questionnaire needs be provided to all schools. The ASC school report shall be signed by the school head and co-signed by the head of the SBMC. A sample accuracy spot check shall be conducted at the state and national levels. Schools/LGAs that provide inaccurate data will be notified for revision. States will analyze the school ASC data collected to produce the state ASC annual report. The report should present the number of schools reporting and missing data (if any). 22. UBE is a shared responsibility of the three tiers of government in Nigeria and the successful implementation of the UBE program calls for intergovernmental coordination and policy alignment. Half of the UBE Intervention Fund managed by UBEC is transferred to the states under a matching grant mechanism and utilized by state governments to build and refurbish school facilities. To access the UBE matching grant, states have to submit to UBEC an investment plan and subsequently report on its implementation. The other half of the UBE 12 Intervention Fund is directly spent and executed by UBEC but on the condition that benefitting states also plan for its uptake. 23. Planning requirements at state level for the implementation of the UBE program require strengthening and increased commitment by state and federal actors. The UBEC template for states to submit investment plans for the utilization of the UBE matching grant is not sufficiently comprehensive, which in turn means that plans’ approval process by UBEC is not sufficiently strategic. Very few states submit need-based and strategically focused plans. UBEC limits its monitoring of project implementation at state level to on-site inspection of physical outputs. Therefore, UBEC intends to strengthen the implementation framework of the UBE program through adequate planning at state level and rigorous oversight at federal level. SUBEBs will be required to submit state plans at the beginning of each fiscal year. Such plans will fulfill the following requirements:  To be consistent with the state education sector plan;  To be informed by baseline information contained in the state ASC;  To be funded through federal, state and local budget appropriations to basic education (over the time frame of the medium-term expenditure framework [MTEF]);  To lay out quantitative objectives on access and quality of basic education, using information captured by the ASC and other official sources of information (such as household surveys);  To be grounded on a sound planning methodology (including for project appraisal, selection, and prioritization); and  To factor in feedbacks from stakeholders including SBMCs on school performance. 24. For lack of adequate M&E of the UBE program outputs and outcomes, executive agencies in charge cannot be held accountable for their effectiveness in program implementation. Although UBEC is legally mandated to “monitor federal inputs into the implementation of basic education” and “present periodic progress reports on the implementation of the UBE to the President through the Minister,” it only reports on financials, activities, and deliverables under its direct interventions. Under the matching grant mechanism, UBEC can only transfer funds to the states when “satisfied that the earlier disbursements have been applied in accordance with the provisions of this Act.” For that purpose, UBEC first approves action plans submitted by state governments, and monitors the implementation of projects to ensure compliance with rules and regulations by SUBEBs. At state level, SUBEBs are required to track fund utilization and monitor project implementation to ensure value for money. However, this monitoring mechanism over the utilization by SUBEBs of the matching grant is in need of reform. A recent joint report by UBEC and the National Planning Commission on the uptake of the UBE matching grant observes that: 13  Information on key outputs and outcomes to monitor and evaluate are not available: UBEC only records states’ planned outputs (projects) in the form of annual action plans, but it has no records of actual state-level outputs.  UBEC has no clear set of agreed output and outcome indicators. 25. Fiscal accountability on expenditure effectiveness also requires strengthening. There is no validated consolidated information on public expenditure allocated to and effectively spent on basic education in Nigeria, which prevents an informed evaluation of financial resource mobilization. There is no information either on the effectiveness of allocated resources under the UBE program for the achievement of its objectives. The UBE Intervention Fund is not captured into the Medium-term Sector Strategy of the FME and the execution rate of the UBE Intervention Fund is not reported on in the federal budget execution audit report or under the M&E system for capital expenditure run by the Federal Ministry of Budget and Planning. Accordingly, the Operation will incentivize:  Reporting on value for money both at federal and state levels through joint strategic planning; and  Review of the allocative efficiency (across economic categories of expenditure and across the states as a result of the distribution formula) of the UBE Intervention Fund. 26. Social accountability has been institutionalized at school level for the implementation of the UBE program but is yet to be fully operationalized. In 2005, the National Council of Education decided that an SBMC should be established in every primary and secondary school across the country and UBEC issued guidelines in 2011 to frame the organization, role, and activities of SBMCs. Subsequently, most states issued policies and guidelines with UBEC and donor TA, including from DFID under the Education Sector Support Programme in Nigeria. By 2012, all Nigerian states had adopted a regulatory framework for SBMCs in alignment with federal guidelines. But SBMCs are yet to become functional in many instances. For that purpose, the FME is drafting a national school-based management policy to “provide communities with capacity and mechanisms to demand accountability and transparency from duty bearers (including school managers, LGEAs, SUBEBs, State Ministries of Education [SMEs], and FME)” and “a legal framework for involving all stakeholders, including the local communities, to participate in the planning, monitoring, and evaluation of educational outcomes at the school level.” For that purpose, UBEC guidelines for the Development of School-Based Management System in Nigeria mandates SBMCs to:  “Set goals and targets for their schools and define strategies for achieving the goals towards improving the learning outcome of learners;”  “Report the progress and challenges confronting the school, and submit an annual report (progress and financial) to LGEAs, SUBEBs, SMEs, and FME as appropriate;” 14  “Ensure effective utilization of school resources on planned projects and activities” and that “financial information is posted on the school notice board.” Under this regulatory framework, states are also required to “establish communication channels and flow between SBMCs, schools, communities, government (FME, SMEs, SUBEBs/LGEAs) and other stakeholders.” The Operation operationalizes SBMCs for data validation (ASC) on school performance. It also incentivizes the reporting by the states of SBMCs’ functionality. C. Institutional Arrangements 27. The overall institutional framework of basic education has been designed to achieve a number of objectives for effective policy implementation. Its objectives are to (a) institutionalize the separation of policy making vested with line ministries from policy implementation vested with executive agencies; (b) structure horizontal and vertical intergovernmental coordination and policy harmonization; and (c) ensure the representation and participation of stakeholders in policy implementation across the three tiers of Government. But it has faced challenges to (a) ensure the accountability and oversight of established executive agencies (UBEC and SUBEBs); (b) tally the functions of established executive agencies with the whole array of stated policy objectives, including the oversight of private schools; and (c) effectively associate local governments with policy implementation.3 28. UBEC has adequate capacity for Program implementation (which is confirmed by its capacity to provide oversight to the World Bank-funded State Education Program Investment Project and Global Partnership for Education-funded Nigeria Partnership for Education Project, which is supervised by the World Bank). It is managed by an Executive Secretary and has 8 Directors and around 150 staff. It also receives 2 percent of the UBE Intervention Fund to cover its operating expenses. UBEC’s management is keen to improve its monitoring and reporting on the effectiveness of UBE program implementation. UBEC is at a disadvantage when it comes to monitoring funds utilization at state level, primarily for lack of a proper planning framework for resource utilization at state level. The Program will help remedy these institutional weaknesses by strengthening (a) the monitoring of UBE program implementation and (b) the intergovernmental strategic planning framework for basic education. 29. UBEC’s Governing Board was disbanded in 2015 after the national elections (as was the case for all federal parastatals). Its appointment in April 2017 will strengthen UBEC decision making including on politically sensitive dimensions such as the allocation formula of the UBE Intervention Fund. The membership of the Governing Board (which includes representatives of the FME, Teachers’ Union, parents/teachers associations, women’s groups, and of a range of other parastatals, such as the National Teachers’ Institute; National Commission for Colleges of Education; National Commission for Mass Literacy, Adult, and Non-Formal Education; Nigerian Education Research and Development Council; and the National Commission for Nomadic Education) ensures a fair level of stakeholders’ representation, making its role all the more important for decision making and oversight. Once the new Governing Board is appointed, its capacity will be built through a partnership 3 World Bank. 2015. Governance and Finance Analysis of Basic Education in Nigeria. 15 established between UBEC and the Bureau of Public Service Reforms (BPSR) with support from the World Bank and DFID. 30. To strengthen the institutional effectiveness for Program implementation, an institutional assessment of UBEC by the BPSR4 has been initiated to determine areas for improvement of its effectiveness in discharging its mandate. Funding for the assessment is provided by DFID through a trust fund managed by the World Bank. The institutional assessment covers the following dimensions: (a) UBEC institutional environment, for example, legal framework, external oversight by the FME, and external auditing by the Auditor-General of the Federation; compliance with government budget management rules; interagency coordination—horizontal and vertical; (b) UBEC corporate governance (including governance of the Governing Board and senior management team; quality of internal fiduciary systems and controls [internal audit and anticorruption]); quality of reporting on results; information disclosure); and (c) UBEC institutional capacity and performance (for example, business processes, financial management, risk management). On completion of the institutional assessment, the BPSR will continue its support to UBEC as it implements the assessment’s remediation action plan. 31. At state level, Program implementation is vested with SUBEBs, which are operating through a network of local branches, LGEAs. Overall, SUBEBs employ around 80,000 staff across the country. They have the capacity for Program implementation (at least comparatively to SMEs) while also focusing insufficiently on accounting for results. LGEAs are also not accountable to local governments or to SMEs, which oversee schools through a parallel network of field offices. The Program will strengthen institutional capacity for both planning and M&E at the state level by incentivizing the development of state UBE program plans based on a standardized template and to be submitted to a strengthened review process by UBEC, and the mainstreaming of data collection through the ASC, to be externally validated by SBMCs. 32. Under the Operation, the effectiveness of the institutional framework is further strengthened by the TA component to improve interagency and intergovernmental policy alignment and coordination as well as M&E capacity. At policy level, the National Program Steering Committee (NPSC), to be chaired by the Minister of Education (or the Permanent Secretary on delegation) with UBEC as the Secretariat, and including representation from UBEC and the Ministries of Education, Finance, and Budget and National Planning, as well as the Vice- Presidency,5 SMEs, SUBEBs, and civil society organizations, will ensure horizontal and vertical coordination. At the federal level, a Coordination Support Team established to strengthen UBEC coordination will work closely with the Policy and Monitoring Team at the FME to foster interagency coordination in overseeing the Program. At the state level, Program implementation is supported through SME and SUBEB teams collaborating through a joint Technical Group. M&E capacity is strengthened both at federal and state levels by incentivizing the mainstreaming of ASC across the states, and supporting the annual fielding of the National Education Data Survey and the conduct of the NLA. And by incentivizing the validation of data collection at 4 The BPSR is a Federal Government agency tasked with promoting governance reforms. For its institutional assessments, it is using a methodology that the World Bank has contributed to and which is already used successfully for a range of public entities. 5 The Vice-Presidency manages two relevant programs for basic education: (a) the national HGSFP and (b) the recruitment of graduates as temporary teachers under N-Power. 16 state level by SBMCs, the Program strengthens community-based monitoring of service providers’ performance. II. Description and Assessment of Program Expenditure Framework 33. The financial sustainability and funding predictability of the UBE program are legally secured at federal level by the UBE Act thanks to the earmarking of at least 2 percent of the consolidated revenue of the federal budget. In the appropriation act, the financing of the UBE program at federal level is clearly identified by a single budget line in the category of statutory transfers (that is, separate from the budget of the FME). During budget execution, funds are released to the extent that projected revenue effectively accrues to the budget. This funding mechanism shelters the UBE program against poor budget performance, including historically low execution rate of capital expenditure. As table 4 shows, the UBE Intervention Fund allocation has been on the order of current US$300–450 million, or US$230– 480 million in 2005 U.S. dollars, over the years since 2005. Budget execution data for 2016 are not yet available so only approved budget amounts are provided. Projections using the MTEF of available funds for the UBE program over the lifetime of the Operation average US$320 million annually, which will result in a total of US$1.6 billion over the period 2018–2022 (see table 5). Table 4. UBE Intervention Fund Allocation (2005–August 2016) Year Current U.S. Dollars 2005 U.S. Dollars 2005 295,053,207 295,053,207 2006 295,053,207 295,053,207 2007 454,340,601 483,294,215 2008 454,340,601 483,294,215 2009 285,390,816 247,929,649 2010 285,390,816 247,929,649 2011 410,084,511 342,340,932 2012 410,084,511 342,340,932 2013 464,595,718 382,318,205 2014 464,595,718 382,318,205 2015 337,143,126 227,686,618 2016 approved budget 391,421,320 258,224,817 Source: https://ubeconline.com/Pre/releases%20UPDATE%20AS%20AT%2010TH%20AUGUST%202016.pdf Table 5. Projected UBE Intervention Fund Allocation (2016–2022) 17 Naira/U.S. Dollars Naira U.S. Dollars Exchange Rate 2016 approved budget 77,110,000,000 197 391,421,320 2017 proposed budget 83,518,764,939 305 273,832,016 2018 projected budget 93,281,395,523 305 305,840,641 2019 projected budget 108,885,950,729 305 357,003,117 2020 estimated budget 95,228,703,730 305 312,225,258 2021 estimated budget 95,228,703,730 305 312,225,258 2022 estimated budget 95,228,703,730 305 312,225,258 Total 2018-2022 budget 487,853,457,443 1,599,519,533 Source: MTEF 34. At state level, the financing of the counterpart funding required for states to access the UBE matching grant (that is, half of the UBE Intervention Fund earmarked for the construction and refurbishment of school facilities) is also legally secured. For example, in Edo State, the law mandates the state government to “contribute not less than 50 percent of the total cost of projects as its commitment in the execution of the projects under the UBE scheme in order to qualify for the Federal Government block grant.” Fiscal transfers under the UBE program are also appropriated in state budgets and clearly identified. As table 1.2 in annex 1 shows, disbursements of matching grants to states as of March 31, 2017, amount to 82.7 percent of total allocations since 2005. However, this reflects the fact that state matching grants are almost fully disbursed through 2014, while 22 states have accessed their 2015 matching grant allocation, and only three states accessed the 2016 allocation as of March 31, 2017. 35. UBE Intervention Fund resources for 2016 show only a 14.6 percent utilization rate for the year (see table 6). The table first shows that roughly 71 percent of the UBE Intervention Fund is for capital expenditures, followed by 27 percent on goods and services, and 2 percent on personnel. Several 2016 Intervention Fund allocations, including for instructional materials, teacher professional development, and special education did not disburse in 2016 at all—with states accessing instead 2014 and 2015 allocations in 2016. States only accessed 4.3 percent of matching grants, reflecting the fact that only three states accessed the 2016 allocation, as described above. Apart from personnel expenditures, which were fully disbursed in 2016, the two 2016 Intervention Fund allocations that disbursed in 2016 were the ‘educational imbalance’ and ‘UBE monitoring’ allocations. The first represents spending on construction and rehabilitation of schools; almajiri education projects; girl-child projects; as well as boy-child projects. The second represents spending on special programs in addition to monitoring interventions, such as advocacy visits to states and quality assurance practices in basic education institutions. In other words, the spending in 2016 reflects a focus on interventions addressing out-of-school children as well as quality of education. 36. Budget execution under the UBE program remains opaque for lack of adequate auditing of financial statements. The execution of the budget line earmarked in the federal budget for the financing of the UBE program is not adequately captured by the audit report on budget execution: being categorized as a statutory transfer, its disbursement is deemed completed when the funds are released to UBEC. UBEC’s own financial statements are supposed to be annually audited by a private auditor under the supervision of the Auditor-General of the 18 Federation, but no such financial audit has been conducted in the past four years. In principle, the jurisdiction of the Auditor-General of the Federation extends to the use of the matching grant by the states, but there is no evidence that it has conducted such financial audits. Table 6. UBE Intervention Fund Utilization, 2016 Statutory Releases Achievement Fund Utilization (Naira) (Naira) (% ) Matching grants 38,555,000,000 1,649,876,128 4.3 Instructional materials 11,566,500,000 - 0.0 Educational imbalance 10,795,400,000 7,228,649,644 67.0 Teacher professional 7,711,000,000 - 0.0 Good performance 3,855,500,000 - 0.0 Special education 1,542,200,000 - 0.0 UBE monitoring 1,542,200,000 872,027,751 56.5 UBE implementation 1,542,200,000 1,542,184,336 100.0 Total 77,110,000,000 11,292,737,859 14.6 st Source: UBEC. January 2017. 2016 Budget Performance as at 31 December 2016, presented to the Senate Committee on Education (Basic and Secondary). 37. For lack of program evaluation and adequate monitoring of outputs and outcomes, the efficiency of UBE program expenditure has not been assessed so far. Although UBEC is legally mandated to “monitor federal inputs into the implementation of basic education” and “present periodic progress reports on the implementation of UBE to the President through the Minister,” it only reports on financials, activities, and deliverables under its direct interventions (that is, to the exclusion of the matching grant). 38. The efficiency of expenditure under the UBE program is also undermined by the lack of overall fiscal accountability and transparency on public expenditure in basic education. There is no verified consolidated information on public expenditure allocated to and effectively spent on basic education in Nigeria, which prevents an informed evaluation of financial resource mobilization. Consolidated budget information will require the harmonization of charts of accounts used across levels of government, the functional reclassification of budget expenditure, and systematic and audited reporting on budget execution. For lack of such standardized budget information, it is practically impossible to assess the effectiveness of financial resources allocation in basic education. III. Program Results Framework and M&E 39. The BESDA Program Results Framework is based on the key processes, outputs, intermediate results (IRs), and Program Development Objective (PDO) outcomes reflecting the three results areas of the Program. The indicators follow the standard criteria of being 19 ‘SMART’ (specific, measurable, actionable, realistic, and time-bound). The BESDA Program’s PDO and results areas are fully aligned with the Ministerial Strategic Plan’s areas of concentration—access, quality, and system strengthening—and UBE program objectives. The first results area focuses on improving access for out-of-school children and reducing the number of out-of-school children in focus states. The second results area supports system-level improvements to improve literacy. The third results area encompasses interventions to strengthen accountability for results, including the availability and use of basic education data and associated public expenditures for planning and reporting. Each of the results areas has a corresponding PDO indicator that is expected to be achieved through a comprehensive and interlinked set of interventions. Based on the Results Framework, the DLIs were identified for disbursement of BESDA funds. A. Disbursement-Linked Indicators 40. Program DLIs include a mix of IRs and outcomes that are a part of the results chain (figure 1). The results chain presents the Program logic, illustrating how activities supported by the Program may contribute to achievement of outcomes. To achieve increased access for out-of- school children, potential interventions supported by the Program include the construction and expansion of schools, training and recruitment of teachers, CCTs, school feeding programs, and the integration of non-formal schools into the system through the introduction of basic education core subjects into their curricula. Support for intensive literacy programs is expected to contribute to improved literacy rates in focus states. Efforts to strengthen accountability for results nationally revolve around interventions to enhance transparency through increased availability of improved quality data for planning and reporting. Enhanced transparency and reporting are critical steps along the continuum of accountability for results. 41. The disbursement-linked rewards to be financed under BESDA have been selected in accordance with the national strategy and priorities to ignite intense efforts to enroll out- of-school children in basic education, improve literacy rates in focus states, and strengthen accountability for education results. The five DLIs that cover the five-year period will pivot the system to put greater emphasis on results and enhance evidence-based planning and management of the UBE program. DLI 1 measures reduced number of out-of-school children in focus states by gender—the key outcome under Results Area 1 on increasing equitable access for out-of-school children. DLI 2—schools adopting primary 1–3 intensive literacy programs— measures a key step in the process toward achievement of DLI 3—improved literacy rates, by state—which is the outcome for Results Area 2. The remaining DLIs represent together the essential ingredients for strengthened accountability for results in basic education in Nigeria, as emphasized by Results Area 3. DLI 4 measures implementation and publication of the ASC validated by SBMCs; and DLI 5 measures annual state plans and progress reports on UBE program implementation, including ASC data, budget data and information on SBMCs submitted on time. The rationale and justification for each of the DLIs is presented in table 7. 20 Figure 1. Results Chain Results Area Interventions Outputs and Intermediate Outcomes Outcomes  Expand existing school infrastructure,  Schools expanded including mobile classrooms for nomadic  Schools rehabilitated  Mobile classrooms provided Reduced number of children, school rehabilitation, and so on  out-of-school children  Provide teaching and learning materials Class size optimized  Average pupil/class ratio, Increase  Provide CCTs for education costs of  Schools equipped with teaching in focus states, by and learning materials by state (IR1.1) equitable children from poor households  Average pupil/latrine ratio, gender (DLI 1/ PDO  Students benefitting from CCT access for out-  Implement school feeding programs and school feeding programs by state (IR 1.2) 1) of-school  Recruit additional teachers to support  Additional teachers recruited to children increasing enrollment match increasing enrollment and  Recruit female teachers in targeted areas school attendance  Integrate basic education curriculum in non-  Female teachers recruited programs  Schools adopting primary 1-3 formal schools intensive literacy programs in  focus states (DLI 2/IR 2.1)  Teachers trained (IR2.2) Improved literacy rates  Students benefiting from direct interventions to enhance in focus states, by state  Intensive primary 1–3 literacy training program  Teacher assessments learning (IR2.3) (DLI 3/ PDO 2) Improve established conducted  States develop literacy improvement plans (rapid   Student assessments literacy situation assessments; selection of conducted schools/teachers; selection of trainers)  Teacher training programs  Training modality developed updated: pre-service, in-  service Implement literacy improvement programs  NLA conducted (IR 3.1)  Teachers trained (teacher training; school-level implementation;  States’ literacy improvement  Total states coaching and monitoring; assessment of results) plans developed implementing and  Impact evaluation of literacy program publishing the ASC (DLI  Literacy results assessment conducted 4/IR 3.2)  Total states completing annual state basic Publicly disclosed  Implement ASC program with school-level education plan and UBE program data validation by community  States implementing ASC progress report on UBE strategic plan  Public disclosure of information (school with data validation by program implementation, reflecting annual performance, baseline information, strategic SBMCs approved by UBEC (DLI progress reports and plans, progress and evaluation reports)  States with education sector 5/IR3.3) Strengthen program evaluation  Conduct national student learning plans  Publicly disclosed UBE accountability program annual progress findings (PDO 3) assessment  NLA conducted for results  Conduct UBEC institutional assessment and  Annual progress report on report (IR 3.4) develop remediation action plan UBE interventions  UBE program evaluation  Submit evidence- and need-based state  Enhanced transparency of and report completed (IR Education Sector Plans UBE program 3.5)  Adopt remediation action plan to findings implementation of progress reports and program evaluation 21 Table 7. Nigeria BESDA Justification for DLI Selection Results Area Disbursement-linked Rationale Why it is Transformational Result (DLR) Increase DLI 1 Nigeria stands out globally in its DLI 1 is transformational because equitable access Year 0: Mapping number of out-of-school children. the results-focused nature of the for out-of-school completed of out-of- The states with the highest out-of- funding allows for customized children school children by LGA school rates are concentrated in the approaches in each state to attract and type of population; north. In these areas, out-of-school children who have never gone to and target set for rates are higher among girls, and school or those who attended at one reduction in number of among the poorest in society. The point but dropped out. Further, the out-of-school children by statistics suggest that differential payment scheme offers LGA approximately one-third of kids an additional incentive to focus on Year 1: Decrease in the have not gone to school in the interventions designed to encourage number of out-of-school north. Thus, BESDA aims to girls to go to school as they will children relative to NEDS achieve increases in access across receive more money for each 2015 baseline the board in focus states. As part of additional girl brought into the Year 2: Decrease in the this effort, it seeks to improve educational system. The way the DLI number of out-of-school enrollment among girls that are at is calculated it not only rewards the children relative to year 1 least as high as those for boys. reduction in out-of-school children Year 3: Decrease in the but also their retention. It is for this number of out-of-school reason it is transformational—that is, children relative to year 2 it not only rewards bringing kids into Year 4: Decrease in the school, it rewards them for staying in number of out-of-school school. children relative to year 3 Year 5: Decrease in the number of out-of-school children relative to year 4 Improve literacy DLI 2 The quality of basic education, DLI 2 is critical to expanding on Year 0: States’ selected measured in terms of student existing intensive literacy programs literacy program learning outcomes, is low on in states which have not yet been approved, selection of average in Nigeria, and is a major able to adopt them in their schools participating schools by contributing factor to the high out- due to lack of resources and TA. It is LGA completed, and of-school rates among both boys transformational because the final quantity of student and girls. The focus on literacy is DLR will reward states where more textbooks and teacher’s important for many reasons: than half of the schools implement guides determined for literacy is fundamental to all areas the program with the hope that procurement and of learning, as it unlocks access to achieving a critical mass will distribution the wider curriculum; and improve literacy in the state. Without Year 1: Students and increased literacy will demonstrate the DLI, disparate and inconsistent teachers adopt intensive the benefits of education to efforts dependent on fragmented literacy program in families and encourage continued donor resources would continue and primary 1 school enrollment. At the same fewer schools would be able to adopt Year 2: Students and time, increasing literacy rates the program and there would be less teachers adopt intensive entails improvements along several likelihood of observing improved literacy program in dimensions, including teaching literacy. primary 1 and 2 quality and learning materials such Year 3: Students and as textbooks for students and DLI 3 incentivizes not only adoption 22 Results Area Disbursement-linked Rationale Why it is Transformational Result (DLR) teachers adopt intensive instruction manuals for teachers. of the program but the end goal of literacy program in improved learning outcomes. Thus, primary 1–3 there is a very strong incentive to Year 4: States with more implement a highly concentrated than 50% of primary effort to teach kids to read. This is schools adopting intensive transformational because reading is literacy programs in fundamental to all other learning and primary 1–3 improved life quality. Increased literacy will hopefully have a DLI 3 demonstration effect where the Year 3: Improved literacy benefits of education to families will rates in focus states as encourage continued school measured by percentage enrollment. point increase above baseline Year 5: Improved literacy rates in focus states as measured by percentage point increase above baseline Strengthen DLI 4 Data discrepancies and DLI 4 is transformational because it accountability Year 1: States implement inaccuracies (for example, pupil provides financial incentives for for results and publish the ASC enrollment and teacher collection of data for the ASC using verified by SBMCs deployment) have seriously appropriate methodologies. Without Year 2: States implement hampered policy making and the DLI, only 18 states collected the and publish the ASC, with dialogue, including between the ASC and published the ASC report at least 10% of school federal and state governments, a using the enumerator approach. reports verified by key requirement for effective UBE LGAs collected information in the SBMCs greater than the program implementation. The aim other 19 states without using the percentage in year 1 of this indicator is to ensure enumerator approach. The DLI will Year 3: States implement implementation in all states of the not only incentivize the use of a and publish the ASC, with ASC under NEMIS, external uniform approach to data collection at least 10% of school validation of administrative data across schools and states but also reports verified by through feedback provided by improved data quality. The DLR SBMCs greater than the SBMCs requires validation of the report by a percentage in year 2 non-school member of the SBMC, Year 4: States implement with the proportion of schools and publish the ASC, with needing to meet this requirement at least 10% of school increasing over the duration of the reports verified by program. The DLI is also SBMCs greater than the transformational because it creates a percentage in year 3 culture where SBMCs are expected Year 5: States implement and encouraged to participate in and publish the ASC, with monitoring schools and participate in at least 10% of school the data collection process as a third- reports verified by party verifier. The collaboration in 23 Results Area Disbursement-linked Rationale Why it is Transformational Result (DLR) SBMCs greater than the this process should encourage closer percentage in year 4 links and relationships with the There is no validated consolidated schools and enhance SBMC authority ____________________ information on public expenditures and participation. DLI 5 on basic education in Nigeria, Year 1: UBEC approves which prevents an informed DLI 5 is transformational because it states’ submitted annual evaluation of financial resource establishes the foundation for operational plan using mobilization. Information evidence based policy and decision agreed template. It will previously provided by the Central making by enhancing the availability include data from the Bank of Nigeria (CBN) is based on of data for strengthened data systems ASC, budget data, and a survey and not on the necessary for planning and reporting data on SBMC consolidation of certified budget at the state level. Regular annual functionality including execution reports (CBN 2013). state planning and reporting is progress report on UBE Consolidated budget information fundamental to building transparency interventions will require the harmonization of and accountability for results. DLI5 Year 2: UBEC approves charts of accounts used across will institutionalize this process and states’ submitted annual levels of government, the publication of education spending operational plan using functional reclassification of and accounting for implementation agreed template. It will budget expenditure, and systematic progress. include data from the and audited reporting on budget ASC, budget data, and execution. For lack of such data on SBMC standardized budget information, it functionality including is practically impossible to assess progress report on UBE the effectiveness of financial interventions resource allocation in basic Year 3: UBEC approves education. This DLI will states’ submitted annual incentivize regular annual planning operational plan using for implementing UBE activities agreed template. It will and reporting related progress on include data from the implementation. The DLI not only ASC, budget data, and requires states to plan and report data on SBMC but also that UBEC approves them. functionality including In essence, this is requiring the progress report on UBE states to be accountable to UBEC interventions for implementation of the UBE Year 4: UBEC approves interventions. states’ submitted annual operational plan using agreed template. It will include data from the ASC, budget data, and data on SBMC functionality including progress report on UBE interventions Year 5: UBEC approves 24 Results Area Disbursement-linked Rationale Why it is Transformational Result (DLR) states’ submitted annual state operational plan using agreed template. It will include data from the ASC, budget data, and data on SBMC functionality including progress report on UBE interventions B. Data Sources for Disbursement-Linked Indicators 42. Data sources vary in terms of quality, rigor, and risk. An important strength of the Results Framework and DLI matrix is that many of the indicators are designed to leverage and build existing country systems. 43. Results Area 1. The first DLI is based on the NEDS conducted by the National Populaton Commission (NPC). The NEDS is a household survey which provides information on education of children ages 5–15 and is representative at the state level. The NPC of Nigeria was established by the Federal Government in 1988. It has the statutory powers to collect, analyze, and disseminate demographic data in the country. It is also mandated to undertake demographic sample surveys; compile, collate, and publish migration and civil registration statistics as well as monitor the country's Population Policy. The NPC has received technical support for the NEDS survey from USAID. In addition, there are several donors supporting the improvement of data collection and general M&E for basic education, including UNICEF (ASC) and USAID (literacy assessment). In close collaboration with these donors, the program will work to support the enhancement of the NEDS to ensure that the questionnaire is further improved to adhere to international standards for basic education, especially regarding the wording of questions related to out-of-school children and literacy. Further, the M&E members of the National Policy and Monitoring Team (NPMT) will work closely with the NPC to ensure that appropriate sampling is conducted to have representative data for the Program. 44. Results Area 2. For purposes of verification of DLI 2, states will be required to maintain records on all aspects of the adoption of the intensive literary program, including in-service teacher training on literacy; evidence of the distribution of books to children and instructional handbooks to teachers (for example, photos of students holding up their books in which they have written); classroom observations of teachers’ performance in applying the techniques for literacy by mentors; coaches and quality assurance officers; and timetables of teachers teaching. DLI 3 will use the NEDS as the data source for literacy rates of children ages 5–15. The NEDS is based on a simple methodology of presenting children with flash cards to assess whether they can read one of three words in English or one of the three official languages (that is, Hausa, Yoruba, and Igbo). 45. Results Area 3. DLI 4 will rely on data collected by the schools and submitted to the state EMIS team to be consolidated, aggregated, and analyzed in a computerized database. Data 25 collection will use a single ASC template to be completed by school heads who have been trained on how to fill out the form. The form will be signed by the school head as well as a non- school member of the related SBMC to assure data quality. NEMIS will consolidate the state data and the information will be published on the FME website within six months of data collection. Finally, DLI 5 will rely on an agreed unified template which will capture planning and reporting at state levels on UBE program implementation, using data from the ASC, budget data, as well as data on SBMC functionality. The M&E member of the NPMT will need to coordinate across various data sources to collect the necessary data for Program monitoring across the three results areas. 46. UBEC has the mandate to monitor and evaluate the outcomes of federal-level inputs into basic education and implementation of the UBE Intervention Fund including developing a database for basic education and conducting research . The current M&E system is implemented by the FME, UBEC, SUBEB, LGEA, and SBMCs. Although, UBEC took over the responsibility of M&E of UBE in 2012 in an effort toward improving data quality, interagency coordination has not improved. For instance, duplicate efforts exist by the FME and UBE in both conducting NLAs. Redundant efforts also revolve around the national ASC and UBEC’s quality assurance processes and personnel audits. UBEC together with SUBEBS and LGEAs survey schools and conduct site visits to assess the quality of schools, teaching performance, and community satisfaction with schools and teachers through dialogue with parents and SBMCs. In this exercise they collect information on enrollment, teachers, and Parent Teachers Association (PTA) involvement. In addition, they conduct a personnel audit which captures information duplicating the ASC. 47. Taking into consideration the key constraints and gaps in the current M&E system, BESDA will support the strengthening of data collection and utilization in key areas of the Program results chain. Key DLIs have been identified to incentivize improvements in data collection. This will include (a) administration of the ASC based on a uniform template verified by SBMCs and its timely publication on the FME website and (b) production of UBEC-approved annual state plans on basic education expenditures and progress reports on implementation of UBE program activities by states to enhance the data available to assess the efficiency of financing for basic education. In addition, TA will be provided to support design and implementation jointly by UBEC and the FME of an NLA, as well as implementation of the Program M&E system. C. Managing Risks and Improving Capacity 48. A potential risk that may hamper disbursement under Results Area 1 is the timeliness of the NEDS survey, which will be conducted annually for the first time. Another potential issue is the capacity of the NPC to conduct the NEDS on a yearly basis without the support of international consultants, as previously used in 2015. These risks will be mitigated by providing technical support through local private consultants or academic institutions. Once the importance of the NEDS results for disbursement becomes clear it will shift from a low stakes survey to become a high stakes survey. While this may produce incentives to alter households’ responses to the NEDS in terms of inflating the number of children in school, practically, it would be difficult to influence all respondents given the sheer size of the survey and the nature of random sampling. 26 49. Another possible risk is potential discrepancies between the NEDS findings and other national surveys coming out in 2017 and 2018, including the Multiple Indicator Cluster Survey and the National Living Standards Survey. Differences could be due to different survey dates, or slight variations in the wording of questions to households. The risk of the NEDS lacking credibility in the face of contradicting information will be mitigated through the use of robust TA that provides credible quality assurance on survey implementation. These additional efforts include conventional survey methods such as routine monitoring of interviews by team supervisors during data collection, but also technological innovations allowing automated data quality checks based on anomalies flagged in the system. 50. The risk of redundant efforts and poor quality and inconsistent data resulting from a lack of coordination across the FME, UBEC, and SUBEBs will need to be addressed to develop a NEMIS which consolidates national- and state-level representative data from the ASC on schools, teachers, students, learning outcomes, education budget resources, and expenditures. The Program will leverage the existing Steering Committee, peer learning, and technical working groups to facilitate opportunities for communication and cooperation between the FME and UBEC in delineation of tasks. Other risks associated with generating timely data for DLI reporting will be mitigated by technical support provided by the World Bank and other development partners as needed to improve instruments and processes for data collection, analysis, reporting, and dissemination. IV. Program Economic Evaluation 51. In Nigeria, any additional year spent in school matters: even having an incomplete primary education means having a total monthly income 22 percent higher than people who never attended school (figure 2). 6 Returns of secondary, technical and vocational education and training (TVET), and tertiary education are high. People with complete upper secondary education or TVET can expect to earn more than 80 percent and 100 percent more, respectively, than people who never attended school. The highest return is for people with tertiary education (130 percent). The education premium in earnings increases as the level of education increases; the premium is manifested prominently in the services sector with high- value-added services like banking and ICT services, and lowest in the agriculture sector— although the premium is still high even there. 52. Beyond private returns to education, the economic rationale for public intervention in basic education under this Program is twofold: (a) the social benefits and externalities associated with education and (b) on redistributive and equity grounds. With regard to social benefits and externalities resulting from education, the evidence on girls’ education is particularly strong: better-educated women enable better health care and education for their own children. In other words, by focusing on educating girls, this Program is not only increasing these girls’ productivity, but leading to higher personal incomes as well as healthier lives. In addition, by educating girls, this Program is fostering a virtuous cycle leading to a next generation of healthier and better-educated children. 6 World Bank. 2015. Nigeria: Skills for Competitiveness and Employability. 27 Figure 2. Private Returns to Education in Nigeria, by Economic Sector (2013) Source: World Bank. 2015. Nigeria: Skills for Competitiveness and Employability, Figure 1.2. 53. Additionally, from a redistributive and socioeconomic equity point of view, by focusing on educating out-of-school children, this Program is focusing on poorer segments of the Nigerian society. Evidence has been provided in earlier sections to the effect that out-of- school children are disproportionately poor, as well as being perhaps those who became recently impoverished through the conflict in the North East. In addition, potential integration of core basic education subjects into the education of almajiri children, who also tend to be poor, would mean that this Program would be further improving equitable provision of education services. 54. Increased access to education alone is not enough, as a better quality of education implies a stronger impact of education on personal productivity, and thereby on national economic growth. Therefore, student learning, as opposed to simply student enrollment, is key to garner the economic benefits of education investments. Teacher quality, in turn, is a key driver in achieving student learning.7 It is true that students’ family background (parent education, socioeconomic status, and conditions at home such as access to books) is the strongest predictor of students’ learning outcomes. But once children get to school, no single factor is as critical as the quality of teachers. Research in the United States on the ‘value added’ of individual teachers over the course of a single school year has shown that students with a weak teacher may master 50 percent or less of the grade curriculum; students with a good teacher advance by one year; and students with great teachers advance 1.5 grade levels or more. Therefore, by enhancing teachers’ skills under the intensive literacy programs, this Program is making an important contribution toward student learning and thereby increasing the economic impact of the Program. 55. Finally, this Program fosters a results orientation of the UBE program, thereby contributing to improving the efficiency of spending under the Government program. 7 This section draws extensively on Bruns, Barbara, and Javier Luque. 2014. Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. World Bank. 28 Given the commitment of Nigeria to achieving UBE and the continued substantial financing of the UBE program, increased availability of data on outputs and outcomes, coupled with associated public spending, will lead to a better understanding of the efficiency of spending and potential reorientation to a more productive allocation of the UBE Intervention Fund further down the line. 29