WPS7021 Policy Research Working Paper 7021 The Permanent Input Hypothesis The Case of Textbooks and (No) Student Learning in Sierra Leone Shwetlena Sabarwal David K. Evans Anastasia Marshak Education Global Practice Group & Africa Region Office of the Chief Economist September 2014 Policy Research Working Paper 7021 Abstract A textbook provision program in Sierra Leone demonstrates to students. At the same time, the propensity to save books how volatility in the flow of government-provided learn- was positively correlated with uncertainty on the part of ing inputs to schools can induce storage of these inputs by head teachers regarding government transfers of books. The school administrators to smooth future consumption. This evidence suggests that schools that have high uncertainty process in turn leads to low current utilization of inputs with respect to future transfers are more likely to store a for student learning. A randomized trial of a public pro- high proportion of current transfers. These results show that gram providing textbooks to primary schools had modest reducing uncertainty in school input flows could result in positive impacts on teacher behavior but no impacts on higher current input use for student learning. For effective student performance. In many treatment schools, student program design, public policy programs must take forward- access to textbooks did not actually increase because a large looking behavior among intermediate actors into account. majority of the books were stored rather than distributed This paper is a product of the Education Global Practice Group and the Office of the Chief Economist, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank. org. The corresponding authors may be contacted at ssabarwal@worldbank.org and devans2@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team The Permanent Input Hypothesis: The Case of Textbooks and (No) Student Learning in Sierra Leone Shwetlena Sabarwal David K. Evans Anastasia Marshak World Bank World Bank Tufts University Keywords: Education, Impact Evaluation, Sierra Leone, Smoothing JEL Codes: O15, I21, I28, J13 Acknowledgements: The team is extremely grateful for the support of the World Bank’s Education Program Development Fund, which provided financing. Keiko Inoue co-designed the evaluation, and Eunice Yaa Brimfah Ackwerh supervised implementation of the intervention. The team benefitted from research assistance from Anna Popova, and comments and guidance from Deon Filmer, James Habyarimana, Peter Holland, Luis Benveniste, Owen Ozier, Christopher Thomas, Ritva Reinikka, Arianna Legovini, Muna Meky, Jee-Peng Tan, and participants in seminars at Brigham Young University, the World Bank, and the Pacific Development Economics Conference. Corresponding authors: Shwetlena Sabarwal (ssabarwal@worldbank.org) and David Evans (devans2@worldbank.org). 1. Introduction In the macroeconomic literature, volatility in aid flows has been linked to reductions in economic growth (Kodama 2012) and significant decreases in overall welfare (Arellano et al. 2009). This paper explores similar patterns within one sector, education, at a micro level. We provide evidence that unpredictable variability in flows of school inputs interacts with the desire of actors (in this case, school administrators) to smooth input utilization over time, resulting in low current utilization of these inputs for student learning. In education, a growing body of empirical evidence shows that public provision of school inputs does not always lead to improved learning outcomes. In fact, there is very wide variation in estimated coefficients of school inputs on test scores (Glewwe et al. 2011; Kremer, Brannen, and Glennerster 2013; Krishnaratne, White and Carpenter 2013; McEwan 2014). This may be puzzling, especially with regard to inputs like textbooks which are both demonstrably scarce in developing country schools and at the same time believed to be vital in the education production function. Clearly, a failure to detect the impact of input provision on student learning could arise due to several reasons. It could be because the inputs are ineffective due to their inherent nature or their quality, resulting in a zero impact on student performance. For instance, a randomized evaluation from Kenya showed that providing textbooks did not raise average test scores, possibly because textbooks were written in English, which is a third language for most students, making it difficult for them to use textbooks effectively (Glewwe, Kremer, and Moulin 2009). Lack of impacts from input provision could also be because beneficiaries re-optimize their allocation of resources when receiving an input. For example, Das et al. (2011) show in both Zambia and India that when school grants for student learning inputs were anticipated by households, the households compensated by reducing private education spending. Alternatively, a study from Kenya shows that insecticide-treated bed-nets distributed to children under age 5 for use over their beds were actually being used for fishing and for drying fish in Lake Victoria (Minakawa et al. 2008). These and other studies provide evidence that behavioral responses of agents to public programs can mitigate program impacts on ultimate beneficiaries. In this paper we present evidence of another scenario where the behavioral response of intermediate agents to a public program strongly influenced program outcomes on 2 beneficiaries. In this case the agents in question are school administrators, an often neglected set of actors in empirical work. Analyzing the results from a randomized evaluation of a textbook provision program to schools, we show that the program failed to have any impacts on student outcomes (attendance or performance) because few textbooks reached the students. Instead, publicly provided textbooks were stored by school administrators. Why should publicly provided school inputs to schools be stored rather than distributed to students? We analyze the correlates of book storage and find that head teachers who exhibit high uncertainty with respect to government transfers of textbooks are significantly more likely to store them. Higher uncertainty is measured by the answers to two queries: if head teachers indicate that they knew how many books the government was to allocate to their school in the previous year, and whether they were able to provide a specific number. This storing behavior in the face of greater uncertainty makes sense from the perspective of consumption theory: Consumption of current government transfers will be dictated in part by expectations regarding future transfers. If current transfers are seen as a one-time (or transitory) shock then their impact on current consumption will be limited. In other words, when there is uncertainty, school administrators have incentives to store current transfers in order to smooth consumption over time, particularly for inputs like textbooks that depreciate quickly when used. As mentioned earlier, this paper sheds light on the importance of behavioral responses of intermediate agents like school administrators in determining the overall impacts of public input provision programs to schools. In particular, it provides an empirical basis to question the assumption that publicly provided inputs are deployed effectively within a short time period. It also puts at center stage the question of how future uncertainty regarding government transfers and policies impacts the current behavior of service providers. On the whole, this analysis adds to our understanding of why such input provision programs might be less effective than expected. Broadly speaking, our results are consistent with the premise that merely providing inputs to schools while ignoring the incentive structure within schools and other systemic challenges is unlikely to yield the desired gains in student learning outcomes. The remainder of the paper is structured as follows: Section 2 provides a broad outline of the education system in Sierra Leone; Section 3 lays out the experimental design. 3 Statistics from baseline, actual implementation of the program, and the empirical strategy are summarized in Sections 4, 5, and 6. Section 7 presents the overall impacts of the program. Results are discussed in sections 8, 9, and 10. Finally, conclusions and policy implications from this empirical work are laid out in Section 11. 2. Context Over a decade ago, Sierra Leone emerged from a brutal, decade-long civil war (1991– 2002), but the impacts linger. The civil war severely impacted the country’s education system leading to large-scale devastation of school infrastructure, severe shortages of teaching materials, overcrowding in many classrooms in safer areas, displacement of teachers, frequent disruptions of schooling, psychological trauma among children, poor learning outcomes, weakened institutional capacity to manage the system, and a serious lack of information and data to plan service provision (World Bank 2007). The education sector is still in the process of being rehabilitated. The post-war “catch-up” phase and the accompanying construction boom supported by donors, together with the Government’s Education Act of 2004 which abolished primary education fees for all, have all contributed to this rehabilitation, driving a sharp increase in access and gross enrollment. Nonetheless, in 2011 more than half of the working age population in Sierra Leone had no formal education, with an average attainment of just 4 years of schooling (World Bank 2014). The Education Act of 2004 stipulates universal basic education – 6 years of primary school and 3 years of junior secondary school. Enrollments in primary school more than doubled since 2000, with the gross enrollment rate (GER) reaching 115.7 percent in 2011. Two-thirds of children enrolled in primary education in Sierra Leone attended government schools, 31 percent attended religious schools, and only 6 percent (composed predominantly of children from the two richest quintiles) attended private schools. The primary completion rate was 69 percent in 2011, 11 percentage points below the average for sub-Saharan Africa (World Bank 2014). Several challenges to the provision of basic education persist, many of them supply- related. As of 2008, sixth-grade pupils from 5,000 primary schools competed for places in fewer than 300 junior secondary schools, for example. Although the government abolished school fees, primary education is still not completely free because schools impose a variety of charges on their students, such as informal tuition fees and auxiliary costs such as school uniforms and books. Our baseline data (see Section 3.2 for more details) show that around 28 percent of the schools charged some degree of fees from parents. As many as 70 percent of the schools received subsidy payment for school fees 4 from the government, while 28 percent of the schools also received support from the community, but there is no systematic correlation between the two sources of support – one is not a substitute for the other. Most schools in Sierra Leone have very poor classroom conditions and still lack sufficient learning materials and adequately qualified teachers (World Bank 2007). The government and development partners have made efforts to provide textbooks, but significant challenges remain. One study reported that, in 2004, a ratio of 1 set of textbooks to 3 pupils in urban areas and 1 set to 6 pupils in rural areas had been reached, whereas the official policy was a ratio of two students for one set of four core- subject textbooks (World Bank 2007). With the Local Government Act of 2004, the Government of Sierra Leone commenced a national decentralization process: Primary education stood among the first functions scheduled to devolve to the Local Councils. Under the policy of decentralization, local councils would have full control and supervision of all primary schools including such functions as the recruitment and payment of teachers, the provision of textbooks and teaching materials, and the rehabilitation and construction of schools. Our baseline data show that the process of decentralization was still ongoing in the education sector of Sierra Leone as of early 2008. District Education Officers (DEO) in the centralized system and local council officers in the decentralized system were both active, leading to some confusion on the exact chain of command and roles and responsibilities of different agents. 3. Experimental Design 3.1 The Intervention This paper evaluates a basic textbook distribution program of the Government of Sierra Leone in the year 2008. Under the program, textbooks were provided by the Ministry of Education, Youth, and Sports (MEYS) to primary schools based on student enrollment numbers as captured by the Education Management Information System – or EMIS. The actual transportation of books from central warehouses to schools was undertaken by local service providers who were competitively selected by the Government. The main objective of the textbook distribution program was to provide a set of core textbooks for every child in the treatment schools. 1 1 A set of core textbooks is made up of English, Mathematics, Integrated Science and Social Studies textbooks. 5 While there were just over 5,000 primary schools in Sierra Leone at the start of the program, the impact evaluation (IE) only focuses on schools that were registered with MEYS as having up to grade 6, the year when the National Primary School Examination is taken. The program included government, government assisted, and community schools, but not private schools. The IE focused on grades 4 and 5, as these are the standards by which education is principally in English, the language of the textbooks. 2 3.2 Sampling The IE relies on a randomized experimental design using a two-step process. First, 4 out of 19 local councils, stratified at the regional level, were randomly selected. All 19 local councils were interested in participating in the study, thus one council from each of Sierra Leone’s four regions was chosen: Kailahun (Eastern region), Kambia (Northern region), Pujehun (Southern region), and Western Urban and Western Rural (who agreed to enter the randomization jointly). The locations of these councils are illustrated in Figure 1. Within the randomly selected councils, 360 program schools were randomly selected using the EMIS data. From the universe of schools in these councils, schools that were already being targeted for textbook interventions by other organizations were excluded. Also, to avoid providing textbooks to schools that already had many, this project included only schools which had a 3:1 student-textbook ratio or higher. From the reduced sample, 90 schools were selected in each district (except for Western Rural and Western Urban which together amounted to 90 schools). Within the local council, sample schools were assigned with equal probability to one of three treatment groups (30 schools in each group in each district): receiving textbooks, receiving textbooks plus teacher training, and a control group. Ultimately, due to multiple implementation challenges, the teacher training component was not undertaken. Therefore, for the IE, we are left with schools that received textbooks (treatment schools) and schools that did not (control schools). Baseline data were captured on 341 schools (out of a total of 360). The 19 schools that were not included in the sample were mainly due to the inability of enumerators to access the schools due to either poor road conditions or flooded rivers. For a small portion of these schools, the enumerators were able to access the location, but there 2 Education at earlier grades is sometimes carried out in the Krio language. 6 was no school at the premises. This could partly be attributed to the large share of “ghost schools” in Sierra Leone. 3 The total number of schools for which data are available at baseline and endline, by district, is summarized in Table 1. Comparing baseline characteristics for treatment and control schools, we find that the sample is balanced across a wide range of characteristics including infrastructure, student-teacher ratio, school fees, test scores, and many others (Table 2). Of 18 such characteristics, a significant difference (at the 90% level or higher) between treatment and control schools is found only for the student-classroom ratio. Thus, the two groups were well balanced at baseline. 3.3 Data Baseline data collection took place in May 2008 and final or endline data collection took place in December 2009. The data collected included a head-teacher survey, a classroom teacher survey, a student survey, and student exams. The data on the school as a whole were obtained through direct observation (e.g., the condition of the buildings, the number of classrooms and other facilities) and a comprehensive interview with the head teacher addressing school finances, record keeping, community participation, management practices, etc. Data were also collected from teachers for grades 4 and 5 through teacher interviews. One hundred students were selected randomly at each school and were given a written numeracy and literacy test. At baseline (May 2008) the student tests were administered to students in Grades 3 and 4 and at endline the same tests were administered again to the same cohort, which was now in Grades 5 and 6 (Dec 2009). Although we have a large number of tested students, only a small fraction were able to be definitively matched from baseline to endline. Therefore, to measure impacts on student performance, our preferred estimates rely on endline scores only, controlling for school- cohort-subject mean at baseline and – beyond that – relying on the assumption of balance on average across schools at baseline. We also report results for the change in score of those students who were matched for baseline and endline, but these results may be affected by selective attrition in test participation. 3 A 2008 news item highlighted the challenge of “ghost schools,” which the Government was seeking to identify and shut down (Fofana 2008). In 2011, the Government of Sierra Leone announced that they had discovered over 5,000 “ghost teachers,” teachers that do not exist but are used to collect payroll on behalf of corrupt principals and officials (Coker 2011). 7 At endline, we also interviewed students from Grades 4 and 5, in order to characterize actual use of textbooks; the books distributed by the program were targeted to Grades 4 and 5. We were unable to gather data on 5 percent of the schools originally designated to treatment and control groups. Table A1 shows the correlates of attrition at endline. The dependent variable is a dummy equal to 1 if the school attrited by the time of the endline survey, and we include a number of control variables. We estimate the attrition regressions using a linear probability (ordinary least squares, or OLS) model. We estimate the relationship between attrition and both assignment to treatment (column 1) and actual treatment (column 2); results are consistent across both estimates. In both specifications standard errors are clustered at the school level. Table A1 shows that neither being assigned as a treatment school nor actually receiving additional books predicted attrition. This is robust to controlling for a number of baseline household characteristics: number of classrooms, electricity, distance to the DEO’s office, and number of students. From both models we see that having a greater number of classrooms at baseline decreases the probability of attrition, while having electricity increases it, although none of the other controls are significant predictors (except the dummy variable for having missing data on the distance to the DEO’s office). Overall, these balanced rates of attrition across treatment and control groups suggest that the impact evaluation results are unlikely to be driven by attrition. 4. Education Service Delivery in Sierra Leone: Evidence from the Baseline Survey One issue that emerged from the baseline survey in the surveyed councils (Kailahun, Kambia, Pujehun, Western Urban and Western Rural) is the stark supply-side disadvantage that remote areas face with respect to education service delivery. Pujehun, which is one of the more remote councils in Sierra Leone, was significantly worse off compared to the other regions in terms of the supply of schooling inputs. In this region, less than 50 percent of all schools were considered to be in good condition, with almost 20 percent falling under the category “no roof, walls are heavily damaged, needs complete rehabilitation.” Further, almost one-third of the schools did not have a working toilet. During the baseline survey (May 2008), unannounced classroom visits were carried out in Grade 4 and 5 classrooms. Out of 828 such classrooms, teachers were found to be actively teaching in 54 percent of cases. If we include classroom management activities 8 (organizing the class, setting the agenda, or disciplining students) then the corresponding number was 77 percent. Once again, the remote councils performed worse than others: Only half of all teachers were engaged in teaching or classroom management in Pujehun, while the corresponding numbers for some other councils were 90 percent. As a part of baseline data collection, students in Grades 3 (8,740 students) and 4 (8,815 students) were given multiple-choice mathematics and literacy tests. For mathematics, 49 percent of Grade 3 students and 32 percent of Grade 4 students could not answer the single-digit subtraction question “Calculate 9-4”. Similarly for literacy, when presented with the word “bird”, 45 percent of Grade 3 students and 35 percent of Grade 4 students could not identify which one of four four-letter words provided was identical to it. Samples of results from the math and literacy tests are summarized in Figures 2a and 2b. A student in grades 4 or 5 requires four textbooks: math, English, social studies, and science. At baseline, students only had 2 textbooks on average, far from the government stipulated level of 4 textbooks per student. Further, only about 40 percent of the teachers claimed that they allowed students to take textbooks home, which implies that student access to textbooks at home is very low. To complement this, at endline, around 36 percent of the sampled schools reported parents complaining about lack of textbooks in the last six months. The likelihood of parents complaining about a lack of learning materials is not systematically related to the existence of a school management committee or to schools charging fees. 5. Intervention Implementation To determine how well the intervention was implemented, two sources of data are used: (i) information from the head teacher at endline and (ii) data on textbook distribution collected by the IE team from district education officers and service providers. 4 Using these data we determine how many schools actually received books from the project. As a second step, head teacher data are used to determine if the school received textbooks from other sources (such as from local government, non-government 4 Head teacher data are given precedence; administrative data are used when head teacher data are missing. In case head teacher and administrative data contradict each other, head teacher data are used, except in 26 cases where the head teacher claims the school received books from the project but is unable to provide information on the number of books received or date books were received. 9 organizations, or from MEYS but outside the project). This allows us to create a variable for actual treatment, i.e., schools in the program that actually received textbooks in 2008-09. Contrasting the measure of actual treatment with the original, randomized allocation, we observe some non-compliance. Fifteen percent of schools assigned as treatment did not receive any textbooks, and 46 percent of schools assigned as control received textbooks from one source or another. Still, assignment increased the probability of receiving books from 46% to 85%. This information is summarized in Table 3. 6. Empirical Strategy Given the imperfect compliance to random assignment, the evaluation first reports Intent-to-Treat (ITT) analysis for identification of program impacts. In ITT analysis, participants are analyzed as if they received the treatment to which they were assigned (Begg 2000). This analysis yields an unbiased estimate only of being assigned to a treatment and not of actually receiving the treatment. Hence, using this methodology we estimate the impact of being assigned to receive textbooks and not of actually having received textbooks. These are likely to be underestimates of the true impact of receiving textbooks. Outcomes are tested using OLS regression models that use assignment to the treatment group as the explanatory variable and thereby calculates how much an outcome of interest – e.g., test scores – differs between schools that are assigned as treatment schools and those that are assigned as control schools. = 0 + 1 + (1) where is the outcome in school i, and Assignment = 1 if the school was randomly assigned as a treatment school and 0 otherwise. Outcomes are tested at the classroom and student levels when appropriate. At the classroom and student level the specification is as follows: = 0 + 1 + (2) where is the outcome in classroom i and school s for the classroom level regression, and it is the outcome for student i and school s for the student level regression. The variable Assignment = 1 if the school was randomly assigned as a 10 treatment school and 0 otherwise. The error term is clustered at the school level in these specifications to account for intra-school correlation of outcomes. The parameters of interest are 1 and 1, which are the average effect of being assigned as a treatment school in the textbook provision program. To supplement this analysis, treatment-on-the-treated (TOT) estimates are also presented. Because actual provision of textbooks to schools outside the randomized assignment could be a function of observed and unobserved school characteristics which may also be correlated with the outcomes of interest (i.e., endogenous), we cannot simply compute the difference between outcomes of schools that received textbooks with those that did not. To correct the potential endogeneity, we employ instrumental variables estimation, using the randomly assigned treatment or control status of a school as an instrument for actual school treatment. The treatment status of a school has a high correlation with actual provision of textbooks to schools (satisfying the first stage requirement for instrumental variables), and it is orthogonal to school characteristics because it is randomly assigned (satisfying the exclusion restriction). For TOT estimation we rely on a two stage least squares model. In the first stage we estimate: = 0 + 1 + (3) where is an indicator of whether school i actually received textbooks during the � – are intervention period. In the second stage, the predicted values of – labelled included as an independent variable: � = 0 + 1 + (4) The key parameter of interest is again 1, which represents the average impact of the program for those schools that received books due to the random assignment. We calculate both ITT and TOT estimates for all outcomes and present both: Panel A contains ITT estimates and Panel B contains TOT estimates throughout the results tables. In general, the pattern of results is highly consistent across ITT and TOT estimates, and we discuss the estimates separately only in cases where the direction or significance varies by estimation technique. However, in the discussion and conclusion 11 sections we rely more heavily on the ITT analysis, as the cleanest estimate of the impact of being assigned as a treatment school for receiving textbooks. 7. Overall Impacts At the school level, we find no impact of the program on student enrollment. We do find, however, that the likelihood of parents complaining to head teachers about the lack of textbooks is substantially lower in treatment schools (Table 4). Information on teaching practices was collected through interviews with teachers for Grades 4 and 5 at endline. These impacts are summarized in Table 5, wherein each teacher interview is a distinct observation and standard errors are clustered at the school level. We find that the program had no impact on the likelihood of a teacher assigning homework from textbooks (Table 5, Columns 1 and 2). This is most likely because the reported propensity to assign homework from textbooks is fairly high even in control schools (93 percent). One may question how teachers are able to assign homework from textbooks if very few students have access to the textbooks. Anecdotal evidence suggests that this is very common: Teachers copy textbook questions onto the blackboard and ask students to copy them down and answer them at home as a part of their homework. Likewise, the program did not influence the likelihood of physical punishment for students found mistreating textbooks (Columns 3 and 4). We also find that the program did not impact the likelihood of a teacher producing pamphlets (informal notes or study guides) for students (Columns 7 and 8). However, we do find significant positive impacts of the program on the likelihood of a teacher having a lesson plan (Columns 5 and 6). Student attendance data were collected through unannounced visits to classrooms (Grades 4 and 5) at endline. We see no pattern of improved attendance, although we do observe a positive significant effect on girls’ attendance in grade 5. No other discernible program impacts on student attendance by grade or by student gender are found (Table 6). These unannounced classroom visits also yielded data on teacher activity (Grades 4 and 5) in class. The program did not affect the likelihood of finding a teacher in class but did lead to an increased likelihood of finding a teacher in class teaching (Table 7). Note that in the control group about 18 percent of the teachers were absent from the classroom at the time of the visit; however, of the teachers in class, only 57 percent were found to 12 be actually teaching. TOT estimates suggest that the program dramatically increased in- class teaching, to about 90 percent (Table 7 Column 2). However, we find no program impacts on learning outcomes as measured by the literacy and mathematics exams (Table 8). As noted above, impacts on student performance are measured by comparing student test scores in literacy and mathematics at endline between control and treatment schools, controlling for baseline school-cohort-subject average scores. Standard errors are clustered at the school level. 5 Therefore, on the whole we find that the textbook provision program had some positive impacts on teacher behavior (increased presence of lesson plan and increased likelihood of being in class and teaching) but no discernible impacts at the student level (either on attendance or performance). 8. What explains the lack of program impacts on students? In order to fully understand the reasons for absence of program impacts at the student level, we exploit student survey data to examine how the program impacted textbook use by students. These data come from interviews conducted with students in Grades 4 and 5 at endline. We find that the government textbook provision program did not significantly impact students’ use of textbooks. Table 9 summarizes program impacts on indicators of student textbook use and related behaviors. We see that the program did not increase the likelihood that students in treatment schools were using a textbook in class for a core subject, have a new textbook, or use a textbook given to them by a teacher. Likewise, the program did not impact the number of students with whom textbooks were shared, or the likelihood of receiving pamphlets (informal notes or an exam guide) from teachers. Information was also collected on materials used by students for exam preparation, including books, pamphlets, and notes (Table 9 Columns 6-8). On these indicators as well, the program had no discernible impacts. 5 The test estimates do not use matched scores for individual students between baseline and endline because of extensive difficulties in matching, mainly because certain first and last names occurred with high frequency and students used inconsistent placement of first, middle, and last names. As a result, less than one-third of students could be matched, and we cannot rule out selective attrition in the matched sample. Nevertheless, we estimated program impacts on test scores even in that sub-sample and found no evidence of impact. 13 Student use of textbooks is often heavily contingent on teacher behavior. We examine program impacts on teacher behaviors with relation to textbooks in Table 10. This information also comes from student interviews. We see that while the program had modest impacts on the likelihood of teachers encouraging the use of textbooks by students, it did not impact any of the more direct predictors of textbook use, such as students reading from textbook in class, students being allowed to take textbooks home, and homework assignment in general. Given the lack of program impacts on student textbook use, it is not surprising that we observe no impacts of the textbook provision program on student performance. We are, however, faced with a puzzle – when schools, by their own admission, received textbooks from the government, why then did the students’ actual use of these textbooks did not increase? What happened to the government-provided textbooks? To answer this question, we again turn to directly observed data collected at endline. During unannounced classroom visits, data were collected on the number of textbooks per student in class. We find that, at endline, textbooks per capita in classrooms were not significantly higher in treatment schools than in control schools (Table 11 Columns 1 and 2). During data collection, enumerators were instructed to request access from the head teacher to the place where textbooks were stored and to count the textbooks (of the 4 core subjects for grades 4 and 5) in storage. We find that in treatment schools the number of core textbooks stored per students present in school is significantly higher than in control schools (Table 11 Columns 4-6). Comparing books stored per capita at endline and baseline, we find that the program led to a strong increase in the number of books stored in treatment schools (Table 11 Columns 7-10). Therefore, a major reason for lack of student access to textbooks either in the classroom or at home could be that many books are stored by schools and not distributed to students. Presumably, this lack of access did not extend to teachers, given that the program positively impacted teacher behavior on a number of dimensions. 6 This suggests that teachers may have used a few books to facilitate teaching (hence the significantly higher rates of teaching in treatment schools) but without providing broad access to students. 6 These results suggest that it would be of interest to derive impacts of textbooks on student performance after controlling for storage. However, book storage is endogenous to other factors (e.g., school needs, teacher capacity) and the authors were unable to find a credible instrument for storage. 14 9. Why are textbooks stored and not used? If school administrators exhibit a high propensity to store inputs instead of providing them to students, input provision programs to schools are likely to have only limited effectiveness. Therefore, it becomes imperative to understand what prompts school administrators to save government provided inputs instead of using them. To investigate underlying factors that predict the propensity to store, we estimate a cross-sectional model in which book storage (per student present) is regressed on a set of school level controls. = + 1 + 2 + (5) where is the number of books stored per student present in school i at endline, is a dummy for whether school i expected to receive books in the academic year 2008-09, and includes the number of books stored at baseline as well as other school level controls that could influence propensity to store. These include the presence of the head teacher at the time of interview, remoteness of the school (distance from nearest paved road and from DEO’s office), parental pressure (whether parents complain to head teacher about lack of textbooks), whether the school charges fees, and whether the school received books from the government in the previous academic year.7 As noted above, among independent variables we include a proxy for whether the school administrators “expected” to receive books or not ( ). The proxy for expectations of receiving inputs from government is created using two head teacher questions at endline: • Expectations proxy 1: If the head-teacher answers yes to the question: “Do you know how many books the government was to allocate to your school for Grades 4 and 5?” • Expectations proxy 2: If the head-teacher is able to answer the question: “How many books was the government to allocate to your school for Grades 4 and 5?” 7 In our sample of 341 schools, 71 schools reported receiving textbooks from the Government in 2008 (the previous academic year). 15 The underlying assumption in treating these variables as proxies for expectations is that head teachers who claim to know whether books were allocated to their schools or, more stringently, who claim to know the number of books that were allocated – must have had some expectations of receiving textbooks from the government in the current academic year. This proxy is not perfect due to the possibility of rationalization: Schools that actually received books may be more likely to claim that they expected to receive books. Results are shown in Table 12. Columns 1 and 3 show results with expectations proxy 1 for ITT and TOT, and Columns 2 and 4 show results with expectations proxy 2. Book storage is significantly higher in treatment schools that did not expect to receive books. This behavior makes sense if we assume that current expectations are positively correlated with future expectations. Head teachers who have low current expectations (and hence low future expectations) of government transfers are more likely to store a significant portion of what they see as a transitory infusion of inputs, so that they can smooth “consumption” in future periods. In this context, consumption is defined as “use of textbooks by students.” The level effect of expecting books then represents the impact for comparison schools. The positive, significant coefficients are consistent with updating of expectations: Schools that expected books but did not receive them reacted by increasing their book storage. Although the direction of these – updating of expectation effects for schools that expected books but did not receive them – is consistent for the TOT estimates and their magnitude is greater, only the ITT estimates are significant. This interpretation calls for the following assumptions, all of which are plausible in this context: (a) the head teacher’s (or class teacher’s) performance is measured by the average performance of students in her school (or class); (b) the head teacher’s and class teacher’s objective function is to maximize their performance over their tenure period; (c) the tenure period extends for more than one year; and (d) head teachers (or class teachers) believe increased student access to textbooks leads to better student performance. In this scenario, low expectations and high uncertainty mean that current transfers are seen as a transitory positive shock by head teachers or teachers with a multi-year planning horizon or objective function. 8 To maximize their performance over a multi- 8 This further assumes that current transfers are not sufficient to lead to an updating of head-teacher or class teacher expectations from “no transfers” to “likely transfers every year.” 16 year period it is therefore optimal for school administrators to store part of the current transfers of textbooks to smooth future use. There are other interpretations of this result. For instance, 1 could simply be capturing unobserved components of head-teacher quality that are correlated with the propensity to store books. It is possible, for example, that head teachers who store books are also those head teachers with less access to information on future transfers from government, potentially because of a lack of information channels. To this point, Table 12 highlights the role of road accessibility in influencing the propensity of input storage in schools. More remote schools are indeed significantly more likely to store books. But even after controlling for road accessibility, the impacts of head teacher expectations remain significant. Similarly, it is possible that head teachers who store books are also those who have worse information regarding the benefits of distributing textbooks to students, potentially as a result not of limited information but because of poor motivation. However, it is not clear that less competent or motivated head teachers would be more likely to store books. Distributing books to students is not a time or labor intensive exercise, and storing books requires space and at least some level of organization, especially when at least a few of the books are being distributed. Note that we do not observe complete storage: “books in class” rises positively but insignificantly with treatment (Table 11). Note also that our results are robust to controlling for at least one – admittedly imperfect – proxy for head teacher motivation, the likelihood of finding the head teacher in school at the time of the unannounced data collection visit. Furthermore, we observe that for the comparison group, greater expectations actually led to higher storage (i.e., unfulfilled expectations led to storage), which is inconsistent with expectations being a simple indicator of head teacher quality. There are of course other omitted variables that could be correlated with head-teacher expectations. In the absence of reliable time-series data on past transfers of textbooks from government to schools, it is difficult to make strong claims regarding expectations and storage behavior on the part of school administrators. That said, the interpretation of rationing or “consumption smoothing” is consistent with the available data. The “consumption smoothing” hypothesis, with few books distributed to classrooms, is particularly striking in conjunction with the observed positive impacts of treatment on teacher behavior. As seen above, the treatment – provision of textbooks to schools – leads to a significant increase in the likelihood of teachers having lesson plans (Table 5) 17 and the likelihood of teachers being in class teaching (Table 7). This behavior is consistent with teachers in treated schools receiving textbooks. Because the number of teachers in a school is much lower than the number of students, distributing books to teachers will still leave enough books in storage for future “smoothing.” Distributing a few books to teachers but not to students may well be rational behavior on the part of head teachers. Teachers have more organizing and bargaining power within schools, which increases their likelihood of gaining access to “rationed” resources. Also, head-teachers might believe that within a smoothing framework, distributing books to teachers is the most effective way to maximize current learning outcomes of students because a teacher is likely to use information from the textbooks to impact student learning. 10. Expectations of schools with regard to government transfers Why should expectations of school administrators regarding current and future transfers of textbooks from government be low? One of the challenges that societies face as they move from conflict and fragility towards development is that trust in the state has to be re-built. Lant Pritchett (2009) argues: “In seeking to build legitimacy the state will be under pressure to create confidence in the state, in part by delivering on identifiable ‘quick wins’ but also at the same time to build and transform institutions that are capable of delivering on development.” While a government transfer program for textbooks can be seen as a “quick win,” the system that is expected to deliver these transfers to schools is still far from perfect. There are inefficiencies and uncertainties in the functioning of government systems in Sierra Leone as evidenced by the fact that in 2009, 17 percent of the head teachers and 36 percent of classroom teachers reported not receiving their full pay in the last year. In addition, relating specifically to the textbook provision program, process data from the impact evaluation reveals that none of the DEOs or Local Councils had a completely clear picture of who was responsible for book pick-up and distribution, official signatories, or monitoring of textbook delivery. Likewise with regard to the actual disbursement of books, knowledge was spotty and written records were rarely found. Furthermore, only in 20 percent of the cases (out of 325) were textbooks ever delivered directly to the school (in other cases, presumably, textbooks are delivered to a central location in the district, like the DEO office) and in as many as 25 percent of the cases the 18 head teacher claimed that she had to pay for textbook retrieval from her personal resources with no expectation of reimbursement by the government. These factors indicate that school administrators might view transfers from government as a one-time positive shock, with little expectation of further replenishment in the short-run. This would explain some of the observed storing behavior. 11. Conclusions and Policy Implications There is considerable preoccupation in the development community with the disappointing impacts of input provision programs on beneficiary outcomes. This paper provides one possible explanation for this phenomenon. We show that public provision of textbooks to schools failed to have any impacts on student outcomes because a large share of the publicly provided textbooks were stored by school administrators instead of being distributed to students. At the same time, improved teacher access to textbooks had significant positive impacts on teacher behavior. One pathway for this might be through enhanced teacher motivation. It has been argued that having a textbook makes teaching easier. It can allow for better planning and measurement of student progress, and it can provide ready-made content for copying to the blackboard and reading aloud. The link between teachers’ access to textbooks and their classroom behavior has important policy implications: Ensuring that teachers have materials is important for their being willing and able to teach. However, the lack of test score results suggests that this is insufficient, by itself, to translate into student learning. We find that school administrators who did not expect to receive books were more likely to store them. Based on this result, we hypothesize that administrators are forward looking and that in times of high uncertainty they store part of current transfers to smooth future textbook consumption. Therefore, in line with the Permanent Income Hypothesis, when current transfers are seen as transitory they will have only limited impact on current consumption of textbooks in schools. Clearly, the behavior of intermediate agents, like school administrators, is likely to exercise a strong mediating influence on the impacts of public programs providing inputs on ultimate beneficiaries. However, the types of intermediate agent behavior that are usually linked to program failure include issues like organizational constraints, problems of human error, and program capture for private gain. In this paper we provide empirical evidence of intermediate agent behavior that is rational and not 19 necessarily driven by private gain, but that nonetheless subverts intended program objectives. These results have clear policy implications. Public (or non-government organization) provision of inputs that depreciate quickly when used, like textbooks, will be fully consumed only if agents have expectations of replenishment. For public programs that are envisaged as recurring transfers, there is a need to establish a reputation of consistent delivery and to reliably communicate the timing of the next transfer to intermediate agents, with the aim of optimizing the use of current transfers. As in many other empirical studies, we see here that mere input provision does not guarantee results without adequate attention to actual input use. For the latter, it is the incentives and expectations of intermediate agents and ultimate beneficiaries that can make all the difference. 20 Works Cited Arellano, Cristina, Aleš Bulíř, Timothy Lane, and Leslie Lipschitz. 2009. "The dynamic implications of foreign aid and its variability." Journal of Development Economics 88, no. 1: 87-102. Begg, Colin B. 2000. "Ruminations on the intent-to-treat principle." Controlled Clinical Trials 21, no. 3: 241-243. Coker, Femi. 2011. “Over 5000 Ghost Teachers Unearthed In Government Pay Vouchers” Sierra Leone News Hunters. October 11th. Das, Jishnu, Stefan Dercon, James Habyarimana, Pramila Krishnan, Karthik Muralidharan, and Venkatesh Sundararaman. 2011. “School inputs, household substitution, and test scores.” National Bureau of Economic Research, Working Paper no. 16830. DIVA-GIS. 2014. Sierra Leone administrative areas data. http://www.diva-gis.org/gdata (Accessed August 8th, 2014). Glewwe, Paul W., Eric A. Hanushek, Sarah D. Humpage, and Renato Ravina. 2011. “School resources and educational outcomes in developing countries: a review of the literature from 1990 to 2010.” National Bureau of Economic Research, Working Paper no. 17554. Glewwe, Paul, Michael Kremer, and Sylvie Moulin. 2009. "Many children left behind? Textbooks and test scores in Kenya." American Economic Journal: Applied Economics 1, no. 1: 112-135. Fofana, Lansana. 2008. “Sierra Leone: Ghost schools, phantom progress on education.” Inter Press Service News Agency. http://www.ipsnews.net/2008/09/sierra- leone-ghost-schools-phantom-progress-on-education/ (accessed June 22nd, 2014). Kodama, Masahiro. 2012. "Aid unpredictability and economic growth." World Development 40, no. 2: 266-272. Kremer, Michael, Conner Brannen, and Rachel Glennerster. 2013."The challenge of education and learning in the developing world." Science 340, no. 6130: 297-300. Krishnaratne, Shari, Howard White, and Ella Carpenter. 2013. “Quality education for all children? What works in education in developing countries.” New Delhi: International Initiative for Impact Evaluation (3ie), Working Paper no. 20. 21 McEwan, Patrick J. 2014. "Improving Learning in Primary Schools of Developing Countries: A Meta-Analysis of Randomized Experiments." Wellesley College Working Paper. Minakawa, Noboru, Gabriel O. Dida, Gorge O. Sonye, Kyoko Futami, and Satoshi Kaneko. 2008. "Unforeseen misuses of bed nets in fishing villages along Lake Victoria." Malaria Journal 7, no. 165.10: 1186. Pritchett, Lant. 2009. “Basic Systems for Service Delivery in South Sudan” Unpublished Manuscript. World Bank. 2007. Education in Sierra Leone: Present Challenges, Future Opportunities. Washington DC. World Bank. 2014. Education Profile for Sierra Leone. Washington DC. 22 Table 1: Distribution of Schools Sampled and Surveyed Western Western District Kailahun Kambia Pujehun Total Area Rural Area Urban Originally selected for IE 90 90 90 18 72 360 Baseline data collected 87 88 82 18 66 341 Endline data collected 87 88 82 18 66 341 Table 2: Comparing Baseline Characteristics of Treatment and Control Groups Assignment P Value Significant Variable Control Treatment difference difference Number of classrooms 5.4474 5.2389 0.5686 No (3.2125) (3.1206) Number of teachers 7.9558 8.7956 0.2477 No (5.7808) (7.207) Number of students 321.9381 341.1875 0.3710 No (183.3082) (192.2288) Student teacher ratio 46.9755 46.9421 0.9896 No (21.0945) (23.7679) Student classroom ratio 68.0851 82.9568 0.0259 Yes (39.515) (82.265) Number of working toilets 3.1053 3.1422 0.9164 No (3.1239) (2.9379) Number of separate working toilets for girls 1.0965 1.0357 0.6820 No (1.3824) (1.0833) Electricity 0.0654 0.0829 0.5701 No (.2484) (.2764) Access to tap water 0.1696 0.2318 0.1737 No (.377) (.423) Distance to nearest motorable road 0.354 0.5315 0.4691 No (1.3557) (3.1156) Distance to DEO office 19.4867 17.7318 0.3072 No (14.4285) (15.5984) Parents pay fees 0.2857 0.2805 0.9215 No (.4538) (.4503) School received fee subsidy in first term 0.2679 0.2884 0.6944 No (.4448) (.4541) School received textbooks this academic year 0.3333 0.2687 0.2474 No (.4737) (.4444) Grade 3 math score 0 -0.0661 0.5755 No (1) (1.0416) Grade 4 math score 0 -0.164 0.1592 No (1) (1.0132) Grade 3 English score 0 -0.1442 0.2169 No (1) (1.0097) Grade 4 English score 0 0.0081 0.9434 No (1) (.9728) Observations 115 226 341 Note: Standard deviations in parentheses. Math and English scores are normalized on the baseline distribution of control schools in a given grade-subject. Significance refers to statistical significance at the 90% level. 23 Table 3: Compliance to Randomized Design Actual Treatment Control Treatment Total % Treated Assignment Control 62 53 115 46% Treatment 34 192 226 85% Difference in Actual Treatment 39% Note: "Actual Treatment" is defined as the school receiving books, regardless of the source. Table 4: Impact of Receiving Textbooks on Student Enrollment & Parental Complaints (1) (2) (3) (4) Number of Number of Parents complain Parents complain students students about lack of about lack of Variables enrolled enrolled textbooks textbooks Panel A: Intention to treat ITT -6.130 -8.320 -0.099* -0.104* (15.752) (15.869) (0.057) (0.057) Baseline enrollment 0.609*** 0.566*** (0.056) (0.071) R-squared 0.389 0.404 0.010 0.025 Estimation method OLS OLS OLS OLS Panel B: Treatment on the treated TOT -15.349 -20.700 -0.253* -0.266* (39.417) (39.289) (0.151) (0.150) Baseline enrollment 0.609*** 0.565*** (0.056) (0.071) R-squared 0.387 0.401 Estimation method 2SLS 2SLS 2SLS 2SLS Control group mean 329.1 329.1 0.423 0.423 Observations~ 325 325 333 333 District fixed effects~ No Yes No Yes Note: Baseline variables other than student enrollment were not gathered and so are not controlled for. Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. *** p<0.01, ** p<0.05, * p<0.1. 24 Table 5: Impact of Receiving Textbooks on Teaching Practices (1) (2) (3) (4) (5) (6) (7) (8) Teacher assigns Teacher assigns Student found Student found Teacher Teacher Teacher has Teacher has homework from homework from mistreating mistreating showed lesson showed lesson created a created a Variables textbook textbook book book plan plan pamphlet pamphlet Panel A: Intention to treat ITT 0.027 0.025 0.010 0.009 0.105** 0.098* 0.064 0.072 (0.025) (0.024) (0.057) (0.046) (0.052) (0.050) (0.052) (0.048) Teacher showed lesson 0.130*** 0.120** plan at baseline (0.044) (0.046) R-squared 0.003 0.029 0.000 0.180 0.027 0.068 0.004 0.080 Estimation method OLS OLS OLS OLS OLS OLS OLS OLS Panel B: Treatment on the treated TOT 0.071 0.066 0.027 0.023 0.297** 0.279** 0.171 0.193 (0.065) (0.063) (0.150) (0.123) (0.138) (0.135) (0.142) (0.127) Teacher showed lesson 0.126*** 0.113** plan at baseline (0.044) (0.045) R-squared 0.021 0.003 0.179 0.024 0.066 0.084 Estimation method 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS Control group mean 0.929 0.929 0.381 0.381 0.525 0.525 0.270 0.270 Observations~ 782 782 788 788 643 643 787 787 Number of schools~ 325 325 328 328 299 299 326 326 District fixed effects~ No Yes No Yes No Yes No Yes Note: Baseline variables other than teacher showing lesson plan at baseline were not gathered and so are not controlled for. Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. *** p<0.01, ** p<0.05, * p<0.1. 25 Table 6: Impact of Receiving Textbooks on Student Attendance (1) (2) (3) (4) (5) (6) Grade 4 Girls Grade 4 Boys Grade 4 Total Grade 5 Girls Grade 5 Boys Grade 5 Total Variables Attendance Attendance Attendance Attendance Attendance Attendance Panel A: Intention to treat ITT 0.010 -0.019 0.004 0.057* -0.005 0.020 (0.030) (0.027) (0.027) (0.031) (0.032) (0.030) R-squared 0.222 0.151 0.221 0.213 0.176 0.232 Estimation method OLS OLS OLS OLS OLS OLS Panel B: Treatment on the treated TOT 0.025 -0.048 0.010 0.140* -0.011 0.051 (0.073) (0.071) (0.067) (0.076) (0.079) (0.075) R-squared 0.230 0.136 0.223 0.213 0.174 0.234 Estimation method 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS Control group mean 0.671 0.715 0.690 0.660 0.672 0.663 Observations~ 303 301 299 239 236 231 Controls for corresponding Yes Yes Yes Yes Yes Yes baseline attendance~ District fixed effects~ Yes Yes Yes Yes Yes Yes Note: Controls for corresponding (grade-gender) baseline attendance are included throughout but not reported. All coefficients on baseline attendance are positive and significant at the 1% level. Even though we see one significant result for attendance, the coefficients are jointly insignificant. Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. *** p<0.01, ** p<0.05, * p<0.1. 26 Table 7: Impact of Receiving Textbooks on Teacher Activity in Class (1) (2) Teacher found Teacher found Variables in class teaching Panel A: Intention to treat ITT 0.039 0.119** (0.035) (0.049) Teacher found in class at baseline 0.020 (0.043) Teacher found teaching at baseline -0.006 (0.045) R-squared 0.063 0.068 Estimation method OLS OLS Panel B: Treatment on the treated TOT 0.112 0.339** (0.102) (0.147) Teacher found in class at baseline 0.019 (0.042) Teacher found teaching at baseline -0.019 (0.049) R-squared 0.052 0.013 Estimation method 2SLS 2SLS Control group mean 0.815 0.571 Observations~ 656 656 Number of schools~ 292 292 District fixed effects~ Yes Yes Note: Controls for corresponding teacher activity at baseline are included throughout. Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. *** p<0.01, ** p<0.05, * p<0.1. 27 Table 8: Impact of Receiving Textbooks on Student Performance, using individual endline data only (1) (2) (3) (4) (5) (6) (7) (8) Grade 5 Grade 5 Grade 6 Grade 6 Grade 5 Grade 5 Grade 6 Grade 6 Variables Math Math Math Math English English English English Panel A: Intention to treat ITT -0.063 -0.085 0.003 -0.009 -0.104 -0.116 -0.006 -0.010 (0.086) (0.082) (0.092) (0.088) (0.089) (0.088) (0.092) (0.090) Baseline school-cohort-subject mean 0.015** 0.013** 0.007 0.006 0.002 -0.002 0.013 0.010 (0.006) (0.006) (0.007) (0.007) (0.009) (0.010) (0.009) (0.009) R-squared 0.012 0.048 0.002 0.026 0.003 0.014 0.004 0.019 Estimation method OLS OLS OLS OLS OLS OLS OLS OLS Panel B: Treatment on the treated TOT -0.144 -0.194 0.006 -0.019 -0.234 -0.263 -0.013 -0.022 (0.197) (0.187) (0.199) (0.195) (0.204) (0.204) (0.199) (0.200) Baseline school-cohort-subject mean 0.015** 0.012** 0.007 0.006 0.003 -0.001 0.013 0.010 (0.006) (0.006) (0.007) (0.007) (0.009) (0.009) (0.009) (0.009) R-squared 0.006 0.041 0.002 0.026 0.003 0.019 Estimation method 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS Observations~ 7,746 7,746 7,028 7,028 7,746 7,746 7,028 7,028 Number of schools~ 295 295 276 276 295 295 276 276 District fixed effects~ No Yes No Yes No Yes No Yes Note: Test scores are normalized on the endline distribution of control schools, and controls are included for baseline school-cohort-subject mean. The regressions were also conducted without these controls, giving similar results. Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. *** p<0.01, ** p<0.05, * p<0.1. 28 Table 9: Impact of Receiving Textbooks on Textbook and Other Material Use by Students (1) (2) (3) (4) (5) (6) (7) (8) Use textbook Have a Use book No. of Received Use Use Use for one core new given by students book pamphlet book for pamphlet notes for Variables subject book teacher is shared with from teacher exam for exam exam Panel A: Intention to treat ITT 0.022 0.039 0.020 -0.332 -0.045 0.024 -0.025 0.002 (0.045) (0.032) (0.047) (0.307) (0.034) (0.042) (0.048) (0.051) R-squared 0.083 0.061 0.077 0.040 0.054 0.064 0.066 0.035 Estimation method OLS OLS OLS OLS OLS OLS OLS OLS Panel B: Treatment on the treated TOT 0.056 0.095 0.049 -0.939 -0.111 0.058 -0.063 0.006 (0.109) (0.078) (0.114) (0.860) (0.086) (0.102) (0.120) (0.125) R-squared 0.091 0.069 0.088 0.041 0.028 0.068 0.056 0.034 Estimation method 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS Control group mean 0.508 0.163 0.412 2.214 0.164 0.430 0.265 0.590 Observations~ 3,150 3,193 3,193 1,685 3,193 3,131 3,130 3,142 Number of schools~ 332 337 337 255 337 332 332 332 District fixed effects~ Yes Yes Yes Yes Yes Yes Yes Yes Note: Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. *** p<0.01, ** p<0.05, * p<0.1. 29 Table 10: Impact of Receiving Textbooks on Teacher Behavior with Respect to Student Textbook Use (1) (2) (3) (4) (5) (6) (7) (8) (9) Teacher Teacher made Student allowed Teacher Teacher made Teacher Teacher publicly Homework Homework encouraged book student read to take any core encouraged use students work used the rewarded student for assigned at least included questions Variables use during class out from book textbook home of excercise book in groups blackboard good performance 3 times a week from textbooks Panel A: Intention to treat ITT 0.056* 0.046 -0.013 -0.002 0.009 0.002 0.031 0.024 -0.015 (0.034) (0.041) (0.035) (0.007) (0.034) (0.004) (0.041) (0.029) (0.028) R-squared 0.040 0.037 0.089 0.002 0.116 0.002 0.085 0.044 0.021 Estimation method OLS OLS OLS OLS OLS OLS OLS OLS OLS Panel B: Treatment on the treated TOT 0.140* 0.113 -0.030 -0.005 0.023 0.006 0.076 0.059 -0.036 (0.082) (0.099) (0.085) (0.018) (0.083) (0.009) (0.100) (0.072) (0.068) R-squared 0.044 0.038 0.086 0.003 0.121 0.001 0.087 0.045 0.018 Estimation method 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS Control group mean 0.767 0.668 0.240 0.985 0.810 0.992 0.606 0.852 0.196 Observations~ 3,150 3,151 2,732 3,149 3,153 3,150 3,148 3,155 3,106 Number of schools~ 332 332 311 332 332 332 332 332 330 District fixed effects~ Yes Yes Yes Yes Yes Yes Yes Yes Yes Note: Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. *** p<0.01, ** p<0.05, * p<0.1. 30 Table 11: Impact of Receiving Textbooks on Books in Class and Books Stored (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Books in class per capita^ Books stored per capita^ Change in books stored Missing values as Missing values as Missing values as Missing values as missing zeros missing zeros Variables Grade 4 Grade 5 Grade 4 Grade 5 Grade 4 Grade 5 Grade 4 Grade 5 Grade 4 Grade 5 Panel A: Intention to treat ITT 0.205 0.126 1.057** 1.477*** 0.966** 1.342*** 0.784* 2.449*** 0.731* 2.286*** (0.180) (0.183) (0.484) (0.445) (0.433) (0.388) (0.454) (0.910) (0.409) (0.817) R-squared 0.187 0.183 0.098 0.118 0.088 0.096 0.079 0.073 0.082 0.066 Estimation method OLS OLS OLS OLS OLS OLS OLS OLS OLS OLS Panel B: Treatment on the treated TOT 0.503 0.299 2.565** 3.582*** 2.489** 3.459*** 2.026* 6.643** 2.038* 6.120*** (0.431) (0.428) (1.187) (1.127) (1.122) (1.041) (1.209) (2.646) (1.176) (2.319) R-squared 0.211 0.192 0.082 0.065 0.084 0.051 0.023 0.012 0.029 0.011 Estimation method 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS 2SLS Control group mean 0.995 1.068 2.704 2.137 2.328 1.728 1.190 0.739 0.908 0.469 Observations~ 296 281 299 286 341 341 240 185 271 209 District fixed effects~ Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Note: Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ~ refers to information that is the same for both ITT and TOT. ^ refers to per capita students present in school. *** p<0.01, ** p<0.05, * p<0.1. 31 Table 12: Predicting the Propensity to Store Books (1) (2) (3) (4) Panel A: Intention to treat Panel B: Treatment on the treated ITT TOT Books stored per Books stored per Books stored per Books stored per Variables capita^ capita^ capita^ capita^ Expected books (proxy 1) 2.425*** 8.953 (0.850) (8.756) Expected books (proxy 2) 2.615*** 11.970 (0.831) (11.169) Treatment school 1.150** 1.159** 2.893** 2.907** (0.465) (0.464) (1.204) (1.270) Treatment*Expected books (proxy 1) -1.930* -10.026 (1.009) (10.195) Treatment*Expected books (proxy 2) -2.010** -13.370 (1.013) (13.081) Headteacher in school -0.548 -0.665 -0.667 -0.741 (0.451) (0.448) (0.510) (0.537) Distance to motorable road 0.462*** 0.463*** 0.485*** 0.496*** (0.083) (0.083) (0.091) (0.097) Distance to DEO office -0.018 -0.014 -0.010 -0.004 (0.015) (0.015) (0.018) (0.019) Parents complain about lack of textbooks -0.428 -0.425 -0.578 -0.565 (0.413) (0.411) (0.446) (0.474) Parents pay fees -0.618 -0.683 -0.535 -0.574 (0.436) (0.435) (0.552) (0.586) Books received from government last year -0.561 -0.508 0.132 0.431 (0.578) (0.567) (0.938) (1.104) Books stored at baseline 0.297*** 0.308*** 0.260*** 0.271*** (0.070) (0.069) (0.081) (0.085) Control group mean 1.838 1.838 1.838 1.838 Observations 214 220 214 220 R-squared 0.329 0.330 0.155 0.037 Estimation method OLS OLS 2SLS 2SLS District fixed effects Yes Yes Yes Yes Note: The expected books proxies are dummy variables for: if the head-teacher answers yes to the question “Do you know how many books the government was to allocate to your school for Grades 4 and 5?” (proxy 1), and if the head-teacher is able to answer the question“How many books was the government to allocate to your school for Grades 4 and 5?” (proxy 2). Control for books stored at baseline and include district fixed effects throughout. Standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. ^ refers to per capita students present in school. *** p<0.01, ** p<0.05, * p<0.1. 32 Table A1: Attrition at Endline (1) (2) Variables Attrition Attrition Assignment to treatment -0.037 (0.054) Actual treatment 0.016 (0.057) Number of classrooms -0.032*** -0.032*** (0.007) (0.007) Electricity 0.233** 0.231** (0.102) (0.103) Distance to DEO office -0.001 -0.000 (0.002) (0.002) Number of students -0.000 -0.000 (0.000) (0.000) Dummies for missing data Number of classrooms 0.393 0.440 (0.376) (0.363) Electricity -0.078 -0.083 (0.091) (0.090) Distance to DEO office -0.227** -0.233** (0.110) (0.112) Number of students 0.295 0.298 (0.361) (0.356) Observations 341 341 R-squared 0.083 0.082 Estimation method OLS OLS Note: Robust standard errors in parentheses. ITT refers to intention to treat; TOT refers to treatment on the treated. OLS refers to Ordinary Least Squares; 2SLS refers to Two Stage Least Squares. *** p<0.01, ** p<0.05, * p<0.1. 33 Figure 1: Map of Districts that Participated in the Impact Evaluation Koinadugu Bombali Kambia Port Loko Kono Western Urban Tonkolili Moyamba Kailahun Western Rural Bo Kenema Bonthe Pujehun Participating district Non-participating district Source: DIVA-GIS (2014) 34 Figure 2a: Student Performance in Mathematics by Grade 8/2 22x8 5x5 Sample math questions 86-14 Grade 4 9-4 Grade 3 6+7+8 25+63 3+6 0 20 40 60 80 100 Percentage of students that answered question correctly Figure 2b: Student Performance in Literacy by Grade add: abb, add, adb, dad Sample literacy questions bird: dirb, bird, brib, drib breathe: breethe, dreathe, breathe, breahte Grade 4 Mammy gave Musa a gift: job, Grade 3 brush, banana, present She saw a tiny bird: little, tired, sick, poor The thieves ran away: chickens, boys, robbers, rabbits 0 20 40 60 80 100 Percentage of students that answered question correctly Note: In the first three literacy questions, children were asked to identify which of the words in the list is identical to the first word presented. In the last three literacy questions, children were asked to identify the word in the list which is a synonym for the word underlined in the sentence. 35