ABCD 69681 The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees PSIA 2008 Survey Final Report March 2009 This report contains 28 pages The World Bank-PSIA Survey Report Final reviewed © 2009 KPMG is a Swiss cooperative of which all KPMG firms are members. KPMG International provides no services to clients. Each member firm is a separate and independent legal entity and each describes itself as such (e.g KPMG Auditores e Consultores, SARL, a Mozambique limited liability company, is a member of KPMG International). All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Contents List of Tables ii Executive Summary iii Acknowledgement v 1 Introduction 1 1.1 Context 1 1.2 General Objectives 1 1.3 Specific Objectives 2 2 Research Design and Methodology 3 2.1 Research Design 3 2.2 Research Methodology 3 2.3 Sampling Size 4 2.4 Data Collection Methods and Instruments 5 2.4.1 Data Collection Methods 5 2.4.2 Questionnaire Structure, Length and Content 5 2.4.3 Questionnaire Development and Changes 7 2.4.4 Database Development and Changes 7 2.4.5 Training materials 7 3 Fieldwork Targets and Outcomes 9 3.1 Field Staff Structure and Composition 9 3.2 Training 10 3.3 The Pilot Exercise 11 3.3.1 The Need for Pilot II 11 3.3.2 Critical Issues 11 3.4 The National Phase 12 3.4.1 Sample Performance 12 3.4.2 Field Work Performance 15 4 Lessons Learned and Challenges 21 The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 i © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 List of Tables Table 1: Overall sample of EAs and Households within each province ..................... 4 Table 2: PSIA 2008 Overall Attrition Rate and by Province, ..................................................... 13 Table 3: Response Rate and Key Reasons for Non-response, PSIA 2008.................................. 16 Table 4: Other Reasons for no Interview .................................................................................... 17 Table 5: Summary Display of Field level indicators for PSIA 2008 .......................................... 17 Table 6: Field performance indicators for PSIA 2008 ................................................................ 18 The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 ii © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Executive Summary The PSIA 2008 survey general objective is to analyze the barriers to school enrolment and retention and particularly the effect that fees and other schooling costs paid at the household level, particularly households with the poorest and most vulnerable children, including girls and orphans. Of interest, is the impact that such costs have on: (a) household decisions with respect to child enrolment and dropout /repetition/completion in primary and secondary school education; (b) school level decisions on budget and expenditures, and (c) perceptions of school quality and educational outcomes at the school and community levels. The PSIA 2008 was a complex exercise, not only on coverage, but particularly on architecture content. The questionnaire was subtle and complex and underwent many changes from its initial formulation. The manual is very long and detailed. The sample was drawn from a data set not meant for a panel study and this introduced a number of complexities and challenges that influenced heavily its implementation in the field. In assessing performance of the PSIA 2008 survey implementation it is shown that the overall attrition rate is 25%. At the provincial level, the highest attrition rates were found in Maputo Province (46%), Sofala (38%), Manica (35%), Maputo City (34%), Manica (30%) and Niassa (24%). The lowest attrition rates were found in Zambézia and Inhambane (12%, each) provinces. In general, it has been very difficult working in urban areas, particularly in Maputo and Sofala, due to mobility, but also incomplete addresses references from the INE database. In some sites it was necessary to replace non-located households when no single one was found after series of attempts at getting hold of the households. Thus the teams visited nationwide a total of 2,234 households. Of those visited only 1,954 households were eligible. Among the eligible households, the teams were able to interview 1,478 households. Note that among those not interviewed, the attrition rate only counts eligible households that were not found after tracking efforts have been exhausted with local leaders, and other key informants. These households could neither be located nor was relevant information for locating them found. In sum, the overall Completion Rate1 is 100%. The success2 and the interview3 rates are 78% and 75% respectively. The response rate for the whole PSIA 2008 sample is 73.6%. Among the non-responses, about 60% the main reason was that the households could not be identified either by the local authorities or any other key informant in the site; 11% could not be located in the enumeration area either because the address did not help to find them or the local authority/key informant could not find them in their area of jurisdiction or had no specific information on where they were at the time of interview. Among the 16.3% of households not interviewed for other reasons were those that moved outside the district – most of which moved for work related reasons (mainly transfers), and; among the death were mostly where either both spouses passed away 1 The completion rate is the ratio of the total number of households visited (confirmed as non-eligible; households interviewed and did not generate tracks; households interviewed and generated tracks, and those not interviewed for a number of reasons) over the total number of households expected for interview. 2 The success rate is the ratio of the number of households actually interviewed over those eligible (both original and generated tracks). 3 The interview rate is the ratio of total number of households interviewed over the total number of sampled households. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 iii © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 and the remaining members moved outside the district or the household member that made the household eligible passed away. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 iv © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Acknowledgement PSIA 2008 survey is a consortium led effort between KPMG Mozambique and Manitou Inc. of the USA. KPMG and Manitou, Inc. formed an alliance in order to assist the World Bank with the collection and preparation of a database with the intent to estimate the impact of lowering the costs of primary education and baseline analysis of the poverty and social impact of general secondary schools fees. We recognize the strategic importance of this project to the World Bank and to the Government of Mozambique. Thus, the strategic alliance formed between KPMG and Manitou, Inc is a blend of complementary skills and resources that are collectively provided to the World Bank with a leveraged solution to the expressed concern of developing a sound Policy for Education Sector and Economic Growth in Mozambique. Under Mozambique´s current circumstances and characteristics we have made all the effort to provide an experienced team led by our best people. During all stages of the project implementation we have benefited from World Bank staff inputs, both from Mozambique and Washington DC. We are thus grateful for all the discussions around questionnaire content and design, training, piloting and monitoring of the actual implementation. In that regard we would like to thank Rui Benfica, Melissa Wells and Louise Fox for their endless patience. KPMG top management would like to extend thanks to its technical team that include Paulo Mole, Binit Varajidas, Marcelino Botao, Momed Jamu, Ricardo Adamo, and all other internal staff, enumerators, field supervisors and provincial coordinators that provided their inputs during project implementation. From Manitou Inc. thanks extend to Tristi Nichols and Joseph Cadwell for their inputs during survey instrument and database design processes. KPMG subcontractors also contributed to the project, and we would like to extend gratitude to Chris Hill, Luis Seródio, Olivia Sigauque, Raul Pitoro, and Fausto Maquina, for the provincial coordination of the fieldwork. We were pleased that the World Bank had selected us on the basis of required skills and capacity to deliver a high quality service that met the required standards. We look forward to being your continuous strategic business partner that offers you the highest level of service and accessibility. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 v © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. 1 Introduction 1.1 Context On behalf of the Ministry of Education and Culture (MEC) of Mozambique, in 2004 the World Bank and a number of donors commissioned a team of consultants to analyze schools costs for the primary education in Mozambique. The study is known as the Poverty and Social Impact Analysis (PSIA) including both quantitative and qualitative analyses. This study led to policy recommendations to the Ministries of Education and Planning and Finance. One such policy decision was the September 2004 Ministerial Diploma that abolished school fees such as the ASE (social action fee) in all public primary schools (from 1st to 7th grade), effective as of school year 2005. Time has elapsed since the decision to drop out the school fees from households´ education budget lines was made. Thus in the course of 2007, MEC with the support of the World Bank and the group of donors involved in the education sector decided to implement a follow up study to the PSIA of 2004 with the aim of estimating the impact of lowering further the costs of primary education as well as to undertaking a baseline study of the cost impact of school fees on the general secondary education. It is in that context that the World Bank and the group of donors commissioned KPMG to implement in 2008 the survey to gather the necessary data for the analysis. This report is a general documentation of the PSIA 2008 survey process, the main survey objectives, approach and methodology, the data collection methods and instruments, the fieldwork targets and outcomes, as well as lessons learned and challenges. 1.2 General Objectives The PSIA 2008 general objective is to analyze the barriers to school enrolment and retention and particularly the effect that fees and other schooling costs paid at the household level, particularly households with the poorest and most vulnerable children, in particular girls and orphans. Of interest, is the impact that such costs have on: (a) household decisions with respect to child enrolment and dropout /repetition/completion in primary and secondary school education; (b) school level decisions on budget and expenditures, and (c) perceptions of school quality and educational outcomes at the school and community levels. The above objective would be met by analyzing the gathered data at two education levels: primary and secondary levels. In primary education PSIA would analyze the role of recent reforms on achieving PARPA education goals, namely: (a) increasing net schooling rate at EP1+2 school level; (b) increasing the first grade girls enrollment level; (c) decreasing dropout rates and repetition levels; (d) increasing graduation rates, and (e) decreasing student-teacher ratios. In the secondary education level, the objective is to collect baseline data to inform policy makers on measures that could support the achievement of the PEEC secondary education goals, ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 particularly with respect to education fees. At the level of the school and the community, the objective is to collect initial data on (a) school budgets and the role of fees and (b) perceptions of barriers to the achievement of the strategic goals in the PEEC. 1.3 Specific Objectives The specific objective of the PSIA 2008 survey was, as commissioned by the World Bank, for KPMG to design, and pilot test, review the survey instruments and fully implement the survey nationwide, in order to produce a high quality data set for its panel analysis. The survey would cover a sample of approximately 2,000 households that were part of the IAF 2002/03 from the Instituto Nacional de Estatística (INE). KPMG should match school data to household’s data by identifying which schools the children in the household attended. On the basis of the MEC´s school database, a household would be linked to a specific school, and provide a data set that would include school characteristics for each of the households. Note that the initial draft of the questionnaire was provided by The World Bank, the basis from which KPMG adapted the instrument to Mozambique’s conditions. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 2 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 2 Research Design and Methodology The following sections present the research design and methodology followed throughout in collecting, recording and archiving the PSIA 2008 data into a database for future analysis. The raw data in CSPRO, as well as conversions to SPSS and STATA are also provided to the World Bank. 2.1 Research Design Further to the general objective, data gathered under the PSIA survey result from one common sample of Households that provide a substantial sample for children attending primary schools, especially in rural areas, and a limited sample of secondary school students. For purposes of measuring the impact of lowering the costs of primary education, the World Bank decided to use a panel survey design on households drawn from the National Household Budget Survey (Inquérito aos Agregados Familiares sobre o Orçamento Familiar 2002/03, IAF) conducted by the Instituto Nacional de Estatística (INE). Data collected in 2007/8 would then be compared with same household data from the IAF 2002/3 survey. Furthermore, it is important to note that the World Bank has used the approach for the qualitative part, PSIA 2004, and therefore would also like compared it with that from the PSIA 2008, in order to picture the resulting differences about relationships among (1) household expenditures, (2) educational enrollment/retention, and (3) education quality. As the issues of interest are which variables inhibit and support the achievement of the PEEC education goals, together with the likelihood that benefits of decisions taken over the education sector take time to be observed, a longitudinal design is appropriate for examining how households respond to subtle impacts over time. 2.2 Research Methodology As indicated above, PSIA 2008 is follows a panel research design. As such, all members of the original household (as interviewed in IAF 2002/2003) were re-interview together with any new members. Survey instruments were designed in a way that an important task during its implementation required the enumerators to first identify positively all household members interviewed in 2002/2003, before the current household members were all listed. Furthermore, all individuals meeting the eligibility criteria in the household in 2002/3, but were no longer resident in the household in 2008 were followed/tracked for interview, if they were still resident in the district they have been interviewed in before. In addition to household survey data, PSIA 2008 survey used a database from the Ministry of Education and Culture for the identification of schools attended by household members. Where The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 3 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 the MEC database did not have information on a specific school, survey teams went to the schools and collected the necessary data. 2.3 Sampling Size The sample size for the PSIA 2008 was set to 2,230 selected from the 8,700 households interviewed in 2002/3 for the National Household Budget Survey, (IAF 2002/03). INE sources indicated that IAF´s sample coverage was 99.7% of which 91.8% were included in the original sample and 7.6% were replacement households. The 2002/3 sample was stratified by urban and rural areas in each province, with approximately equal samples taken from each stratum. The primary sampling units (PSUs) were enumeration areas from the 1997 census. In each primary sampling area a fixed sample of 15 households was taken in urban areas and 12 in rural areas. In all, there were approximately 660 primary sample units. The households for PSIA 2008 were selected on the basis of the following criterion: a household interviewed in 2002/3 should have children aged 17 years or younger in 2002/2003. Households meeting the criteria should amount to between 80% and 90% of all households in the sample. This meant that an average of approximately 12 urban and 10 rural households per primary sampling unit included children aged 17 years or younger in 2002/3. These parameters set limits on the distribution of the PSIA 2008 sample. At one extreme the sample could include all the 660 PSUs with approximately 3 households each. At the other extreme the sample could cover about 220 PSUs or less with an average of 9 households each. Table 1 shows the overall sample size per province, and Enumeration Areas (EAs) both Urban and Rural, as well as the number of households in each of the selected EAs. Table 1: Overall sample of EAs and Households within each province Urban Rural Total Provinces Enumeration Enumeration Enumeration Areas Households in Areas Households Areas Households in the sample the Sample in the sample in the Sample in the sample in the Sample Niassa 11 132 10 90 21 222 Cabo Delgado 2 24 17 152 19 176 Nampula 4 48 17 152 21 200 Zambézia 3 36 17 152 20 188 Tete 6 72 15 135 21 207 Manica 11 132 10 90 21 222 Sofala 10 119 10 90 20 209 Inhambane 1 12 20 179 21 191 Gaza 7 84 11 99 18 183 Maputo Província 10 120 8 72 18 192 Maputo Cidade 21 244 0 0 21 244 Moçambique 86 1,023 135 1,211 221 2,234 The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 4 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Note that in addition to the original target, eligible households members were to be tracked as long as they have moved out from the original households, and remained within the geographical limits of the districts where they have been interviewed in 2002/2003. 2.4 Data Collection Methods and Instruments This section provides an overview of the data collection methods and instruments developed and used for PSIA 2008 survey. 2.4.1 Data Collection Methods PSIA 2008 was carried out using the traditional pen-and-paper interviewing at the household level through a formal questionnaire. Collected data was computer entered, initially in the field and later re-entered centrally for quality consistency check. The survey used GPSs for the geo-location of households and schools. Data entry was undertaken with resort to CSPro. CSPro is a software system specifically designed for survey data entry developed by the US Bureau of the Census for use in developing countries. CSPro has been used over a number of years both by INE and for the Agriculture Surveys (TIA) in the Ministry of Agriculture. The TIA surveys used CSPro for direct data entry in the field. Members of our team have worked with this application. 2.4.2 Questionnaire Structure, Length and Content The PSIA 2008 questionnaire is complex. It covers a diversity of topics within one interview. Some sections involve collection of large amount of detail. Other sections attempt to collect with precision information that is not precise or varies greatly in structure from person to person or household to household. The majority of the questions are at the household member level with pre-coded responses. However, there are sections with questions referring to occupation by household member, goods, expenditure categories, food consumption, and expenditures on semi- and non-durable goods per household. The questionnaire consists of two main parts: (i) Pre-filled information from the 2002/2003 IAF survey; and (ii) sections to be filled during the 2008 interviews. The two parts make a distinction between an original household to that required for a member to be tracked. Overall, the questionnaire contains 12 sections. Below is a brief description of the content of each of the questionnaire’s sections: Section A – The Questionnaire Cover, which consist of two components: (i) pre-filled component with information from 2003; and (ii) a component to be filled by the enumerator. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 5 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 • The pre-filled component identifies the household, and include geographical information, the name of the household head, the home address where available, and the household eligibility criteria – that is, if the household had either children 17 years old or younger, or people studying full time during the year 2003; and • The component to be filled by the enumerator with 2008 geographical information (if different from that of 2003), as well as the 2008 household eligibility criteria, household location, confirmation of whether the household was or not interviewed in 2008. Furthermore, the enumerator should indicate if the household has moved out of the district, the current location, the name of the enumerator filling the questionnaire, the data and time of interview and the geographical coordinates of the household house. Section B – This section shows the household composition in 2003 and 2008. It includes basic bio-data for each of the household member. The section helps in identifying positively the household members from 2003 whose information is required in other sections. Section C – Questions included in this section explore the maximum education level attained by each of the household members, and register the reasons members may have not go to school, transport means used to attend school [Sub-section C0]; education profile for the 1998-2008 time period [Sub-section C1]; and education expenses, both in cash and in kind for the last school year [Sub-section C2]. Section D – This section determines whether the household members that 5 years or older are involved in any economic and other activities. It includes four different employment activities: (i) domestic activities – economic and others [Sub-section D0 Part B], (ii) wage employment [Sub-section D1], (iii) self-employment [Sub-section D2], and (iv) agricultural activities – agricultural and livestock [Sub-sections D0 Part C and D3]. Section E – This included a single page with questions related to housing characteristics and transport means used to access services. Sections F and G – These sections contain questions related to household consumption. There are five parts: (i) durable assets ownership [Sub-section F], (ii) household food, beverage and tobacco for the last 7 days [Sub-section GA], (iii) verification of food purchases for the past 30 days [Sub-section GB], (iv) more frequent non-durable goods and services purchased in the last 30 days [Sub-section GC], and (v) Durables, Semi-Durables and services paid in the 12 months [Sub-section GD]. Section H – This section refer-se to respondent’s perspectives and perceptions about the education system, the quality of the education provided, and the level of involvement of the civil society in school decision making. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 6 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Section I – The questions in this section record transfer received and sent in the last 12 months, its frequency and sources. It registers transfers from both public and private sources. Section J – This is the last section of the questionnaire. It contains only one question on the household poverty status. It asks if the household holds any Poverty Certificate. 2.4.3 Questionnaire Development and Changes The time required to perfect survey instruments to get to an agreeable form and state was huge. The observations made above on changes made during training on both pilots are an example of the complexity of the PSIA questionnaire. This had an obvious impact on the process of developing complementary instruments for the training and testing. Every change in the questionnaire introduced other changes including: The possibility of needing to make other changes within the questionnaire; The retraining of field staff, in particular field staff needed to “unlearn� what they have been previously practiced; The manual needed substantial revisions; and The data entry system needed substantial revisions. The training stressed the importance of asking the questions as given in the questionnaire. This required discipline on the part of the interviewers, but also on the part of the survey designer. Questions need to be explicit, but not over elaborated. 2.4.4 Database Development and Changes During the life of this project there was some inconsistency in the requirement as to whether the data entry should take place in the field or in the regions. The initial proposal was that the data entry should take place in the field. At some stage during negotiations this was modified to doing data entry in the provinces rather than actually with the field team. At the end the plan reverted to doing the data entry in the field. The quality of the final data is clearly improved by data entry in the field. This practice allows the interviewer to return to households and make corrections immediately. It also by giving immediate feed back improves the overall standard of work of the interviewers. It of course can only pick up errors of incompleteness and inconsistency that fail edits. In practice the most common errors are failure by the interviewer to complete the many detailed items. Many of these items are in fact items with a “no� response. Ensuring completeness in these cases ensures that the occasional “yes� response is not missed. 2.4.5 Training materials PSIA 2008 training materials included two manuals: (1) A general instructions manual containing mostly guidelines for enumerators, supervisors, and coordinators duties and The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 7 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 responsibilities, (2) a questionnaire manual with detailed instructions and filled questionnaire shots as examples on how to answer questions and fill responses. The main manual contained all the complementary forms such as the tracking and control forms, and the industry and activity code lists. In addition, a full module power pointer presentation was prepared to be used during the training for both pilots and national survey trainings. The latter also included a training agenda. The training power point presentation was prepared to be as detailed as possible allowing delivery of the training in an interactive form, with examples on how to fill the questionnaire. An additional manual was also developed for the training of the d-entry staff. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 8 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 3 Fieldwork Targets and Outcomes 3.1 Field Staff Structure and Composition The PSIA 2008 survey team had three layers of technical skills and responsibility levels. The figure below shows these different levels and the team structure and composition. KPMG Partner Project Manager Team Leader Field Work Data Entry Expert Expert Data entry Assistant Coordinator Coordinator Coordinator Coordinator North Centre A Centre B South 1 1 1 1 1 1 1 1 1 1 1 Sup Sup Sup Sup Sup Sup Sup Sup Sup Sup Sup NPL NSA CDL ZMB TET SFL MNC IBN GZA MPC MPP 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 3 1 4 1 3 DE Int DE Int DE Int DE Int DE Int DE Int DE Int DE Int DE Int DE Int DE Int Legend: Sup – Field Supervisor; DE – Data Entry person; Int – Interviewer; NPL – Nampula province; NSA – Niassa province; CDL – Cabo Delgado province; ZMB – Zambezia province; TET – Tete province; SFL – Sofala province; MNC – Manica province; IBN – Inhambane province; GZA – Gaza province; MPC – Maputo City; MPP – Maputo Province At the Central level, a team composed of a Senior Partner, and a senior manager took stock of the whole responsibility of the project. This team managed all the contacts with the World Bank team responsible to oversee the implementation of the project. At the technical level another team composed of a Team Leader, a survey fieldwork specialist4, a Database specialist and an assistant to data entry oversaw all the process of instrument design and development, database design and management and field support. The third layer is a team at the provincial field level, one for each province consisting of 1 supervisor, 3 enumerators and 1 field data entry person. 4 This specialist also was involved in the training sessions The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 9 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Each of these teams is led by a Coordinator who may be responsible to oversee one or more provincial teams. Coordinators reported directly to the Field Work Expert, who jointly with the Team Leader oversaw the overall technical implementation of the survey. 3.2 Training Training for PSIA 2008 occurred in three moments. First training was undertaken during before the first pilot in Xai-Xai city, province of Gaza. The second training took place in Maputo city before the second pilot. It became clear that neither the training for the first nor the second pilot training sessions fully met expectations. After the first pilot, there was a decision to allocate additional preparation time before the second training. As a result, there were more training material available and a much improved training manual. To improve further the quality of field work during the National phase a third training took place in Chimoio city, province of Manica in which sessions only team members from the Central and Northern region participated5. After the first join session (for the questionnaire, and for the d-entry), d-entry staff were separated from the other survey team members for a specific discussion. It is important to note that in all training sessions substantial discussions took place on the correctness of the content of the questionnaire and many changes were made to the questionnaire. Furthermore, all training sessions had practice periods of 1 to 2 days before the field work. In particular, the training during the National phase for each module presented the participants practiced interviews among themselves. After each module and practice feedback was given in plenary sessions. It was only after, that participants left for field practice over a sample of households in two sites in Chimoio and Gondola districts. At the end of the field practice there was a report back session of a day. Participants also mentioned that the quality of the training delivery was high. Both modules and presenter were well prepared and presented. However, there were modules such as education, employment, and consumption considerate as the core of the whole survey should have had more time due to their complexity. Furthermore, participants indicated that the practice introduced before the pilot field work was very helpful. The practice introduced after each of the more practical modules was crucial to the mastering of the instruments before fieldwork. With respect to training materials, participants indicated that earlier distribution of this material allowed better familiarization with them. The manual contained relevant information and necessary explanations to the content of the training. However, the questionnaire structure was considered very complex. The proposed sequence was the most complex of all the issues in the manual, and one with which participants had to coop with. 5 Only survey team members from the Southern region participated on both pilot exercises. Due to time span between the pilot exercises and the actual National phase, after the training in Chimoio, team members from the Southern region had a two days refreshing session on the survey instruments before they started the fieldwork. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 10 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Overall, it seems that the absorption capacity was high, though participants had shown different levels of survey exposure and experience. Participants indicated that due to the questionnaire complexity and required rigor, the training was very intensive and therefore more time should have been allocated to each session. This is even so, if one takes into account changes that had to be made in the instruments during the training. These circumstances have affected enumerators’ ability to immediately complete interviews with minor errors. However, later on, productivity and quality increased considerably. 3.3 The Pilot Exercise PSIA 2008 included a period of piloting the survey instruments before moving towards the implementation of the survey at the national level. The pilot was conducted in two stages, and later complemented with a field practice before the national phase take off. The first stage occurred in Gaza province during the period of June 10th-18th, 2008; whereas the second pilot occurred in Maputo City and province during the period of July 15th-26th. The field practice during the training of for the national phase occurred August 29th-30th . All the pilot exercises were preceded by training and in house practices. 3.3.1 The Need for Pilot II After the first pilot exercise, there was a need for a second pilot. The decision was based on observations made on the outcomes of the first training and its impact on the field pre-testing. Although, it was felt that the first pilot was not a complete failure, critical issues were identified in the questionnaire and important lessons drawn. The issues were both of logistical and questionnaire content nature6. The solutions to these issues were important for the training at the national level. 3.3.2 Critical Issues For the majority of the time before the agreement on the final set of survey instruments, a number of critical factors had an impact on the survey implementation process. These included an underestimation and thorough understanding of the complexities of the survey instruments, the changes introduced in these instruments, the flexibility of the trainers to deliver within this context, and few logistical issues. There are important lessons to be drawn from those factors, namely: due to the complexity of the survey, the development of the training material, a necessary stage, has shown to be time consuming. In the process, changes in the instruments particularly during training have been a major constraint for the whole piloting exercise. 6 See Relatório da Fase de Pilotagem for further details. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 11 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 These changes had influenced the process of developing the training material, namely the final questionnaire and related forms (tracking and control forms). The lessons learned from this process was that constant changes in the questionnaire have, among others, two major implications: • Changes made during the training phase do not allow for a complete absorption of the training material. It is always the case that trainees get confused at the end of training because they did not have sufficient time to get acquainted with the newly developed instrument; and • In terms of logistics, changes in the questionnaire require changes in other instruments such as the manual and forms (tracking and control). These additional instruments are also required during the training. Furthermore, late changes require additional effort to get instruments ready before field work starts. These changes quite often meant decisions to retrain teams, which increased implementation costs. 3.4 The National Phase 3.4.1 Sample Performance In the sections that follow, a summary indicator is presented for the PSIA 2008 overall sample and by province. The summary indicator used in assessing sample performance is the overall attrition rate. It is defined as the ratio of the number of households not interviewed plus the number of tracks not interviewed over the sum of the number of eligible households and the number of tracks recorded/generated. That is, the proportion of households that could not be interviewed for different reasons among those those were eligible for interview. Critical issues influencing such ratio are also presented. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 12 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 3.4.1.1 Attrition rates Table 2 below, shows the overall attrition rate, together with those from each covered province in Mozambique. Table 2: PSIA 2008 Overall Attrition Rate and by Province, Provinces Attrition Rates Niassa 25% Cabo Delgado 18% Nampula 16% Zambézia 12% Tete 20% Manica 35% Sofala 38% Inhambane 12% Gaza 13% Maputo Province 46% Maputo City 34% Mozambique 25% The PSIA 2008 overall attrition rate is 25%. At the provincial level, the highest attrition rates were found in Maputo Province (46%), Sofala (38%), Manica (35%), Maputo City (34%), Manica (30%) and Niassa (24%). The lowest attrition rates were found in Zambézia and Inhambane (12%, each). In general, it has been very difficult working in urban areas, particularly in Maputo and Sofala, due to mobility, but also incomplete addresses references from the INE database. In some sites it was necessary to replace non-located households when no single one was found after series of attempts at getting hold of the households. Under the field work performance section below, we present complementary indicators such as the completion, success rate, and the interview rates. 3.4.1.2 Critical Issues The main critical issues arise from the limitations of IAF 2002/2003 data as the basis for a panel survey. The IAF was not designed to be used as the basis for a panel survey. It also became apparent that the work of some of the IAF interviewers was of poor quality. These factors created many difficulties for the PSIA 2008 panel survey. The timing of the survey was also limited by the fact that it was taking place after the National Census and when a number of other projects were under way. Any further information obtained from the National Institute of Statistics (INE) did not help in clarifying the location of households. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 13 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 PSIA 2008 Sample As mentioned earlier, PSIA 2008 sample is a sub-sample of the IAF 2002/3. Three months were selected covering period of the year close to the time period planned for the survey. In fact, with the delays in the project the actual months of the 2008 survey were quite different from the 2003 survey. This change raises a question concerning the design of the 2008 sub-sample. The design of AIF was such that every province had a representative sample over the entire year with approximately half the households being from the urban areas and half from the rural areas. The AIF did not however design the spread of the survey over the year to be balanced. Some months may have had a much higher proportion of urban or rural household than other months. Although the overall National PSIA 2008 sample has a reasonable urban rural distribution, it does need to be looked at carefully. It is not representative at the provincial level and its overall coverage of the country may not be very representative. Pre-recorded Household Members Information There appeared to be a great deal of variation in the quality of work undertaken by the IAF interviewers. Where the work of the IAF interviewer had been done properly there was little problem in establishing the relationship between the 2008 household members and those in 2003. Where the work had not been well done there were many discrepancies. Among the most common problems found were: Very incomplete listing of household members, Ages of members total erroneous, and Errors in sex of members. In these situations it was sometimes difficult for the PSIA 2008 interviewer team to determine or identify positively whether or not the persons in 2003 were the same persons in 2008. Geographic Information IAF used names rather than geographic codes for the identification of geographic areas. There is no standardization in the spelling of geographic areas. Most areas with names that include an accent were spelt in two different ways; once with the accent and once without the accent. Most cities were also written in different ways. For example, in one case Matola City had been used in five different versions. The City of Maputo has five urban districts. Many enumeration areas in Maputo were very poorly identified. The district was not specified or was in error and was spelt in various different ways. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 14 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Address Information Generally, in cities and large towns IAF had a record of the address of each household and the name of the household head. In rural areas, the only information available was normally the name of the household head. Using the name of the head of household as the sole means of identification is not a problem in rural areas. Community leaders in rural areas normally know all households and could easily identify the households. In a few cases where the head of household had changed it was usually possible to find the household. In urban areas finding households was often much more difficult. In many cases IAF did not provide complete addresses and the local officials did not know the names of many household heads. Typical examples of addresses were “Flat 1� or “4th floor Right� without any indication of which building. In some cases, there was no information on which quarter within a “bairro� of the city was referred to. Considerable effort was needed to locate household in these situations. However, IAF cannot be criticized for this lack of information. The limitations of using the IAF as the basis of a panel survey were underestimated. In general, any intention to use a survey for a follow up survey should be incorporated into the design of the original survey. 3.4.2 Field Work Performance The sample profile is a critical quality of the sampling frame and the assumptions made to draw the sample. Fieldwork performance is heavily influenced by these factors. For PSIA 2008 the other elements for the field performance include: • The initial time frame was about 4 weeks. However, estimation based on the sample size and a reasonable enumerator’s productivity levels of 1.5 days in each of the EA, the time frame required would have been 5 weeks. Furthermore a number of holidays in between the training and actual fieldwork had an effect on the actual amount of time in the field due to involvement of local authorities in some of the political activities. • Interviewer performance within the above time frame; and • The willingness of households to participate in the survey. In effect, apart from the above the complexity of the questionnaire and the time to make contacts and tracking households resulted in important factors to the overall fieldwork performance. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 15 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 3.4.2.1 Survey Response Rates Overall, field level performance is summarized by a number of indicators. Among these are the completion rate, survey response rate and the interview rate. While the completion and interview rates report directly on team effort to visit and actually interview households, the response rate report the outcomes for sampled households regardless of when they were interviewed. Thus, the response rate for the whole PSIA 2008 sample is 73.6%. Table 3 that follows shows the different outcomes from the households included in the survey. Table 3: Response Rate and Key Reasons for Non-response for PSIA 2008 Valid Cumulative Frequency Percent Percent Percent Refusal by respondent 6 0.2 0.9 0.9 Not Available for Interview 10 0.4 1.6 2.5 Absent and never found home 49 2.0 7.6 10.1 Other non-response 578 23.7 89.9 100.0 Total 643 26.4 100.0 Complete Interview 1,792 73.6 Total 2,435 100.0 Among the non-response category, the distribution of reasons for non-interview is presented in Table 4. As shown above, about 60% of the households among those not interviewed the main reason was that the households could not identified either by the local authorities or any other key informant in the site; 11% could not be located in the enumeration area either because the address did not help to find them or the local authority/key informant could not find them in their area of jurisdiction or had any specific information on where they were at the time of interview. Among the 16.3% of households not interviewed for other reasons were those that moved outside the district – most of which moved for work reasons (transferred), and; the death were mostly where either both spouses passed away and the remaining members moved outside the district or the household member that made the household eligible passed away. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 16 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 Table 4: Other Reasons for no Interview Valid Cumulative Frequency Percent Percent Percent Difficult access to household location 1 0.2 0.2 0.2 Mentally ill 2 0.3 0.3 0.5 Death 35 6.1 6.1 6.6 Moved outside of Village 1 0.2 0.2 6.7 Moved outside District 94 16.3 16.3 23.0 Not Eligible 33 5.7 5.7 28.7 Not identified 348 60.2 60.2 88.9 Not Located 64 11.1 11.1 100.0 Total 578 100.0 100.0 3.4.2.2 Success Interviews and Fieldwork targets PSIA 2008 sample expected coverage of 221 EAs with a total number of households for interview of 2,234. Tables 5 and 6 that follow show the different outcomes from the survey. Table 5: Summary Display of Field level indicators for PSIA 2008 Indicators Total A Total number of Sampled Households 2,234 B Total number of Eligible Households 1,954 C Total number of Non-Eligible Households 280 D Households Interviewed without Tracks 1,268 E Households Interviewed with Tracks 210 F Households not Interviewed 476 Household Tracking G Number of Tracks Generated 283 H Number of Tracks followed and Interviewed 204 I Number of Tracks not Interviewed 79 As Table 5 above shows, the teams visited nationwide a total of 2,234 households. Of those visited only 1,954 households were eligible. Among the eligible households, the teams were The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 17 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 able to interview 1,478 households. Note that among those not interviewed, the attrition rate only counts eligible households that were not found after tracking efforts have been exhausted with local leaders, and other key informants. These households could neither be located nor was relevant information for locating them found. Table 6: Field Performance Indicators for PSIA 2008 Completion Success Interview Province Rate Rate Rate Niassa 100% 76% 75% Cabo Delgado 100% 85% 82% Nampula 100% 85% 85% Zambézia 100% 87% 88% Tete 100% 81% 80% Manica 100% 73% 65% Sofala 100% 66% 62% Inhambane 100% 92% 88% Gaza 100% 90% 88% Maputo Província 100% 65% 54% Maputo Cidade 100% 73% 66% Moçambique 100% 78% 75% Table 5 above presents summary indicators for field performance by province. The table shows that the overall Completion Rate is 100%. That is, the total number of households visited (confirmed as non-eligible; households interviewed and did not generate tracks; households interviewed and generated tracks, and those not interviewed for a number of reasons) over the total number of households expected to be interviewed. The success and the interview rates are 78% and 75% respectively. The success rate is the ratio of the number of households actually interviewed over those eligible (both original and generated tracks). The interview rate is the ratio of total number of households interviewed over the total number of sampled households. 3.4.2.3 Critical Issues Tracking of households One of the key elements in the PSIA 2008 was the requirement that households should be tracked if they had moved within the same district. During the survey implementation this requirement was refined to take account of the fact that in many cases households divide with one or more members moving out to join other households7 or to form their own households. In fact tracking of households is much more often tracking of some household members rather than tracking the entire household. During the pilot phases of the project a great deal of effort went into rigorously defining under what conditions households should or should not be tracked. A set of detailed rules were 7 Some of which were eligible in the same enumeration area. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 18 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 establish to define clearly when a household should be tracked or not tracked. The rules for tracking were discussed at great length and went through a number of changes and refinements. An important note is that five years is a long period over which to track persons, particularly persons in the upper end of the eligible age range near to the limit of 17. A substantial proportion of adolescents will have moved out of their original household over this time period. In fact, the original perception of split households as being due to divorce and separation was revised as it is clear that most separations are in fact due to the evolution of families with youngsters moving out. Tracking has proved to be a significant part of the work of the teams. In some situations the first stage tracking resulted in a second stage when eligible members where found to have moved to a household different from the household head. Field data entry The implementation of data entry in the field raised two types of problems: Logistic problems concerning the charging and management of computers in the field; and The delivery of a correct final version of the software application. The many and late changes in the questionnaire created a major challenge for the specialist designing the data entry system. When the field work started there were still some corrections to be made to the application. The delivery of these corrections was a major challenge. Most provinces in Mozambique have very few internet cafes where the data entry person could download changes. Also some of the data entry personnel had problems loading the revised application. Due to the above logistical considerations, the d-entry in the field did not occur at the expected pace, which resulted in second entry at the central level. To ensure completeness, after the second d-entry a list of errors was generated leading to a revisit period in which survey team were deployed to the field with the following tasks: (a) re-interview households whose questionnaires had missing information; (b) revisit enumeration areas, seeking to find non- located/identified households, and (c) interview households in replacement areas. The new data was later computer entered at the central level8. Time in the field As mentioned earlier, the reduced time span spent by the teams in the field did not allow completion of d-entry in the field. An effort was made, in both training and fieldwork a given enumerator should complete a minimum number of questionnaires per day, as well as the minimum d-entered questionnaires per day. 8 Find progress report for this period in the Annexes. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 19 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 However, experience from the field had shown that while enumerators could reach, on average, the minimum number of questionnaire per day, review by supervisors could not keep pace because of daily advanced contacts with local authorities to locate households for the following day9. As a result, d-entry staff could not meet their minimum requirements. 9 The following was the strategy set for time spent in a given EA. Enumerators were instructed to start in a given urban EA (1), and as each interview is completed and reviewed by the supervisor the questionnaire would be given to the d-entry person. He/she would start the d-entry process while the enumerator went for another household within the EA (1). All the interviews should have been done by the end of the day. The d-entry for EA (1) would be completed throughout the night, if necessary. If this was the case, errors would be generated and given to the supervisor and enumerators for discussion in the following morning and decision to go back to households made. For questionnaires that would require returning to households, corresponding enumerators would do so while others moved to the following EA (2). The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 20 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 4 Lessons Learned and Challenges PSIA 2008 was a complex exercise, not only on coverage, but particularly on architecture content. The questionnaire was subtle and complex and underwent many changes from its initial formulation. These factors have a major impact on all aspects of the survey including the delivery of the training program, the production of field manual and the data entry system. We learned that the training of interviewers should take place in a stable environment. Discussion on changes to the design of the questionnaire should be minimized or avoided during training. In fact there were extensive discussion about changes during the first pilot training and these discussions continued to a lesser extent during the second pilot training. The manual is very long and detailed. From an initial very incomplete document it has been transformed into a very complete document. Experience with other surveys indicates that in practice interviewers make very limited use of their manual. It should be noted however that during the training frequent reference was made to the manual and the interviewers were often directed to make reference to it. The constant changes in the questionnaire did in fact create a great deal of work for those responsible for drafting the manual. The fact that PSIA 2008 sample was drawn from a data set not meant for a panel study introduced a number of complexities and challenges that influenced heavily its implementation in the field. Just to mention that the decision to replace households in areas where not a single household could be identified or located is one of the clear sign of the burden teams had to bear resulting from lack of complete information from IAF 2002/03. There are other lessons as well. For instance, teams learned that (i) in urban areas, weekend contacts are not useful. In rural areas such contacts may result fruitful thought they may not be extended to Sundays; and (ii) relying on administrative authorities for communication with lower level authorities is not useful. Direct contacts with community authorities are better, though district level authorities should not be ignored. Thus, survey stakeholders should be prepared for: • Allowing more time for contact with local authorities before field work starts. A reasonable time for such work to take place should at least be done a week before field work start, probably during the national training phase. The first contact with authorities should take place during working days. Contacts must be done down to much lower levels of authority. If possible households must be met to set interviews; and • More effort should be put into explaining households the purpose of the survey to attract attention and interest, which may take extra time. Another challenge is the d-entry in the field. The ability to d-entry data in the field is mainly constrained by three factors, namely (a) availability of the alternative sources of energy to keep The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 21 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved. ABCD The World Bank Estimating the Impact of Lowering the Cost of Primary Education and Baseline Analysis of the Poverty and Social Impact of General Secondary School Fees March 2009 computers operating all day long, (b) the time the first filled questionnaire come into d-entry staff hands for entry, and (c) the time field work team remain in the same EA. Thus a careful assessment of these factors should be undertaken, and preparation done for it. The Final World Bank-PSIA 2008 Survey Report - 9 April 2009 22 © 2009 KPMG Auditores e Consultores, SARL. All rights reserved.