Document of The World Bank Report No: ICR00003637 IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF-98729) ON A GRANT IN THE AMOUNT OF US$19.2 MILLION TO THE INDEPENDENT STATE OF PAPUA NEW GUINEA FOR A READING EDUCATION (READ-PNG) PROJECT June 27, 2016 Education Global Practice East Asia Pacific Region       CURRENCY EQUIVALENTS (Exchange Rate Effective May 31, 2016) Currency Unit = PNG Kina 1.00 = US$0.37 US$1.00 = Kina 2.7 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CDAD Curriculum Development and Assessment Division CDD Curriculum Development Division CLRG Classroom Library Reference Group CPS Country Partnership Strategy DA Designated Account DL Disbursement Letter DoE Department of Education EGRA Early Grade Reading Assessment EFA-FTI Education for All-Fast Track Initiative EDPACC Education Development Partners Advisory Coordination Committee EMIS Education Management Information System ENBP East New Britain Province FM Financial Management FODE Flexible and Open Distance Education Project GoPNG Government of Papua New Guinea PNGEI Papua New Guinea Education Institute GPE Global Partnership for Education GST Goods and Services Tax ICB International Competitive Bidding IFR Interim Financial Reports ICR Implementation Completion and Results Report ISM Implementation Support Mission LEG Local Education Group MDG Millennium Development Goals M&E Monitoring and Evaluation MYOB Mind Your Own Business NCB National Competitive Bidding NCD National Capital District NDoE National Department of Education NEP National Education Plan ii      NEP2 National Education Plan for 2005-2014 ‘Achieving a Better Future’ NLS National Library Service PAD Project Appraisal Document PDO Project Development Objective PNG Papua New Guinea PSC Project Steering Committee READ-PNG Reading Education Project-Papua New Guinea SBC Standards Based Curriculum UBE Universal Basic Education Plan (2010-2019) UNICEF United Nations Children’s Education Fund VSO Voluntary Service Overseas WB World Bank Senior Global Practice Director: Claudia Maria Costin Practice Manager: Harry Anthony Patrinos Project Team Leader: Binh Thanh Vu   ICR Team Leader: Binh Thanh Vu iii      PAPUA NEW GUINEA READ PNG CONTENTS   A. Basic Information ........................................................................................................ v  B. Key Dates ...................................................................................................................... v  C. Ratings Summary ........................................................................................................ v  D. Sector and Theme Codes ............................................................................................ vi  E. Bank Staff .................................................................................................................... vi  F. Results Framework Analysis...................................................................................... vi  G. Ratings of Project Performance in ISRs ................................................................... x  H. Restructuring (if any) ................................................................................................. xi  I. Disbursement Profile .................................................................................................. xi  1.  Project Context, Development Objectives, and Design ........................................... 1  2.  Key Factors Affecting Implementation and Outcomes ............................................ 7  3.  Assessment of Outcomes ....................................................................................... 14  4.  Assessment of Risk to Development Outcome ...................................................... 21  5.  Assessment of Bank and Borrower Performance ................................................... 21  6.  Lessons Learned..................................................................................................... 23  7.  Comments on Issues Raised by Grantee/Implementing Agencies/Donors ............. 24  Annex 1. Project Costs and Financing .............................................................................. 25  Annex 2. Outputs by Component ...................................................................................... 26  Annex 3. Economic and Financial Analysis...................................................................... 32  Annex 4. Grant Preparation and Implementation Support/Supervision Processes .......... 36  Annex 5. Beneficiary Survey Results................................................................................ 38  Annex 6. Stakeholder Workshop Report and Results........................................................ 39  Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR ........................... 40  Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ............................ 49  Annex 9. List of Supporting Documents ........................................................................... 50  MAP.................................................................................................................................. 51    iv      A. Basic Information Country: Papua New Guinea Project Name: READ PNG Project ID: P105897 L/C/TF Number(s): TF-98729 ICR Date: 06/27/2016 ICR Type: Core ICR The Independent State Lending Instrument: Sector Investment Loan Grantee: of Papua New Guinea Original Total USD 19.20M Disbursed Amount: USD 19.20M Commitment: Revised Amount: USD 19.20M Environmental Category: C Implementing Agencies: Department of Education Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 05/27/2010 Effectiveness: 11/31/2011 11/31/2011 03/21/2014 Appraisal: Restructuring(s): 05/28/2015 Approval: 03/03/2011 Mid-term Review: 09/16/2013 11/04/2013 Closing: 06/30/2014 12/31/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Substantial Bank Performance: Moderately Satisfactory Grantee Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: v      C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry Yes None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Primary education 79 79 Public administration-Education 21 21 Theme Code (as % of total Bank financing) Education for all 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Victoria Kwakwa James W. Adams Country Director: Mona Sur Ferid Belhaj Practice Harry Anthony Patrinos Eduardo Velez Bustillo Manager/Manager: Project Team Leader: Binh Thanh Vu James A. Stevens ICR Team Leader: Binh Than Vu ICR Primary Author: Sandra Beemer, Kris McDonall F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The principal development objective of the proposed READ-PNG project is to improve the reading skills of elementary and primary education students. Revised Project Development Objectives (as approved by original approving authority) The revised project development objective is to promote better teaching and learning of reading skills in elementary and primary education. vi      (a) PDO Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Proportion of primary school teachers using supplementary readers during Indicator 1 : reading instruction as a sample of 1,800 classrooms with classroom libraries, as identified by classroom observation checklist Value 100% 62% quantitative or 0 N/A 1,800 1,940 qualitative) Date achieved 3/21/2014 12/31/2015 12/31/2015 Survey sample exceeded the target. Observations of primary teachers showed 62 Comments percent using supplementary readers. Although the target was missed by 38 percent, (incl. % conducting the survey was a substantial achievement. Indicator revised at the time achievement) of the March 2014 restructuring. Source: Survey Data Improvement in students’ average scores in three EGRA subtests (phonemic Indicator 2 : awareness, letter-sound identification, and familiar word reading) in pilot provinces (by gender) Madang Province Madang Province Madang Province Baseline 2011 Target Endline 2013 Phonemic awareness Not Applicable for Phonemic awareness Boys: EGRA results Boys: G3=53 G3=85 G4=59 G4=89 Girls Girls: G3=46 G3=85 G4=56 G4=89 Letter-Sound ID Letter-Sound ID Boys: Boys: G3=28 G3=73 G4=36 G4=88 Value Girls Girls: quantitative or G3=26 G3=82 qualitative) G4=30 G4=86 Familiar Word Familiar Word Boys: Boys: G3=31 G3=34 G4=45 G4=53 Girls Girls: G3=30 G3=41 G4=47 G4=57 Western Highlands Western Highlands Province Province Baseline 2013 Endline 2014 Phonemic awareness Phonemic awareness Boys: Boys: vii      G3=43 G3=80 Girls: Girls: G3=43 G3=82 Letter-Sound ID Letter-Sound ID Boys: Boys: G3=24 G3=49 Girls Girls: G3=21 G3=57 Familiar Word Familiar Word Boys: Boys: G3=17 G3=32 Girls Girls: G3=17 G3=39 Date achieved 2/28/2011 12/31/2015 12/31/2015 Comments There was significant improvement in the scores for boys and girls in all subtests. (incl. % Annex 2 has full details. New at the time of the March 2014 restructuring. achievement) Students in elementary and primary school will demonstrate improvements in Indicator 3 : their literacy skills a defined by early grade reading assessments Value quantitative or qualitative) Date achieved Comments Dropped/restated as PDO indicator 1 above at the time of the March 2014 (incl. % restructuring. achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Number of students in elementary and primary schools with access to Indicator 1 : learning and reading materials in their classrooms, provided by the project Value E=0 E=375,000 E=827,517 (quantitative P=0 P=720,000 P=856,510 or qualitative) Date achieved 02/28/2012 12/31/2015 12/31/2015 Comments Target exceeded. Number of students in elementary increased by approximately (incl. % 121 percent and primary students by approximately 19 percent. Source: achievement) NDoE/CDD Number of teachers in elementary and primary schools who have Indicator 2 : participated in professional development activities provided by the project Value E=0 E=4,500 E=0 (quantitative P=0 P=25,000 P=25,120 or qualitative) viii      Date achieved 03/25/2014 12/31/2015 12/31/2015 Comments Partially achieved. Primary teacher training target was achieved. However, it was (incl. % agreed that elementary teachers would receive user-friendly training materials achievement) with learning kits which were delivered to 22,254 teachers. Source: NDoE/CDD At least two post-assessment reading intervention pilots designed and Indicator 3 : completed, including their evaluation reports for dissemination Value (quantitative 0 2 2 or qualitative) Date achieved 04/05/2013 12/31/2015 12/31/2015 Comments Target met. Madang baseline 10/2011 and endline 2013 and final report 12/2015. (incl. % Western Highlands baseline 5/2103 and endline 2014 and final report 12/2015. achievement) Indicator modified during the 2014 restructuring. Source: NDoE/CDD Classroom libraries established in approximately 18,000 grade 3-8 classrooms Indicator 4 : (each classroom library containing 60 titles of supplementary readers) Value (quantitative 0 18,000 21,711 or qualitative) Date achieved 02/01/2012 12/31/2015 12/31/2015 Comments (incl. % Target exceeded by approximately 20.6 percent. Source NDoE/CDD achievement) Learning material kits and teacher guidelines on the use of kits in Indicator 5 : approximately 15,000 elementary classrooms (grades EP-E2) Value (quantitative 0 15,000 22,254 or qualitative) Date achieved 02/01/2012 12/31/2015 12/31/2015 Comments Target was exceeded by 48.4 percent. Added at the time of the March 2014 (incl. % restructuring. Source: NDoE/CDD achievement) Indicator 6 : Vernacular reading materials in 2,500 elementary classrooms (EP-E2) Value (quantitative 0 2,500 15,027 or qualitative) Date achieved 02/01/2011 12/31/2015 12/31/2015 Comments (incl. % Target was exceeded by 501 percent. Source: NDoE/CDD achievement) Two teams of coaches and 22 teams of provincial library trainers trained by Indicator 7 : the project to facilitate classroom library training to teachers at the district level in each province. Value 2 teams coaches 2 teams coaches (quantitative 0 22 prov. teams 22 prov. teams or qualitative) Date achieved 03/25/2014 12/31/2015 12/31/2015 Comments Target met. Two provincial teams were formed in 22 provinces, and (incl. % approximately 1,334 individuals were trained to train teachers in classroom library achievement) use and management. Source: NDoE/CDD ix      Indicator 8 : At least one national public reading event during the project Value (quantitative 0 1 1 or qualitative) Date achieved 03/25/2014 12/31/2015 12/31/2015 Comments Target met. There was a national book week event in 2014. In addition, there was (incl. % a public launch of the EGRA reports in 2015 and the READ-PNG website was achievement) launched. Source: NDoE/CDD Up to four EGRA provincial surveys and results reports completed from Indicator 9 : baseline assessments in selected provinces. Value (quantitative 0 4 4 or qualitative) Date achieved 02/01/2011 12/31/2015 12/31/2015 Comments Target met. Madang, Western Highlands, National Capital District, and East New (incl. % Britain participated in EGRA baseline assessments with published reports. Source: achievement) NDoE/CDD Up to four events to disseminate results and findings from provincial EGRA Indicator 10 : surveys to NDoE, provincial officers and other stakeholders Value 4 (quantitative 0 4 or qualitative) Date achieved 03/25/2014 12/31/2015 12/31/2015 Comments Target met. Madang (2012), Western Highlands (2014), National Capital District (incl. % (2013), and East New Britain (2014) participated in dissemination workshops. achievement) Added at the time of the March 2014 restructuring. Classroom teaching practices reflect enhanced knowledge in reading Indicator 11 : instruction and use of learning materials Value (quantitative or qualitative) Date achieved 06/05/2012 7/31/2015 7/31/2015 Comments (incl. % Dropped during the 2014 restructuring achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 05/23/2011 Satisfactory Satisfactory 0.00 2 01/24/2012 Moderately Satisfactory Moderately Satisfactory 0.93 3 07/18/2012 Moderately Satisfactory Moderately Satisfactory 1.10 4 11/04/2012 Moderately Satisfactory Moderately Satisfactory 1.10 5 01/15/2013 Moderately Satisfactory Moderately Satisfactory 1.47 6 06/18/2013 Moderately Satisfactory Satisfactory 2.19 7 12/31/2013 Moderately Satisfactory Moderately Satisfactory 9.01 8 06/29/2014 Moderately Satisfactory Moderately Satisfactory 11.25 x      9 12/11/2014 Moderately Unsatisfactory Moderately Unsatisfactory 13.07 10 02/22/2015 Moderately Unsatisfactory Moderately Satisfactory 13.97 11 07/24/2015 Moderately Satisfactory Moderately Satisfactory 16.96 12 12/29/2015 Moderately Satisfactory Moderately Satisfactory 19.04 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions PDO simplification to better reflect intended outcome of project; indicators revised to be Yes better aligned with the modified 03/21/2014 MS MS 9.87 GPE Board PDO. Reduction in EGRA provinces from 15 to 4. Closing date extended to allow for completion of project activities. Extension of closing date. 05/28/2015 No MU MS 14.78 Reallocation of funding across components. I. Disbursement Profile xi        1. Project Context, Development Objectives, and Design 1.1 Context at Appraisal 1. Country Context. Papua New Guinea (PNG) is an ethnically and socially diverse nation of 7 million people speaking over 840 languages and is spread across some of the world’s most difficult terrain. PNG was, and continues to be, rich in natural resources, but subject to extreme weather events and natural disasters with many remote and difficult to reach areas. An estimated 88 percent of the population lives in rural areas, often isolated with poor communications. The agriculture and oil/minerals sectors dominated the economy which led to markedly improved economic fortunes since the mid-2000s. The global commodity price boom and stronger macroeconomic management helped the country avoid the declining real incomes and macroeconomic volatility that the country had experienced in the past. The global downturn of 2009 had only a mild impact on PNG’s economy, which expanded by 5.5 percent, outperforming its regional peers and most other developing economies. This followed several years of 6 to 7 percent annual Gross Domestic Product (GDP) growth. While sharply higher global commodity prices strongly supported incomes (average export prices doubled between 2005 and 2008, and were still about 50 percent higher in mid-2012), the non-minerals sector led the resurgence, benefiting from private investment in future minerals production capacity as well as in the newly liberalized telecommunications and aviation sectors. However, even after several years’ of broader growth, PNG’s economy remained dominated by two sectors: (i) the agricultural sector, which engages most of the labor force; and (ii) minerals and energy extraction, which earns the largest share of export receipts. It should be noted that this is still the case in PNG. In 2014, the mining and petroleum sectors were estimated to account for about a quarter of GDP, 83 percent of exports and around 7 percent of total employment. In contrast, 87 percent of Papua New Guineans live in rural communities, engaging in traditional subsistence and semi-subsistence agriculture in the informal sector. At the time of appraisal, growth in demand was outpacing that of productive capacity, and inflationary pressures built and became increasingly domestically focused. PNG’s consumers were buffeted by the global energy and food price rises of 2007 and 2008, which lifted year-on-year urban average inflation above 13 percent and urban food price inflation to above 20 percent. The subsequent retreat in global prices slowed local price growth, but only to around 5 percent year on year. As domestic demand pressures built through 2011, reported inflation approached 10 percent. 2. Sector Context. Modern education in PNG effectively began when independence was achieved in 1975. PNG experienced fundamental changes in the education sector from 1995-2008 with a dramatic expansion of its education system. The most profound change was to nine years of basic education (three years of elementary schooling and six years of primary) followed by four years of secondary schooling.1 Elementary schools were introduced for the first time together with a new curriculum aimed at improving the quality of teaching and learning and developing life skills. Crucially, initial literacy began to be instructed in the vernacular mother tongue. The expansion of the education system was supported by relatively generous budgetary allocations. But PNG's educational achievements                                                              1   Basic Education in PNG is divided into elementary education (grades Prep, 1, and 2 for children aged 6-8) and primary education subdivided into lower primary (consisting of grades 3-6) and upper primary (consisting of grades 7 and 8). English is the national language of instruction beginning in elementary school.   1   have remained considerably below those of Asian countries in general, and the question of quality of education has been a recurring theme in the donor programs.2 3. In 2004, the PNG National Department of Education (NDoE) (formerly the Department of Education (DoE)) published the National Plan for Education for 2005-2014 entitled “Achieving a Better Future” (NEP2). Following the NEP2, PNG developed the Universal Basic Education (UBE) Plan (2010-2019) which was endorsed in May 2010 by PNG’s education donor partners for purposes of the Education for All - Fast Track Initiative (EFA-FTI). Although the 10-year UBE was not a comprehensive sector plan, it represented a significant step in terms of strategic orientation in comparison with the Government of Papua New Guinea’s (GoPNG) NEP2, which was produced in 2005. Progress had been made in three main areas: (i) the plan identified performance indicators against which progress could be measured; (ii) it provided a general analysis of spending in the education sector along with projections for future desired enrollment increases; and (iii) it outlined implementation constraints and broadly described approaches to executing the government’s programs and building institutions in the education sector. The UBE Plan provided details on the status of access and retention and, to a lesser extent, quality. It was recognized that PNG would not achieve the Millennium Development Goal (MDG) of universal primary completion by 2015. NDoE calculated a national net enrollment rate for grades 1-6 of only about 45 percent and a gross enrollment rate of about 78 percent, although there was significant regional variation. 4. In order to improve access, the NDoE focused on programs that promoted on-time enrollment of children at the prescribed age of school entry – age six. Retaining children who start school was also a priority performance indicator. Of the children who began in first grade in school year 2002/2003 only 62 percent completed six years of education. NDoE was keenly aware that getting children into school and keeping them in school would achieve little unless an equal priority was given to improving teaching and learning. But basic information on learning outcomes was the weakest part of the UBE plan. Existing grade 8 exams showed disappointing results for literacy, numeracy, general skills, and written expression, and these reflected the performance of the relatively small share of youth who began school and continued to grade 8. There were some important steps that had been taken to promote quality. A modern outcomes-based curriculum was introduced in 2005, and related teacher training was being slowly rolled out. A complementary series of curriculum standards assessments for grades 2, 5, and 7 had also been developed and were set to be introduced at the beginning of the UBE plan period. The UBE plan also contained a series of centrally-funded programs, to be supported by government and donors, aimed at improving teacher skills and expanding access to textbooks, libraries, and other learning materials. In addition, a school subsidies program was introduced to increase resources and support activities that are included in the School Learning Improvement Plans which were controlled directly by schools. This program was aimed at promoting school-level planning, getting communities more involved in schools, and allocating increased resources to schools so they can pursue their own priorities and promote community involvement.                                                              2 In 2011, literacy statistics were extremely low: the overall literacy rate (literate in at least one language) was 68 percent, an increase of 11 percent from the 2000 Census. Tok Pisin showed the highest literacy rate of 57 percent, followed by local languages Tok Ples at 56 percent and English at 49 percent (PNG Census, 2011).   2   5. In 2010, the GoPNG began implementing the centerpiece of the UBE program: abolishing fees for elementary schools. An equally important signal of the wider government support for UBE was the fact that funding for subsidies to replace parental fees in elementary education was increased significantly to about K143 million, for the first time, for distribution to schools at the beginning of the school year. Funding for education in the 2010 budget increased by 6 percent overall from 2009, and basic education was the beneficiary of nearly all of this increase. NDoE and development partners were committed to making sure that UBE implementation continued. 1.2 Original Project Development Objectives (PDO) and Key Indicators 6. The original principal development objective of the project, in the Project Appraisal Document (PAD), was to improve the reading skills of elementary and primary education students. The PAD also states that a secondary objective of the proposed “funding” was to build the policy, monitoring, and evaluation capacity of the Curriculum Development Division (CDD) (formerly the Curriculum Development and Assessment Division (CDAD)) to select, procure, and distribute books and reading support materials and to undertake assessment of early grade reading together with provincial officials. 7. The Grant Agreement consolidated the PAD statements into a PDO that was stated as: (i) to improve the reading skills of elementary and primary education students and (ii) to build the policy, monitoring, and evaluation capacity of the Curriculum Development and Assessment Division in the Recipient’s Department of Education. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators and reasons/justification 8. The project development objective was restated as a part of the March 21, 2014, restructuring as: “to promote better teaching and learning of reading skills of elementary and primary education students.” The key indicators were also modified at this time as follows: Table 1: History of Project Indicators Original PDO Indicator Adjustments at the time of March 2014 Restructuring Students in elementary and primary schools will Proportion of primary school teachers using demonstrate improvements in their literacy skills as supplementary readers during reading defined by early grade reading assessment (EGRA) (to instruction in a sample of 1,800 classrooms with be reported by gender). This indicator was modified at classroom libraries, as defined by classroom the time of restructuring. observation checklist. Added at the time of restructuring. Improvement in students’ average scores in three EGRA subtests (phonemic awareness, letter sound identification, and familiar word reading) in pilot provinces (reported by gender). This indicator reflects the modifications made to the original indicator. Intermediate Indicators Number of students in elementary and primary schools Same as original that have access to learning materials and improved teaching practices   3   Number of teachers in elementary and primary schools Same as original who have participated in project professional development Classroom teaching practices reflect enhanced Dropped during restructuring knowledge in reading instruction and use of learning materials Number of provincial EGRA assessment reports Modified to “at least two post-assessment completed reading intervention pilots designed and completed, including their evaluation reports for dissemination” 3001 new children’s books will be available on the Modified to “vernacular reading materials in local market, either written by local authors or 2,500 elementary classrooms (grades EP-E2)” translated into vernacular languages from international publishers Classroom libraries established in 32,402 grade P+1 to Modified to “classroom libraries established in six classrooms in 9,624 schools2 approximately 18,000 grade 3-8 classrooms (each library containing 60 titles of supplementary readers) Learning material kits and teacher guidelines on the use of kits in approximately 15,000 elementary classrooms (EP-E2). Added during the restructuring. 860 teachers from 86 districts trained and supported as Two teams of coaches and 22 teams of peer mentors provincial classroom library trainers trained by the project to facilitate classroom library training to teachers at the district level in each province. At least one national public reading event during the project. Added during the restructuring. Two visits per school from 860 peer mentors to 9,624 Dropped during the restructuring schools At least 15 provinces complete EGRA baselines Modified to “up to four EGRA provincial surveys and results reports completed from baseline assessments in selected provinces” EGRA assessments repeated in at least two provinces Modified to “up to four events to disseminate results and findings from provincial EGRA surveys to NDoE, provincial officers and other stakeholders 1 Sixty titles would be provided for elementary schools and 40 titles for each of grades 3-8. 2 These calculations were made on the basis of figures given as: 5,939 elementary schools (17,817 classrooms); 3,281 primary schools (19,686 grade 3-8 classrooms). 1.4 Main Beneficiaries 9. The direct beneficiaries of the project were approximately 856,510 students in grades 3-8 with classroom libraries, 827,517 elementary students in grades EP-E2 with access to learning kit resources, and 450,810 elementary students in grades EP-E2 with access to vernacular reading materials. In addition, approximately 25,120 grades 3-8 teachers benefitted from the project through the training, as well as 1,334 provincial trainers and 18 national teacher trainer coaches. A total of 19 (47 percent female) NDoE officers directly benefited from Early Grade Reading Assessment (EGRA) capacity building activities, including the development of the EGRA diagnostic instrument, supervising data collection in schools, and marking and coding the assessments. There were 141 student teachers (71 percent female) from four primary teacher colleges trained as EGRA enumerators. In addition, there were 1,952 students in grades 3 and 4 in Madang that received the intervention   4   (58 teachers in treatment schools) in 28 schools, and in Western Highlands, 2,320 grade 3 students received the intervention in 46 schools (75 teachers in treatment schools). 1.5 Original Components 10. Part 1: Increasing the Availability of Books and Learning Materials by Establishing Classroom Libraries (US$14.3 million). This component provided support for the central selection and procurement of illustrated children’s early grade reading materials, including supplementary reading books and simple bookshelves that together were to form classroom libraries. The goal was to quickly increase the supply of literacy materials in schools and to provide child-friendly circumstances in which children could develop their reading skills as part of the daily curriculum. The component was to support the distribution of some teacher’s guides to classroom teachers along with training materials designed to help teachers to encourage reading habits. Simple bookshelves were also to be procured provincially from province- or district-level suppliers. The National Library Service (NLS) was to produce and distribute a catalogue of books and materials that had been made available to schools under the classroom libraries initiative. 11. Part 2: Promoting Reading through Professional Development, Professional Teacher Networks, and Public Reading Campaigns (US$1.7 million). This component was to support professional development of teachers in the use of classroom libraries and activities that built upon children’s literacy skills. Resources for strengthening and expanding existing school clusters to promote effective teaching practices in literacy development was a main goal of the professional development.3 Networks of teachers from elementary and primary education grades (EP, E1, E2 + P3 to P8) were to be supported to promote teaching strategies in the use of learning materials and improved reading instruction, improved reading fluency and comprehension, improved student writing, and critical thinking. The strengthening of professional communities would center on child-friendly practices of reading books and student writing and projects that could be inspired by the learning materials. The Classroom Library Management Unit (CLMU) would work through the Papua New Guinea Education Institute (PNGEI) as the professional development provider to implement the professional development program with support from international technical assistance. 12. Part 3: Strengthening Early Grade Reading Assessment (US$2.6 million). This component was to support the EGRA initiative by: (i) producing a baseline of early grade reading performance by province during the project period (2011-2013); (ii) building local capacity to replicate early grade reading assessment at the central and provincial levels and to monitor and report on progress against the established baseline; (iii) analyzing the implications of EGRA findings for instructional and curriculum improvements, policy directions, and future investment opportunities; and (iv) using the EGRA findings as an input into a reading information campaign aimed at mobilizing parental support for reading.                                                              3  Many districts in PNG organize school clusters for purposed of training and delivery of material support.   5   13. Part 4: Project Management and Monitoring (US$0.6 million). This component was to support the management and coordination of the day-to-day implementation of the project, including financial management, accounting, procurement, monitoring supervision activities, and reporting with the support of technical assistance. 1.6 Revised Components 14. During the March 2014 restructuring, Part 3: Strengthening Early Grade Reading Assessment, was modified. The related modifications were: (i) reduction in the number of provinces from 15 to four (one per region of the country) where the EGRA diagnostic surveys would be carried out and (ii) introduction of a post-EGRA intervention to improve teaching practice in a small number of selected provinces that would be used to monitor improvements in learning outcomes against the new PDO. During the May 28, 2015, restructuring, it was also agreed that the project would not finance bookshelves, but that the schools and villages would finance bookshelves from their own budgets. 1.7 Other Significant Changes 15. On February 19, 2013, the original Grant Agreement was modified, at the request of the GoPNG, to reflect changes to the administration of the GoPNG’s revenue laws which were related to exemption of tax, duty, or imposition under the revenue laws administered by the Customs Department. 16. On April 5, 2013, the Disbursement letter was amended to add the Special Commitment disbursement method which allowed the government to process applications for Special Commitment issuance. 17. The Office of the Minister for Treasury sent a letter on January 23, 2013, to the Bank requesting a restructuring of the project based on findings and recommendations from the November 2012 implementation support mission (ISM). On May 30, 2013, the restructuring was endorsed by the United Nations Children’s Fund Papua New Guinea (UNICEF-PNG) as the coordinating entity of the Global Partnership for Education (GPE) in PNG after receiving responses from members of the Education Development Partners Advisory Coordination Committee (EDPACC). This is a prerequisite for GPE financed projects. Upon Bank approval of the restructuring documentation, the Bank sent the package to the GPE in January 2014 for approval by the GPE Board. The Board ultimately approved the restructuring in March 2014. The March 21, 2014, Level 1 restructuring of the project included: (i) a restatement of the PDO; (ii) modifications to the results framework to restate the original PDO indicator and add an additional outcome indicator related to EGRA; (iii) reduction in the scope of Component 3 activities by reducing the focus of the EGRA activities from 15 provinces to four provinces and introducing post-EGRA interventions to improve teaching practice in a small number of selected provinces which were to monitor improvements in learning outcomes against the PDO; and (iv) extend the closing date by one year to June 30, 2015. 18. On May 28, 2015, there was a Level II restructuring that extended the project closing date from June 30, 2015, to December 31, 2015, to allow time for the government to: (i) distribute the elementary learning kits from the provincial centers to the elementary schools and (ii) distribute the vernacular reading materials in Tok Pisin to all elementary schools in   6   the country as well as a selected number of books in nine vernacular languages to 190 schools in eight provinces. The extension of closing date was also to allow the government to collect all data and information through the relevant monitoring and evaluation (M&E) activities to report against the project results framework and provide evidence of achievement of the PDO. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design, and Quality at Entry 19. Project Preparation. In 2010, the Reading Education Project (READ-PNG) was prepared as a GPE-supported project,4 and the Bank was the designated “supervising entity” for the project. The year 2010 also marked a period of the Bank’s re-engagement with PNG during which the GoPNG, for the first time in more than a decade, borrowed for an investment credit in education.5 The project was also consistent with the Bank’s draft Country Partnership Strategy (CPS) that had the aim of ensuring girls and boys benefited equally from PNG’s development goals. The preparation team used the Sector Specific Investment Grant instrument given the targeted nature of program interventions. The preparation team included project activities based primarily on the extensive review and analysis conducted as part of the UBE (2010-2019) development. Project preparation benefited from working through the NDoE systems and collaborating with UNICEF, which was designated as the GPE coordination agency and the Education Development Partners Advisory Coordination Committee (EDPACC). The preparation of the UBE and lessons learned from other countries,6 which had implemented similar types of projects, all contributed to assisting the team in preparing and appraising the project in five months to meet the GPE’s timeline for its Board approval in November 2010. The Bank approval by the East Asia Pacific Regional Vice President was March 3, 2011. 20. Project preparation was collaborative and benefitted from a deliberative process between NDoE, donor partners and the Bank. The Bank preparation and appraisal teams consisted of technical experts that were appropriate for the development of the program. They included peer reviewers with knowledge and experience in developing and implementing EGRA activities, reading interventions, and targeted teacher education programs, as well as GPE projects, and provided sound guidance on the project design. 21. Project Design and Quality at Entry. The project design was fully aligned with the government’s UBE Plan (2010-2019), which provided the platform for sectoral planning and financing, and the GPE goals of meeting the 2015 MDGs, as well as the draft CPS. The project design took into consideration comments from the peer reviewers during the concept                                                              4 At the time of project preparation, the Global Partnership for Education was named the Education for All-Fast Track Initiative (EFA-FTI) Catalytic Fund, and the funds approval committee was referred to as the Catalytic Fund Committee. 5 PNG borrowed US$5 million for a Flexible and Open Distance Education project for which the GoPNG made a US$1 million equivalent contribution. This was the first project in over a decade that the GoPNG had chosen to borrow from the Bank for education. 6 The project lessons included, but were not limited to, some from the Mongolia: Rural Education and Development Project (2006-2013) related to classroom libraries; the EGRA experience in Peru; and the preparation experience of the Malawi EFA-FTI project.   7   and appraisal review meetings which included: (i) simplifying the PDO by dropping a focus on improving capacity; (ii) provision of a system for documenting the delivery of inputs; (iii) expanding the role of the project steering committee, established for the Flexible and Open Distance Education Project (FODE), to include the READ project; and (iv) ensuring that the project had strong risk mitigation measures related to procurement and financial management. 22. The project design included a three-year implementation period which is the standard for GPE projects. The overall project design was focused on quality rather than access and school construction which was unusual for GPE projects at the time. This was because the GPE mandate at the time was to increase access to education to meet the MDG of universal access and completion. The READ project, instead, introduced reading-centered activities as well as the EGRA to help the NDoE assess the reading and learning challenges associated with the diverse regions of the country. This was a highly appropriate design given that GPE has subsequently migrated most of its grant funds to projects focusing on improving education quality. The project design also covered all elementary schools in the country with the provision of reading materials to develop classroom libraries and promote reading. This focus was significant because it gave early recognition to the importance of quality in improving educational outcomes. Moreover, it was the first time in 30 years that primary schools had received supplementary readers. The team also used the re-engagement period to mainstream project implementation within the NDoE’s CDD to build capacity within the NDoE. This design feature included: (i) a project steering committee (PSC) to oversee overall project implementation; (ii) a Classroom Library Reference Group (CLRG) to assist in the design and implementation of the project which was chaired by the Assistant Secretary of the CDD; and (iii) a national project coordinator that reported to the CDD. Again, this was a significant design feature because it recognized the importance of mainstreaming implementation to build local capacity. 23. The original “principal” PDO, as specified in the PAD, was simply stated and the key indicator that used the EGRA tool was appropriate for measuring progress toward achieving the principal PDO. However, while simply stated, in retrospect, the PDO was ambitious because it is difficult to measure improvements in reading skills over a three-year period, and the initial design did not include interventions, such as a targeted booster reading program and a follow-on assessment to measure reading improvement, that could have led to improved reading. It should also be noted that the wording in the PAD of the “secondary objective of the proposed funding was to build policy, monitoring and evaluation capacity…” led to some confusion over the exact wording of the PDO. The preparation documentation was specific that the project PDO was “to improve the reading skills of elementary and primary education students,” but the final documentation was less clear. The inclusion of the EGRA activities was highly relevant; however, the attempt to produce EGRA baselines in 15 provinces during the project period was ambitious due to capacity limitations and staff availability to carry out the survey. In addition, the design team underestimated the costing for the 15 provincial EGRA activities due to large logistical challenge of carrying out the assessments in PNG. The inclusion of provision of library books, vernacular translations, and teacher professional development were all highly relevant. The overall project design had many positive aspects with a primary focus on quality. However, the PDO and some project targets were ambitious.   8   24. The 2014 Level I project restructuring, which was initiated in 2013, provided the opportunity to enhance the potential effectiveness of the project design by restating the PDO to ensure that the objectives could be measured and achieved in three years. It also eliminated the confusion in the documentation, reduced the EGRA targeted provinces to four which was a more manageable number of provinces both from a cost and implementation perspective, introduced a pilot reading booster program in two provinces where the project teams had already administered the EGRA baselinet, and added a rigorous evaluation of the reading program with treatment and control groups to assess the overall impact. These modifications are judged to have been highly appropriate. 2.2 Implementation 25. Implementation context. There were exogenous factors that impacted project implementation, which were: (i) security concerns due to unemployment and lawlessness throughout the country that limited the Bank and implementation teams’ mobility and contributed to increases in operational costs due to security teams needed when traveling; (ii) inflationary factors between 2010 and 2013 related to increases in global commodity prices; and (iii) a 2014 downturn in country macroeconomic conditions caused by lower global commodity prices, the suspension of production at a major mine, and the unfavorable weather conditions that have affected food and commodity crops which is estimated to have affected 2.5 million people (over a third of the population). 26. Implementation details. Between April 2011 and October 2011, and prior to the November 2011 project effectiveness, the Bank team conducted three technical missions to PNG to begin preliminary implementation work with CDD. These activities included: (i) conducting a project launch workshop to inform the various stakeholders of the project activities and development objectives; (ii) preparing bidding documents for English language readers and instructional materials to support reading; (iii) providing details on the order of events for procurement of the reading and learning materials; (iv) laying the ground work for the first EGRA baseline gathering activity in the National Capital District (NCD); and (v) identifying an additional three provinces, Madang, East New Britain, and Western Highlands, for the EGRA baseline. It was during this period that the Bank and NDoE also agreed to use the Voluntary Service Overseas (VSO) group to help support CDD in managing the EGRA survey activities. All of these preliminary activities provided a foundation for implementation after the project became effective. 27. After effectiveness, and during the first official implementation support mission (ISM) in March 2012, it was recognized that implementation of some activities was proceeding as expected with: (i) procurement for the supplementary primary readers on schedule; (ii) detailed plans for teacher professional development underway with the recognition that training needed to be conducted once the supplementary readers were available; (iii) EGRA activities in East New Britain Province (ENBP) and NCD on schedule; (iv) discussion on procurement of vernacular materials underway; and (v) recognition of the need to procure classroom library shelves locally. However, there were also areas of implementation weakness related to financial management and procurement mainly due to limited capacity within NDoE CDD and the need for better management of the responsible consultants. Financial management at this time was rated moderately unsatisfactory because: (i) interim financial reports (IFRs) had not been summited; (ii) monthly reports for   9   the designated account (DA) had not been submitted; and (iii) the reporting and accounting systems, processes, and procedures were not in place. Procurement was also rated moderately unsatisfactory because procurement staff could not provide updates on the status of preliminary procurement actions. In an effort to improve this area, the Bank agreed that procurement for the FODE and READ projects would both be supported by a national and international procurement advisor to streamline procurement support and to hold a workshop to increase the awareness on procurement and a better understanding of roles and responsibilities. It was also at this time that there was a recognition by the Bank and CDD teams that the design of the EGRA activities were ambitious and a restructuring would be needed to reduce the number of provinces from 15 to four. As indicated, this was due primarily to an underestimation of the actual costs as well as implementation capacity to conduct the EGRA activities in a country with such remote and difficult to reach areas. 28. Between March 2012 and March 2013, the Bank team conducted two additional ISMs reporting on implementation progress. The CDD team was making progress with the: (i) signing of contracts with 16 successful bidders for the supplementary readers totaling 50 percent of project funds; (ii) drafting of a teacher training manual for classroom library management; (iii) continuation of EGRA activities in four provinces; (iv) endorsement of the Madang province EGRA findings and preparation for a dissemination workshop; (v) development of plans for a post-EGRA reading booster trial intervention in Madang; (vi) recruitment of a national consultant as well as and international procurement advisor; (vii) recruitment of a national financial management consultant as well as an internal auditor; (viii) purchase and installation of accounting software acceptable to the Bank, Mind Your Own Business (MYOB); (ix) all required IFRs had been summited and accepted; and (x) monthly reporting of eligible expenditures paid from the DA had commenced. Addressing the financial management (FM) issues resulted in upgrading of the FM rating to moderately satisfactory. The project restructuring had also been endorsed by senior management within the NDoE and the Bank. However, it was acknowledged that the GPE secretariat approval would be difficult due to the fact that project disbursements were only US$2.19 million, or approximately 11 percent. The limited disbursements were related to the fact that the CDD could not make payments to the 16 book suppliers because they needed a letter of credit which had not been provided. The letter of credit could not be issued because the Bank disbursement letter (DL) needed to be modified to include the special commitment method to finance the purchase of imported goods under the project. The Bank modified the DL in February 2013. By September 2013, the project had disbursed 43 percent of project funds. Although there was good overall project implementation progress, procurement delays related to the contracts for warehouse management, consolidation, and distribution of the primary readers were delaying the actual distribution of the primary readers which, in turn, was delaying the teacher training programs because they were to be coordinated with the distribution of the readers. 29. The November 2013 mid-term review confirmed that the project remained relevant but that a restructuring was needed to ensure the project could reach the overall development objectives. It was recognized that the restructuring should include: (i) clarification of the PDO; (ii) adjustments and restatement of the PDO and intermediate indicators; (iii) confirmation of the reduction of EGRA provinces; and (iv) extension of the project closing date to ensure implementation of activities that had been delayed. The restructuring was finally completed in March 2014 after delays. There were three areas that contributed to the   10   restructuring delays. These were: (i) the disbursements were low due to the problems associated with the letters of credit mentioned previously; (ii) an October 2013 FM review found some FM issues pertaining to possible ineligible expenditures and unaccounted funds that needed to be addressed before the restructuring could be processed (these issues were subsequently addressed to the satisfaction of the Bank); and (iii) there was an organizational change in structure of GPE that delayed GPE secretariat approval of the restructuring. By the time of project restructuring, approximately 50 percent of the project funds had been disbursed. 30. The post-restructuring period (April 2014-December 2015), is marked by improved overall implementation. Achievements include, but are not limited to: (i) distribution of more than one million supplementary reading materials in all 22 provinces in PNG; (ii) distribution of learning material kits to approximately 7,418 elementary schools (22,254 classrooms) in all 22 provinces; (iii) distribution of vernacular reading materials in nine vernacular languages; (iv) training for provincial facilitators and teachers on management of classroom libraries; (v) completion of four EGRA diagnostic assessments; (vi) dissemination of EGRA findings for all four assessment; (vii) implementation of a pilot booster reading training program in Madang and Western Highlands with impact evaluation completed, findings published, and PDO indicator reported; and (viii) completion of classroom observations in a sample of more than 1,800 classrooms that received libraries so that monitoring and evaluation related to the PDO could be reported. Moreover, capacity within the CDD had improved to the extent that the READ-PNG implementation team had become the “go-to” team for guidance and inputs to other NDoE departments and offices. This was particularly evident in the support that the READ team provided to the new standards-based curriculum (SBC) team based on EGRA findings. These are all substantial achievements of the project. (See Annex 2 for more details.) 31. There were also challenges: (i) delays in procurement and delivery of supplementary readers to the provinces led to a modified delivery plan for schools which meant that teachers took the readers back to their classrooms upon completion of their training—in the end a highly effective plan because books arrived with no loss during delivery; (ii) increased enrollments as a result of the government’s “tuition fee free” policy for elementary and primary schools led to student numbers that far exceeded the number used for project planning, and procurement of books thereby led to a shortage in classroom libraries— meeting project targets but missing some classrooms; (iii) delays in procurement of vernacular language materials which were related to the need to clarify certain copyright issues prior to purchasing the books7; and (iv) financial management issues, which were progressively resolved with support from the Bank and CDD teams so that there were no outstanding FM issues by the end of the project. 32. There were three technical missions prior to effectiveness and 11 Bank review and implementation support missions during the project implementation period. The makeup of the Bank teams was consistently appropriate and well balanced. Each team composition reflected the needs of the respective mission and included specialists from the areas of                                                              7 It should be noted that, during the delays, the project experienced exchange rate benefits which were then used to cover the higher than expected costs of distribution of the learning kits and the vernacular materials.   11   education, financial management, procurement, social, and monitoring and evaluation specialists as necessary. In addition, implementation support missions consistently reported to UNICEF and others within the donor community to keep them informed of project progress. The task team leader during implementation was based in Sydney which allowed for regular interaction between the Bank, donor community, and NDoE. This also provided continuity for implementation of the program and enhanced the Bank’s ability to quickly respond to issues/challenges as they arose, further contributing to the project’s implementation. By project closing, the project had disbursed US$19.2 million or 100 percent of the grant. 2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization 33. Design. The original M&E design was simple and three-fold. The first aspect of M&E design were the EGRAs that would provide the principal evaluation tool for determining the impact of the project. Secondly, the CDD was to monitor and report data related to the project results framework. Finally, the project was to include a delivery mechanism so that beneficiary classrooms directly confirmed, to the provincial education administration, that they received books and other materials supported by the project. The 2014 restructuring did not modify the overall design of the M&E structure. However, as indicated, the restructuring did: (i) modify the results framework to include a PDO indicator that would monitor the use of the new reading materials in a sample of 1,800 classrooms; (ii) reduce the number of provinces where the EGRAs would be implemented from 15 to four; and (iii) introduce an impact evaluation of the reading booster program in two pilot provinces. All were appropriate amendments to the results framework. 34. Implementation and Utilization. The implementation of the EGRA activities was highly successful. An EGRA diagnostic assessment was carried out in Madang, the National Capital District, East New Britain, and the Western Highlands provinces. The project produced four detailed assessment reports: one for each province.8 Based on the findings in 2014, the project developed a pilot reading booster program9 for Madang and Western Highlands. The associated impact evaluation10 of the pilot program registered remarkable results that supported the achievement of the PDO (see Section 3.2 for details). Moreover, the results have been used in the further development of the new standards based curriculum and associated teacher training activities related to the rollout of the new curriculum for elementary and primary schools. This was the first time that results for an evaluation/assessment had been used in the development of curricula in PNG. The CDD reported data related to the results framework. However, in the early stages of                                                              8 The four provincial reports are live in the EDDATA Global Website which is a global repository of data and reports on early grade reading and mathematics assessments and interventions. 9 Teachers in the treatment group received the following support: (i) training; (ii) instructional materials showing teachers what to teach and when, supported by a simple instructional model; (iii) class sets of student readers directly linked to lesson plans; (iv) developmental lesson observation with feedback from a coach; and (v) ongoing remote support using mobile phone text messages. Key elements of the training included: (i) new pedagogical knowledge and skills developed; (ii) relevant and participatory training; (iii) teachers given opportunities to practice new skills and instructional approaches; (iv) feedback and support provided to teachers; and (v) monitoring of teachers as they apply new skills in the classroom. 10 The impact evaluation was considered a paired, balanced study of four independent groups: control and treatment groups. The sample was balanced between treatment and control groups.   12   implementation, there were challenges with reporting because of: (i) limited availability of data due to slow implementation of inputs needed to measure progress; (ii) limited central and provincial understanding of indicator definitions, and (iii) limited capacity to collect data. After clarification of the indicators during the 2014 restructuring and, with M&E technical assistance support, the CDD team was able to provide the needed data for the results framework and use the data to support other departments as indicated above. The development of a system for reporting on the delivery of books and materials to schools was more challenging. This was partially due to delays in procurement of the books and materials which resulted in the need for quick distribution in the months immediately prior to project closing. The reporting system was also challenged by incorrect enrollment and classroom data related to free fee policy for elementary and primary schools as mentioned above. The project worked to overcome these challenges by: (i) giving the reading materials to teachers to take back to their classrooms after training and (ii) for the first time, empowering provinces by transferring the distribution resources directly to them to deliver the learning kits and vernacular books. After the 2014 restructuring, the team developed a classroom observation checklist that was used by the NDoE-trained team to determine if teachers were actually using the new reading materials in a sample of 1,800 classrooms. The survey was completed and the results were used to inform the achievement of the PDO. As indicated, this was an important addition to the project M&E because the reading books and materials were the largest percentage of the overall project investment. 2.4 Safeguard and Fiduciary Compliance (focusing on issues and their resolution, as applicable) 35. Safeguards. The project was rated a Category “C” operation. The indigenous peoples safeguard, OP/BP 4.10, was triggered, but a separate Indigenous People’s Plan (IPP) was not required due to the fact that all project beneficiaries were considered indigenous peoples and all elementary schools in the country were to be covered by the project. Project safeguards were consistently rated satisfactory. 36. Financial management. Financial management ratings fluctuated through the implementation period between moderately satisfactory and moderately unsatisfactory. As mentioned, this was primarily due to capacity constraints of CDD’s FM staff and the limited knowledge of Bank requirements stemming in part from the limited involvement of the Bank in the education sector during the preceding decade. This led to challenges in the timely submission of withdrawal applications and timely provision of the unaudited interim financial reports (IFRs). In addition, this led to challenges in ensuring that the Goods and Services Tax (GST) was not paid from the Designated Account (DA) in accordance with GoPNG law and as stipulated in the grant agreement. With the continued support of the Bank teams, capacity was improved to the extent that at the time of project closing, all FM documentation was in order including IFRs being up-to-date and the October 2015 audit being unqualified. This is a substantial achievement, and the project closed with a moderately satisfactory FM rating.   37. Procurement. Procurement was consistently rated moderately satisfactory. This, again, was primarily due to limited capacity related to Bank procurement methods. As previously indicated, implementation was often delayed because of the challenges faced by the implementation team related to the large-scale procurement of the supplementary readers,   13   vernacular materials, and learning kits.  It was the first time the CDD had done an international competitive bidding (ICB) which involved selecting large numbers of book samples from 20 international bidders. It also involved establishing a team that comprised approximately 10 people to do evaluation and selection of books, with assistance from the Bank’s expert in books evaluation and distribution which was a lengthy process. Once the international procurement advisor and national procurement consultant were in place, the procurement process improved which, in turn, led to improved project disbursements. The Bank also consistently provided support during the ISMs by ensuring that procurement staff were included to offer support. In late February 2013, the Bank also responded quickly to the request to add the special commitment disbursement method to the DL to ensure the large book procurement could go forward. Again, by project closing the CDD team had been able to complete all the large-scale procurement activities and demonstrated improved capacity for future projects. 2.5 Post-completion Operation/Next Phase 38. The NDoE is fully committed to continuing support for early grade reading as is evidenced by the use of the EGRA data and experience in the development of the elementary and primary school SBC. Moreover, the pilot booster reading program material has also been incorporated into the PNG pre-service and in-service teacher training programs. As a result of the project, the NDoE now has strengthened its capacity to: (i) observe reading interventions and then use the information to inform lesson plan development for the SBC language syllabi; (ii) develop and write training resources related to the use of classroom libraries; (iii) serve as national coaches to train provincial facilitators for classroom library training; (iv) support provincial officers in the training of teachers at cluster-based schools; and (v) support provincial officers in processing donor-funded projects in a more efficient manner. All of this represents capacity development within the NDoE curriculum development department to develop curricula and training programs, assess impact of programs, and use the results of the assessment to modify curricula and training programs. In 2016, the NDoE, with support from the GPE, will begin updating the ESP. Once the ESP is updated, the NDoE confirmed that preparation will begin for a follow-on GPE-supported project that will be developed using lessons learned from the implementation of READ-PNG. The details of the new project have not been defined. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design, and Implementation 39. Relevance of Objectives. The objective of improving literacy skills and quality of education was highly relevant to the country’s sectoral needs when the project was developed. The objectives fit into PNG’s UBE (2010-2019) that focused on the need to improve teaching and learning as well as the Bank’s draft CPS that focused on improving the quality of life for girls and boys. The 2014 restated project objective remained highly relevant to the UBE and CPS. Moreover, it continues to fit into the goals of the UBE (2010- 2019) and second pillar of the CPS (2013-2016) which remained the same as the draft CPS. 40. Relevance of Design. The relevance of the original project design was substantial. The focus of the overall project on quality was highly relevant. As indicated, the primary   14   PDO of improving reading skills was relevant, albeit difficult to measure within a three-year implementation period. Although the PAD documentation added a secondary objective, the primary objective of improving reading skills was the main goal and, as preparation documentation indicates, was the agreed overall objective. The original project activities and inputs were suitable for supporting achievement of the development objectives, and using the EGRA tool to measure baseline and endline reading skills was appropriate. However, the inclusion of 15 provinces in the EGRA activities was ambitious given the remoteness of parts of PNG. In addition, the preparation team underestimated the costs of administering EGRA in all 15 provinces. The restatement of the PDO during the 2014 restructuring was highly relevant because it could be measured within the project period. The modifications made to the results framework were also highly appropriate for measuring achievement of the PDO. Moreover, the addition of an intervention to improve teaching practices and student reading skills through the rigorous piloting of the reading booster program was very relevant, and the design allowed for direct attribution to the project intervention. The project risks and mitigation measures were also appropriate. As mentioned previously, the organizational structure was appropriate particularly for building NDoE capacity after a decade of limited Bank engagement in the sector. All-in-all, the adjustments made at the time of the 2014 restructuring were highly relevant. 41. Relevance of Implementation. The relevance of project implementation is judged to be substantial for both pre- and post-restructuring. As indicated above, there were implementation challenges largely related to procurement and financial management throughout the project period. Despite the challenges, the CDD was able to complete the procurement of the reading and vernacular books and learning materials kits. ISMs, provincial education offices (POEs), and the survey of 1,800 classrooms indicated that the libraries had been established. Of significant note was the implementation of all the quality- related EGRA activities with: (i) completion of the baseline assessment in four provinces with full reports available on the EDDATA Global Website; (ii) development of the pilot reading booster program with an impact evaluation built in that included treatment and non- treatment schools; (iii) full analysis and reporting of the pilot results; and (iv) inclusion of the findings in the teacher training program and SBC. Finally, capacity development, related to on-the-job learning for the CDD staff, led to the team becoming advisors for other NDoE departments and was another significant achievement. 3.2 Achievement of Project Development Objectives 42. This section evaluates the outcomes against the project development objective and the results framework that was adjusted during the March 21, 2014, formal restructuring. The development objective was to promote better teaching and learning of reading skills in elementary and primary education. More details on project outputs can be found in Annex 2. 43. Promotion of better teaching of reading skills was to be measured by the PDO indicator added during the restructuring: proportion of primary school teachers using supplementary readers during reading instruction in a sample of 1,800 classrooms receiving classroom libraries, as defined by classroom observation checklists. The target was to have 100 percent of teachers using the supplementary readers in the 1,800 classrooms. The project was able to conduct a survey in 1,940 classrooms using an observation checklist which   15   exceeded the target of 1,800.11 This was a substantial accomplishment because it was the first large survey of observed classroom teacher behavior conducted by the NDoE/CDD. Findings from the data analysis showed that 81 percent of the teachers interviewed reported having participated in the training activities (97 percent in Jiwaka), and 62 percent said they had used at least one of the main instructional strategies covered in the training (i.e., story tagging (strategy for retelling a story), hot seat (a comprehension activity), character profiles, role plays, and book reviews). The survey also found that 28 percent of the classrooms had a class reading timetable, and 61 percent of the teachers reported they allocated time for class reading sessions with times ranging from 10 to over 30 minutes. Sixty percent of the students interviewed (5,500 students were interviewed) reported that they had read or used books from their classroom libraries. The 100 percent target for the proportion of teachers using supplementary readers was missed by 38 percent. However, the findings are significant because they show that: (i) over 60 percent of the teachers trained were using teaching strategies from their training; (ii) teachers did acquire new knowledge and skills from their training; and (iii) training had made a difference to their classroom practices. The teaching practices for using the supplementary readers in the classroom have been incorporated into the pre-service and in-service teacher training programs, which is a positive sign for sustainability of the program. It is also significant that the overall teacher training programs benefited from knowledge acquired from the development of the pilot booster reading program material. 44. The intermediate indicators that supported the promotion of better teaching of reading skills were: (i) number of students in elementary and primary schools with access to learning and reading materials in their classrooms, provided by the project; (ii) number of teachers in elementary and primary schools who participated in professional development; (iii) number of classroom libraries established in grades 3-8 classrooms; (iv) number of learning material kits and teacher guidelines to use in elementary classrooms (EP-E2); (v) two teams of coaches and 22 teams of provincial classroom library trainers trained by the project to facilitate classroom library training to teachers at the district level in each province; and (vi) vernacular reading materials in 2,500 elementary classrooms (EP-E2). There were 827,517 elementary students and 856,510 primary students that benefited from the materials which exceeded the targets of 375,000 and 720,000, respectively. There were 25,120 total primary teachers trained exceeding the target of 25,000. Because delivery of the supplementary readers was delayed due to procurement delays, the project was not able to train 4,500 elementary schools teachers. The solution was to develop a teacher manual that was user- friendly and include it in the package of learning and teaching materials to enable teachers to follow instruction.  The project did provide 22,254 elementary classrooms with teacher guidelines on using classroom libraries as well as learning material kits thereby meeting the target for provision of learning kits. The teacher guides were important because they did contain the library training materials that had been developed for teacher training and were designed with simple exercises. The project also trained two teams in 22 provinces for a total of 1,334 trainers that will continue to provide classroom library training in the provinces. The project established 21,711 classroom libraries in grades 3-8, exceeding the target of                                                              11 It was not possible to use a fully representative sample due to delays in delivering classroom library books to some provinces. Therefore, the methodology used was a purposive sampling, known as judgmental, selective, or subjective sampling, based on reports from provincial project steering committees. There were six provinces visited: East New Britain, Jiwaka, Manus, Morobe, New Ireland, and the National Capital District, i.e., the city of Port Moresby. In each province the schools were randomly selected by the monitoring team.   16   18,000 classrooms, and delivered vernacular reading materials to 15,027 elementary classrooms far exceeding the target of 2,500. 45. Promotion of better learning of reading skills in elementary and primary education was to be measured by an increase in the average score in three subtests (phonemic awareness,12 letter-sound identification,13 and familiar word reading14) in pilot provinces (by gender). These three skills were selected for two reasons: first, these three skills are predictors of more complex reading skills; and second, these skills are expected to change given the intensity and duration of the intervention (one school year), as research from other EGRA pilots suggest other (higher-level) skills are not likely to observe changes in such a small amount of time. These three skills gave the EGRA team and NDoE an opportunity to observe the impact of the reading booster program and the time to have relevant discussions related to inclusion of findings in the new curriculum standards. The EGRA endline results were taken after the pilot reading booster program had been provided to teachers in a treatment group. The measures below only represent improvements for these three areas for the treatment group. The results below show significant improvement in the three subtests given in the two pilot provinces of Madang and Western Highlands. Also, the results from Western Highlands shows the proportion of fluent readers with good comprehension increased from 4 percent to 25 percent in treatment schools. In addition, means and standard deviations were calculated with weights, and analysis of the Western Highlands data showed that there were large positive effect sizes in the target sub-tests of initial sounds and letter sound recognition of 1.09 and 1.14 respectively. There were also moderate spillover effects onto other sub-tests with invented word reading and listening comprehension both having effect sizes greater than 0.2. These are highly significant findings. (See Annex 2 for additional EGRA results.) While there were some potential data problems with the Madang sample,15 the below results are still significant. Another significant aspect of the EGRA interventions is the fact that the improvements in reading outcomes, specifically related to Western Highlands, can be directly attributed to the pilot reading booster program. Moreover, the final results actually show improved reading skills which was the goal of the original project development objective.                                                              12 Phonemic awareness (initial sound understanding). The EGRA and subtest that focuses on manipulating, breaking apart, and putting together sounds orally. 13 Phonics (letter-sound identification). The EGRA subtest that focuses on linking written letters to their sounds and forming spelling patterns. 14 Familiar wording reading. The EGRA subtest that provides simple and common one- and two-syllable words and measures familiar word fluency in terms of correct familiar words read per minute. 15 It should be noted that the analysis of the data for Madang province had to take into consideration that : (i) initial analysis suggested that one of the subtest was not measuring the same thing for the baseline and endline; (ii) the sample had to be rebalanced due to the closing of some schools prior to endline collection and inaccessibility of other schools due to dangerous weather; and (iii) the baseline was conducted in 2011 almost two years prior to the beginning of the treatment. This was compared to Western Highlands where the baseline was conducted in 2013 and the endline in 2014.   17   Table 2: Madang EGRA Results Subtest Areas Boys Boys Girls Girls Baseline Endline Baseline Endline Phonemic Awareness (%) Grade 3 53 85 46 85 Grade 4 59 89 56 89 Letter Sound Identification Grade 3 28 73 26 82 Grade 4 36 88 30 86 Familiar Word Reading Grade 3 31 34 30 41 Grade 4 45 53 47 57 Source: Madang EGRA Report. Table 3: Western Highlands EGRA Results Subtest Areas Boys Boys Girls Girls Baseline Endline Baseline Endline Phonemic Awareness (%) Grade 3 43 80 43 82 Letter Sound Identification Grade 3 24 49 21 57 Familiar Word Reading Grade 3 17 32 17 39 Source: Western Highlands EGRA Report. 46. The intermediate indicators that supported the promotion of better teaching of reading skills were: (i) two post-assessment reading intervention pilots designed and completed including their evaluation reports for dissemination; (ii) at least one national public reading event during the life of the project; (iii) up to four EGRA provincial surveys and results reports completed from baseline assessments in selected provinces; and (iv) up to four events to disseminate results from provincial EGRA surveys to NDoE, provincial officers, and stakeholders. The project supported two post-assessment interventions in Madang and Western Highlands, and the results were published thereby meeting the target. There were also three national public reading events during the project period: (i) a book week event involving the project; (ii) an EGRA launch event for the EGRA reports attended by the Minister of Education in 2015; and (iii) the launch of the READ-PNG website (http://rpng.education.gov.pg/). In addition, there were four provincial EGRA baseline surveys and four EGRA report dissemination activities. As noted above, all four of the EGRA reports have now been live in the EDDATA Global Website which is a global repository of data and reports on early grade reading and mathematics assessments and interventions. 3.3 Efficiency 47. A high return to the classroom library investment is expected given its extremely low cost per student and the international evidence on similar programs’ impact on human capital. The link between cognitive ability and earnings and consequently labor productivity is well researched (Patrinos and Psachoropolous 2010). Similar classroom library programs studied in Fiji, Sri Lanka and Singapore (Elly et al. 1996) found strong impacts on children’s   18   cognitive outcomes. The estimated cost per student of the project’s classroom library investment is PGK 30 (USD 12) per student. If the impact of the classroom library investment on children’s cognitive outcomes were as strong as the lowest results in Fiji and if the link between cognitive ability and future earnings were around average for developing countries, then an estimate of the internal rate of return of the investment would range from 49 to 53 percent. This high rate of return results from the low cost of classroom library programs for young children, and emphasizes the importance of investing in human capital at an early age. 48. The scientifically rigorous evaluation of PNG’s reading booster program establishes it as a safe and high return investment. The Reading Booster program provided remedial courses in these skills for grade 3 and 4 students. A follow-up early grade reading assessment revealed varying effect sizes across different reading skill domains with an average effect size of 0.24 standard deviations. If the link between cognitive ability and future earnings were around average for developing countries, an estimate of the internal rate of return of this investment would range from 23 to 25 percent. The randomized-controlled trial evaluation of this program provides conclusive evidence of its impact on cognitive ability and subsequent human capital which is often not available for many types of public investment. Consequently, further investment in the program is a safe investment in addition to having a high expected rate of return. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 49. The overall outcome rating of the project is moderately satisfactory. The relevance of the objective pre-restructuring is rated substantial. This is because the PDO is judged to have been ambitious. However, the relevance of the post-2014 restructuring objective is high. The design, as indicated, was fully aligned with the UBE for both pre- and post- restructuring. This design of the project prior to restructuring is rated substantial primarily because the PDO, PDO indicator, and inclusion of 15 provinces in the EGRA activities were ambitious for a three-year implementation period. However, because the fundamental focus of the project was on quality and the inputs were all appropriate for achieving the objective of improving literacy skills, more weight is given to this aspect of original design. The design of the post-2014 restructuring is high. The restructuring of the PDO, modification of PDO indicators to measure the restated PDO, and inclusion of the booster reading program with associated teacher training and a rigorous evaluation were highly appropriate modifications. The relevance of implementation for both periods is substantial. Although implementation faced many challenges, the project activities were implemented, and there is documented capacity development from the implementation of these activities. Overall efficacy is rated moderately satisfactory. During the pre-restructuring period efficacy is rated moderately satisfactory. Although the PDO and PDO indicators were modified, the project had already begun the baseline EGRA activities in the four provinces. Moreover, the implementation teams had already identified the need for a reading booster program and designed the impact evaluation for the program. The Madang results, reported in section 3.2 above, were already available prior to the formal restructuring. It should also be noted that the lessons learned from the Madang experience were used in improving the instrument used in the Western Highlands province impact evaluation. Finally, the sample schools had already been selected for conducting the survey of 1,800 classrooms and the data collection instrument was being developed. Efficacy post restructuring is rated satisfactory. This is because the documented   19   achievements from the rigorous evaluation in reading are so impressive that more weight is given to these in determining the overall achievement of the development objectives post restructuring. In addition, the design of randomized control trial impact evaluation of the reading pilot program shows that the results can be measured directly attributed to the project interventions. Finally, as indicated above, the overall project efficiency rating is substantial. Again, this is related to the safe and high return to the investment of the reading booster program which provides conclusive evidence of its impact on cognitive ability and subsequent human capital which is often not available for many types of public investment. Table 4: Project Rating Original Project – 3/3/2011 – 3/21/ 2014 – 51.41 percent disbursement-net project funds1 Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating Substantial Moderately Satisfactory Substantial Moderately Satisfactory Original Project – 3/22/2014 – 12/31/2015 – 48.59 percent disbursement-net project funds1 Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating High Satisfactory High Satisfactory Overall Project Ratings – 100.00 percent disbursement of total grant as of 12/31/2015 Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating Substantial Moderately Satisfactory Substantial Moderately Satisfactory2 1 The net amount for the total project is US$19.2 million. 2 Overall rating = (4)(.5141) +(4)(.4859) = 4.00. 3.5 Overarching Themes, Other Outcomes, and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development Not Applicable (b) Institutional Change/Strengthening 50. The project resulted in strengthening the capacity and relationships between the NDoE and provincial education offices. The project contributed, in a substantial way, to supporting the development of the new SBC, which attests to the relevance of project activities and usefulness of the collected EGRA data for informing curriculum planning. The project activities also contributed to strengthening institutional capacity in curriculum development related to reading resource development, teacher training, and the monitoring of teachers. There was additional strengthening in: (i) adopting ideas from observing reading interventions to inform lesson plan development to support SBC language syllabi; and (ii) developing and writing training resources for classroom library training. Capacity strengthening at the provincial level was also significant. The NDoE and project staff worked with provinces to establish provincial project steering committees (PPSC) in all 22 provinces and to establish PPSC bank accounts for the transfer of funds to support training activities and delivery of resources to schools. The PPSCs took ownership of the project activities within their provinces in preparing provincial teacher training and book distribution plan and worked hard to ensure that the training was completed, primary school classroom libraries were established, and elementary school resource kits delivered. For activities implemented by the provinces, the CDD and provincial education offices signed contracts based on   20   implementation plans endorsed by the CDD and specified activities to be completed within a specific timeframe. Upon completion of the agreed deliverables, the project reimbursed funding. The introduction of these procedures helped to improve transparency and accountability in spending project funds and, as indicated above, increased ownership of project implementation. This mechanism will continue to be used for future operations. All are significant developments. (c) Other Unintended Outcomes and Impacts (positive or negative) Not Applicable 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not Applicable 4. Assessment of Risk to Development Outcome Rating: Substantial 51. The original project preparation identified the overall risk rating as substantial. The ratings can be summarized as: (i) high at the institutional level due to the limited capacity and limited experience with Bank processes given the decade of no Bank lending; (ii) substantial for fiduciary because of no direct experience with Bank procurement or financial management; and (iii) moderate for technical aspects because there was technical expertise in country. As indicated, the high risk associated with the NDoE and provincial capacity was correctly identified particularly as it related to financial management and procurement, especially considering the implementation challenges associated with these areas. The moderate risk related to technical expertise was correctly identified. Going forward, the institutional risks should be moderate. The NDoE has an increased understanding of the requirements to implement Bank and/or Bank-supervised operations. In addition, there is increased capacity to manage projects; develop curricula; and conduct assessments, analyze data, and procure and distribute large quantities of books and learning materials. The overall risk related to financial management and procurement should remain as substantial. While there was considerable improvement in these areas, given the overall low capacity levels at the beginning of the project, it will take more time to develop this capacity. It is expected that future projects will continue to build capacity within the NDoE. Finally, the risk for sustainability of overall project investments is rated substantial. This is due to the macroeconomic conditions that include a budget deficit in 2015 that increased to 5 percent of GDP compared with an estimated 4.5 percent in the 2015 Budget. This has led to a decline in public expenditure, which is affecting all government departments, including education. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 52. The project preparation team ensured that the project design was closely aligned with the UBE (2010-2019) and the Bank’s draft CPS 2013-2016. The original design also focused   21   completely on quality improvement in the delivery of educational services which was unusual for GPE projects at the time of design. As indicated, this was of high relevance. The original design was simple. However, there was confusion over the PDO, some of the indicators had ambitious targets particularly related to the EGRA activities, and project costing related to the EGRA activities was underestimated. The 2014 restructuring modified the design to support and ensure achievement of the PDO. All modifications were highly appropriate. The preparation team identified the appropriate risks and incorporated design features to mitigate them and included the relevant technical specialists to develop the project. The overall quality at entry for the READ-PNG is judged to be moderately satisfactory primarily because of the ambitious targets and costing associated with the EGRA activities. (b) Quality of Supervision Rating: Satisfactory 53. The Bank conducted 11 technical, review, and implementation support missions, which included a mid-term review. The project team was actively engaged in supporting the government in its efforts to implement the project. Whenever implementation challenges arose, the Bank team worked with government to find solutions. The Bank team recognized early in the implementation period that a restructuring would be necessary and provided constant and consistent support to ensure that the restructuring was completed despite the various institutional processes and procedures associated with a restructuring. The supervision teams consistently reported on financial management and procurement progress and worked with the NDoE teams to build their capacity in these areas. The EGRA technical team was instrumental in ensuring the integrity of the EGRA activities which played a critical role in ensuring achievement of the PDO. The team also systematically documented project progress in aide-memoires, back-to-office reports, and ISRs, all of which kept Bank management informed of progress and provided the foundation for the ICR analysis. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 54. Based on the implementation analysis above, overall Bank performance is rated moderately satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 55. At the time of preparation, the government was fully committed to the program. NDoE led the UBE development exercise with the support of the donors and promptly submitted an application for EFA-FTI Catalytic Fund (EFA/FTI CF) grant. In consultation with other donors, the government identified the Bank as the “supervising entity” and closely worked with the Bank team to develop and successfully secure the US$19.2 million grant from the EFA/FTI CF program. All parties actively participated in the program launch workshop and remained committed to the overall goals of the project. There were ongoing challenges associated with provision of government resources needed to pay for the distribution of elementary books and learning materials kits that led to implementation   22   delays. Based on this, the overall rating of the government performance is rated moderately satisfactory. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 56. The CDD was responsible for managing all aspects of implementation. When faced with implementation challenges, the CDD was able to work effectively with the Bank to resolve them. The capacity building activities related to EGRA, curriculum development, administration of surveys, and data analysis were very successful and fully documented in other sections of the ICR. Implementation was challenged by the capacity constraints related to FM and procurement which delayed aspects of implementation. Based on these aspects, the implementation agency performance is rated moderately satisfactory. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 57. Based on the implementation analysis above, overall Borrower performance is rated moderately satisfactory. 6. Lessons Learned 58. Lesson 1. Learning outcomes can be measured under education projects when project designs are simple and include components and activities that are linked to the project outcomes with rigorous evaluations to measure impact. The READ project was able to achieve measureable increases in students’ average scores in three EGRA subtests. The increases in reading were achieved because the project: (i) conducted the EGRA baseline that identified weaknesses; (ii) analyzed the results and developed a reading booster program to address the weaknesses; (iii) trained teachers on teaching methodologies to improve reading in the weak areas; and (iv) conducted a rigorous evaluation of the interventions to measure the results of the intervention. 59. Lesson 2. The inclusion of EGRA activities, when administered well, can have substantial impact on informing education policies. The results of the EGRA baseline led to development of the booster reading program. The endline analysis showed measurable progress that could be attributed to the intervention. Moreover, the EGRA activities had a substantial influence on the development of the SBC as well as teacher training learning materials. 60. Lesson 3. Surveys included in project designs are valuable tools for assessing the impact of project interventions. The survey of more than 1,800 classrooms was important for understanding the impact of the project-supported teacher training as well as the new classroom libraries. 61. Lesson 4. Large book procurements need proven capacity for in-house book assessment and selection as well as distribution systems to ensure on-time delivery of materials. The procurement of the books and learning materials was a challenge due to limited capacity for large-scale international competitive bidding and in-house book   23   assessment and selection. There were also challenges with the distribution of books because there was not a tested distribution system in place. Through the project, the NDoE/CDD helped to build the system by establishing provincial project steering committees in the 22 provinces and provincial bank accounts for the transfer of funds to support training activities and delivery of resources to schools. In addition, the project used the support of international technical assistance to build procurement capacity within NDoE. 7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors (a) Grantee/Implementing agencies Not Applicable (b) Cofinanciers/Donors Not Applicable (c) Other partners and stakeholders No additional comments provided.   24   Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million, equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Component 1: Increasing the 14.3 13.76 96.22 Availability of Learning Materials by Establishing Classroom Libraries Component 2: Promoting Reading 1.7 2.79 164.12 through Professional Development, Professional Teacher Networks, and Public Reading Campaigns Component 3: Strengthening 2.3 1.49 64.78 Early Grade Reading Assessment Component 4: Project 0.6 1.17 195 Management and Monitoring Total Baseline Cost 19.2 19.2 Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 0.00 0.00 Project Preparation Costs 0.00 0.00 .00 0.00 0.00 .00 Total Financing Required 19.2 19.2 (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Trust Funds 0.00 0.00 Education For All –Fast Track 19.20 19.20 100.00 Initiative   25   Annex 2. Outputs by Component 62. Initial progress towards achieving project outcome and meeting output targets was measured by one PDO indicators and 10 intermediate indicators. Following the restructuring in 2014, the revised results framework included two PDO indicators and a revised 10 intermediate indicators. This annex uses the combined indicators as listed in the revised results framework to assess output achievements. Component 1: Increasing the availability of books and learning materials by establishing classroom libraries in elementary and primary schools (US$14.3 million) 63. This component was to provide support for the central selection and procurement of illustrated children’s early grade reading materials, including supplementary reading books and simple bookshelves that together were to form classroom libraries. The goal was to quickly increase the supply of literacy materials in schools and to provide child-friendly circumstances in which children could develop their reading skills as part of the daily curriculum. The component was to support the distribution of some teacher’s guides to classroom teachers along with training materials designed to help teachers to encourage reading habits. Simple bookshelves and classroom furniture were also to be procured provincially from province- or district-level suppliers. The National Library Service (NLS) was to produce and distribute a catalogue of books and materials that had been made available to schools under the classroom libraries initiative. During the second restructuring it was decided the provision of bookshelves would be left to schools and communities to fund and arrange due to the complexities of arranging small grants to schools and accounting for the purchase. 64. The intermediate indicators that were to measure achievement of this component were: (i) number of students in elementary and primary schools with access to learning and reading materials in their classrooms, provided by the project; (ii) number of classroom libraries established in grades 3-8 classrooms; (iii) number of learning material kits and teacher guidelines to use in 15,000 elementary classrooms (E-E2); and (iv) vernacular reading materials in 2,500 elementary classrooms (EP-E2). There were 827,517 elementary students and 856,510 primary students that benefited from the materials which exceeded the targets of 375,000 and 720,000, respectively. The project established 21,711 classroom libraries in grades 3-8, exceeding the target of 18,000 classrooms. The project also provided 22,254 elementary classrooms with teacher guidelines on using classroom libraries as well as learning material kits thereby meeting the target for provision of learning kits. Finally, the project provided vernacular materials to 15,027 classrooms in elementary schools exceeding the target of 2,500. 65. Activity 1 - Supplementary reading materials to primary schools (Grades 3 – 8). This activity planned to deliver sets of 60 grade-appropriate English language supplementary readers to all primary classrooms across the country. As indicated, the project was able to deliver 21,711 supplementary readers to classrooms in 3,794 primary schools, providing books to an estimated 856,510 students across the country. The target of 18,000 primary classrooms and 720,000 students with access to supplementary reading materials was achieved. This is a remarkable achievement given the difficulties in procuring, sorting, and   26   delivering over 1,100,000 books to all 22 provinces of a geographically challenging country that relies on boat, plane, and helicopter to reach the high number of schools in remote locations. The mode of delivery of books to the provinces, districts, and, ultimately, schools took lessons from previous donor-funded projects that had faced difficulties accounting for many deliveries. The READ-PNG approach built upon local systems already in place, including provincial education offices (PEOs), to successfully ensure books reached their intended destinations. In most cases, the PEO’s gave the books to teachers to take back to their classrooms upon completion of their training. This proved to be a highly efficient and effective method for ensuring books reached the classrooms. 66. Activity 2 – Elementary Learning Materials (Prep and Grades 1 – 2): Elementary Learning Kits included grade-appropriate learning materials16 and a set of wall charts.17 The kits also included a teacher guide developed by NDoE’s Curriculum Development Division (CDD) which provided guidance on how to use the contents of the kits in classroom activities. One kit was delivered to each elementary school, with 22,254 elementary classrooms and 827,517 students now having access to these kits. The development and writing of the teacher’s guide played an important role in building the capacity of the CDD team, who subsequently used these skills to develop SBC which is being rolled out to elementary schools in 2016. 67. Activity 3 – Elementary Vernacular Reading Materials: With over 840 languages spoken in PNG, and orthographies developed in around 400 language groups, this activity planned to provide reading materials to children in a familiar language. Having vernacular reading materials available in elementary schools has been shown to increase learning opportunities for children who do not speak English at home and who often belong to more disadvantaged groups. In addition, evidence from the project’s EGRAs showed that learning to read in a familiar language improved reading outcomes. This finding was an important driver behind the decision to target all elementary classrooms with vernacular reading materials. Fourteen titles in nine vernacular languages were selected for distribution (see Table 5). Tok Pisin was given to all classrooms while the other vernacular books were delivered to areas where the languages were spoken. Delivery of vernacular reading materials to elementary schools across the country was combined as much as possible with delivery of the elementary learning materials kits because the same schools were to receive both deliveries. The vernacular books were delivered to 15,027 classrooms in elementary schools nationwide, exceeding the target of 2,500 classrooms.                                                              16 Learning materials included small chalkboards, flashcards, beads, dice, pattern blocks, linking cubes, pocket display board, and silkscreen printers. 17 Wall charts included grammar, punctuation, weather, health, and a map of PNG.   27   Table 5: Vernacular Material Distribution Language Province Estimated Number of Schools Tok Pisin/English Nationwide, all schools 5,009 Biga Kilivila Milne Bay 54 Kiwai Western 12 Naliku New Ireland 10 Yom Kawac Morobe 15 Maututu West New Britain 10 Motu Central/NCD 10 Telei Autonomous Region of 50 Bougainville Mekeo Central 34 Source: CDD. Component 2: Promoting reading through professional development, professional teacher network, and public reading campaign (US$1.7 million)   68. The goal of this component was to improve children’s reading habits through: (i) professional development of teachers on the use of classroom libraries; (ii) strengthening cluster-based teacher networks to encourage teacher development; and (iii) conducting a public campaign to promote reading, publicize the project’s investment in reading materials to schools, and encourage family participation in schools. 69. The intermediate indicators that were to measure achievement of this component were: (i) number of teachers in elementary and primary schools who participated in the project professional development activities as provided by the project; (ii) two teams of coaches and 22 teams of provincial classroom library trainers trained by the project to facilitate classroom library training to teachers at the district level in each province; and (iii) at least one national reading event during the life of the project. Training for provincial facilitators and teachers on management of classroom libraries using a cluster-based approach was completed before project closing. There were a total of 1,334 provincial trainers comprising head teachers and standards officers trained as well as 25,120 primary teachers in all 22 provinces across the country, exceeding the target of 18,000 primary teachers. As indicated below, the project was not able to train elementary teachers. However, 22,254 elementary teachers received the simple teacher guides on classroom library use and management, far exceeding the target of 4,500. Finally, there were three reading events that took place thereby exceeding the target. 70. The cluster-based training modality using the Training of Trainers (TOT) approach built on and improved existing systems in place for annual in-service teacher training. The improvements include the use of head teachers and standards officers as the trainers, which has built capacity at the local level and enables teachers to more easily to seek guidance and receive further training later. The effectiveness of this training model has informed NDoE’s rollout of the new SBC in elementary schools in 2016, which will use this modality to train elementary teachers on the new curriculum. Elementary teachers received teacher guides   28   with the elementary learning materials kits that were delivered under Component 1.2, which provided them with suggestions on how to use the materials in the kit, including sample activities and games. The guides were intended to be simple and user friendly. While the original project design intended elementary teachers would also receive face-to-face training in the use of the kits and vernacular reading materials in the classroom, this did not take place under the project due to the delays in procuring and delivering the materials to schools. 71. The project aimed to carry out at least one national public reading event, which took place with demonstration activities on teaching and learning strategies and materials and coincided with the 2014 National Book Week. In addition, support was also provided to Provincial Education Officers to carry out similar events in their provinces. The project also had a national launch event for the EGRA reports that was attended by the Minister of Education in 2015 and the READ-PNG website was launched. Component 3: Strengthening Early Grade Reading Assessment (EGRA) (US$2.6 million) 72. The original project design proposed to carry out diagnostic EGRAs in at least 15 provinces in PNG, with the aim of producing a baseline of reading levels by province; developing local capacity to replicate the EGRA and monitor improvements; and informing instructional and curricula improvements, policy directions, future investment opportunities, and a reading campaign. Under the restructuring in 2014: (i) the number of provinces was reduced from 15 to four (one province from each of the four regions of PNG); and (ii) a pilot booster reading intervention in two of the four EGRA provinces was introduced, using endline assessments to measure the impact of the pilots. 73. The intermediate indicators that were to measure achievement of this component were (i) up to four EGRA provincial surveys and results reports completed from baseline assessments in selected provinces; and (ii) up to four events to disseminate results and findings from provincial EGRA surveys to NDoE, provincial officers, and other stakeholders. 74. Activities under this component were successfully completed as scheduled. Between 2012 and 2014, four reading diagnostic assessments were carried out using EGRA instruments, developed in English for Madang, National Capital District, and Western Highlands Province, and in Kuanua (local language) and English in East New Britain province. During this time, a total of 19 NDoE officers (47 percent female) received capacity development training in (i) the development of the EGRA diagnostic instrument; (ii) supervision of data collection in schools; and (iii) marking and coding the assessment. In addition, 141 student teachers (57 percent female) from four primary teacher colleges were trained as EGRA enumerators. The full diagnostic results reports for the four provinces were launched by the minister of education in September 2015. 75. Using results from the baseline survey, the NDoE was able to identify reading areas where teacher training could be improved. Based on these findings, the project developed a reading booster program for piloting in Madang and Western Highlands provinces in 2013 and 2014, respectively. In total, 116 primary school teachers were trained (56 Madang and 60 Western Highlands). In both provinces, there was a treatment group and a control group.   29   In Madang, approximately 1,952 students benefited from the booster program and 2,320 students in the Western Highlands. Between April and May 2015, the READ-PNG team carried out a complementary qualitative evaluation in both provinces to supplement the original pilot evaluation reports. The evaluation provided endline results for Western Highlands and Madang provinces. The results for Madang show that the data was somewhat problematic, which limited the team’s ability to clearly analyze the data.18 However, the results from Western Highlands were of good quality and showed that there were substantial differences in zero scores between the control and treatment groups. The PDO EGRA data was provided above; however, there are additional results that show substantial achievements in Western Highlands which are provided below. Table 6: Percentage of Students with Zero Scores in Western Highlands Subtest Areas Treatment Treatment Control Control Baseline Endline Baseline Endline Letter sound knowledge 22 2 21 26 Initial sound 35 4 35 32 identification Listening comprehension 29 25 33 37 Reading comprehension 34 14 34 21 Oral reading fluency 11 3 8 3 Source: Western Highlands Intervention Analysis. 76. The survey also looked at fluency and comprehension by dividing students into the following typology: (i) non-readers: students who were not able to read any works of a passage; (ii) readers without comprehension: students able to read some words of a passage but unable to answer any questions related to the passage correctly; (iii) readers with some comprehension: students able to read at least some of the passage and answer some question(s) about the passage but less than 80 percent; and (iv) readers with good comprehension: students able to read a passage and answer at least 80 percent of comprehension questions about the passage correctly. Table 7 shows substantial and positive results between the treatment and control groups that can be directly attributed to the interventions piloted under the project. Table 7: Percentage of Reading Comprehension and Fluency in Western Highlands. Type of Reader Treatment Treatment Control Control Baseline Endline Baseline Endline Readers with good 4 25 3 14 comprehension Readers with some 62 61 63 66 comprehension                                                              18 The data for Madang province was of poor quality because: (i) initial analysis suggested that one of the subtests was not measuring the same thing for the baseline and endline; (ii) the sample had to be rebalanced due to the closing of some schools prior to endline collection and inaccessibility of other schools due to dangerous weather; and (iii) the baseline was conducted in 2011 almost two years prior to the beginning of the treatment. This was compared to Western Highlands where the baseline was conducted in 2013 and the endline was 2014.   30   Readers without 24 11 25 18 comprehension Non-readers 11 3 8 3 Source: Western Highlands Intervention Analysis. 77. The project planned to hold up to four events to disseminate results and findings from the provincial EGRA surveys to NDoE, provincial officers, and other stakeholders. This target was achieved, with events carried out in each of the four EGRA provinces to disseminate results from that province, a workshop for education stakeholders to disseminate results from all four provincial EGRA surveys, and a wrap-up policy workshop to present the project's final results. As indicated above, the four reports are also now available on EDDATA Global Website which is a global repository of data and reports on early grade reading and mathematics assessments and interventions. In addition, evaluation reports of the two post-EGRA interventions were to be finalized for dissemination, which have been endorsed by the government. Component 4: Project management and monitoring (US$0.6 million) 78. Project activities were managed, implemented, and monitored by a Classroom Library Management Unit (CLMU), which was housed within NDoE’s Curriculum Development Division (CDD). The CLMU included a Project Coordinator, who reported directly to the Assistant Secretary CDD, and national consultants responsible for EGRA, teacher training, and monitoring of and reporting on progress of project activities. The CLMU also hired a national procurement consultant and an accountant, both of whom were supported by international procurement and financial management consultants to build fiduciary capacity within NDoE. All planned procurements were successfully completed, all project funds were disbursed, and all audits are up to date and approved by the Bank without qualification. 79. CDD and the project team made considerable efforts to ensure that project activities were implemented fully prior to project closing on December 31, 2015. The project planning, budgeting, and monitoring and evaluation activities consistently improved over the duration of the project. It is commendable that the READ-PNG team has earned a reputation as the team that other divisions/offices of NDoE go to for guidance and inputs. The READ-PNG team provided support for the development of the new SBC, which attests to the relevance of project activities and usefulness of the collected EGRA data in terms of informing curriculum planning.   31   Annex 3. Economic and Financial Analysis 80. The economic benefits of the project’s PDO arise through strengthening human capital and subsequently higher productivity. The project’s development object (PDO) is to promote better teaching and learning of reading skills in elementary and primary education. Reading skills are crucial to a child’s future educational outcomes and subsequent labor market outcomes as well as his or her contribution to society. This project supports four major interventions: (1) establishment of classroom libraries and associated teacher training in grades 3 to 8, (2) provision of elementary school (EP to E2) literacy materials, (3) provision of elementary school vernacular language reading materials, and (4) a pilot remedial course on reading in Madang and Western Highland provinces. Table 8 presents estimated costs per student.   Table 8. Estimate costs per student of components 1 and 2 (PGK) Cost / Component 1 Classrooms Students Cost student Classroom libraries 21,711 856,410 18,400,631 21 Elementary literacy materials 22,254 827,517 6,010,816 7 Elementary Vernacular Language Books 15,027 558,780* 4,462,637 8 Cost / Component 2 Teachers Students Cost student Professional development 25,120 856,410** 7,708,675 9 Total classroom libraries (grades 3 to 8) 856,410 26,109,306 30 Cost / Component 3 Students Cost student Reading Booster Pilot Program 4,272 794,243 186 * Estimated number of students based on number of classrooms and average students per classroom receiving literacy materials. ** Number of students equated to the number of students receiving classroom libraries. 81. Educational attainment, especially completing primary school, is strongly associated with earnings in PNG. Earnings differentials between levels of schooling represent the value the labor market places on education as well as incentives for individuals and society to invest in education. In the 2010 Papua New Guinea Household Income and Expenditure Survey (HIES), one year of schooling corresponds to 7.7 percent higher earnings which are slightly lower than the international average of 9.7 percent (Figure 1). As with other developing countries, the association between a year of schooling and earnings is especially high in primary school; the difference in earnings between an individual who has completed primary and who has not is equivalent to 43 percent for each year of primary school.   32   Figure 1. Association between Schooling and Individuals Earnings 50 Percent increase in earnings 40 43.2 East Asia and Pacific All countries 30 PNG 20 10 10 7.7 0 1.8 Primary Secondary Post‐secondary Average % increase in Average % increase in earnings for an additional year by level of earnings for an schooling* additional year of schooling Source: Montenegro and Patrinos 2014. The study calculated the association between years of schooling and earnings using 819 datasets in 139 economies. *This is the increase in earnings for completing each level of schooling annualized to the number of years needed to complete the level.   82. International evidence establishes the link between cognitive ability (e.g.: test scores) and earnings. Primary enrolment rates have grown substantially in Papua New Guinea as with other developing countries since the Education for All declaration in 2000, but what matters for productivity, and more generally human capital, is not a child’s attendance in school but rather the skills that he or she acquires in school. Recent research decomposes human capital into of cognitive, non-cognitive and technical skills. The link between earnings and cognitive skills has been studies in many countries, a recent review by Patrinos and Psacharopolous (2010) finds that a standard deviation increase in cognitive ability corresponds to approximately 17 to 22 percent higher earnings on average (Table 9). In Papua New Guinea, a 17 to 22 percent increase in a 6 year-old’s future annual earnings would have a net present value of PGK 11,426 to 15,174 (USD 3,605 to 4,789) discounted at 5 percent, under certain assumptions and modeling19. Using this benchmark, an education investment that increases cognitive skills by a standard deviation could cost up to PGK 11,426 (USD 3,605) and still be worthwhile.                                                              19 A 6 year-old’s future annual earnings are predicted using a Mincer earnings function for age 18 to 65 assuming the child has an average years of schooling. No earnings are assumed before age 18.   33   Table 9. Summary of returns to cognitive ability Estimated Country Effect Source Chile 0.17 Patrinos and Sakellariou (2007) Ghana 0.14–0.30 Glewwe (1996) Ghana 0.05–0.07 Jolliffe (1998) Kenya 0.19–0.22 Boissiere et al. (1985), Knight and Sabot (1990) Pakistan 0.12–0.28 Alderman et al. (1996) Pakistan 0.25 Behrman et al. (forthcoming) South Africa 0.34–0.48 Moll (1998) Tanzania 0.07–0.13 Boissiere et al. (1985), Knight and Sabot (1990) Average 0.17-0.22 Source: Table 7 in Patrinos and Psachoropolous 2010   83. The Reading Booster pilot program in component 3 was evaluated using a randomized controlled trial; under certain modeling assumptions, this program would yield an internal rate of return of 23 percent. The Reading Booster program provided remedial courses in these skills for grade 3 and 4 students. The program included teacher and student support materials as well as training, monitoring and evaluation of teachers as the implemented the program. 4,272 students were randomly selected to participate in the program. A follow-up early grade reading assessment revealed varying effect sizes across different reading skill domains with an average effect size of 0.24 standard deviations (Figure 2). The total cost of materials and training was PGK 794,243 (USD 250,549) or PGK 186 (USD 59) per student. If a standard deviation increase in cognitive ability were associated with a 17 to 22 percent increase in earnings, and if this impact on cognitive ability persists, then these findings imply that the program has an internal rate of return of 23 to 25 percent20. 84. Classroom libraries and associated teacher training constitute the majority of this project’s investment; international evidence suggests a strong impact on a child’s cognitive ability. For example, Elly et al. (1996) review three interventions in Fiji, Singapore, and Sri Lanka. In Fiji, the intervention exposed year 4 and 5 students in 8 rural schools to an “abundant supply of high interest books” and compared outcomes between a treatment and control group. Tests in the second year of the program revealed an effect size for the intervention of 0.76 to 1.18 standard deviations, and in the year 6 external exams, those who were exposed to the books were twice as likely to pass as before. A similar program in Sri Lanka from 1995 to 1996 revealed an increase in scores from pre-test to post- test of 11 percent for year 4 students exposed to books while only 4 percent for the control group; for year 5 students the increase was 9 percent and 3 percent, respectively. Finally, in Singapore in the late 1980s, students exposed to classroom libraries performed better than students in the control group on 53 out of 65 reading tests performed over three years. The                                                              20 The internal rate of return is estimated assuming the cost of the program occurs at age 6 and that earnings increase from age 18. See foot note 19 for methodology on estimating a child’s future earnings   34   effectiveness of book flood programs like these rests on how children (and teachers) ultimately interact with the reading materials. The classroom library and associated teacher training emphasizes increasing the time spent reading which is seen as crucial for book flood programs to be effective. For example, Anderson, Wilson and Fielding (1988) found that time spent reading was the best predictor for gains in reading achievement between 2nd and 5th grade. Figure 2. Average effect sizes on Early Grade Reading Assessment sub-tests of the reading booster randomized control trials in WHP and Madang Province Average 0.51 Familiar word reading 0.04 Letter sound knowledge 0.58 Initial sound identification % 0.91 0 0.2 0.4 0.6 0.8 1 Source: WHP and Madang Reading Booster pilot programme reports. Figures are averages of the two programmes weighted by the number of treatment group children in the enndline reading assessment. 85. The classroom library investment has extremely low costs per student and consequently a high return to the investment is expected. The classroom library program is financed through component 1 (provision of books and materials) and component 2 (provision of teacher training). An estimated 856,410 students benefited from classroom libraries and 25,120 teachers from teacher training. The total cost of the classroom library component is PGK 26,109,306 (USD 8,236,374) or PGK 30 (USD 12) per student (Table 8). While there was no rigorous evaluation of the program’s impact on test scores, there is evidence to believe that the impact could be substantial. For example, the Fiji program increased cognitive ability by 0.76 to 1.18 standard deviations. The test score on a familiar words test for children in the control group of the reading booster program (see above) increased by approximately 0.8 standard deviations. If a standard deviation of cognitive ability increases earnings by 17 to 22 percent, then a 0.76 standard deviation increase in cognitive ability, given the very low cost of the program, would yield an internal rate of return of 49 and 53 percent, respectively21.                                                              21 The internal rate of return assumes that the program costs are all incurred at age 7 and that earnings begin after age 18 to age 65 as in footnote 20.   35   Annex 4. Grant Preparation and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Name Title Unit Specialty s Lending/Grant Preparation James A. Stevens Senior Operations Officer GEDDR Task Team Leader Dennis Davis Consultant GEDDR Economist Douglas Drew Consultant GEDDR Mgmt. Information David Hill Consultant GEDDR Economist Stephen David Close Human Development Specialist GEDDR Education Stephen Paul Hartung Financial Management Specialist GGODR Financial Ellen Kulumbu  Human Development Officer EACGF Country Operations Myrna Machuca-Sierra Education Specialist GEDDR EGRA Julian Watson Publishing Consultant GEDDR Publishing Sabrina Gail Terry Program Assistant GHNDR Program Support Michelle Lee Program Assistant EACNG Admin Support Evelyn Villatoro Senior Procurement Specialist GGODR Procurement Nanda Gasparini Social Specialist EASNS Safeguards Kevin Macdonald Economist GEDDR Economic Analysis Hanneke van Tilberg Senior Counsel LEGES Legal Counsel Supervision/ICR Binh Thanh Vu Senior Education Specialist GEDDR Task Team Leader Cristiano Nunes Senior Procurement Specialist GGODR Procurement David Bruce Whitehead Financial Management Specialist GGODR Financial Myrna Machuca-Sierra Education Specialist GEDDR M EGRA Franco Russo Senior Operations Officer GEDDR Operations Julian Watson Publishing Consultant GEDDR Publishing Ellen Kulumbu Human Development Officer EACGF Country Operations Danielle Merrett Human Development Economist GEDDR Impact Evaluation Stephanie de Silva Econometrician GEDDR Impact Evaluation John Kalu Human Development Specialist GHNDR Country Operations Kerry Pagau Senior Human Development Spec. GHNDR Country Operations Dodi Doiwa Operations Analyst EACGF Procurement Joy Sagati Operations Analyst EACGF Financial Tasha Sinai Program Assistant EACGF M Program Support Summerville Kave Team Assistant EACGF Admin Support Michelle Lee Program Assistant EACNF Program Support Jane Sprouster Implementation Support Specialist EACNF Operations Support Sandra Beemer Consultant GEDDR ICR Author Kevin Alan Macdonald Economist GEDDR ICR Econ. Analysis Kris McDonall Program Assistant EACNF ICR Co-Author   36   (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only*) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY07 1.23 38.52 FY08 5.23 91.34 FY09 12.82 111.31 FY10 0.00 7.22 FY11 13.54 108.89 Total: 344.86 Supervision/ICR FY11 0.40 7.06 FY12 4.78 98.98 FY13 22.26 74.34 FY14 4.94 36.59 FY15 8.38 78.46 FY16 5.05 48.23 Total: 343.66 * Trust fund resources were also used to support lending (US$213,478.93) and supervision (US$472,911.54).   37   Annex 5. Beneficiary Survey Results Not Applicable   38   Annex 6. Stakeholder Workshop Report and Results Not Applicable   39   Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR Government of Papua New Guinea Department of Education READ PNG PROJECT IMPLEMENTATION COMPLETION REPORT Overview of the READ PNG Project (2012-2015) funded by a grant from the Global Partnership for Education (GPE) 1. Achievement of Program Development Objectives 86. The READ PNG Grant Agreement clearly identified the expected beneficiaries through the Program Development Objective which was: To promote better teaching and learning of reading skills in elementary and primary education. The project supported the goals of the National Education Plan (NEP) and the Universal Basic Education (UBE) Plan. Achievement of the two project outcome indicators is shown below: 87. Indicator 1: Proportion of primary school teachers using supplementary readers during reading instruction in a sample of 1800 classrooms receiving classroom libraries, as defined by classroom observation checklists. In total, 1940 classroom libraries and training surveys were conducted in six provinces (target sample = 1,800). The selection of provinces was based on reports from Provincial Project Steering Committees (PPSCs) indicating that teacher training activities and the delivery of books to schools had been completed. The provincial breakdown is shown below: Province Sample East New Britain 450 Jiwaka 296 Manus 229 Morobe 560 New Ireland 277 National Capital District 128 TOTAL 1,940 88. Main preliminary findings:  81 percent of teachers interviewed reported that they had participated in the READ PNG Classroom Libraries Training Program. The participation rate was particularly high in Jiwaka Province, where 97 percent of teachers interviewed had taken part in the training.  62 percent of teachers reported that they had used at least one of the main instructional strategies covered in the training: Story tagging (a strategy for retelling a story); Hot seat (a comprehension activity); Character profiles; Role plays; Book reviews.   40    Class reading timetables were observed in 28 percent of classrooms visited, with 61 percent of teachers reporting that they allocate for class reading sessions, with times ranging from 10 minutes to over 30 minutes.  60 percent of the 5,500 students interviewed in the six provinces reported that they read or used books from their classroom libraries. 89. The survey shows that classroom libraries have been established in almost 61 percent of the classrooms visited in the six provinces. However, it was apparent that in some schools the classroom library books were being used but were not observed in classrooms. This was due to several reasons. The monitoring team reported that when they visited schools in the afternoon, some teachers had already packed up their classroom library books and placed them in secure overnight storage. It is clear that the security of books is a concern for many teachers. In other schools, teachers were under the misapprehension that book shelves were required before establishing their classroom libraries. Teachers were informed that this is not the case, and were shown the Teacher Guide which includes suggestions on setting up a classroom library without shelves. 90. There are encouraging signs from the classroom library training program, with over 60 percent of teachers reporting that they had used one or more of the teaching strategies from the training. This shows that many teachers had found ways of using the classroom library books for instructional activities, and not just as supplementary resources. It also indicates that teachers are applying the new knowledge and skills acquired at the training, and the training has made a difference to their classroom practice. This is a very positive sign for the sustainability of the program. 91. Indicator 2: Increase in the proportion of students’ scores in 3 EGRA sub-tests (phonemic awareness, letter-sound identification and familiar word reading) in pilot provinces (to be reported by gender). In 2014, the DoE used the results from the EGRA survey in Western Highlands to pilot a reading intervention. The purpose of the intervention was to improve the basic reading skills of Grade 3 children. The pilot used a treatment and control design: teachers in treatment schools were provided with training and resources, while control school teachers were only monitored. 92. The results showed big improvements in children’s reading over a period of just 6 months. The table below shows the increases in students’ scores in the 3 EGRA sub-tests (phonemic awareness, letter-sound identification and familiar word reading) from baseline to endline. The results show that in all three sub-tests, boys and girls in treatment schools showed more improvement than students in control schools.  Boys Girls Treatment  Sub‐test Measure   Baseline Endline Baseline Endline group Initial  sound  % correct initial  sounds Treatment  43% 80% 43% 82% identification Control 46% 48% 40% 47% Letter sound  Correct number of  letter  Treatment  23.83 49.13 21.09 56.59 knowledge sounds per minute 23.64 24.80 23.71 25.54 Control Familiar word  Correct number of   Treatment  17.50 31.57 16.60 39.09 reading familiar words per minute 16.48 27.22 16.14 28.92 Control   41   93. The intervention had a significant impact on students’ reading skills. The program’s impact can be defined as ‘the changes between baseline and endline in the treatment schools which can be directly attributed to the program’. The impacts over a short period of time included:  The proportion of fluent readers with good comprehension increased from 4 percent to 25 percent in treatment schools.  Big improvements in basic reading skills. For example, on the Initial Sound Identification test, average scores in treatment schools improved by 87 percent from baseline to endline, compared to 10 percent in control schools. On the Letter Sounds test, average scores in the treatment group increased by 134 percent from baseline to endline, compared to just 6 percent in the control group.  The program helped children with very weak reading skills. By the end of the program, just 4 percent of children in the treatment schools failed to identify any letter sounds, compared to 32 percent of children in control schools. 2. Progress by Components/Sub Components: 2.1. Component 1: Increasing the Availability of Books and Learning Materials 94. Classroom Libraries established in all primary classrooms (Grades 3-8). Sets of supplementary readers have been provided for primary schools in order to help teachers establish classroom libraries. The sets include 60 titles of graded English-language readers, containing a mixture of fiction and non-fiction texts. Around 21,711 classroom libraries have been established against a Results Framework target of 18,000 grade 3-8 classrooms. An estimated 856,510 students in grades 3-8 now have access to classroom libraries books, against the target figure of 720,000 students. 2.1.2. Elementary Literacy Materials 95. Under the READ PNG project, all elementary schools in the country will receive:  One resource kit containing small chalkboards, flashcards, beads, dice, pattern blocks, linking cubes, pocket display board, silkscreen printer and a Teacher Guide  A selection of Wall Charts for their classrooms showing grammar, punctuation, weather, health, map of PNG, etc. 96. Service providers have been contracted to deliver the resources to elementary schools in each province. Delivery of the resource kits to schools commenced in October 2015. Learning resource kits are available for use in 22,254 classrooms against a target of 15,000 classrooms, grades EP, E1 and E2. 2.1.3. Elementary Vernacular Local Language Books 97. READ PNG has provided vernacular (local language) reading materials for elementary schools. It is estimated that over 840 languages are spoken in PNG with orthographies developed in about 400 language groups. NDoE’s new language policy states that elementary schools may teach vernacular as a subject, provided the orthography is approved and the use of vernacular is supported by the BoM, local community and the PEB (see National Policy Statement No. 01/2013).   42   98. All elementary schools will receive sets of bilingual readers (English and Tok Pisin) while selected other schools will be given sets of local vernacular language readers. The readers are appropriate for students in elementary grades and all the books have been published by local PNG publishers and writers. Delivery of the readers to elementary schools commenced in October 2015. Readers are available for use in approximately 15,027 classrooms against a target of 2,500 classrooms in grades EP, E1 and E2. 2.2. Component 2: Promoting Reading through Professional Development, Teacher Networks and Public Reading Campaigns 99. READ PNG has provided funding for the training of primary school teachers on establishing classroom libraries. A Training of Trainers (ToT) model has been used, with a team of national trainers utilized to train teams of provincial trainers, who take responsibility for facilitating teacher training at the cluster or school level. To support the training, a standardized training package was developed including a Facilitator Guide, Teacher Guide and training video. After completing the training, Provincial Project Steering Committees (PPSCs) submitted reports and acquittals to the READ PNG project office. 100. A summary of teacher training activities completed by the READ PNG project is shown below: Level Number trained Target National coaches 2 teams, comprising 18 2 teams of coaches coaches Provincial trainers 22 provincial teams, 22 teams comprising 1,334 trainers Teachers (primary grades 3-8) 25,120 teachers 25,000 2.3. Component 3: Strengthening Early Grade Reading Assessment (EGRA) 2.3.1. Building Capacity to plan and assess early grade reading 101. The Early Grade Reading Assessment (EGRA) is a tool for monitoring reading performance in the crucial early years of education. EGRA is administered through face-to- face oral interviews and is designed to report on foundation levels of student reading including: recognizing letters, reading simple words and understanding short stories. NDoE and the READ PNG project have completed four provincial EGRA surveys in PNG: in Madang, NCD, East New Britain and Western Highlands. The full Diagnostic Results Reports were launched by the Minister for Education in September 2015. 102. A total of 19 (10M/9F) NDoE officers directly benefited from EGRA capacity building activities, including the development of the EGRA diagnostic instrument, supervising data collection in schools and marking and coding the assessments. In addition, 141 student teachers (60M/81F) from four Primary Teacher Colleges were trained as EGRA enumerators.   43   2.3.2. Building capacity to plan and pilot reading interventions to improve reading instruction 103. The EGRA results have helped NDoE to identify instructional needs. Two reading interventions were piloted in Madang and Western Highlands Province. Again, NDoE officers from the Curriculum Development Division (CDD), Standards Division and Teacher Education Division (TED) benefited from capacity development activities. In total, 13 NDoE officers (5M/8F) participated in the Madang intervention, with 15 NDoE staff (8M/7F) participating in the WHP intervention. NDoE officers benefited from various capacity building activities, including resource development, in-service teacher training and monitoring activities. This has had a direct impact on the development of the new Standards Based Curriculum (SBC) with elementary CDD officers reporting that EGRA and the interventions provided the basis for the promotion of teaching English as a subject in elementary schools, with the skills tested in EGRA embedded in the new SBC English curriculum. 104. In total, 116 primary school teachers were involved in capacity building activities in Madang (11M/45F) and Western Highlands (13M/47F) provinces. Teachers were given training and resources to support reading instruction as part of a reading booster program. The ultimate beneficiaries of this program were students who benefited from improved reading instruction. In Madang, an estimated 1,952 students (1,037M/915F) participated in the program while 2,320 students (1,168M/1,152F) in Western Highlands benefited. 2.4. Component 4: Project Management and Monitoring 105. READ PNG established Provincial Project Steering Committees (PPSCs) in all 22 provinces to coordinate project activities at the sub-national level. The project established PPSC bank accounts in order to provide funds for project activities such as teacher training. PPSC representatives from the READ PNG Implementation Completion Workshop (12-13 November 2015) commented that they felt that they had strong ownership of the project and the establishment of PPSC bank accounts had generally worked well. PPSC members were given training on how to acquit funds and report to READ PNG. 106. Monitoring and evaluation systems have been strengthened at provincial level. For example, the PPSCs in East New Britain and West New Britain requested to use unspent READ PNG funds to conduct classroom libraries monitoring activities in their provinces. Meanwhile, the Provincial Education Advisor in Madang allocated an additional PGK30,000 of provincial funds for classroom libraries monitoring. This is positive news for sustaining the project as these provinces have used their initiative to conduct these monitoring activities. 107. Financial Management systems have also been strengthened at provincial level. The PPSCs, the TOTs and classroom teachers have all been trained to appropriately receive and acknowledge project funds, acquit and report on the expenditure within the required reporting timeframe. READ PNG Project Office holds all records of reports which include acquittals and supporting documents that fully comply with the financial requirements of the project. In addition to this, the project has undertaken internal and external audits as part of the implementation requirement which has involved capacity building not only at the project and divisional level but most importantly at the provincial level.   44   3. Key Factors Affecting Implementation of Project Activities 108. The project has experienced numerous challenges in the implementation of activities, particularly in components 1 and 2. It is important that NDoE learns from these experiences to avoid similar issues in the future. 3.1. Component 1: Increasing the availability of books and learning materials 109. A range of procurement and financial management issues affected the implementation of project activities some of which are listed below:  Lack of GoPNG counterpart funding has delayed implementation of critical project activities, particularly teacher training and delivery of materials to elementary and primary schools.  Increased school enrollment due to the government policy of eliminating school fees resulted in shortage of classroom library books for schools in Southern region and two districts in Eastern Highlands Province because the original plan was made prior to the free education policy.  Delay in awarding of contracts by the Central Supply and Tenders Board (CSTB) for the provision of the elementary resource kits.  Delay in approval for customs clearance of materials delaying the delivery of materials to schools.  Procurement of vernacular reading materials delayed for various reasons, including: (i) evaluation committee unable to convene; and (ii) very few publishers of vernacular early reading books. 3.2. Component 2: Promoting reading through Professional Development, Professional Teacher Networks and a Public Reading Campaign 110. Similar to the other components, this component also faced a number of challenges, a few listed below:  Identifying an effective mode of training modality to use. The modality finally agreed upon is shown below:   45      Communicating with provinces to organize and arrange logistics for training to take place.  Mobilizing participants from schools, districts to attend training at nominated central location 3.3. Component 3: Strengthening Early Grade Reading Assessment (EGRA) 111. No major challenges were faced in terms of conducting the 4 EGRA diagnostic surveys. However, some challenges were experienced in implementing the pilot intervention in Madang province in 2013. Due to problems of an inadequate sample size (caused by school closures and problems in accessing some remote schools) there are concerns over the validity of the data. Nevertheless, the Madang pilot provided some valuable lessons which informed the successful intervention in Western Highlands. 3.4. Component 4: Project Management and Monitoring 112. In managing READ PNG, some of the challenges faced included:  Lack of procurement and financial management at commencement of project implementation which delayed major procurement activities and submission of withdrawal applications   46    The lengthy procurement processes for recruitment of consultants caused delays in the engagement of specialist expertise and technical input for the Project. Additional delays caused by lack of capacity demonstrated to not only recruit suitable key personnel but also lack of capacity to procure goods and services. Lack of support by existing key personnel contributed to delays in implementation within work plan and timeframe.  From project commencement, there was a lack of capacity to establish Financial Management systems and controls which contributed to great delays, in particular payment of goods and services, non-compliance to World Bank and GoPNG financial management systems. The limited capacity regarding financial management procedures caused difficulties in the acquittal of project funds. Delays inhibited and prevented the implementation of scheduled tasks on the project until later on during the 3rd year of implementation when the current Project Accountant was recruited. 4. Cross-cutting themes  Gender - Results from EGRA diagnostic surveys and post-assessment interventions disaggregated by gender.  Indigenous representation - The East New Britain EGRA survey was conducted in a vernacular language (Kuanua). The project also provided vernacular language elementary reading materials for 8 vernacular language groups: Motu, Telei, Mekeo, Maututu, Yom Kawac, Naliku, Kiwai, Biga Kilivila.  Environmental issues - One of the criteria for the evaluation of classroom library books was environmental sustainability. 5. Sustainability  There is evidence of substantial capacity development at NDoE, particularly in the Curriculum Development Division (CDD) where the findings from the EGRA surveys and the pilot reading interventions have informed the development of syllabus and teacher support materials for the new Standards Based Curriculum (SBC). The training of elementary teachers has followed the Training of Trainers model used by READ PNG, and numerous provinces have used their PPSC bank accounts to manage funds for SBC training. These are major success stories and bode well for the long-term sustainability of systems and processes established by the project.  Well-established initiatives such as School Learning Improvement Plans (SLIP) are a vehicle for sustaining project activities at the school level. Schools can use SLIP to plan and allocate funds to purchase additional classroom library books and for on- going teacher professional development.  Although not a specific project objective, a pre-service teacher training module based on classroom libraries has been developed as part of the Language Support Program. This is a positive sign as it shows that elements of the classroom library concept are becoming institutionalized.  Although there is no GoPNG funding for READ PNG activities to continue in 2016, the Curriculum Development Division budget includes funds to maintain project activities. This shows a strong commitment by the NDoE to sustain the successful elements of READ PNG, such as the classroom library program.   47    All teacher resources developed by READ PNG are available on the project’s website http://rpng.education.gov.pg/ .  Establishing a new project which builds on the lessons and capacity of READ PNG. The results of the WHP reading intervention showed a direct, causal link between the teacher training programs and improved reading scores. There is potential to replicate this pilot program in other provinces in Papua New Guinea. The successful elements of the program – participatory training, feedback and support, monitoring – can be used to establish standardized, national in-service programs throughout PNG. This will support the overarching goal of new National Education Plan (2015-2019) which is Quality Learning for All. 6. Summary of Lessons learned 113. The experiences of the READ PNG project provide many lessons for future projects in the education sector. Participants at the Implementation Completion Workshop held on 12-13 November 2015 also provided valuable feedback and learning. Some of the main lessons learnt from the implementation of the project are shown below. • Initial identification and establishment of projects - Project awareness is important and the objectives of project must be shared with all partners (provinces, government departments, development partners, etc.). Prior to implementation, READ PNG visited every province to discuss plans and strategies with provincial education officers. • Financial management –GoPNG counterpart funding is essential at the start of future projects. The establishment of PPSC bank accounts was a success, although some provinces reported delays in receiving funds. It is also important that PPSCs acquit their funds in a timely manner. • Procurement - Clear, step-by-step processes for all procurement activities need to be established before the project begins implementation. Good quality, technical support from a procurement specialist was very helpful to the READ PNG project. 7. Conclusion 114. Despite facing some challenges, the achievements of the READ PNG Project are impressive. The project has provided reading resources and learning materials to children across the country, while the pilot intervention in Western Highlands showed significant improvements in children’s reading skills which can be directly attributed to the project. Capacity building at NDoE has been significant, with the project informing the development of a new national curriculum and the accompanying teacher guides. The successful elements of the program (e.g. classroom library program, PPSCs) can become institutionalized into NDoE systems, while the pilot interventions have the potential to be scaled up across PNG. The project has provided valuable lessons to guide NDoE on how to successfully implement future education projects.   48   Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders No additional comments have been provided.   49   Annex 9. List of Supporting Documents Project Appraisal Document, World Bank, 2011 Aides-memoire and Implementation Status Reports 2010 – 2015 World Bank Country Assistance Strategy 2013-2016 Project Paper for a proposed restructuring, World Bank, 2014 World Bank Country Partnership Strategy 2012-2016 Madang Early Grade Reading Assessment Survey: 2011 Diagnostic Results Report, World Bank, 2011. National Capital District (NCD), Early Grade Reading Assessment Survey: 2012 Diagnostic Results Report, World Bank, 2012. East New Britain (ENB), Early Grade Reading Assessment Survey: 2012 Diagnostic Results Report, World Bank, 2012. Western Highlands Province, Early Grade Reading Assessment Survey: 2013 Diagnostic Results Report, World Bank, 2013. Anderson, R., P. Wilson, P., and L. Fielding (1988). Growth in reading and how children spend their time outside of school. Reading Research Quarterly, 23: 285-303. Elly, W., B. Cutting, F. Mangubhai, and C. Hugo (1996). Lifting Literacy Levels with Story Books: Evidence from the South Pacific, Singapore, Sri Lanka, and South Africa. Paper presented at the World Conference on Literacy (Philadelphia, PA, March 12-15, 1996). Montenegro, C. and H. A. Patrinos 2014. “Comparable estimates of returns to schooling around the world.” Policy Research Working Paper Series. No. 7020. Washington, D.C.: The World Bank Patrinos , H. A. and Psacharopoulos, G. 2010. “Returns to education in developing countries”. In: P. Peterson, E. Baker, and B. McGaw, (Eds), International Encyclopedia of Education. Volume 2, pp. 305-312. Oxford: Elsevier.   50   MAP     51