OFFICIAL DOCUMENTS Supplemental Letter No. 2 REPUBLIC OF KAZAKHSTAN 2017 International Bank for Reconstruction and Development 1818 H Street, N.W. Washington, D.C. 20433 United States of America Re: Loan No. 8709-KZ (Education Modernization Project) Performance Monitoring Indicators Dear Sirs and Mesdames: This refers to paragraph A. 1 of Section II of Schedule 2 to the Loan Agreement of even date herewith between the Republic of Kazakhstan (the Borrower) and the International Bank for Reconstruction and Development (the Bank). The Borrower hereby confirms to the Bank that the indicators set out in the attachment to this letter shall serve as a basis for the Borrower to monitor and evaluate the progress of the Project and the achievement of the objectives Very truly yours, REPUBLIC OF KAZAKHSTAN Aut rized epresentative Attachment Attachment to the Supplemental Letter No. 2 (Education Modernization Project) Performance Monitoring Indicators PDO Statement The project development objective (PDO) is to improve quality and equity in primary and secondary education, particularly in rural and disadvantaged schools These results are at Project Level Project Development Objective Indicators Cumulative Target Values Indicator Name Baseline YRI YR2 YR3 YR4 YR5 End Target By subjects: a statistically Grade 4 EALA significant (reisd) coes n o b masuedinincrease in By subjects: a statistically (revised) scores inTmemesrdi project schools year 3 when the scores as significant increase in scores as iproe byReadi assessment tool compared to compared to baseline improved, by Reading 0.00 0.00TD0.0 bsln (i.e., EALA) is ands/girls Mahmtcrevised in line with Boys/girls: reduction in disparity boys/girls the new standards. Boys/girls: for reading only sreduction in disparity for reading only Percentage of students from project schools benefiting from instruction materials 0.00 0.00 30.00 100.00 100.00 100.00 100.00 and multimedia equipment (Percentage) Percentage of teachers in project schools who demonstrate improved 0.00 0.00 0.00 0.00 60.00 60.00 60.00 pedagogical approaches (Percentage) 2 Intermediate Results Indicators Cumulative Target Values Indicator Name Baseline YRI YR2 YR3 YR4 YR5 End Target Evaluation of the new currculum pilot No Yes Yes Yes Yes Yes Yes completed (Text) Percentage of textbooks that are appraised by National Textbooks Center using revised 0.00 33.00 41.00 66.00 100.00 100.00 100.00 quality assurance system (Percentage) Assessment instruments are designed in line with EALA: Yes EALA: Yes EALA: Yes EALA: Yes 0.00 0.00 0.00 the new standards UNT: Yes UNT: Yes UNT: Yes UNT: Yes (Text) Number of students enrolled in the new model of pre-service 0.00 0.00 0.00 280.00 280.00 280.00 280.00 teacher training (Number) Number of teachers who pass the proficiency test 0.00 0.00 450.00 450.00 450.00 450.00 450.00 in English after training (Number) Percentage of teachers from project schools trained and certified in 0.00 20.00 40.00 60.00 80.00 100.00 100.00 new pedagogical approaches, by gender (Percentage) Percentage of project school administrators who exercise 0.00 0.00 30.00 70.00 90.00 90.00 90.00 pedagogical leadership, by gender 3 (Percentage) Number of regions that: (i) publicly display and discuss school budget in meetings with citizens, 0.00 0.00 4.00 8.00 12.00 16.00 16.00 and (ii) publicly report on meeting discussions and actions to be taken (Number) Percentage of teachers and administrators of project schools received 0.00 0.00 0.00 50.00 75.00 90.00 90.00 support post training (Percentage) Number of project schools equipped with new multimedia 0.00 0.00 5,400.00 5,400.00 5,400.00 5,400.00 5,400.00 equipment (Number) Third party and/or beneficiary monitoring findings disclosed, Actions Actions Actions included in 2 national discussed at open forum 0.00 0.00 included in Actions included in Actions action plans and implemented in and agreed actions national implemented national implemented included in national action plan 1 action plan action plans (Text) Percentage of schools that formulate School Improvement Plans (SIPs) reflecting feedback provided by 0.00 0.00 0.00 35.00 0.00 75.00 75.00 parents and beneficiaries in project schools (Percentage) Percentage of project schools in which parents 0.00 0.00 0.00 35.00 0.00 75.00 75.00 report an improvement in the openness and 4 responsiveness of school teachers and management (Percentage) Indicator Description Project Development Objective Indicators Indicator Name Description (indicator definition, etc.) Frequency Data Source CMethodology Responsibility for D Collection Grade 4 EALA (revised) Measures the extent to which students in Twice, at The External Assessment of NTC/PIU scores in project schools project schools have mastered the Grade 4 baseline and Learning Achievements improved, by Reading and curriculum (revised) in Reading and in Years 3 and (EALA) is administered by the Mathematics: boys/girls Mathematics. The EALA will be revised to 5. National Testing Centre measure the knowledge and skills, particularly (NTC). Pnor to establishing a non-traditional and higher-order cognitive baseline, the instrument will skills, targeted by the revised curriculum. be revised (with project Higher grades are not selected for the indicator support under Subcomponent because the roll-out schedule is such that no 1.2) to be aligned with the higher grade will have been exposed long revised curriculum. The enough to the new curriculum for one instruments will be designed reasonably to expect an impact on learning to discriminate at the lower outcomes, within the project's time frame. (See end of achievement range, so Annex 2, Box 1). The indicator measures the as to pick up on improvements effectiveness of the new curriculum and its achieved by lower-achieving delivery in bringing about higher learning students. Sample-based, with a outcomes particularly in relation to the new, disaggregated structure to nontraditional cognitive skills targeted by the determine an overall value and new curriculum. That is, using instruments a value for rural and constructed with equated items, the baseline disadvantaged schools assessment will test students following the old supported under Component 2. curriculum while the follow-up assessment will test students who have been following the new curriculum. Given the new curriculum's rollout schedule, it is not possible within the Project's time frame to test two different Grade Four cohorts who follow the revised curriculum, with one cohort being tested prior to project interventions and the other being tested after project interventions have been going on sufficiently long to expect learning baeie5h ntuetwl impacts. The target will be measured by incremental values in language and mathematics in national assessments rather than targets. This is because there is no reliable national student assessment trend data that provides information on language and mathematics to inform the setting of targets, particularly given that the indicator measures performance against a new curriculum. Percentage of students from Measures access to learning resources by Twice: Years Administrative data PIU project schools benefiting project schools before and after the project. To 2 and 3 from instruction materials and be measured prior to the start of the school multimedia equipment year. All the resources will be supplied by the end of Year 3. Percentage of teachers in Measures the number of teachers observed to Twice: Years Sample-based. Observations Contracted evaluation project schools who follow revised curriculum and have good- 2 and 3 will be conducted by a party firm/PIU demonstrate improved quality teaching-learning practices, as a independent of agencies pedagogical approaches percentage of all teachers observed. The delivering any of the teacher criteria that constitute a satisfactory level of training supported by the pedagogical skills will be defined by the project. The instrument to be teacher training program development TA. used will be validated and internationally recognized (e.g., class or similar), and will be designed to capture the quality teaching-learning practices targeted by the teacher training. Intermediate Results Indicators Indicator Name Description (indicator definition etc.) Frequency Data Source / Methodology Responsibility for Data Collection Evaluation of the new Evaluation report submitted including Year 1 Based on formal resolutions of MOES/PIU curriculum pilot completed recommendations for system-wide NAE and MOES implementation new curriculum based on pilot results Percentage of textbooks that Based on textbooks review records kept by the Years 3, 4, Independent, sample-based Contracted third party are appraised by National National Textbooks Center, which will be and 5 review of textbooks evaluation Textbooks Center using analyzed to determine if the textbook was reports revised quality assurance reviewed in compliance with the revised QA system system designed by the TA contracted under 6 this component. Denominator: all the textbooks submitted for consideration by the National Textbooks Center. The analysis will be undertaken by an expert and independent third party familiar with the new QA system. Assessment instruments are Two assessment instruments will be designed Years 3, 4, Project records PIU designed in line with the new in line with the new standards: EALA and and 5 standards UNT Number of students enrolled in Refers to the number of students enrolled in Years 3, 4, Project records PIU the new model of pre-service the programs under the new model developed). and 5 teacher training Number of teachers who pass Refers to the number of teachers who complete Years 3,4, and Project records PIU the proficiency test in English the course by passing the proficiency test. 5 after training Percentage of teachers from Denominator is all teachers working in a Years 2, 3, 4, Orleu (the network of national PIU project schools trained and project school. Numerator is number of and 5 in-service teacher training certified in new pedagogical teachers certified in new pedagogical colleges), based on training approaches, by gender approaches records Percentage of project school Denominator is all general secondary School Years 2, 3, 4, Contracted TA. TA will be PIU administrators who exercise Directors in a representative, randomly and 5 responsible for sampling, pedagogical leadership, by selected sample. Numerator is number of design of data collection gender project School Directors in that sample who (instruments and mechanism), are observed to practice improved pedagogical and reporting leadership. The disaggregation refers to those School Directors already working in schools, e.g., "the percentage of female School Directors from project schools that exercise pedagogical leadership" refers to the percentage of those female School Directors already working in the schools. The indicator does not set a target that 90% of School Directors exercising pedagogical leadership will be female (or male); rather it refers to gender parity in terms of results. The criteria that constitute improved leadership will be defined by the TA, contracted under the project. Number of regions that (i) Numerator is the number of schools disclosing Years 3 and 5 Survey PIU publicly display and discuss both budgets and meeting minutes (containing school budget in meetings both discussions and decisions), denominator 7 with citizens, and (ii) publicly is total number of schools. Data derived report on meeting discussions survey-sample-based will be conducted by an and actions to be taken independent third party. Percentage of teachers and Teachers and administrators of project schools Years 3, 4, Resource centers, NAE PIU administrators of project who receive support post training. and 5. schools received support post- training Number of project schools No description provided. No description No description provided. No description provided. equipped with new multimedia provided equipment Third party and/or beneficiary A third-party report will be completed and Years 3 and 5 Third party survey PIU monitoring findings disclosed, publicly disclosed in Year 3 and Year 5, discussed at open forum and discussed at a public forum and a national agreed actions included in action plan produced. The national action plan national action plans is implemented in the subsequent year. All results where appropriate will be gender disaggregated by gender and disadvantaged schools. Percentage of schools that Sample based evaluation. Denominator is all Years 3 and 5 Third party survey PIU formulate School project schools. Numerator is the project Improvement Plans (SIPs) schools which have improvement plan reflecting feedback provided approved by the Board and show evidence of by parents and beneficiaries in including community feedback. project schools Percentage of project schools Sample based evaluation. Denominator is Years 2, 3, 4, Contracted TA. TA to PTU in which parents report an project schools surveyed. Numerator is the and 5 mobilize and train school improvement in the openness number of project schools where parents community members. TA will and responsiveness of school surveyed reported both openness and be responsible for sampling, teachers and management responsiveness of (i) teachers and (ii) design of data collection management. Openness and responsiveness (instruments and mechanism), will be defined through a basket of actions by and reporting. teachers and management. 8