ICRR 12925 Report Number : ICRR12925 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/20/2008 PROJ ID : P051838 Appraisal Actual Project Name : VN-Primary Teacher US$M ): Project Costs (US$M): 35.75 32.2 Development Country : Vietnam Loan/ US$M ): Loan /Credit (US$M): 19.84 15.9 (SDR15.6million) (SDR11.7million) Sector Board : ED US$M ): Cofinancing (US$M): 10.29 9.5 Sector (s): Primary education (57%) Tertiary education (40%) Central government administration (3%) Theme (s): Education for all (100% - P) L/C Number : C3594 Board Approval Date : 12/20/2001 Partners involved : DFID Closing Date : 12/31/2005 06/30/2007 Evaluator : Panel Reviewer : Group Manager : Group : Gayle Martin Fernando Manibog Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: The objective of the project was to "lay the foundation for a nationwide program to upgrade the quality of the primary teaching service" (PAD, p2), and is generally consistent with the objective as stated in the DCA (DCA, p12). b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): A. Development of Teacher Professional Standards (appraisal estimate: US$1.01 million/3% of estimated cost; actual cost: US$0.65 million/2% of actual cost) for the development of teacher professional primary education standards (teacher profile); support for a Profile Coordination Team; application of the profile to training courses; application of the profile to teacher assessment; and action research to refine the profile . B. Training Programs for Teachers and Education Managers (appraisal estimate: US$26.15 million/73% of estimated cost; actual cost : US$25.95 million/81% of actual cost) support for a Training Coordination Team; needs assessment and specification of the content of training packages; preparation of trial training packages; improving the capacity of teacher training institutions (TTIs), provinces and districts; appraisal of training packages and publishing, and delivery of training modules . C. Quality Assurance (appraisal estimate: US$3.68 million /10% of estimated cost; actual cost : US$1.90 million/6% of actual cost) for: the support for a Primary Education Quality Assurance (QA) Team; improvements in the personnel development information system (PDIS); development of accreditation and teacher classification procedures; communication and training for school improvement . D. Revision of terms of service for teachers (appraisal estimate: US$0.5 million /1% of estimated cost; actual cost : US$0.4 million/1% of actual cost) for technical assistance and research; preparation and discussion of options and supporting changes; and consensus building activities . E. Project Coordination, Capacity Primary Education Strengthening, and Monitoring (appraisal estimate: US$4.41 million /12% of estimated cost; actual cost : US$3.30 million/10% of actual cost) for project oversight; creation of a project coordination unit (PCU) and professional provincial and district teams; monitoring and evaluation; and preparation for program expansion . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Co st. st Actual project cost (US$35.75 million) was 90% of cost at appraisal (US$32.2 million). The component with Cost the largest variance between appraisal and actual was Component C : Quality Assurance for Primary Teachers (actual component cost was US$ 1.9 million, 52% of appraisal cost). Financing . At closing the IDA credit was 80% disbursed (actual: US$15.9 million). Borrower contribution was 98% of the appraisal estimate, and the DFID's contribution was 92% of the appraisal estimate. Disbursement rate was slow---by the original closing date (December 2005), only 37% of the Credit was disbursed. Dates The project became effective as planned (08/29/2001). By the MTR (November 2004), only US$2.1 million Dates. was disbursed (10% of the credit). The closing date was extended by 18 months (to 06/30/2007)---according to the ICR, primarily due to delays in the development of training materials (under Component B) which in turn resulted in delayed teacher training using the newly developed training materials . 3. Relevance of Objectives & Design: Relevance of Project Objectives . The relevance of the objective is rated as high. high The emphasis on primary education was consistent with government ’s successive national development strategies and education strategies . It was also consistent with the poverty reduction themes of the Bank ’s 1998 and 2003 CASs and remains relevant with the second pillar of the 2007-2011 CAS: "Strengthen social inclusion, by enhancing human resources, assets and opportunities for the poor and vulnerable " (CAS, p23). The focus on primary education complements the priorities of other donors, and the emphasis on quality of education is furthermore appropriate given the gains in expanding access in years preceding the project . The project focused on teacher development, complementing the government of Vietnam (GOV) software inputs (curriculum development) and infrastructure investments in teacher institutions to realize the broader objectives of the education strategies and curriculum reforms (PAD, p14). Relevance of Project Design . The relevance of project design is rated as high. high The design benefitted from lessons from the on-going primary education project and the Bank ’s analytical work (e.g., the public expenditure review ). The design combined DFID and Bank support in a single project, and is said to have reduced the transaction costs to the GOV. The design positioned the reforms within the context of broader civil service reforms, but this also added an additional challenge. The complexity of what otherwise was a relatively simple project design was increased by the interrelatedness of the project components with : other Ministries (e.g., civil service reforms, salary scale adjustments); with numerous departments within the large Ministry of Education and Training (MOET) bureaucracy; and with other Bank and donor -financed projects. Some interventions were highly controversial (e.g., revision of terms of service and salary scales ), and design features were added to mitigate some of this risk : e.g., the multisectoral Steering Committee. In part because the project -financed reforms were quite far reaching, the project focused on 10 of the country’s 64 provinces and large cities. The criteria for the choice of provinces (e.g, regional, social and ethnic diversity; variation in qualification of teachers; variation in quality of TTIs (PAD, p9)) aimed to get sufficient variation to ensure optimal learning from this project . Overall, relevance is rated as high. high 4. Achievement of Objectives (Efficacy): Objective : to lay the foundation for a nationwide program to upgrade the quality of the primary teaching service Outputs . The project financed outputs in the following areas : Professional Standards for Primary Teachers : - Development of Primary Teacher Professional Standards (teacher profile), conducting a trial (involving 2,200 th teachers) and subsequent pilot (involving 25,000 teachers) in 10 project provinces, culminating in the 5 revision which was subsequently approved by MOET . - Drafting of Regulation on Primary Teacher Professional Standards (promulgated by MOET as Decision 14/2007) and development of implementation guidelines for the regulation . - Completion of 30 research projects on the development and application of professional standards . Training Programs for Teachers and Education Managers : - Teacher training curriculum development and revision for College -level (12+3) pre-service training (promulgated as Decision No. 17/2004), upgrading teachers from 12+2 status to College-level (promulgated as Decision No. 4578/2004); pre-service university-level teacher training (promulgated as Decision No. 39/2006); and identifying training requirements for teachers below standard to achieve College -level qualifications. - Teacher training module development : 32 modules (exceeding the targeted 24) were developed including the accompanying books and videos; approval of all 32 modules by MOET (exceeding the targeted 14). - Delivery and use of training modules : training was provided for 276,746 person-modules in project provinces (9,641 trainer-modules, 216,653 teacher-modules, 50,452 principal-modules) exceeding the targeted 250,000 person-modules; 5,982 teachers completed the 18-month Upgrading Program provided at TTIs . - Capacity building in TTIs and the provincial Department of Education and Training (DOETs): only 73 masters degrees successfully completed (of the targeted 150); 343 people completed the one-month structured training in teacher management and teaching methods in Canada, Australia and New Zealand (replaced the intended 56 overseas masters degree training ); equipment packages (including computers, physical education, music and art equipment; etc.) were provided for TTIs and TTIs resource centers were refurbished . - In-service training: Distance education provided for managers and teachers using in -service modules; 154 television programs produced of which 50 were telecast twice weekly; and were replayed by resource centers . - Module printing: 2.7 million units of print and video materials were mass produced and distributed . Quality Assurance: - a QA framework was developed and QA, a new concept at start of project, was applied across the whole project through workshops, training and working collaboratively across components . - PDIS: system development and revision of PDIS based on trials and workshops; database store information on 58,361 primary teachers in the 10 target provinces; PDIS training of 2,198 principals and 50 and 318 officers at DOET-level and Board of Education and Training (BOET)-level, respectively. - Teacher classification: PDIS had the capacity to include data on teacher titles (developed under Component D) and professional standards (developed under Component A) but the two concepts were not successfully integrated . - TTI accreditation: a QA framework was developed for College -level teacher training, criteria and procedures for self and external appraisal were developed . The legal accreditation entity was specified under Decision 75/2006 and submitted to the Department of Testing and Accreditation in MOET for promulgation . - Communications: Information about the project and its reforms were widely disseminated in the professional and public media. Revision in the Terms of Service for Teachers : - Following survey research and extensive consultation with authorities and other stakeholders (including unions) three titles were agreed for primary teachers (promulgated by Ministry of Home Affairs as Decision 61/2005); despite being controversial, modifications to the salary scale were proposed and implemented through Joint Circular 81/2005 of the Ministry of Home Affairs and the Ministry of Finance; - Surveys and other research were conducted on the workload of primary teachers including variations by school (level, size, location, number of sessions ) and class (full day schooling, multi-grade classes, ethnic composition, maternity provision). Following extensive consultation, the project recommended changes that were then implemented through Joint Circular 35/2006, of MOET and Ministry of Home Affairs. - Workshops on options for school -based management were conducted, but new arrangements to devolve decision-making, as anticipated in the PAD, were not proposed . M&E and Planning capacity: The project financed IT training for managers and database maintenance . Activity tracking and monitoring of outputs were implemented using PDIS software as well as qualitative techniques . Outcomes and Impacts . Some of intermediate outcomes were: - TTIs in 10 provinces underwent self-accreditation although only 4 of these TTIs underwent external accreditation . - The proportion of "unqualified teachers" estimated in the PAD to be 20% country-wide, was reduced to 3% in the project schools at closing . - PDIS was reported to be widely used to analyze the data for planning and policy decisions especially at provincial level, although use at BOET-level was modest---only half of BOET-level managers reported use of PDIS . - Equipment use was extensive in TTIs; but only 55% of general facilities reported use of project -financed equipment. - Revision of terms of service included the three -title structure which was well received by teachers . But the titles are not yet linked to professional standards (the profile was in draft form when the decision on titles was promulgated ). - The title structure was also linked to salary scales . New salary coefficients were linked to increments in the scale, with ceilings set by teachers ’ titles. Although all 400,000 primary teachers benefitted from adjustments in the salary scale, the teachers who upgraded to College -level through the project (almost 6,000) were significant beneficiaries. A detailed classroom observation study was conducted that gave some insight into the teacher knowledge and competency levels across 5 categories (ICR, p20) and information on teacher pedagogical skill levels (ICR, p21). While the results may be suggestive of impact, as the ICR asserts, it is hard to attribute these to the project without comparison with non-project schools or another study in the same schools to compare trends over time . Although the ICR presented evidence on Mathematics and Vietnamese Reading test scores, it is mot possible to credibly attribute changes in test scores to the project -financed outputs. (Furthermore, even if relevant, the improvement in test scores is insufficient to demonstrate impact, but rather that the improvement in project provinces exceeded the improvement in non -project provinces.) Overall, efficacy is rated as high. high 5. Efficiency (not applicable to DPLs): No rate of return was calculated (in PAD and ICR). The project design was efficient in the way it implemented the reforms in a subset of provinces before expanding the reforms . The project combined Bank and DFID financing, thereby reducing transaction costs to the government . There were several unintended consequences, contributing to the returns to the project investments in the long run . There were, however, some factors that detracted from the efficiency of the project investments : There were substantial delays in the project (see section 2d). The delay was in the largest component (Component B comprised 81% of project costs), causing disbursement to be well below expected levels . Components were inter-related, and the long delay in Component B implementation detracted from efficient implementation of other aspects of the project. Lastly, one of the sources of sectoral inefficiency identified in the PAD was the relatively low pupil /teacher ratio suggesting under-utilization of teachers (PAD, p17). The ICR did not report on whether this ratio had changed over time. Efficiency is considered substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on the high relevance of the objectives and project design, the high efficacy and substantial efficiency, the project outcome is rated satisfactory . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome is rated low. low There was a high level of ownership of reforms among various stakeholders and a high degree of complementarity between GOV ’s investments in education e .g., into curriculum development and infrastructure investments in TTIs . This, together with the decision to appoint a Steering Committee to oversee the project, secured the acceptance and ownership of the various parties affected by the project reforms, and helped mitigate key risks identified in the PAD (pp22-24). The fact that many reforms were promulgated as regulations suggests high likelihood of sustainability . Further evidence of high ownership and low risk to sustainability is the protection of public education spending from the fiscal pressures experienced during the East Asia crisis, and the commitment by GOV to further increases (from 17% of budget in 1999 to 20% in 2010 (PAD, p15). a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: at -entry . The PAD provided a detailed articulation of the project activities . A lesson incorporated from Quality -at- the on-going primary education project was the need for sequencing of reforms and this guided the design of : teacher training material development, improved pre - and in-service training, and changes in teachers ’ conditions of service linked to qualifications and performance (although some sequencing challenges remained ). The Bank's analytical work (e.g., the public expenditure review ) and studies during project preparation provided the evidence-base for project design. Examples of studies include: national census of school and their teachers; study on pre- and in-service training. Project preparation included extensive consultations with variety of stakeholders. Although the project had a PCU it was highly integrated into the rest of the MOET structures . One weakness of the design was that the implementation arrangements for devolution of decision -making to principals and the political economy thereof could have been better thought through . Quality of supervision . There was a high level of continuity in Bank supervision, and a close working relationship between the Bank team and the PCU . Supervision missions were biannual with continued follow -up and oversight inbetween missions. Missions included technical specialists in selected areas (teacher standards, teacher development, curriculum reform, materials development etc .). The Bank, however, has to share some of the responsibility for delays,especially early in the project . at -Entry :Highly Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Government Performance . The high level of government ownership and commitment to the project was demonstrated by the active participation of the high -level Steering Committee under the leadership of the vice Minister of Education. According to the ICR the active participation of the Steering Committee members materially contributed to the success of the project and contributed to the policy environment for implementation of the teacher quality reforms. Implementing Agency Performance . A PCU was created in the MOET, implementing agency . According to the ICR, the PCU was closely linked with other departments in the MOET, as well as necessary structures outside the ministry. Provincial PCUs were created to facilitate decentralization of selected project functions . The PCU staffing was stable and competent . The PCU had the operational experience from two previous Bank -financed projects, providing support to the provincial PCUs . Nonetheless, there were still some capacity -related procurement delays, and the project had a slow start, with only 10% of credit disbursed at MTR. The PCU did well to implement splitting of the component implementation arrangements for Component B (separating module development from training delivery ). The project management facilitated the links across components. Coordination worked well in some areas, but in some instances did not : e.g., the failure to include professional standards in the award qualifications for teacher titles (under Component D) that were approved through the Ministry of Home Affairs. The beneficiary survey complimented the PCU and provincial PCUs for : coordination across various levels of the education system, flexibility during implementation, the high degree of support provided to the local level facilitating implementation by the provincial authorities (ICR, Annex 5 p7). According to the ICR the MOET had a high level of commitment to the project's objective . MOET approved various policy reforms as anticipated in the project documents ---examples include: approval of the Professional Standards for Primary Teachers and promulgation of the necessary regulation; passing of various regulations pertaining to teacher training standards; regulations revising teachers' terms of service (together with Ministry of Home Affairs, Ministry of Finance, Ministry of Planning and Investment ) were passed. a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: Design . The results framework in the PAD included a high level of detail . The results framework included numerous indicators, and this was subsequently simplified . Because the focus of the project was to develop, test and refine the reforms, the KPIs were understandably output - and process-oriented. The design proposed extensive use of action research and studies to inform the development and revision of the envisaged reforms . The explicit inclusion of a component on quality assurance was a strength of the project design, explicitly incorporating evaluation and evidence-based implementation in the design . Implementation and utilization . The Project supported the strengthening and establishment of monitoring systems (PDIS) and training in its implementation (targeting teachers, managers as well as PCU staff ). Progress reports were regularly produced informing tracking of implementation . In the development of teacher standards (under Component A), two surveys were conducted (respectively involving 2,200 and 25,000 teachers) before finalizing the set of professional standards for primary teachers . According to the ICR the studies financed by the project informed the development of systems and emphasizing evidence -based policy reforms. More detail on how the various studies were used would have been helpful . a. M&E Quality Rating : Substantial 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Environment and social safeguards . The Project had a environmental category C rating, i .e., no specific environmental safeguards were required . Fiduciary safeguards . The PAD identified insufficient financial management capacity as substantial risk (PAD, p23). According to the ICR financial management arrangements at the central level were adequate . Since capacity at the provincial level was limited, hardware, software and training were provided to achieve satisfactory financial management. The project was audited annually and had an external audit at project completion . Early in the project there were procurement delays related to capacity, subsequently resolved . The PCU was proactive to address capacity limitations among potential providers --to enable wide participation in procurement, the PCU provided training to potential providers in submission of bids (resulting in building of capacity to participate in Bank procurements). Unintended outcom es. es There were several unintended outcomes : professional standards are in the process of being developed for other parts of the education system (early childhood development and secondary teachers ); the revision of teacher training curricula and the module development to assist the introduction of the new curriculum were not planned; within the civil service the adoption of a standards -based approach to teacher quality is a new advancement; and the project's communications strategy exposed parents and the public to reforms occurring in schools (e.g., new approaches to teaching and learning ). 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Highly Satisfactory Satisfactory ICR guidelines indicate that for a highly satisfactory rating there had to be "no shortcomings in the operation’s achievement of its objectives, in its efficiency, or in its relevance ." As mentioned earlier, while the achievements were significant, there were some shortcomings, especially in efficiency. Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The lessons below complement the lessons mentioned in the ICR . Reforms are often controversial and there were two key lessons that can be taken from this project : - A proactive communications strategy that provides general information about the project to stakeholders and gradually introduces some of the more controversial aspects of reform to winners and loosers of reform facilitates a balanced public discourse . - A high-level steering group that functions actively and reflects the affected stakeholders in its make -up is a means of gaining acceptance of the reform agenda and securing ownership for the reforms . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provided a detailed assessment and frank distillation of the project’s achievements. The ICR is rated satisfactory with a few caveats : - A correction has to be made before the ICR goes to print and is disclosed : The column "At Approval" (ICR, p ii) in the Data Sheet indicates Alain Barbu as the Country Director "At Approval" in December 2001. - The ICR presented numerous lessons, but many of these were in fact a combination of findings, lessons and recommendations. For examples of findings are: "Teachers appreciated the Professional Profile because it defined their work, set targets for improvement, and offered rewards for better performance; " "Teachers’ attitudes changed through being recognized for their achievements and motivated them to seek self improvement, including through self learning." An example of a recommendation: "Using the PDIS as a starting point, and working with PEDC [primary education for disadvantaged children ] and relevant MOET departments, the Department for Teachers and Managers needs to seek rationalization and effective use of the management information systems currently in use ." a.Quality of ICR Rating : Satisfactory