ICRR 14340 Report Number : ICRR14340 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 05/12/2014 Country : Afghanistan Project ID : P089040 Appraisal Actual Project Name : Afghanistan: US$M ): Project Costs (US$M): 40 55.7 Strengthening Higher Education Program L/C Number : CH162 Loan/ US$M): Loan /Credit (US$M): 40 65 Sector Board : Education Cofinancing (US$M): US$M ): Cofinanciers : Board Approval Date : 05/19/2005 Closing Date : 06/30/2010 06/30/2013 Sector (s): Tertiary education (96%); Central government administration (4%) Theme (s): Education for the knowledge economy (67% - P); Administrative and civil service reform (33% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Erik A. Bloom Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: The legal agreement (page 10) states: "The objective of the Project is to assist the Recipient in restoring the basic operational performance of Higher Education Institutions so as to enhance the quality of tertiary education." The objective in the Memorandum and Recommendation of the President (the equivalent of the PAD, page 5) is similar: "The development objective of the project is to progressively restore basic operational performance at a group of core universities in Afghanistan ." A later formulation in the Project Paper for additional financing (2010) states that the objective was to "assist in restoring the basic operational performance of Higher Education Institutions so as to enhance the quality of tertiary education." This review will use the objective as outlined in the legal agreement . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project had two components . Component 1: Program Development and Quality Improvement in Key Higher Education institutions (appraisal: US$ 38.5 million; final amount: US$ 49.6 million). This component was to focus on support to universities, including a university partnership program between six Afghan universities and foreign universities, and block grants to enable these universities to implement activities prioritized in their individual strategic plans . Component 2: Higher Education System Development (appraisal: US$ 1.5 million; final amount: US$ 6.1 million). This component was to focus on strengthening the higher education system and included support for a new governance system and establishing a quality assurance system . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost and Financing: The project was originally approved for US$ 40 million from an IDA grant. The Afghanistan Reconstruction Trust Fund provided an additional US$ 5 million in August 2008 to support four of the six participating universities to enable improvements in their physical facilities . IDA provided an additional US$ 20 million (from the Crisis Response Window) in June 2010 to cover six additional universities . A total of US$5.7 million was cancelled and US$1.2 million was undisbursed. The ICR does not explain the reason for the cancellation. Borrower Contribution: There was no local contribution . Dates: The project was approved on May 19, 2005, with an expected closing date of June 30, 2010. The project received additional financing In August 2008 and again on June, 2010; the latter extended the closing date by three years. The project closed on June 30, 2013. 3. Relevance of Objectives & Design: a. Relevance of Objectives: The relevance of objective is rated high. high At the time of appraisal, Afghanistan was still recovering from significant internal conflict . A massive skills deficit cut across all sectors of the country . While demand for higher education was increasing, most institutions of higher education had experienced no significant quality or quantity inputs in many years; as a result, they were lacking in most measures of basic operational performance . The goal of strengthening higher education in Afghanistan remains a national priority and is also a priority for the World Bank . The ICR indicates that the Afghanistan National Development Strategy continues to give importance to the project's objectives, focusing on the importance of quality and independence of higher education . Likewise, the most recent World Bank Interim Strategy Note, 2012-2014 (page 12) continues to include higher education in all three of its pillars, and the Bank is preparing a follow-on project. b. Relevance of Design: The relevance of design is rated substantial . Given the straightforward goal of restoring operation of higher education, the project's design was relatively simple and well grounded in literature . For universities, the project promoted partnerships between local and foreign universities, to reincorporate Afghanistani universities into the global academic community, to facilitate curriculum development, and to improve local capacity . It also included grants to universities to focus on their own projects based on local planning . At the national level, it focused on developing a governance framework and on establishing a national quality assurance system . The latter is often seen as central to providing quality in autonomous higher education systems . 4. Achievement of Objectives (Efficacy): Based on the objectives included in the legal agreement, this review will focus on restoring the basic functioning of higher education institutions as well as efforts to ensure a minimum level of quality . In practice, these two objectives are closely related and will be evaluated together . Efficacy is rated substantial . Outputs At the national level, the project supported the creation of a National Development Curriculum Committe e, which was a major priority as the curriculum used in most universities in Afghanistan wa s out-of-date, often dating to the 1970s (page 13 of the ICR). The project also supported the establishment of a Quality Assurance Committee that supervised quality assurance and accreditation at the national level (page 16 of the ICR). The project supported the Ministry by recruiting 25 technical specialists in a variety of fields (page 26 of the ICR). The project supported the training of 1,166 staff at the Ministry and universities in management, which represents about 50 percent of administrative staff . This exceeded the original target of 20 percent and the revised (2010) target of 40 percent. The project also supported the creation of University Partnership Programs with 12 foreign universities. These programs helped to support faculty members receiving overseas Master's degrees as well as setting research committees to strengthen the capacity of universities (pages 13 & 25 of the ICR). In addition, the project sponsored short training courses and exchange programs for faculty both in national and foreign universities . At the university level , the project provided block grants that supported 21 construction projects and 17 renovation projects . This and other support led to the renovation of laboratories , the equipping and refurbishing of computer laboratories , and the establishment of student services departments in all project universities . In addition, the project supported the restoration of basic infrastructure (walls, water supply, and bathrooms) and the construction of dormitories in several universities (pages 25-26 of the ICR). Outcomes Overall, the number of university students increased from 12, 28 ,300 in 2013 and the total 12 ,300 in 2005 to 28, number of graduates increased from 1,500 to 11, 11 ,300 in the same time period. It is plausible that the support to physically rebuilding universities made a significant contribution to increasing both enrollment and graduation rates. Restoring basic functioning The data presented above on the university -level block grants and support to infrastructure are also indicators of the project's contribution to restoring basic functioning of higher education institutions . Before the project, there was a total of 25 libraries, 32 science labs, and 3 computer labs in the targeted universities, and most of these were probably not functional . By the end of the project, there were 37 libraries, 84 science labs, and 50 computer labs . All of these were functional, and each university had at least one . As a result of this investment, all universities had at least one functioning academic building by the end of the project. The ICR reports that at the start of the project, few universities had any functioning infrastructure (ICR Data Sheet, Section F(a)). Quality of instruction and faculty Under the project, 100 percent of selected faculties were using revised curricula , which compares to the original target of 65 percent and the revised target (in 2010) of 100 percent. This suggests a significant contribution to revising capacity of universities to teach . The project supported efforts to improve the education of university instructors . By the end of the project, 49 percent of faculty members in participating universities had Master's degrees , compared to a target of 50 percent. The baseline was 38 percent in the original six universities; although not discussed in the ICR, the actual baseline for all twelve universities (six original and six additional) is likely to be below 38 percent. Restoring university governance By the end of the project, all universities had developed a strategic plan . The ICR reports that all project universities use these plans as a guideline for their operations (page 15). The project also supported the development of a National Higher Education Sector Plan , to provide a strategic vision for the sector. The project supported the creation of a Higher Education Management Information System which was designed to serve a central database for the Ministry of Higher Education to monitor the sector . However, this IT development has been hampered by the lack of IT capacity and infrastructure in a number of universities (page 15 of the ICR). 5. Efficiency: Efficiency is rated substantial . The project was prepared using a simplified format (Memorandum and Recommendation of the President ) that did not include any economic analysis . The ICR contains a detailed economic analysis that focuses on the contribution of higher education to employment and includes a detailed discussion of the assumptions and a sensitivity analysis . It uses a realistic set of assumptions to estimate the impact of the project on increasing the number of graduates as well as increasing the quality and relevance of a degree from higher education . Based on this analysis, the project's rate of return is likely to be in the range of 24 percent . The economic analysis does not any quantification of the social benefits of the projects . In several areas, the project was able to produce additional outputs which also contributed to the project's efficiency. Some examples include the development of a National Higher Education Sector Plan (which was not originally included in the project ) and the increase in the number of faculty with Master's degrees, which was largely financed by the project . The decision to extend the project and provide additional financing also facilitated the government's efforts to expand its coverage to more universities by using the existing project framework . At the same time, the project had a number of delays with procurement . These delays appeared to be largely administrative, such as delays in the internal and external audits, and did not appear to have much long -term material impact on the project's implementation (page 11 of the ICR). ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 24% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project remains highly relevant to both government and Bank strategy, and it had a design that was well constructed to deliver the necessary interventions to support a straightforward objective . The project achieved its objectives and contributed to restoring basic operations at selected universities . The project was efficient and provided good value for its investment . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The project has a high level of support from the government, and the World Bank continues its support to the sector with a proposed new operation (P146184, Higher Education Expansion and Development Project ). This support is likely to continue the work of refurbishing higher education institutions . The economic analysis in the ICR also notes that universities are starting to develop independent sources of income and that the economy is growing rapidly, increasing the likelihood of continued public funding (pages 32-33). However, the country risk in Afghanistan remains high, particularly as the country holds presidential elections in 2014 and as security responsibilities are transferred to the government (page 19 of the ICR). a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The Bank included higher education in its earlier Emergency Education Rehabilitation and Development Project (P077896), which provided the Bank with experience and lessons that were incorporated into this project (Technical Annex, page 5). In addition, the Bank supported a technical study of the education sector as well as a study on the role of the private sector in higher education . The Bank's experience with grants to universities was the basis for the project's Component 1. The technical annex outlines the steps taken to ensure good financial management, particularly for the grants to universities. The grants were provided through Development Budget Imprest Bank accounts, with tranches based on the university's progress in improving procurement (page 18 of the Technical Annex). This provided a mechanism to ensure that funds were used as intended in an environment where fiduciary systems are still in their initial stages . Initially simplified procurement procedures were used, given universities' low level of procurement experience (pages 24-25 of the Technical Annex). There were minor differences between the development objective in the financing agreement and project documents. Likewise, although the results framework was reasonably well developed, there were some areas that were not covered . These related mainly to the impact of the grants on restoring basic infrastructure (pages 19-20 of the ICR). These are relatively minor shortcomings in project preparation . at -Entry Rating : Quality -at- Satisfactory b. Quality of supervision: Initially, there were differences between the development objectives in the legal agreement and the project document. The project team was proactive in adjusting the objective in the project document to match the legal agreement as was required by Bank policy . Given the initial low state of development of the higher education system in Afghanistan, the Bank was quite active in supporting the implementation of the project . This included support for technical aspects of the project, in the form of technical specialists as well as technical studies . The Bank also worked hard to improve the data system; while this was an integral part of the project, the Bank worked to make sure that it was used to improve project monitoring (page 20 of the ICR). The Bank was realistic in its internal ratings and took steps to lower the ratings in response to shortcomings in both procurement and financial management . This was complemented by focused support to help the project and the Ministry improve their performance in both areas (pages 10-11 and 20 of the ICR). Security was a major impediment in supervision, including the verification of data . The project team worked with the Ministry to develop alternative forms of monitoring (page 9 of the ICR). Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government provided strong support to the sector and the project through its policies and development strategies (pages 1 and 20-21 of the ICR; page 1 of the Technical Annex). There were some delays caused by administrative issues within the Ministry of Finance . These led to delays at the provincial level due to antiquated disbursement systems . The ICR argues that while these delays were real, they did not affect the overall outcome (pages 10 and 20-21 of the ICR). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: When the project was approved in 2005, Afghanistan was in the process of establishing the basic institutions of government (page 1 of the Technical Annex). Initially, there was a lack of understanding of the project at the Ministry level and a lack of effective autonomy at the university level . This improved as the project was implemented (page 21 of the ICR). There were number of shortcomings with procurement that led to some delays in the procurement process . This was complicated by the departure of senior procurement staff . This improved as greater central control was introduced (page 11 of the ICR). The project had significant staff turnover at all levels . Although this did create delays, the project probably saw less turnover than other World Bank -financed projects operating in Afghanistan during the same time period (page 21 of the ICR). The ICR reports that the quality of management at the central level increased substantially at later stages, despite the shortcomings with procurement . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: During preparation, M&E posed a major challenge due to the lack of infrastructure to collect and analyze data . The education sector was by no means unique in this problem as most data -collecting institutions were destroyed by conflict (pages 8, 9-10 of the ICR). The results framework was relatively well designed and aligned to the development objectives . The results framework identified five areas that were related to the project's development objective, with a total of eight indicators at the PDO level (pages 8-9 of the ICR). The results framework in the project document did not include baselines and targets . This may reflect the simplified nature of the project document; the baselines and targets were included in the first internal monitoring document (ISR), soon after project approval (page 9 of the ICR; pages 13-14 of the Technical Annex). While the ICR expresses concern that the project could have been more proactive in including additional indicators (pages 9, 20 of the ICR), the results framework provides an adequate and useful summary of the project and its implementation. Many indicators proposed by the ICR were already included in the project's administrative M&E system but were not included in the results framework . b. M&E Implementation: Given the low capacity of the sector, the project included support to improve the capacity of the local system . This was done through project investment as well as technical support (page 9 of the ICR). One challenging area was to collect and verify data from remote sites given the security situation in Afghanistan . This situation improved through the use of technology such as remote conferencing (page 9 of the ICR). The project supported the establishment of university -level M&E systems in nine out of the twelve participating universities (page 10 of the ICR). The project team was proactive in adjusting the results framework to respond to the evolving reality of the sector. This was necessary, given the low starting point of the sector and the uncertainty involved in the sector and the country. This included paring down the original results indicators to a more manageable number to set during implementation (page 9 of the ICR). c. M&E Utilization: Starting from scratch, the project established an M&E system for the Afghanistan higher education system . Although there remain many issues related to the low capacity of users and the lack of national IT infrastructure (page 15-16 of the ICR). The M&E system played a role in implementing the quality assurance system, which was a major part of the project and is generally considered important for higher education systems (page 16 of the ICR). According to the project team, M&E was used to help allocate grants and resources for facilities and equipment . This was a central part of the project that required flexibility to adjust the size of grants . M&E data and analysis were also used to develop the new Higher Education Strategy . M&E Quality Rating : Substantial 11. Other Issues a. Safeguards: The project initially did not trigger any safeguards and was rated Category C on environmental assessment . The ICR (page 10 and 20) states that during implementation, it was clear that the project was supporting significant infrastructure construction, which should have triggered the safeguard policies . The ICR also reports that the project did follow Afghanistan's Environmental and Social Safeguards Framework, and there was no involuntary resettlement (page 10 of the ICR). b. Fiduciary Compliance: The project faced a number of fiduciary challenges . Annual audits were often submitted late . Internal audits also detected weaknesses with expenditures, particularly petty cash (pages 10-11 of the ICR). Throughout the life of the project, there were shortcomings with procurement that were detected by audits . Many of these issues were administrative in nature, although they led to delays in the procurement process . The ICR attributes these issues to the very low existing procurement capacity at both the national and local levels . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately Although Ministry was largely effective Satisfactory in implementing the project, there were some shortcomings related to procurement. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR presents a number of well-thought out lessons from project design and implementation (pages 21-22). Of particular importance are: Government ownership is critical to a successful outcome . The project had strong support from the government, which gave it a high priority . This support led to good integration of the project into the Ministry. Flexibility is needed in a fragile environment . Many aspects of the project changed as the sector evolved and the project was adjusted to reflect new circumstances . The early delays in project implementation may have been avoided if there had been more flexibility in the timing, roll out, and activities which the block grants were contingent on – this lesson was learned and incorporated over time . In addition IEG notes that having a flexible design allowed the project to respond at each stage . The World Bank provided flexibility by supporting additional financing as well as adjusting the M&E framework during implementation. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: This a very well prepared ICR, with a clear and frank account of the project as well as a detailed analysis of its outcomes. The ICR draws on data that are outside of the results framework to analyze outcomes, and it shows a clear causal chain to justify the ratings . The ICR also carefully considers both the Bank Performance and Borrowers Performance ratings, making clear arguments for each . The ICR presents financial data clearly and contains a comprehensive description of the activities supported by the project. It also provides a concise description of how the project was implemented at the component level . The ICR's economic analysis provides a comprehensive discussion of the project's economic impact . The lessons are clear and well thought out . a.Quality of ICR Rating : Satisfactory