ICRR 12860 Report Number : ICRR12860 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/26/2008 PROJ ID : P069326 Appraisal Actual Project Name : Jordan Higher US$M ): Project Costs (US$M): 65.78 41.8 Education Development Country : Jordan Loan/ US$M): Loan /Credit (US$M): 34.7 33 Sector Board : ED Cofinancing (US$M): US$M ): Sector (s): Tertiary education (99%) Central government administration (1%) Theme (s): Education for the knowledge economy (100% - P) L/C Number : L4539 Board Approval Date : 02/29/2000 Partners involved : Closing Date : 06/30/2005 06/30/2007 Evaluator : Panel Reviewer : Group Manager : Group : Helen Abadzi Kris Hallberg Soniya Carvalho IEGSG 2. Project Objectives and Components: a. Objectives: The development objective of the project was t o initiate improvements in the quality, relevance, and efficiency of higher education in Jordan and to support the Government's program to reform sector governance . The project would (i) establish system-wide modem information technology, management information systems (MIS), and library infrastructure for higher education; (ii) support a Higher Education Development Fund that would provide grants to public universities for innovative and economically relevant sub -projects and for improving teaching and learning; (iii) initiate reforms of higher education governance, including the introduction of formula -based allocation of higher education recurrent funding from the Government; and (iv) support the rationalization of the community college sub-sector through the new Al-Balqa' Applied University (BAU). b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (a) Initiate improvements in quality, relevance and efficiency (US$53.8m at appraisal, US$36.4m actual) to support the entire higher education sector and to improve the quality, relevance and efficiency of teaching, administration in universities; a higher education development fund was to provide grants and research . (b) Initiate Improvements in governance (US$1.911m at appraisal, US$0 actual) for strengthening the capacity to implement governance reforms as well as university planning and management capacity . These outcomes would be achieved through (i) a higher education council secretariat for policy formulation, (ii) an education accreditation council to help evaluate and accredit the programs of study in public and private universities, and (iii) improved planning and management capacity at the university level to align process in public universities in line with the upgraded practices at the national level . (c) Reform of the Community College System /Al- Al-Balqa Applied University (US$5.6m at appraisal, US$4.1m actual). The Al-Balqa Applied University was to be responsible for the academic management and administrative affairs of all community colleges. The university was to support the reform process in governance and management, program development, human resources management and facility upgrading . (d) Project Implementation capacity (US$1.5m at appraisal, US$1.3m actual) for project management. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The ICR states that closing date was 12/31/05, while the loan agreement gives the date of 6/30/05. The latter is being used. After extensions totaling 24 months, the project closed on June 30, 2007 and approximately US$3.4 million were canceled. 3. Relevance of Objectives & Design: The development objective is consistent with the Bank ’s current Country Assistance Strategy (CAS) which supports Jordan’s National Agenda to promote knowledge -based private investment, job creation and poverty reduction . The development objective continues to remain relevant to this date . However, there were several problems with project design. These included a lack of a government-endorsed sector strategy or a specific plan of actions to ensure that policy reforms were introduced; a lack of links between the competitive fund and Government priorities and labor market needs; the failure to downsize the project scope in response to the reduced loan size; and the weak M&E system. In addition, the project was expected to operate through overlays of committees and an implementation unit outside the higher education institutions and was not sufficiently targeted to carry out a major implementation reform . 4. Achievement of Objectives (Efficacy): (a) Initiating improvements in the quality, relevance, and efficiency of higher education (modest). This included the following sub-objectives: (i) establishing system-wide modem information technology, management information systems, and library infrastructure for higher education (substantial). Activities included: - New teaching methods and technology were to be adopted, that should result in improved student ratings of faculty teaching behaviors . An information technology program was fully implemented, surpassing numerical targets and deadlines. The universities acquired videoconferencing centers, and an integrated library system became operational. An evaluation concluded that 70% of students used information technology for coursework, but teachers rarely did. - New teaching methods were introduced in 13 new and 11 revised undergraduate programs as a result of a Higher Education Development Fund. Also, 10 new institutes were developed that included capacity for water and environmental research. About 146 Faculty Development Centers were also established in all public universities, which conducted workshop on classroom instruction, methods of teaching, and instructional media, and evaluation; 12,500 teacher evaluations were carried out between 2004 and 2006 in all universities. An evaluation showed, however, that educational media and technology use in classrooms did not become more prevalent or meet the 70% utilization target set during appraisal . Teacher evaluations of faculty performance did not improve as a result of training. (ii) supporting a Higher Education Development Fund (modest) - A competitive mechanism was to be used to fund new or updated programs . However, it was not applied in the first two of three rounds; instead matching funds were given, but this funding method did not ensure that the selected programs were the most relevant from the country perspective . Nevertheless, 2150 students enrolled in the first year of the 11 new undergraduate programs; and 5200 in the 13 restructured programs; 13 Masters programs were introduced. Revised programs reached approximately 5200 students. 30 percent of subprojects made links with enterprises and social organizations . (b) Initiating reforms of higher education governance (negligible overall). Activities included: (i) initiating reforms of higher education governance (negligible) An accreditation system was established that became a model for the region, and all 12 private universities became accredited. However, it was only partly implemented . At least three public universities underwent the accreditation process but none was accredited . - Between 2000 and 2005, the government reduced public financing in university operations from 50% to about 20%; universities introduced fee-paying programs, and about 14% of students got access to targeted and subsidized student loans. However, public universities did not reduce the vast numbers of non -teaching staff in relationship to teaching staff. Actually the ratios worsened during the project from 2.7 non-teaching staff per professor to 2.8 non-teaching staff. At the same time, the ratio between students and faculty worsened from 23 in 2000 to 31 in 2004 and 38 in 2005 in public universities. Data does not exist to show how public resources have been used, but with the compression of expenditures and despite the increase in self -generated resources, universities have accumulated arrears. The ability of universities to finance quality was adversely affected by increased enrollments and reduced funding from the Government. The Management Information System would have been one of the most important mechanisms to improve informed decision making, but unfortunately this was not completed as planned . - A formula for allocating budgetary resources transparently and efficiency was to be established, and public universities were to receive resources through this method . However, the formula was abandoned; the competitive fund mechanism was introduced only in last of three competitive rounds, and at the time the ICR was written there were no data about its effectiveness . (ii) supporting the rationalization of the community college sub -sector (negligible) -Efforts were made to increase enrollment in the country's 21 community colleges from 10,000 to 15,000 students by introducing four new programs. However, enrollment in community colleges did not increase by the end of the project. Employer satisfaction with skills of graduates was to improve over baseline, and the percentage of students competent in English was to increase from 0 to 30%. Although language labs were installed, their degree of utilization and subsequent student performance were not measured . -An organizational plan and procedures manual were prepared to improve operations and lower the high percentage of unemployed graduates in community colleges, but no decisions were made by the end of the project, the procedures were not implemented, and no strategic plan was developed . There were no champions to promote policy change. As a result, community colleges did not gain autonomy from the overseeing Al Balqa Applied University. 5. Efficiency (not applicable to DPLs): Efficiency was modest. The material inputs of the project, such as technological inputs, were obtained efficiently, with satisfactory procurement arrangements, and delivery of these inputs took place on time and exceeded targets . However, the mechanisms established to carry out critical policy changes proved inefficient; studies were conducted, committees met, but decisions could not be made . As a result, efficiency did not improve during the project . The public universities are in financial difficulty due to inordinately high costs of non -teaching staff salaries. One cost effectiveness analysis showed a possible 10% rate of return, but the assumptions were not later met . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The development objectives had high relevance to the needs of the country, but the relevance of the design was modest, did not provide effective means for obtaining the expected benefits; overall relevance is substantial . Overall efficacy is modest; despite much activity related to the acquisition of inputs, policy objectives were not attained, and the financial condition of the universities did not improve . Efficiency was also modest. Although the procurement of materials was carried out efficiently, the university finances and governance did not become more efficiently managed as a result of the project . Thus, the policy project objectives were not attained . - Overall, the project seems to have produced outputs rather than outcomes . For example, the extent to which employers were satisfied by the performance of graduates was unknown . Also, attribution is difficult; for example, increasing enrollments may be due to population increases rather than project benefits per se . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome is rated moderate . As the universities have lost financing and have been indebted, the maintenance and replacement of technological equipment is in doubt . The various procedures and studies developed with the aim of improving governance were not acted upon and may become obsolete in subsequent years. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at entry was unsatisfactory. The project lacked a solid consensus by stakeholders on the strategic and policy framework. The appraisal lacked a clear plan on how governance reforms would be implemented; the specific responsibilities and technical expertise needed to implement reforms were not clarified . Rather, the project used dated covenants which were not understood and never complied with . The risks of lack of technical assistance were underestimated as was the importance of sufficient funding for the sector . During negotiations, the government decided to reduce the budget envelope by 20 percent, but the Bank did not adjust project scope, and indicators were also not revised to reflect the new reality . Supervision missions were conducted regularly, with support from the Lebanon country office in Lebanon . The Bank team provided technical assistance with international experience in accreditation, labor market issues and community colleges to mitigate the Government ’s inability to find resources for the project . However, the Bank could not keep up with policy changes and demographic trends that emerged during the life of the project; it and did not restructure and simplify the project to focus on areas where there was a consensus to reform and formally revise indicators. Despite a lack of progress on important policies, t he Bank focused on disbursements . Supervision consistently rated implementation as satisfactory, with only two ratings as moderately satisfactory . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: The project implementation unit was very competent at delivering outputs and planned the various activities well and within deadlines. However, there were few policy champions in the unit or universities, and the result was implementation delays in the financing and governance reforms . The creation of a separate Ministry of Higher Education and Scientific Research in 2003 improved the policy environment and helped to advance project implementation. However, implementation focused on technological outputs rather than outcomes . Although policy studies were carried out, there was limited emphasis on implementing reforms and monitoring policy implementation and outcomes. Also, the project implementation unit did not have enough staff with the in -depth knowledge and experience to guide reforms, implement a competitive fund and build university management capacity. The government did not assign specific responsibilities and technical expertise to implement reforms, such as instituting the formula funding mechanism or restructuring community colleges . Decisions on specific issues regarding admissions, financing, student fees and appointments were made by entities such as the Royal Court and the central government. But there was little formal coordination among various decisionmakers, and it became difficult to maintain consensus on the reform program . Frequent changes in ministers, political appointments, and the institutional setup also undermined adherence to a sector strategy . a. Government Performance :Unsatisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: The monitoring and evaluation design were modest . Design issues : Some indicators were very complicated and difficult to obtain from universities . Studies of student satisfaction and technology use were conducted, but the extent to which decisions were based on their results was unclear. A management information system that would have improved decision -making was not fully implemented. A tracer study and employer attitude study were carried out in 2004 under the a donor-financed project, but study results were not conclusive and since few college programs had been implemented by the closing date . Utilization issues : The lack of decision-making at key points in the project suggests that either data to make decisions were unavailable or the political will was weak; for example, public universities did not obtain accreditation, despite the exercises they underwent . Overall, the monitoring and evaluation findings were not used for further decision-making. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The implementation unit was competent at managing procurement, project activities, financial management and collecting monitoring indicators. Annual audits were sent as expected . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Quality at entry was moderately Satisfactory Unsatisfactory unsatisfactory. The Bank did not ensure that political will or functional mechanisms would be available for implementation. Emphasis during supervision was placed on disbursements rather than faltering policy implementation process. Borrower Performance : Moderately Moderately The project implementation unit Satisfactory Unsatisfactory performed competently in procuring technology inputs. However, the government had limited political will to carry out policy reforms. As a result, project outcomes were not realized, and governance in public universities did not improve. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: In some political circumstances it may be easier to obtain material inputs through Bank financing than to reform policies. The Bank must seek creative ways to bring about these difficult changes and maintain momentum. In cases where governments are reluctant to undertake important reforms, perhaps the Bank should refrain from lending; To be effective, policy champions must have political support . If policy champions cannot be found, efforts at reform may flounder. Authorities of higher education institutions may hesitate to take political risks and implement innovative reforms without internal support; The Bank must remain vigilant about the extent to which project activities are realistic during changing political environments. For example, governments may decide to downsize a project during negotiations; task teams must decide if the project scope, development objectives, and indicators can be obtained with the financing available. Restructuring must also remain an option during implementation if the project is not making adequate progress or if indicators are found to be irrelevant; and For a competitive fund to add value it needs to be harmonized with government policies and with broader economic priorities. An authorizing political environment must exist for implementation, and guidelines must be developed from authorities of the institutions that will be involved . Without internal safeguards, criteria may be overridden, and the competitive fund may be spent on activities that were not chosen on the basis of merit . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR was clear and presented in critical detail the challenges of the project . However, it did not follow the guidelines for rating Bank performance . It rated quality at entry as moderately unsatisfactory and quality of supervision as moderately satisfactory, but incorrectly rated overall Bank performance as moderately satisfactory . The Guidelines state that, when one rating is in the satisfactory range and the other is in the Unsatisfactory range, overall Bank performance is moderately unsatisfactory if outcome is below the line . a.Quality of ICR Rating : Satisfactory