ICRR 11686 Report Number : ICRR11686 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 03/19/2004 PROJ ID : P002373 Appraisal Actual Project Name : Higher Education I Project Costs 30.9 25.4 US$M ) (US$M) Country : Senegal Loan/ US$M ) 26.5 Loan /Credit (US$M) 23.7 Sector (s): Board: ED - Tertiary Cofinancing - - education (100%) US$M ) (US$M) L/C Number : C2872; CP745 Board Approval 96 FY ) (FY) Partners involved : Closing Date 06/30/2003 06/30/2003 Prepared by : Reviewed by : Group Manager : Group : David Berk Roy Gilbert Alain A. Barbu OEDST 2. Project Objectives and Components a. Objectives The underlying objective of the project was to support continuing wideranging subsector reforms . The project documents are more specific, but displayed different sets of objectives . The ICR discusses the confusion, but does not clarify what the development objectives were . It finds the total set of planned reforms to have been over-ambitious. This OED evaluation is based on a combination of the overlapping Development Credit Agreement (DCA) and Staff Appraisal Report (SAR) long-term objectives, namely: DCA: Establish an efficient system of higher education with greater cost -effectiveness and more relevance to socio-economic development SAR: Long-term objectives: improve graduation rates, gender equity, greater cost -effectiveness, more relevance b. Components 1. Improvement of library services at Universite Cheikh Anta Diop (UCAD) (planned $15.6m or 50% of total costs of $30.9m, actual $16.7m or 66%) -- building expansion, books, journals, and systems especially IT 2. Upgrading of teaching and research (planned $9.4m or 30%, actual $4.2m or 17%) -- mainly buying science equipment 3. Strengthening of management capacity (planned $4.1m or 13%, actual $2.7m or 11%) -- student services, maintenance, PIU The project design was seriously flawed, in that the major project component (the university library) had little direct relationship to the project objectives . c. Comments on Project Cost, Financing and Dates Since the development objectives were not being achieved, $ 1.5 million was canceled and the project was scaled back end-2002 to financing physical inputs (additional library books and journals, science equipment ) and operating costs. The ICR's data on actual costs and their financing (section 5.4 and Annex 2) are incomplete and inconsistent . 3. Achievement of Relevant Objectives: Efficient System of Higher Education -- not achieved: output (measured by graduation rates ) fell, while inputs (measured by unit costs) rose, and the system consumed an increasing share of available resources (education budget), all contrary to planned developments Graduation rates -- not achieved: at UCAD the rates fell from 46% to 30%, vs. target 65%; though much higher initial rates rose further at the second much smaller public university . Gender equity -- achieved: female enrollments rose from 24% to 29% of the total, vs. target 30%. Cost-effectiveness -- not achieved: in public universities, the student -teacher ratio rose from 22 to 29.5, vs. target 16; teaching loads increased 1 hour per week (17-25% increase), vs. target 2 hours; unit costs increased overall . However, private institutions' share of enrollments grew to 29%, vs. target 15%. Relevance -- not achieved: the project provided more and up -to-date physical inputs; but there is no evidence of program changes; a graduate tracer study and employer survey were not carried out . 4. Significant Outcomes/Impacts: The project was successful in its early years, but major policy reversals following a change in government led to a highly unsatisfactory final outcome in relation to the project's important and ambitious objectives . Nevertheless, advances in some politically sensitive areas, comparable to those in some other countries' projects, were sustained despite overall policy reversals : Large number of persistently failing students, and unauthorized occupants of student housing, were removed. Cost-recovery for housing and meals was increased from 4% to 20%. Shares of females, and of private institutions, in total enrollments increased . Teaching loads increased somewhat . Expenditure on scholarships abroad stabilized . Implementation of the library component was highly satisfactory, producing a state-of-the-art functioning library with high usage and the potential to improve greatly the university's effectiveness . 5. Significant Shortcomings (including non-compliance with safeguard policies): The project design was seriously flawed . The implementation arrangements (PIU in ministry) were unsuccessful. Sub-sector reforms The most important reforms were reversed after about 3 years of implementation, due to student violence, faculty resistance, and the opposition's victory in national elections . Anti-reform forces were entrenched. Control of enrollments stopped (at UCAD they rose from 23,000 to 31, 000, vs. target 15,000). Scholarships were provided to all again (the budget share for scholarships rose from 23% to 32%). Increases in teaching loads were limited. The share of the education budget for higher education increased instead of falling, with substantial opportunity costs to primary education . Other reforms were also unsuccessful : The competitive research fund was ignored by faculty : 0 proposals were received, vs. target 400. A maintenance system was not developed . The accreditation system was not developed nor accreditations carried out . Only one policy study was done . Academics The employer survey and tracer study of graduates were not carried out . There is no evidence that programs became more relevant. Quality increases were permitted by more and better inputs and the library's quality and high usage, but there is no direct evidence of better teaching . Pass rates fell overall. Components The second and third components were not fully implemented . 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Unsatisfactory Highly Unsatisfactory The most important objectives were not achieved, in fact most indicators worsened appreciably (efficiency, graduation rates, cost-effectiveness). The relevance of most objectives was substantial, but efficacy (achievement) was mostly negligible, and efficiency is unknown. Institutional Dev .: Modest Negligible The efficacy of ID activities re the enabling environment and governance was negligible, and re improving organizations modest Sustainability : Unlikely Unlikely Bank Performance : Unsatisfactory Unsatisfactory Borrower Perf .: Unsatisfactory Highly Unsatisfactory Despite remarkable pre-project national consultation on the subsector, project design that resulted was flawed . Implementation was highly unsatisfactory : there were major policy reversals, leading to non-achievement of the most important objectives. Quality of ICR : Satisfactory NOTE: NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. 7. Lessons of Broad Applicability: (a) Project Concept 1. Project objectives should be clear and agreed and stated consistently throughout 2. Project components should relate demonstrably to achieving these objectives 3. Avoid promising big loan amounts then scaling down 4. Tertiary education reforms are difficult and face strong vested interests, so : do the political economy analysis up-front; analyze risks realistically and include mitigation measures; involve major stakeholders (faculty, students) including range of political opinion in sector consultations, design (and implementation); don't overload a single operation with reforms; phase reforms and try for early wins, or at least to accompany pain with gain; balance scope of reforms and importance of project investments; and have contingency plans including exit strategies . 5. Define performance indicators and arrange for data collection before implementation . (b) Implementation 1. Stakeholders' commitment to reforms must be sustained throughout implementation by continuous efforts, broad participation, and communication campaigns . 2. Reforms need communication ahead of time, explaining their rationale, rules and benefits . 3. Implementation units need the correct institutional home and clear authority and accountability . 4. When original project objectives cannot be achieved, restructure per reduced objectives or cancel ASAP . 5. Operations supporting difficult reforms require intensive Bank supervision with appropriate expertise . 8. Assessment Recommended? Yes No 9. Comments on Quality of ICR: The ICR brings out the different objectives and indicators, reasons for the project's failure (including QAE, borrower policy reversals, and Bank supervision ), and some project lessons. However, it could have tried to better clarify what the development objectives were, as the basis for evaluation . The data on actual costs and their financing in text section 5.4 and Annex 2 are incomplete (the region commented that this was the only data provided by the PIU )