ICRR 11426 Report Number : ICRR11426 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 05/13/2003 PROJ ID : P063386 Appraisal Actual Project Name : Cl-Millennium Science Project Costs 15.0 15.0 Initiative (LIL) US$M ) (US$M) Country : Chile Loan /Credit (US$M) Loan/ US$M ) 5.0 5.0 Sector (s): Board: ED - Central Cofinancing government administration US$M ) (US$M) (85%), Tertiary education (15%) L/C Number : L4466 Board Approval 99 FY) (FY) Partners involved : N/A Closing Date 03/31/2002 09/30/2002 Prepared by : Reviewed by : Group Manager : Group : David Berk Patrick G. Grasso Alain A. Barbu OEDST 2. Project Objectives and Components a. Objectives The specific objective was to demonstrate significantly improved performance in a highly selected segment of the Chilean science and technology system, through a pilot of the Millennium Science Initiative (MSI). Transparent, merit-based allocation procedures and investigator autonomy would improve the quality and efficiency of scientific research and advanced training . b. Components 1. Management Structure for MSI ($1.5m): (i) Establishment and operations of Board of Directors, (international) Program Committee, and Implementation and Management Unit (IMU); (ii) TA for selection of Science Institutes (SI) and Science Nuclei (SN); (iii) Development of proposal to improve S & T institutional framework; and (iv) M & E studies. 2. Competitive Funds for Scientific Excellence ($12.0m): Funding of research projects at 1-3 Science Institutes and 5-10 Science Nuclei, including infrastructure, equipment, fellowships, advanced training, and outreach activities . 3. Networks for Promotion of Scientific Excellence ($1.5m): (i) Visits; (ii) Exchange programs; (iii) International advanced courses; and (iv) Dissemination. c. Comments on Project Cost, Financing and Dates Appraisal underestimated cost of research proposals . GOC so committed it accepted proposals for 3 SI and 10 SN totaling $27m instead of original 1-3 SI and 5-10 SN for $15m. Bank loan effective on time and project implementation started quickly; loan fully disbursed in 2 years vs. intended 1.5 years. 3. Achievement of Relevant Objectives: The efficiency of R & D, the quantity and quality of advanced training, and the volume of collaborations and networking have all increased . When more time has elapsed, it seems likely that the quantity and quality of R & D will also have increased. Results in PAD's six dimensions of improved performance to be monitored, with their associated indicators, were : 1. Selection Process: MSI confirmed as credible and fair funding source (target was significantly greater confidence in MSI process). 2. Administrative Efficiency: Grants processed within 60 days (target was 50% faster than other research funding organizations). 3. Concentration of Resources : Annual grants averaged $1.3m (SI) and $0.3m (SN), reported consistent with OECD averages for corresponding programs (target was within 33% of OECD). 4. Perceptions Regarding "Stagnation" of S & T: Scientists report improved working conditions and career possibilities; repatriation of researchers apparently increasing (target also included perception of MSI as facilitating high-level scientific research). 5. Collaboration with International Scientists : Number increased at least 50% (target was 20%); reported duration and quality rising (per target). 6. Human Capital Training Opportunities: MSI-funded researchers trained 50% more Ph.D.s and post-docs than colleagues (on target). No information in ICR on whether meeting target of MSI providing opportunities for women and other previously underrepresented groups . ICR also reports on an output: 7. Promotion of Regional Network of Scientific Excellence : 18 Ph.D.s/post-docs from neighboring countries trained under MSI fellowships (target was 12). 4. Significant Outcomes/Impacts: (a) By component: 1. Management Structure: Light and effective administrative structure for science funding created, requiring only 5.5% of project costs. Transparent competitive funding mechanism established and respected, involving strong role for high-level international scientists (successful TA). Intra-GOC dialogue underway on next phase, but planned proposal to improve S & T institutional framework not yet produced. Extensive M & E conducted through (late) baseline study, two external evaluations, and tracking of PAD indicators. 2. Competitive Fund: First round of funding very efficiently conducted despite its complete novelty. Selected 3 SI and 5 SN from 75 applicants. Second round also held (5 SN selected from c. 70). Some of Chile's best scientists (recognized with international awards) selected. MSI by far most productive source of advanced scientific training: over 150 directly funded, including 66 funded graduate students who received 47 Ph.D.s, 25 post-docs; 250 others associated. MSI-funded researchers are 8% of their peer group but responsible for training almost 50% of Ph.D. students in science and engineering. Too early to judge research results, but 3 big discoveries and 6 patent applications so far, and number of publications in international journals looks likely to increase. 3. Network: Big increase in networking and especially outreach; funds non-fungible. Joint research and training programs conducted. Dozens of international courses and seminars held. Participation in regional and international networks increased. Students from neighboring countries funded. Outreach to secondary education in some disciplines. Regional cooperation started among countries considering or introducing MSI (Chile was first). (b) General: S & T is receiving increased attention from GOC and other governments in region. Fair, open, merit-based selection process has been established (under pilot institutional arrangements). Accepted by scientific research community, and has influenced other funding mechanisms in Chile. Initial indications that productivity of Chile's top researchers has improved, due to improved working conditions made possible by MSI's larger grants. Opportunities for and quality of advanced training have improved. Several forms of collaboration have increased significantly. M & E base is being built, including benchmarking for scientific productivity for the first time in Chile. 5. Significant Shortcomings (including non-compliance with safeguard policies): To mount quick and successful pilot, MSI was grafted onto Planning Ministry, expanding its mandate, and capacity it has built is also separate from main S & T funding agency CONICYT . Sustainability of reforms, and their expansion to wider S & T sector, require further thought . GOC has not yet developed proposal to improve S & T institutional framework (component 1.(iii)). 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Highly Satisfactory Highly Satisfactory Institutional Dev .: Substantial Substantial Sustainability : Highly Likely Highly Likely Bank Performance : Highly Satisfactory Highly Satisfactory Borrower Perf .: Highly Satisfactory Highly Satisfactory Quality of ICR : Satisfactory NOTE: NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. 7. Lessons of Broad Applicability: 1. The model of (i) transparent, merit-based allocation of research funds, (ii) investigator autonomy, and (iii) specific focus on training opportunities and on collaborations and networking has quickly increased the efficiency of R & D, the quantity and quality of advanced training, and the volume of collaborations and networking . When more time has elapsed, it seems likely that it will also have increased the quantity and quality of R & D . 2. Nevertheless, capitalizing on the LIL -- by scaling up the successful pilot reforms to change whole sectors /systems -- will require considerable further thought and effort and remains challenging . In particular, decisions are needed on whether to continue or change the special institutional arrangements at MinPlan used for the pilot, and on incentives to get other public S & T agencies to change to the new approach . 8. Assessment Recommended? Yes No 9. Comments on Quality of ICR: Good. Thorough and to-the-point evaluation of each aspect of experience and learning .