Document of The World Bank Report No: 26039 IMPLEMENTATION COMPLETION REPORT (SCL-4043A; CPL-40430; SCPD-4043S; TF-29185) ON A LOAN IN THE AMOUNT OF US$ 65 MILLION TO THE GOVERNMENT OF INDONESIA FOR A HIGHER EDUCATION SUPPORT PROJECT: DEVELOPMENT OF UNDERGRADUATE EDUCATION June 27, 2003 CURRENCY EQUIVALENTS (Exchange Rate Effective February 1996) Currency Unit = Rupiah (Rp) Rp 1 million = US$ 445 US$ 1 = 2,246 Rp Academic Year July 1 - June 30 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS BAN-PT - National Accreditation Board BHE - Board of Higher Education CAS - Country Assistance Strategy CPCU - Central Project Coordinating Unit DGHE - Director General of Higher Education DUE - Development of Undergraduate Education GOI - Government of Indonesia GPA - Grade Point Average IDR - Indonesian Rupiah LPIU - Local Project Implementation Unit MOF - Ministry of Finance MoNE - Ministry of National Education QUE - Quality of Undergraduate Education S1 - Degree equivalent to Bachelor's Degree S2 - Degree equivalent to Master's Degree SAR - Staff Appraisal Report UMPTN - National University Entrance Examination Vice President: Jemal-ud-din Kassum, EAPVP Country Manager/Director: Andrew Steer, EACIF Sector Manager/Director: Emmanuel Y. Jimenez, EASHD Task Team Leader/Task Manager: Christopher J. Thomas, EASHD INDONESIA HIGHER EDUCATION SUPPORT PROJECT DEVELOPMENT OF UNDERGRADUATE EDUCATION CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 2 5. Major Factors Affecting Implementation and Outcome 9 6. Sustainability 9 7. Bank and Borrower Performance 10 8. Lessons Learned 12 9. Partner Comments 12 10. Additional Information 13 Annex 1. Key Performance Indicators/Log Frame Matrix 14 Annex 2. Project Costs and Financing 15 Annex 3. Economic Costs and Benefits 17 Annex 4. Bank Inputs 18 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 20 Annex 6. Ratings of Bank and Borrower Performance 21 Annex 7. List of Supporting Documents 22 Annex 8. Borrower's ICR Contribution 23 Project ID: P004004 Project Name: Development of Undergrad. Educ (DUE) Team Leader: Christopher J. Thomas TL Unit: EASHD ICR Type: Core ICR Report Date: June 27, 2003 1. Project Data Name: Development of Undergrad. Educ (DUE) L/C/TF Number: SCL-4043A; CPL-40430; SCPD-4043S; TF-29185 Country/Department: INDONESIA Region: East Asia and Pacific Region Sector/subsector: Tertiary education (92%); Central government administration (8%) Theme: Education for the knowledge economy (P) KEY DATES Original Revised/Actual PCD: 04/07/1995 Effective: 09/13/1996 09/13/1996 Appraisal: 11/10/1995 MTR: 11/24/1999 11/24/1999 Approval: 06/18/1996 Closing: 09/30/2002 09/30/2002 Borrower/Implementing Agency: GOI/TBD Other Partners: STAFF Current At Appraisal Vice President: Jemal-ud-din Kassum Gautam Kaji Country Director: Andrew D. Steer Marianne Haug Sector Manager: Emmanuel Y. Jimenez Samuel S. Lieberman Team Leader at ICR: Halsey L. Beemer Lauritz B. Holm-Nielsen ICR Primary Author: Halsey L. Beemer; Sandra F. Erb; Christopher James Smith 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome:S Sustainability:L Institutional Development Impact:SU Bank Performance:S Borrower Performance:S QAG (if available) ICR Quality at Entry: S S Project at Risk at Any Time: No 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: Context. When the Government of Indonesia (GOI) and Ministry of National Education (MoNE) initiated the Development of Undergraduate Education (DUE) project in 1996 the priorities were to: (i) train a highly skilled labor force to compete internationally in the production of high value-added products and services and (ii) ensure greater efficiency of the university sector in the use of public resources and create financial incentives for improvement. To achieve these priorities the GOI introduced the New Paradigm, which reduces the role of the GOI to one of shaping regulations and directing resources to high-priority areas, while also decentralizing planning and management responsibilities to individual institutions. The DUE project was designed to promote these priorities through support to the New Paradigm. The three project objectives were to: (a) improve the quality of undergraduate education; (b) improve the efficiency of the educational process; and (c) to improve the relevance of the study programs offered. These objectives were consistent with the Bank's Country Assistance Strategy (CAS), which called for human resource development through a higher quality of education services, and an active policy dialogue designed to redefine and refocus the GOI's role in the education sector with a shift towards more decentralization. The objectives also reflected the second priority supported by the New Paradigm to give greater attention to the role of incentives in guiding quality improvement, the use of competition to motivate institutions and staff, and increase accountability to enhance performance. 3.2 Revised Objective: Not Applicable 3.3 Original Components: The project had five components in support of the three objectives given above: Component 1: Provision of block grants to six universities known as the University Development Program ($63.3 million) Component 2: Institutional Capacity Building in Higher Education ($6.4 million) Component 3: Implementation of a National Accreditation System for Higher Education ($10.0 million) Component 4: Competitive Domestic Fellowship Program ($ 9.9 million) Component 5: Project Administration ($2.4 million) 3.4 Revised Components: Not Applicable 3.5 Quality at Entry: Quality at entry is rated satisfactory based on the strength of: (i) consistency of objectives with the CAS, the priorities, and the New Paradigm of the MoNE and the DGHE; (ii) the incorporation of successful experience and lessons learned by the five previous Higher Education Projects; (iii) demonstrated capacity of the DGHE to implement previous projects and its own policy to improve the role of incentives in guiding quality education; and (iv) project design that anticipated decentralization and university autonomy. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: The outcome of the project is judged to be satisfactory having achieved most of the major objectives with only a few shortcomings. The project has achieved improvements in quality, efficiency and relevance and has put into place a block grant mechanism to fund higher education. This overall rating is based on: (i) - 2 - the originally agreed on Staff Appraisal Report (SAR) performance indicators; (ii) additional institutional level indicators agreed during project implementation, and (iii) the degree of success in introducing and institutionalizing new models of block grant funding which supports the New Paradigm. A review of performance indicators agreed at the appraisal and in the SAR has demonstrated improvement in the quality of undergraduate education in 100 percent of the project-supported study programs, all of which achieved or exceeded a 10 percent increase in the mean scores on benchmark tests. Additionally, in the fifth year of the project, 62 percent of the students taking the National University Entrance Examination (UMPTN) scored at least 5 percent higher than students taking the same examinations five years earlier. The increase in the mean test score by 5 percent was a project performance indicator which proposed to measure increased quality by the attractiveness of a study program. Improvement in the efficiency of the educational process, as indicated in the decrease of the average time to graduate from 5.75 years at appraisal to 5.25 years at completion occurred in 87 percent of the study programs. This is impressive given that the Asian economic crisis caused a downturn in the job market for new graduates in Indonesia. Finally, improvements in relevance of the study programs as indicated by a decrease in the mean job search time of graduates from the estimated time at appraisal of 1.75 years to 1.5 years was achieved in 95 percent of the study programs. These indicators suggest improvement, but some universities questioned the validity of baseline data and pointed out that exogenous factors may also impact UMPTN scores. In response, each university and study program established its own set of institutional baseline and target indicators, which was consistent with the bottom-up planning methods of the block grant mechanism and New Paradigm. Despite continued measurement problems, two quality measures appeared to have validity and substance at the study program level. The average grade point average (GPA) and the number of skripsi topics (skripsi is the final research requirement for graduating undergraduate students) both showed unambiguous and significant increases. Additionally, the improvement in faculty qualifications as measured by the large numbers of teachers (4400+) who received training during the project is unambiguous and significant. Both sets of indicators are reflected in Annex 1. A new model of block grant funding was successfully introduced supporting the GOI's priorities and New Paradigm. By project closing the new model was fully internalized in the eight project supported universities and institutionalized within the funding system of Indonesian higher education. This new model has been replicated in a follow-on Bank-supported project (Quality of Undergraduate Education-QUE) as well as two domestically funded projects. An increasingly larger percentage of the national budget supporting higher education uses the block grant method. However, a few problems arose during project implementation: (i) there was a lack of evidence that data from the performance indicators and outputs were used by universities to make management decisions. The systematic use of data is a major aspect necessary to establish responsible autonomy and quality education in the New Paradigm; (ii) low utilization levels for libraries, computer centers and language laboratories persisted in some universities. Additionally, some universities planned to sustain those activities through user fees, but fees provided a disincentive for improving utilization; and (iii) financial management at both the national and institutional level has been disappointing ­ there has not been a professionalization of accounting practices in the universities and internationally recognized standards have not been widely accepted at the university level. Despite project shortcomings the major result of DUE that makes this project satisfactory is that higher education delivery in Indonesia has become more efficient and funding has become more performance-based. A key assumption is that the new institutional arrangements inspired by DUE and the New Paradigm (e.g. the block grant scheme) will contrive to improve the quality education over the longer term. - 3 - It should be noted that many newly autonomous universities increased tuition. Even though original university entrance and tuition levels were quite low, the rapid fee increases in some universities have become a cause of concern for people. Tuition fees, and more generally income and expenditure management, are important considerations for higher education regulatory bodies. The BHE was not established with particular expertise in this area, but now recognizes the need to focus on financial management and is strengthening its advisory role in this area with assistance from the Bank-financed QUE project. 4.2 Outputs by components: Component 1: University Development Program (US$ 63.3 million SAR, US$ 61.96 million Actual) Overall this component is rated as satisfactory. The component originally provided 5-year block grants to six universities. Savings realized in the first round of proposals were used in mounting a second round of competition during the second year of implementation and this provided three-year block grants to an additional two universities employing the same selection process agreed at Appraisal. The primary objective was to enable less developed universities to carry out development programs to strengthen undergraduate education. A secondary objective was to pilot and establish a transparent, competitive, peer review process for the award of funding linked to performance targets and coordinated by the newly established Board of Higher Education (BHE). Achievement of the first objective as evidenced by civil work improvements, new equipment and books and journals, fellowships for visiting teaching staff, for staff development and technical assistance, for teaching and research grants, and management support for the project at the institutional level. The second objective was also achieved as evident at the system level where the DGHE adopted the selection criteria and peer review process piloted by the project to award an increasing amount of its development budget. At the institutional level, the eight universities report increasing use of the principles of competition; domestic and international peer review; and linking funding decisions to performance targets, recipient absorption and management capacity. a) Civil Works As detailed in their respectively agreed 5-year and 3-year development plans, the grants were intended to provide recipient universities with resources for civil works to rehabilitate and upgrade existing classroom, laboratory and student service facilities. During the project 46,900 m were2 rehabilitated, 31.4 percent more that the planned 35,700 m . Three main factors contributed to exceeding 2 the estimated physical targets: (a) frequently, the original "owner estimates" were too high, (b) several universities reported that a benefit of employing World Bank NCB guidelines introduced a more competitive process thereby lowering construction costs, and (c) the effect of the Asian economic crisis led many contractors to lower their profit margins to attract work. Also, the block grants provided universities with flexibility to meet emerging needs, such as repairing the widespread damage from the 1998 earthquake in Sumatra. Students and staff interviewed during supervision missions reported that building new and upgrading existing classrooms and laboratories facilitated better staff/student interaction and as a consequence increased the time spent on campus by both staff and students. b) Equipment The grants included funds for teaching and laboratory equipment. During the course of annual reviews by the BHE and CPCU, the estimated budget for equipment was increased in order to meet the needs of the individual institutions. A total of 165,000 units of teaching and laboratory equipment were purchased thereby exceeding the original target of 73,800 units by 223 percent. Equipment purchases surpassed targets because of cost savings in the civil works sub-components: (a) - 4 - additional upgraded classrooms and laboratories created a demand for further investment in equipment to complete the upgrade of facilities, and (b) the efficiency savings allowed the BHE and the Central Project Coordinating Unit (CPCU) to approve revised investment requests from grant universities for the procurement of additional equipment with saved funding. This resulted in study programs being able to provide increased research topics for students, thereby decreasing the amount of time necessary to obtain degrees. It also increased the research opportunities for faculty, provided large class sets of instrumentation allowing for student participation in experiments rather than demonstrations only, and generally broadened the scope of core and elective course offerings. Additionally, the upgraded facilities provided some universities with the opportunity to strengthen links with local industry, both providing services for income generation and increasing the industrial relevance of research topics. c) Books and Journals The grants provided the recipient universities with resources to increase university and faculty holdings of books and journals. Universities were able to purchase 30,100 titles (75,100 books), exceeding the target of 27,800 titles (70,700 books) by 6 percent; as well as 997 journals, exceeding the target of 966 by 1 percent. The major contributing factors to over-achievement of targets for both books and journals: (a) original price estimates were too high as they were generally based on book-list prices from Indonesian suppliers with high transaction costs, and (b) during the course of the project LPIUs established links with publishers through the internet and procured books and journals directly from a more competative group of publishers. The impact of the books and journals was to increase access to more current academic knowledge for use by student and faculty research. When surveyed during supervision missions, students consistently mentioned the increased library holdings as something that had significant positive impact on the level of their satisfaction with the education they were receiving. These new materials also indirectly contributed to revision of the curriculum and the introduction of new teaching and research topics. While it has been noted elsewhere in this report that in some universities the utilization and transaction rates remain low, the consolidated report prepared by the BHE indicates that for universities in Jami, Lampung and Solo, both the number of visitors to the university library and the number or transactions per day increased by approximately 200 percent from baseline to project close. d) Staff Development Staff development allocations were to provide international and domestic fellowship opportunities to upgrade staff post-graduate qualifications through PhD and Masters programs. Overseas and domestic specialist non-degree training was also provided. During the course of annual reviews by the BHE and CPCU, the budget for Staff Development decreased but planned targets were generally exceeded: 119 overseas PhDs planned, 128 realized; 37 domestic PhDs planned, 26 realized; 130 overseas Masters planned, 112 realized; and 167 domestic Masters planned, 198 realized. Non-degree training targets were also exceeded: 177 overseas planned, 203 realized; 2093 domestic planned, 3419 realized. Three main factors contributed to the general over-achievement of staff development targets: (a) original estimates were based on unit costs which would allow fellows to undertake studies in countries with relatively high education costs (USA, UK and Europe); in practice many fellows undertook studies in countries with lower education costs (Australia, Singapore, Malaysia); (b) fellowship costs were estimated on the basis of 24-30 months for a Masters Degree and 30-42 months for a PhD; in practice approximately 78 percent of the overseas Masters Degree fellows completed their studies in less than 30 months. Similarly, about 25 percent of overseas PhD fellows completed their studies in less than 42 months; and (c) in response to the Asian economic crisis, host universities in a number of countries reduced or waived student fees for Indonesian students. These investments contributed to one of the most important project achievements. The increased numbers - 5 - of teachers and faculty trained helped to improve the quality and attractiveness of study programs. It allowed universities to attract additional research funding through the government's own domestic programs, to increase academic publications, and to expand course offerings. The program supported strong linkages between participating universities and the international scientific community which will help sustain the quality of the study programs. e) Technical Assistance The grants provided the recipient universities resources for domestic and overseas visiting scholars who contributed to curriculum review and development, quality assurance through international benchmarking, and improved teaching practice in the classroom and laboratories. The project also provided resources for technical assistance. In total the development plans indicated 46 person-months for Overseas Visiting Scholars, (of which 35 person-months were actually delivered) and 255 person-months for Domestic Technical Assistance, (223 person-months delivered). The Asian economic crisis and especially the political turmoil in Indonesia were the major contributing factors to the under-achievement of planned targets for both overseas and domestic technical assistance. During 1998-2000 travel advisory alerts dissuaded many overseas consultants from taking up assignments, similarly the political situation made internal travel for domestic consultants difficult. The university in Kupang was particularly affected by United Nations travel restrictions to West Timor. Overall the technical assistance sub-component is rated as satisfactory (except for Kupang where realization was unsatisfactory) as the universities report numerous examples of on-going cooperation agreements established with foreign universities and with regional and premier domestic universities as a result of technical assistance. Such cooperation agreements relate to continuing quality assurance, benchmarking, staff exchange and joint research. In each of the recipient universities (with the exception of Kupang, West Timor) there are examples of revised curricula, the introduction of new teaching and research topics and the development of new teaching materials and practices attributable to technical assistance activities. f) Staff Incentives The grants provided the universities with funds to establish an internal competitive peer review program for the award of teaching and research grants. The planned targets of 560 Teaching Grants and 517 Research Grants were more or less achieved, with a realization of 488 Teaching Grants and 568 Research Grants. The BHE played a strong role in ensuring that these grants were awarded on a competitive basis. Research Grants were generally more attractive to university staff and helped many students reduce the time taken to complete their final research task (Skripsi). On average, three students participated in each Research Grant, bringing the total number of beneficiaries to more than 1,700 students. More significantly in terms of sustainability, the Research Grants broadened the range of Skripsi topics available to students. For example at UNILA, Lampung the number of topics increased eight-fold during the life of the project allowing students to undertake an increased breadth and depth of research. Although competition for Teaching Grants was lower, the quality for selection remained high. The grants proved to be a good mechanism for disseminating good teaching practices by supporting improvements in lesson plans, development of teaching aids and handouts, and the introduction of innovative teaching methodologies. g) Program Management ­ Local Project Implementation Units (LPIUs) Finally, the Grants gave the recipient universities funds to establish and maintain LPIUs in each of the project universities. The IDR portion of this sub-component provided stipends for LPIU staff, covered materials for the LPIU office and travel, while the USD portion supported the cost of office equipment and international technical assistance for an annual review of performance. Overall the performance of the LPIUs is rated as satisfactory. A significant factor in this assessment is the decision of all grant universities to maintain, with university - 6 - funding, a restructured LPIU to continue monitoring the academic performance and quality of educational processes and educational outcomes of Study Programs supported by the project. They intend to broaden this monitoring function over time to encompass all educational programs offered by the universities. Further there is substantial evidence from supervision missions and from the consolidated report of the BHE that the project design, which required LPIUs to be staffed mainly with young academics, provided a mechanism which empowered this new generation of academics and accelerated cultural change within the universities. Component 2. Institutional Capacity Building in Higher Education (US$ 6.4 million SAR, US$ 1.59 million Actual) Overall this component is rated as satisfactory, with some aspects considered highly satisfactory. The objective of the component was to support capacity development of the recently created Board of Higher Education (BHE). As a result, the BHE provided sound and often innovative advice to DGHE which ultimately led to legislative change and advancement of the New Paradigm. In coordination with the selection and peer review process for university grants, the BHE mirrored international best practice, again drawing on international technical assistance to contribute greatly to the perception of a fair and transparent process. The number of staff training and study tours was limited by the GOI in response to the Asian economic crisis, but the quality of activities undertaken was high and the impact substantial. The BHE was less successful in contracting policy studies domestically, at least partly because of the disruption to the domestic consulting market due to social and political turmoil during the middle years of the project. National and regional workshops and seminars were conducted during the project by the BHE. International and domestic technical assistance coordinated through the BHE was provided to monitor implementation progress and provide quality assurance. With project support the BHE offices were equipped and furnished by the project. The DGHE has undertaken to continue the BHE role after project close with GOI funds. Component 3. Implementation of a National Accreditation System for Higher Education (BAN-PT) (US$ 10.0 million SAR, US$ 0.61 million Actual). Overall this component is rated satisfactory. The National Accreditation Board (BAN-PT), with funding from DUE, other donor projects, and domestically supported projects, provided quality assurance during the development and implementation of the block grant funding system. BAN-PT assessed the quality of academic programs using international standards for program accreditation. Under DGHE guidelines, accreditation of a study programs was a prerequisite to apply for block grant funding. During implementation of the DUE project, the assessment process of all Indonesian tertiary level study programs was accelerated so that more than 5500 S1 programs were accredited. This accounts for approximately 87 percent of all the S1 undergraduate programs. Maintenance of the quality assurance system is critical for he institutionalization of the block grant system within the higer education system. Component 4. Competitive Domestic Fellowship Program (US$ 9.9 million SAR, US$ 4.09 million Actual) Overall this component is rated partially satisfactory. The primary objective was to address the low qualifications of teaching staff in the 27 least developed universities by supporting a competitive domestic fellowship program. The program was designed to enable 1,000 recent S1 (undergraduate), graduates, or teaching staff with S1 qualifications to pursue Master degrees at Indonesian universities. Fellowships were provided to 849 candidates, 85 percent of the original SAR target. At project completion, 739 or 87 percent have completed studies and taken up teaching duties at their home university. Of the remaining 110 candidates, 77 are expected to commence teaching duties during the second semester of 2002 and 33 in the first semester 2003. Less than 2 percent of fellows undertook studies in their home university and more than 50 percent completed studies at the four most prestigious Indonesian universities. A secondary objective was to support a comparative analysis of the remuneration and incentives structure for university teachers. The Board of Higher Education undertook the study. - 7 - Component 5. Project Administration (US$ 2.4 million SAR, US$ 1.26 million Actual) The overall rating for this component is satisfactory. The objective of the component was to support the Central Project Coordinating Unit (CPCU) in its roles to a) monitor and assist LPIUs in implementing their respective projects; and (b) to collect and analyze data from the target universities. The initial budget from Loan funds was US$ 1.1 million. During the GOI initiative to reduce the level of borrowing across all sectors in 1999 this budget was cut to US$ 750,000. However, at project close actual expenses paid from the Loan were US$ 1.01 million. Several factors contributed to the budget overrun: (a) monitoring costs for the 3-year Grants at the two universities added to Component 1 were higher than anticipated; (b) in response to the Asian economic crisis the disbursement ratio was revised from 50/50 IBRD/GOI to 80/20; and (c) the number of standing Discipline Commissions (KDI) supported under the BHE was increased from seven to thirteen for a trial period of 20 months before the DGHE restructured the BHE. The CPCU performance was satisfactory in terms of monitoring and assisting LPIUs to implement their respective projects. Throughout the project the CPCU: (a) coordinated bi-annual meetings at which LPIUs shared examples of best practice and developed strategies to address shared implementation problems; (b) organized procurement and financial management workshops in conjunction with Bank staff from the Resident Mission; (c) played a central role in annual procurement planning; and (d) provided procurement and financial support to LPIUs on specific issues. In its other major task of collecting and analyzing data from the target universities, the CPCU was less successful. While the CPCU ensured that annual reports were provided by each LPIU and that each LPIU prepared and submitted for Bank prior review annual procurement planning the CPCU was not fully successful in providing consolidated reports. The overall level of analysis of raw data was low, and fragmented by university. It did not provide a comprehensive overview of the project. This was particularly apparent in the financial reporting. 4.3 Net Present Value/Economic rate of return: Information on graduate enrollment status and wage level is needed to calculate rates of return on investments. Such data was requested of the eight universities and three responded but with differing levels of detail and completeness. This incomplete data does not agree in every instance with the data contained in study program final reports which were far more detailed and qualitative in their presentation and were used for the analysis in Section 4. Achievement of Objective and Outputs. Nevertheless, a partial analysis of the data from the three universities was possible and preliminary results point to efficiency gains of the project in terms of a reduction in the length of study for one university in the order of about two months while the initial rate of return calculations assumed six months. GPA scores generally show a positive trend. One university was able to reverse a negative GPA trend into a positive one and thus seems to have achieved the salary increase that was expected in the appraisal report, but fell short of the reduction in search time for the first job by two months. For another university there was a small positive trend in test scores, but it is difficult to attribute this to the project. 4.4 Financial rate of return: Not applicable. 4.5 Institutional development impact: The project introduced a number of innovations which, after being successfully trialed, have been institutionalized and disseminated more broadly in the Indonesian higher education community. The introduction of the block grant program with its transparent, international standards of peer review and - 8 - selection has helped change the resource allocation process as well as the academic culture in project supported study programs. Individual university final reports indicate that the universities are distributing their own research funds to other study programs based on the peer review process. In addition, during discussion about the project they often refer to improvements in staff attitudes due to the transparency in resource allocations. At the national level, the MoNE has chosen to use the block grant funding process piloted under the project in its own government supported higher education projects. As a consequence, 28 percent of the higher education budget was distributed by the DGHE in 2002 up from less than 1 percent in 1992. BAN-PT has been able to assess and accredit approximately 87 percent of the S1 programs over the past five years. The core values of the project were also incorporated into legislation which has now established the four major universities as autonomous institutions. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: During project implementation, the East Asian financial crisis struck Indonesia and had a substantial impact on all services provided by the GOI as well as the economic environment within which the project was being implemented. Ties with industry weakened as financial concerns superseded the needs for research and development. 5.2 Factors generally subject to government control: Midway through the implementation of the project, the DGHE decided that the model of new block grants at the core of the DUE loan were successful and as a result established two domestically supported block grant schemes. It was also at this point that the DGHE showed its strong support and commitment to the block grant mechanism by not cutting the grants during the financial crisis. 5.3 Factors generally subject to implementing agency control: The project experienced early problems in implementing World Bank procurement procedures. The CPCU took steps to build its procurement planning and monitoring capacity to overcome the unfamiliarity with World Bank Guidelines and also acted as a mentor to LPIUs that were experiencing difficulties. Financial management capacity within the CPCU and LPIU was an issue that was not addressed until later in the project implementation. However, the Bank's Indonesia disbursement office did provide financial management training for the CPCU. The LPIUs generally devoted sufficient resources to the collection of data performance indicators and in response to supervision mission comments developed ancillary performance indicators. Howver, there was little evidence that sufficient resources were devoted to analysis of data and the incorporation of lessons learned in routine decision-making. 5.4 Costs and financing: The project became effective September 12, 1996 and the project costs were estimated to be US$65.0 million. There were three loan cancellations during the life of the project, reducing the overall loan funds to US$57.75 million. Exchange rate savings as well as the difficulty of the Indonesia government in meeting the Rupiah counterpart commitments contributed to the decision to cancel a portion of the loan. The counterpart ratio was also modified for the overall project to an 80:20 split for all disbursement categories. However, a strong commitment to the DUE Grant program was maintained and the budget for this category of the Loan was not reduced. 6. Sustainability 6.1 Rationale for sustainability rating: - 9 - Sustainability of many aspects of this project is likely. The DUE project provided financing to support the introduction and institutionalization of models of new practice to support the New Paradigm. The MoNE, and especially the DGHE, recognizing that a good policy framework (the New Paradigm) alone was insufficient for sustainable change, supported a socialization process to develop a climate of acceptance of the framework's ideas as well as sound and verifiable implementation experience to establish a body of supporting evidence. In addition, they established the legal and regulatory framework necessary for the change to be sustainable. Thus sustainability should be seen as financial and institutional and both are likely, given the success of the project and the policy environment now in place within the Indonesian higher education community. Positive indications of both institutional and financial sustainability of the new model can be found in the following actions. At the national level, the DGHE has established two new domestic block grant funding mechanisms using the same selection and quality assurance bodies initially setup and supported by the DUE project. The percentage of GOI funding for higher education now using block grant funding mechanisms has risen from less than 1 percent in 1992 to 28 percent. The GOI has established a target of 50 percent by the year 2006. The DGHE, in anticipation of project closing and in order to address the issue of sustainability, has: (i) reorganized and streamlined the BHE, divesting it of the thirteen standing disciplinary commissaries and replaced them with a more flexible ad hoc mechanism (the new, less costly, BHE will be funded from GOI sources); (ii) directed existing LPIUs at the institution level should reduce their size and focus attention on monitoring and evaluating the quality of the DUE investments and assure that equipment and civil works are adequately maintained; and (iii) directed the annual amounts of self-raised funds (DRK) which were used for counterpart funds to sustain project investments. Universities have devised a number of ways to continue financing core activities (e.g. bench fees for equipment use and user fees for computer and language centers). Some tuition revenues were used for research and computational equipment, and services were sold to other universities. Although not large in scale, demonstrate a commitment to institutionalize the innovations made under the project. 6.2 Transition arrangement to regular operations: Most of the project activities have been institutionalized as a part of the higher education system and the distribution mechanism for university funding within the Indonesian academic community. Models of new practice to support the first phase of the implementation of the New Paradigm are supported now by the DUE-like and semi-QUE projects put in place by the MoNE and the DGHE in late-1999 and mid-2000 respectively. The BHE and its quality control responsibilities has been restructured to serve the continuing needs of the currently implementing Bank-supported Quality of Undergraduate of Education (QUE) and the ADB-supported Polytechnic project as well as the DUE-like and semi-QUE projects. The BAN-PT continues its work and has a widening responsibility of study program and institutional accreditation. 7. Bank and Borrower Performance Bank 7.1 Lending: The project design built on a substantial base of analytical work in higher education projects in Indonesia and other large East Asian countries, as well as the work of Indonesian academic Synergy was achieved between best practices the Bank had to offer and senior Indonesian academics' analysis of the issues the sector. 7.2 Supervision: Bank supervision in the early days of the project maintained the originally envisioned schedule although budgetary restrictions limited the composition of the supervision teams. Supervision was interupted during - 10 - the East Asian financial crisis and the turmoil that followed. However, the policy and implementation review mechanisms put into place before the financial crisis allowed for continued implementation of the project at a reasonable pace and with fundamentally good results. As a result, when the Bank was able to resume full scale supervision, supervision reports and aide-memoires indicated that the project was on track and yielding good results. 7.3 Overall Bank performance: The Bank's overall performance is satisfactory. There was reasonable continuity of staff throughout preparattion and supervision. Supervision missions included an appropriate mix of expertise providing regular guidance on sectoral issues, as well as procurement and financial management procedures. With members of the Bank team on three continents, procurement clearance procedures proved cumbersome and time consuming, and resulted in many delays early on in the process. The introduction of more robust electronic mail services the process speeded up and by the end of the project delays in the issuance of NOLs had decreased considerably. Borrower 7.4 Preparation: Borrower involvement and participation in the preparation of the project was excellent. There was a clear understanding of the problems based on a thorough analysis of lessons learned from the implementation of past projects, and the project was seen as a major mechanism to facilitate desired changes in the DGHE 10-year development plan. The borrower approached the preparation of the project in an exemplary manner, assigning experienced and highly motivated staff to the preparation team. 7.5 Government implementation performance: Overall implementation performance by the Borrower was satisfactory across the three levels at which the project operated; National ­ BHE, Central ­ CPCU, and Institutional ­ LPIU. The BHE was established to provide policy advice to the DGHE, to coordinate the peer review and selection of university grant recipients, conduct benchmark testing, to review annual reporting by the LPIUs, and to provide review, clearance of variations in the university grants. The BHE performed the first two tasks very well. The BHE examined long standing structural and governance problems in Indonesian higher education and provided often innovative advice to DGHE. In coordination of the selection and peer review process for university grants, the BHE mirrored international best practice and helped to create a fair and transparent process. In other tasks the BHE was less successful. Benchmark testing was often not carried out on schedule and the BHE was slow in communicating results to the LPIUs. The capacity of the BHE to review and clear university grants was limited particularly in terms of financial management and record keeping. Capacity building in these areas should be a central concern for DGHE re-structuring of the BHE if the BHE is to exercise its full potential in coordinating university grant programs. The CPCU was responsible for: (a) disbursement of university grant funds following review of LPIU bi-annual reports of financial and physical progress; (b) monitoring overall loan expenditures; (c) providing progress reports for the Bank; (d) monitoring procurement activities; and (e) collecting data for monitoring and evaluation. Overall the CPCU ensured the timely review and disbursement of university grant funds in a commendable manner. Monitoring loan expenditures, providing progress reports to the Bank, monitoring procurement and collecting monitoring and evaluation data were carried out in a satisfactory manner. Two areas not fully mastered are: (a) consolidated reporting, (providing an overview of project performance rather than the performance of individual institutions) and (b) financial reporting and analysis. With support from the financial management unit at the Resident Mission, significant gains were made in the last - 11 - 18 months of project implementation to standardize financial reporting formats and to make sound cash flow forecasts. Throughout the project the CPCU took steps to build its own capacity for procurement planning and monitoring and to support the LPIUs in procurement matters. The LPIUs were established in each recipient university with responsibility to: (i) implement the grant contract with the CPCU, including management of financial and procurement activities; (ii) establish baselines and routinely collect data on the achievement of performance indicators and (iii) provide monthly, quarterly and consolidated annual reports to the CPCU and the Board of Higher Education (BHE). Generally the LPIUs undertook these tasks in a commendable fashion and achieved considerable success given the staff of the LPIUs had academic rather than financial and implementation backgrounds. In most cases the collection of data on performance indicators was carried out although there is little evidence of systematic analysis of the data and its use in management decision making. The DGHE took an active interest in the implementation of the project and was quick to intervene decisively when his assistance was sought. The DG played a significant role as a champion of the university grant mechanism and of the New Paradigm approach of decentralizing decision-making, insisting that accountability goes hand-in-hand with greater autonomy. 7.6 Implementing Agency: As discussed in detail in 7.5 above overall performance of the implementing agencies, BHE, CPCU and the LPIUs, was satisfactory. 7.7 Overall Borrower performance: Overall Borrower performance was satisfactory. With regard to institutionalizing the New Paradigm and the principles of transparent merit-based selection, performance was highly satisfactory. Support from the Ministry of Finance (MOF) was also apparent in that the block grant component was not reduced during major GOI initiatives to reduce overall GOI borrowing in 1998 and 1999. However, further socialization of the nature and benefits of the block grant process needs to be undertaken throughout MOF if MoNE is to fulfill its plan to continue expanding block grant programs. Additionally, financial management capacity at the LPIU and CPCU levels and within the BHE remains a weak point. 8. Lessons Learned l Transparent grant programs, based on agreed international standards or quality assessment and peer review, can change academic culture and resource allocation; and can contribute to improving the quality and efficiency of education outcomes. l Block grant funding mechanisms need to be supported by socialization to develop a climate of acceptance, recording of implementation experience to establish a body of supporting evidence, and finally reform of the legal and regulatory framework. l A CPCU is essential to support decentralized implementation, and to be fully effective the CPCU procurement and financial management capacity needs to be strong. l National level institutions, using international standards, are necessary to ensure the integrity of the selection process, the high quality of the proposals, and effective monitoring and evaluation block grant programs. l Dissemination and feedback mechanisms must be incorporated into project design in order to develop examples of best practice and support systemic change. l Early and continuous training in procurement and financial management from the Bank is necessary for improving response time in project implementation. l Project management training builds capacity for implementation of local institutions' own projects. - 12 - l Reforming the structure and governance of institutions of higher education necessitates, among other things, changing the way the money flows in the institution. In this case project financing provided the incentive for such change at the study program level and allowed the universities to experiment with innovative ways in restructuring the study programs. 9. Partner Comments (a) Borrower/implementing agency: See Annex 8 for borrower portion of the ICR. (b) Cofinanciers: Not Applicable (c) Other partners (NGOs/private sector): Not Applicable 10. Additional Information - 13 - Annex 1. Key Performance Indicators/Log Frame Matrix Achievement of Final Performance Targets by Study Programs Supported in the Eight University Grants Under Achievement of Achievement in excess of Primary No Final Target (X) Final Target (Y) Performance As per Data Indicators X > 10 % 9%5 Target % 4%1% 4%1% 9%5% Y > 10 % Independent Benchmark Score 11 5 3 3 1 1 Entrance Examination Score 1 4 2 3 1 6 4 3 Grade Point Average 1 3 6 2 6 6 Reduced Length of Study 1 2 5 3 7 4 2 Reduced Waiting Time to First 4 2 2 1 6 1 1 7 Employment TOTAL 17 9 12 18 19 20 12 13 For the Study Programs for which data is available: 8 of the 13 (62%) met or exceeded the final performance target for the Independent Benchmark Score 14 of the 23 (61%) met or exceeded the final performance target for the Entrance Test Score 12 of the 24 (50%) met or exceeded the final performance target for the improvement in Grade Point Average 13 of the 23 (57%) met or exceeded the final performance target for reduction in time to complete studies 15 of the 20 (75%) met or exceeded the final performance target for reduction in waiting time to first employment - 14 - Annex 2. Project Costs and Financing Project Cost by Component (in US$ million equivalent) Appraisal Actual/Latest Percentage of Estimate Estimate Appraisal Component US$ million US$ million Universities Development Program 63.30 61.96 97.97 Institutional Capacity Building in Higher Education 6.40 1.59 24.84 Implementation of National Accreditation System for 10.00 0.61 6.1 Higher Eduction Competitive Domestic Fellowship Program 9.90 4.09 41.29 Project Management 2.40 1.26 52.6 Total Baseline Cost 92.00 69.51 Physical Contingencies 2.00 Price Contingencies 8.10 Total Project Costs 102.10 69.51 Total Financing Required 102.10 69.51 Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) 1 Procurement Method Expenditure Category ICB NCB 2 N.B.F. Total Cost Other 1. Works 0.00 0.50 0.00 0.00 0.50 (0.00) (0.40) (0.00) (0.00) (0.40) 2. Goods 0.00 0.70 0.30 0.00 1.00 (0.00) (0.50) (0.20) (0.00) (0.70) 3. Services 0.00 0.00 90.00 0.00 90.00 (0.00) (0.00) (63.90) (0.00) (63.90) 4. Incremental Recurrent 0.00 0.00 0.00 10.60 10.60 Costs (0.00) (0.00) (0.00) (0.00) (0.00) Total 0.00 1.20 90.30 10.60 102.10 (0.00) (0.90) (64.10) (0.00) (65.00) - 15 - Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) 1 Procurement Method Expenditure Category ICB NCB 2 N.B.F. Total Cost Other 1. Works 0.00 0.01 0.00 0.00 0.01 (0.00) (0.00) (0.00) (0.00) (0.00) 2. Goods 0.00 0.28 0.00 0.00 0.28 (0.00) (0.16) (0.00) (0.00) (0.16) 3. Services 0.00 0.00 55.46 0.00 55.46 (0.00) (0.00) (13.60) (0.00) (13.60) 4. Incremental Recurrent 0.00 0.00 0.00 0.00 0.00 Costs (0.00) (0.00) (0.00) (0.00) (0.00) Total 0.00 0.29 55.46 0.00 55.75 (0.00) (0.16) (13.60) (0.00) (13.76) 1/Figures in parenthesis are the amounts to be financed by the Bank Loan. All costs include contingencies. 2/Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. Project Financing by Component (in US$ million equivalent) Percentage of Appraisal Component Appraisal Estimate Actual/Latest Estimate Bank Govt. CoF. Bank Govt. CoF. Bank Govt. CoF. Universities Development 50.80 17.30 49.56 12.40 97.6 71.7 Grants Civil Works 0.40 0.10 0.01 0.00 2.5 0.0 Equipment 0.70 0.20 0.29 0.16 41.4 80.0 Domestic Fellowships 5.60 5.60 3.27 0.82 58.4 14.6 Non-Degree Training Overseas 1.80 0.00 0.73 0.00 40.6 0.0 Domestic 0.80 0.80 0.14 0.04 17.5 5.0 Studies 1.60 0.00 0.37 0.00 23.1 0.0 Consultants' Services 1.10 0.30 0.37 0.09 33.6 30.0 Project Management 1.10 1.10 1.01 0.25 91.8 22.7 Incremental Recurrent 0.00 10.60 0.00 0.00 0.0 0.0 Costs Unallocated 1.10 1.10 0.00 0.00 0.0 0.0 Total 65.00 37.10 55.75 13.76 85.8 37.1 - 16 - Annex 3. Economic Costs and Benefits In the project appraisal document the expected rate of return for the higher education project was calculated on the basis of expected improvements in the length of study time, length of time it takes to find a first job, and starting salaries. The assumptions were that the project would: (i) reduce the time needed for students to graduate from 5.75 to 5.25 years; (ii) reduce the time needed for graduates to find their first job from 1.75 to 1.5 years; and (iii) increase starting salaries once employed by 12 percent per month, growing by 3 percent a year. Under these assumptions the project would yield a return of 24 percent. During implementation of the project data from agreed indicators on the performance of study programs were gathered and analyzed but no specific data were systematically collected to track the validity of these expected rate of return assumptions. It would have required data collection through tracer studies and administrative data, both for the universities which were selected into the project, and a comparable set of universities. Therefore, making an estimate of the rate of return at this stage, at the close of the project is difficult. However, universities were asked for individual student records on test scores, length of study, search time to find the first job and starting salaries before and at the end of the project. Three out of eight universities responded to the request. Test scores were requested to estimate the relationship between starting search time/starting salaries and test scores. Since many other factors will have influenced changes in starting salaries and search time over the course of the project, such as the economic crisis, a comparison of these variables at the beginning and the end of the project cannot provide a robust set of results, but rather a useful approximation which can help inform the Implementation Completion Review. Analysis: Comparing the length of study and test scores before and after the project the data for two universities show that the average length of study decreased slightly, by two and three months respectively, somewhat less than the six months that was assumed in the initial rate of return calculation. No difference can be found in the grade point average (GPA) for the first university although during the project the GPA scores dipped and then were coming back up by the end of the project. However, increases in test scores at this university seem to have come at the cost of a rise in study time. At a second university the GPA increased by 0.08 point but had been experiencing a positive trend in GPA scores for a long time and the trend did not change much over time during the project. Data from the third university show a negative trend. Test scores, length of study and starting salaries was also reviewed. Only the data from one of the three responding universities are suitable to provide an answer to this question. These results show that GPA score has a significant positive influence on expected earnings. A one point increase in GPA scores yielded an expected salary increase of 30 percent. The length of study had no significant influence on starting salaries. Females earn on average 17 percent less than their male fellow students. Similar, but opposite results are found for the length of time it takes to find the first job. Higher GPA scores help to reduce search time and females need on average two months more to find their first job. Conclusion: The results point to efficiency gains of the project in terms of a reduction in the length of study for one university in the order of about two months while the initial rate of return calculations assumed six months. At the second university, the length of study fell in the initial phases of the project but increased in recent years. GPA scores generally show a positive trend. The second university has been able to reverse a negative trend into a positive one and thus seems to have achieved the salary increase that was expected in the appraisal report, but fell short of the reduction in search time for the first job by two months. For the first university there is a positive trend in test scores, but it is difficult to attribute this to the project. Over the course of the project the GPA score did not improve more than 0.1 percent. - 17 - A detailed analysis on data provided by three universities is available in the project files. - 18 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, 1 FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation 2/17/95 5 Economist (2); Higher Ed. S S Specialist (1); Enginnering Specialist (1); Operations Officer (1) 9/15/95 5 Economist (2); Higher Ed. S S Specialist (2); Operations Ofiicer (1) 5/21/95 5 Economist (2); Higher Ed. S S Specialist (2); Operations Officer (1) 11/24/95 3 Higher Ed. Specialist (2); S S Operations Officer (1) Appraisal/Negotiation 2/16/96 7 Higher Ed. Specialist (2); S S Education Specialist (1); Human Resource Specialist (1); Economist (1); Operations Specialists (2) 5/14/96 5 Higher Ed. Specialist (1); S S Operations Officer (2); Lawyer (1); Disbursement Specialist (1) Supervision 08/02/1996 4 Operations Officer (1) Economist S S TTL (1); Education Specialist (1); Higher Education Specialist (1) 08/29/1997 3 Sr. Economist (1); Operations S S Officer (1); Consutant (1) 12/04/1998 4 Education Specialist-TTL (1); S S Operations Officer (2) Higher Education Specialist (1) 09/10/1999 2 Operations Officer-TTL (1) S S Procurement Officer (1) 07/28/2000 3 Operations Officer-TTL (1) S S Higher Education Specialist (1); Implementation Specialist (1) 06/19/2001 3 Operation Officer-TTL (1); S S Consultant (1); Education Specialist (1) 01/25/2002 4 Lead General Educator (1); S S Consultant (2); Education/Health Coordinator (1) ICR - 19 - 10/3/02 4 Education Specialist (1); S S Higher Education Specialist (1); Operations Officers (2) (b) Staff: Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation 47.8 184.9 Appraisal/Negotiation 37.6 143.2 Supervision 125.0 330.0 ICR 8.0 30.0 Total 218.4 688.1 These figures are estimates based on the project preparation file information, supervision aide-memoires, 590 forms and staff and salary information that is available. - 20 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA - 21 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU - 22 - Annex 7. List of Supporting Documents Staff Appraisal Report - May 22, 1996 Project Status Reports (PSRs) Mission Aide-Memoires Project Financial Reports Project Correspondence Borrower ICR University Study Program Completion Reports - 23 - Additional Annex 8. Borrower's ICR Contribution IMPLEMENTATION COMPLETION REPORT DEVELOPMENT OF UNDERGRADUATE EDUCATION PROJECT IBRD LOAN 4043-IND DIRECTORATE GENERAL OF HIGHER EDUCATION MINISTRY OF NATIONAL EDUCATION Introduction Development of Undergraduate Education Project (DUE-Project) is the project developed by the Directorate General of Higher Education (DGHE) under the Ministry of National Education (MONE) Republic of Indonesia. The Loan Agreement with Loan Number 4043-IND, dated August 12, 1996 between Republic of Indonesia as borrower and International Bank for Reconstruction and Development (the Bank), is completed on March 31, 2002 and closing date on September 30, 2002. The total amount of the loan is sixty five million dollars ($65,000,000) being used for Higher Education Support. Due to certain circumstances the loan is cancelled 1998 in amount of $ 6,855,000 so that the total loan become $58,145,000. After the closing date the balance in amount of $2,385,455 was cancelled on March 2003 bringing the total cancellation to $9,240,619.65 and a final disbursement of $55,759,380.35. Lesson learned from development project which were implemented during the period before 1995 was encouraged the DGHE to develop an alternative scheme of development project for higher education which ensure sustainability and accountability. Investment projects during those previous period has been done on the basis of investment categories such as fellowship, libraries, and laboratories. Although the specific needs at the investment site had been taken into consideration, the design of those projects was done mainly using a top-down approach where sustainability and accountability were only limited concern. The lack of integration with the university's overall plan created a divergence from the university's mission and objective, resulted in a minimal ownership by the university member particularly academic staff. The focus on specific project had created a problem in applying priorities, and tend to develop overlap with minimal synergy. The long term development strategy of higher education for period 1996-2005 was formulated by the DGHE in the Document of Long Term Higher Education Development Framework 1996-2005 (KPPT-JP III). The basic policies of this development strategy, are: a) expansion of opportunity, b) improvement of relevance and quality, c) improvement of efficiency, and d) new paradigm in higher education management. The latter, the new paradigm concept in higher education management, is the policy that focuses on educational outcome, decentralized planning, performance based as well as competitive based resource allocation, accountability, and sustainability. Through the new paradigm concept, the new approach had been established. The plan in this new paradigm are developed through a bottom up approach, through which stronger ownership, broader participation in the planning stage, and higher commitment that ensure sustainability would be established. Objectives and Components The DUE project was the pilot or pioneer project in field of education of higher education system in introducing the new paradigm concept in higher education management which basically a shift toward more autonomy and decentralized management. The project has three objectives : (a) to improve the quality of undergraduate education, (b) to improve the efficiency of the educational process and, (c) to improve the relevance study programs offered. Under the new paradigm concept, a competitive funding mechanism was applied. The DUE project comprised of five components : 1) Universities Development - 24 - Program which provided a grant to eight universities to enable them carry out their development programs to strengthen undergraduate education, 2) Institutional Capacity Building in Higher Education by establishing the Board of Higher Education (BHE), 3) Implementation of Nation Accreditation System for Higher Education, 4) Competitive Domestic Fellowship Program, and 5) Project Administration. Component 1. Universities Development Program. Universities Development Program was the first competitive project which had been designed to support institutional capacity building of less established public universities in Indonesia. Through this mechanism, eight less established universities had been selected to be a grantees of DUE Project among 17 public universities which batch 1 consists of 6 universities, i.e. University of Riau (UNRI), University of Bengkulu (UNIB), University of Lampung (UNILA), University of Sebelas Maret ­ Solo (UNS), University of Jember (UNEJ), University of Nusa Cendana (UNDANA) for five years of implementation. Batch two consists of University of Jambi (UNJA) and University of General Soedirman Purwokerto (UNSOED) for three years of implementation. In implementing a new paradigm in higher education, a competition of DUE Project was tier competition and provided through a competition funding mechanism. Two stages selection process were carried out in the competition, the first stage was to review on pre-proposal submitted by universities which basically comprises of self evaluation and 5-years strategic plan for Batch I and 3-years strategic plan for Batch II, and the second stage was to review on full proposal submitted by the universities which passed the first stage. The review was the peers review process with the criteria used in both stages was Leadership and Institutional Commitment, Relevance, Academic Atmosphere, Internal Management, Sustainability and Efficiency and Productivity which is abbreviated as L-RAISE. The guidelines for proposal preparation including explanation about the criteria that would be used in the review process as well as submission procedures for each stage were supplemented to the invitation letter for competition signed by the Director General of Higher Education. The participated universities in the DUE project competition and those which finally won the competition can be found in the project files. Target universities received budget for strengthening undergraduate education which was providing fellowship for teaching staff, training programs for teaching and administrative staff, provision of teaching and research grants, provision of technical assistance through visiting scholars, provision of books, journals, teaching materials, and equipment, and rehabilitation and upgrading of facilities. Beneficiary Units A beneficiary units of DUE project in each universities comprised of several study programs and University Wide Programs. Since the selection was an institutional-based all study program and University Wide Programs (UWPs) explained in the pre-proposal and full-proposal of each university were beneficiary units of DUE project of respective university. University Wide Program is the programs designed by the university's services units in which the beneficiary population are students of all study programs offered by the university, not limited to students of DUE project beneficiary study program. Central Library, Computer Centers are some examples of University Wide Programs. The leadership of this service unit usually directly responsible to the university leadership. Performance indicators To measure the achievement in improving the quality of education at each grantee, a set of main performance indicators and auxiliary indicators had been established. The baseline data of those indicators were stated in proposal of each grantee. The main indicators were the same for all grantees, whilst the auxiliary indicators were varied among universities according to activities designed by each university for - 25 - improving quality. The main performance indicators set up were: student entrance examination score (UMPTN score), length of study, GPA score, benchmarking test score, TOEFL score, and waiting time to get a job. Main performance and auxiliary indicators are measured since the beginning of the project so called baseline data, in the middle of the length of the project so called midterm and final year of the end of project. It can be seen that all of indicator tend to increase gradually and are achieving the target design from project proposal except UMPTN score and some target are exceeding the target. Extension The contract between CPCU and LPIU Batch I for 5 years, from December 27,1996 until December 26, 2001 and being extended until 15 July 2002. And contract for Batch II for 3 years starting from June 15, 1999 until June 14, 2002 and being extended until July 15, 2002. Finance The total allocated for Grants $50,800,000 and after cancellation of amount $600,000 the actual disbursement become $49,559,438.67. The remaining budget for this category after closing date in amount of $640,561.33 Component 2 : Institutional Capacity Building in Higher Education Background The Board of Higher Education (originally referred to as Higher Education Council) was established in May 1996 under the decree of Minister of Education Culture No.0121/U/1996. The main responsibility of the BHE is providing recommendations to the Minister of Education through the Director General for Higher Education on policies regarding the operation and the development of higher education. Specifically, the board undertakes three major scopes of duties, i.e. Education, Research and Community Services and Development. The internal structure of the board consists of one chairperson, one Secretary, four Vice-Chairpersons, and members. The first membership of the Board has finished its term, and new members have been appointed for the second membership in 2002. A significant policy change has been introduced in the selection of new members. Currently the Board is chaired by the Director General of Higher Education comprising 17 ex-officio members representing the DGHE, Indonesian Science Academy, Board of National Education Advisory, The Assessment and Application of Technology Agencies, Indonesian Institute of Science, Minister of National Planning, Commission VI of the Parliament, Indonesian Chamber of Commerce, Ministry of Trade and Industries, Ministry Labor and Transmigration, National Research Council, Council of Rectors of the public higher education institutions, and Association of Private Higher Education Institutions. The only non ex-officio members in the Board are the Vice Chairperson for Development and the Secretary of the Board. When the Board was established in 1996, it was supported by 3 councils, the Development Council, the Research Council, and the Education Council and 13 Discipline Based Commissions. Each council is chaired by Vice Chairperson of the BHE. Later on, for the reason of efficiency, the discipline based commissions was dissolved in 2002. Currently, the board is overseeing the three councils only and ad hoc committees and the ad-hoc committees are established to undertake specific task defined by the BHE. The council' task are set forth by the BHE and its members are appointed and discharged by the Director General based on the proposal submitted by the BHE. Undertaking Activities Investment projects under the new paradigm have the main characteristics of being competitive based allocation. There are currently four major projects under this scheme, namely the Development of - 26 - Undergraduate Education (DUE), the Quality of Undergraduate Education (QUE), the Development of Undergraduate Education purely supported by the Government of Indonesia (DUE-Like), and the Technical and Professional Skills Development Project (TPSDP). The first two are supported by the IBRD loan, whilst the last one is supported by the ADB loan. The BHE has been very instrumental in supporting the DGHE in implementing the above four projects. There are essentially three major forms of the BHE involvement, i.e. project preparation, selection process, evaluation and monitoring. During the project preparation, the BHE provides recommendation on the project design and develops selection criteria as well as guidelines for proposal development and submission. In the aspect of evaluation and monitoring of on going projects, the BHE is also supporting the project coordinator by providing necessary instruments and human resources for the monitoring and evaluation process. There are basically four types of monitoring and evaluation, the base-line, annual, mid-term, and final. The BHE has developed appropriate scheme and instruments for each type of the evaluation. By and large, the monitoring process for the above four project schemes has been carried out quite commendably. Almost similar to the selection process, in the monitoring process the BHE recruited reviewers and deployed them based on the specific request from the project coordinator. The monitoring results in the form of consolidated reviewers' comments and recommendations are used as the basis for determining the quality of the overall project implementation at individual grantee, based on which the BHE developed recommendation to the DG. Attempt to develop proper monitoring schemes was initiated by the BHE in 1998 in collaboration with the Quality Assurance Agency ­ the United Kingdom. This includes the development of guidelines for monitoring and evaluation as well as training for domestic peers (evaluators). There are however still some shortcomings encountered with regard to this monitoring activity. It is realized then that the BHE seems to go beyond its intended task, tends to incline itself toward a project implementer rather than a peer group. Furthermore, there are problems whether to monitor quality of programs or implementation progress or both. Of equally interesting question is regarding whether to monitor merely academic issues or to include also financial matters. Since most if not all reviewers are academia, their interest as well as expertise will be more on program quality and academic issues rather than financial and administrative matters. The BHE is however fully realized that the two aspects are of equally important to be monitored to ensure successful implementation of any projects. Contribution to the reform agenda The higher education reform in Indonesia was initiated in early 1996 as an implementation of the Long Term Development Framework of Higher Education (KKPT-JP) 1996 ­ 2005. Responding to this, the BHE took the initiative to hold a scientific event to solicit inputs and lessons learned regarding the higher education reforms from overseas as well as domestic experts. In August 2001 an International Conference on Higher Education Reform was held by the BHE involving 8 distinguished experts from various nationalities as invited speakers, and 16 free papers from domestic and overseas contributors. The conference focused on three main themes, i.e. The global trends and country specific initiative in introducing higher education reform, University's responses to the reform initiative, and Funding mechanism in higher education reform. The conference attracted around 400 participants. One of the key elements of the abovementioned reform agenda is the corporatization of public higher education institutions that is to provide wider autonomy to a public higher education institution. In this - 27 - regard, the BHE has played very important roles during the preparation as well as the process of establishment of the four leading universities (UI, UGM, ITB, IPB) under the new legal status. During the preparation stage, the BHE was involved in developing guidelines for the statutory changes of a public university to become a state legal entity. The BHE was also assigned by the DG to review the documents submitted by the four universities and evaluate their preparedness to embrace the new legal status. Due to limited domestic experiences concerning this statutory change, the BHE recruited overseas experts to work closely with the BHE team during the review process. The BHE also took the initiative to propose to some donor agencies to support financing overseas technical assistants to supervise the four institutions preparing their documents. Upon the establishment of the four leading universities as a legal entity, the BHE took a progressive step by facilitating the four institutions to developing their internal quality assurance system. This initiative has been strongly supported by the DSE (Germany) and the HEFCE (the UK), by providing the opportunities for representatives from the four universities to take short courses pertaining to the QA in higher education. Regulations The BHE has been involved also in preparing academic papers for a number of government regulations pertaining to higher education. To name a few examples are: the Government Regulation No. 60/1999 on Higher Education System, the Government Regulation on Appraisal System for University Lecturer. Recently, the BHE has set up an ad-hoc commission for developing academic papers for higher education act. Policy Studies On November 2000, the BHE took the initiative to carry out a study focusing on two main issues, namely: i) University's Responses to the Strategic Document developed by the DGHE, and ii) The expected roles of DGHE, BHE, and NAB from the perspective of HEIs. Two strategic documents were being addressed in the study, they are the Long-term development framework of higher education (KPPT-JP) 1996 ­ 2005, and the Higher Education Strategies. Substantial findings were resulted from the study, among others are: l majority of respondents agree that the existing development framework and strategies are still relevant and needs sustaining; l there are mix expectations regarding the future roles of DGHE, BHE and NAB; although most are in agreement that the DGHE should be more as facilitator rather than regulator, the BHE shall position itself as a peer, and the NAB shall be more as an independent body rather than under the auspice of the DGHE; l quite reciprocal with the desire to seek greater autonomy, significant portion of respondents still expect the central authority to set the core curriculum as well as establish and terminate programs; In addition, each council has also conducted numerous policy studies pertinent to issues within the scope of the council. Though, it will be too lengthy to include all of them in this report. Comparative studies and linkages Since its establishment, the board has been very keen to observe the overall trend and development of higher education sector locally and globally. In light of the institutional capacity building within the board, a team of key members of the board has been assigned to undertake comparative studies to the United Kingdom and Germany. - 28 - The United Kingdom: develop and carry out institutional collaboration with HEFCE and QAA. The collaboration resulted in the involvement of QAA in providing overseas reviewers to evaluate QUE grantees, involvement of HEFCE in the development of funding formula, and involvement of HEFCE in the review and preparation of corporatizing the 4 public universities. Germany: develop collaboration on institutional capacity building, particularly university management. Institutions involves among others are HRK (German Rectors Conference) and DSE (Institute for International Cooperation). During the collaboration period, the German side provides scholarship for 4 Indonesian young academics to participate in an annual 4-weeks international training workshop on self evaluation. Until 2002, participants from the 4 autonomous universities have been annually involved. Discipline-based Commission As explained earlier, the 13 discipline-based commissions were established to support the BHE especially on matters pertained to specific field of disciplines. The commission itself constitutes close to 200 experts coming from various leading higher education institutions, covering 13 areas of disciplines. Originally, the commission was to serve the function as resource person to advise the councils on specific area of disciplines. Along the way, the commissions were deeply involved in evaluating proposals for establishing new study programs or other academic unit within higher education institution across the nation. It was then realized that such proposal evaluation tasks have gone beyond the expected roles of the commission. Considering this fact and for reason of efficiency, the commissions are no longer established under the new BHE structure that was set up in mid 2002. Support for BHE operation During the duration of the DUE Project, the day-to-day operation of the BHE was mostly supported by the DUE Project. In addition to this support, the respective projects also provide support for the BHE involvement, i.e. selection process, monitoring, studies, and training. After the termination of the DUE Project, the DGHE has taken over the responsibility of providing support for the routine BHE operation. Lessons learned and future directions The expected roles and functions of the BHE as stated in the decree of its establishment, that is to become an advisory board to the DGHE, has along the way been gradually shifted, especially in conjunction with the competitive-based funding (projects under the new paradigm). For instance, there is a tendency that the board plays more as an implementer rather than merely a peer group. This clearly reflects that the board needs to strengthen and clarify its role in the future. One of the most intriguing challenges will be to secure successful implementation of the new paradigm in all aspects. A new long term strategic development framework (2003 ­ 2010) is currently being developed and soon will be completed. The new framework consists of three main development strategies, i.e. Nation competitiveness, Organizational health, and Autonomy. The BHE is expected to play major roles in promoting the aforementioned strategies. In order to promote organizational health at institutions as well as at national level, the board will have a very critical role to play, including developing good practices for and providing supervision to higher education institutions. Such good practices are expected to inspire institutions to be more efficient and creative in achieving their organizational health. Component 3: Implementation of a National Accreditation System for Higher Education (BAN) - 29 - Objectives Based on Loan Agreement for Development of Undergraduate Education, the object of the project are strengthening the capacity of BAN in establishing accreditation mechanism and carrying out of accreditation of institutions of higher education and of study program in such institution. The activities to be funded are including the provision of technical assistance and equipment, staff training in management and program evaluation, and renovation of office space. However not all of activities of BAN funded by DUE project, since the DGHE already set up the project so call BAN project, in which could accommodate the activities of BAN through DIP of BAN project. A Good Coordination has done among DGHE, BAN, CPCU and Directorate of Budged Ministry of Finance, in re-planning the activities to avoid the same activities under BAN project as well as DUE project. As the result some allocated budged such as renovation of office space activity, funding for site visit activity were cancelled from DUE project. Finance Budget allocation for BAN is $1,300,000 and total expenditure in amount of $624,942.08 Component 4 : Competitive Domestic Fellowship Program Objectives The main objective of Domestic Fellowship is helping the state university in achieving the quality of new academic staff in which the minimum qualification is having S2 degree. This program is giving the fellowship to the qualified of fresh graduate (S1) who want to be an academic staff. For five years program, the target is 1000 fellowships which was divided into two batches. The Batch 1, it was planned 500 fellow and the commencement of the academic calendar was in 1997, and the Batch 2, it was planned for 500 the commencement of the academic calendar in 1998. However since the target has not been achieved, the Batch 3 was opened for the extended program. Program priority and socialization The University target, were the universities in which the number of academic staff having S2/S3 qualification below the national average, involved in this program are 26 universities. Since, the Directorate General of Higher Education will increase the number student in certain field, this would lead demand to the academic staff in the field of Engineering, Basic Science, Management and Accounting. The appointed of post graduate programs as a place for pursuing the degree of fellowship are selected based on their university in which has at least one study program priority. Information of this program has been distributed to 26 state universities in all part of Indonesia and 13 universities which offer post graduate program by sending a letter from Directorate General of Higher Education (DGHE) on every January during 1997 until 1999, since the students enrollment of post graduate program is on September. Information distribution to the public was conducted through National News Paper (Kompas News Paper) and National Magazine (Gatra Magazine) on every February for the students enrolling the program on September. Requirement Candidates should be S1 degree with minimum GPA 2.75 in the field of Engineering, Basic Science, Management and Accounting. Maximum 30 years old fresh graduates or part time lecturer, and 35 years old if they have minimum of 5 years job experience. Accepted as candidate of academic staff in one of university target and also accepted as a candidate of student in one of post graduate school (PPS) appointed. Having a minimum of Academic Qualification Test ( TPA - Tes Potensi Academic) score of 500 which was conducted by Overseas Training Office (OTO)-Bappenas. The candidates could be finished their study within 24 month, but with very strong reason could be finished within 30 month. - 30 - Mechanism of selection For Batch I, from the first selection, the candidates proposed application to PPS with three choices of study program with 3 choices of the target universities and did the test. The selected candidates from first selection had to take TPA test coordinated by OTO Bappenas and Education Council of BHE (MP-DPT). The candidates having TPA score higher than 500 will be selected by target University and sent to PPS appointed Since from 295 selected applicants only 183 candidates succeed, the modification of selection process have been applied for Batch II and Batch III. The modification is mainly in the stage I, in which on Batch II and Batch III the selection stage I was conducted by university target instead of the PPS appointed. The total number of candidates joined this program is 849 from target 1000. Finance Allocated budget for this category is $5,600,000 and after cancellation the budget allocation become $3,000,000 and the disbursement is $3,267,421.46 which is exceeding budget allocation after cancellation amount of $267,421.46. Component 5 : Project Administration This component financed the Central Project Coordinating Unit (CPCU) and BHE activities. The CPCU plays the important role in coordinating and implementing the project such as training, monitoring, evaluation. Training on procurement process and procedures as well as financial management under World Bank and Government of Indonesia regulation were conducted every year to LPIU. CPCU also conducted pre-screening activity not only for the General Procurement Notice and Annual Procurement Plant from LPIU which submitted to the World Bank, but also for action to be taken on complaints come from the academic faculty as well as suppliers and contractors. CPCU is the first office that has to explained to the internal and external audit, such as Inspectorate General of the DGHE, BPKP, BPK, Independent Audit and Council of the counsel for the prosecution for every question concerning with implementing the project. CPCU conducted coordination meeting twice a year attended by all LPIU, BHE, CPCU and DGHE representatives. Coordination meeting was held by turn among eight LPIU DUE project and the meeting not only sharing experiences and problems solving but also to show the progress of physical achievement from each LPIU. CPCU together with BHE conducted monitoring and evaluation each year for eight LPIU. Every six months, CPCU accompanied World Bank mission to review selected DUE grantees, regarding program on achievement and physical progress. CPCU every year prepared budget proposal with supporting data and together with DGHE-MONE presented and discussed with National Planning Bureau (Bappenas) and Directorate General of Budget ­ Ministry of Finance to obtain budget allocation for the next fiscal year of implementation. The extension of the contract with 8 LPIU also affected budget allocation on project management. CPCU with BHE conducted monitoring and evaluation to all LPIU concerning their performance and target achievement. As the result some decisions were made such as postpone the activities, cancellation the activities, and confiscated the budget allocation in LPIU. Finance Budget allocation for this activity is $1,100,000 and after cancellation amount of $350,000, become $750,000. The total disbursement is $1,01,731.50 and exceeding the budget allocation after cancellation on amount of $260,731.50. - 31 - - 32 -