The World Bank FOR OFFICIAL USE ONLY Report No: RES19262 RESTRUCTURING PAPER ON A PROPOSED PROJECT RESTRUCTURING OF THE RESEARCH AND INNOVATION IN SCIENCE AND TECHNOLOGY PROJECT (RISET) P121842 - LOAN 82450-IND TO THE REPUBLIC OF INDONESIA JANUARY 12, 2017 EDUCATION GLOBAL PRACTICE EAST ASIA AND PACIFIC REGION This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Page 1 of 33 ABBREVIATIONS AND ACRONYMS BATAN Badan Tenaga Nuklir Nasional (National Nuclear Energy Agency) BPPT Badan Pengkajian dan Penerapan Teknologi (Agency for Assessment and Application of Technology) IDP Institutional Development Plans or RPK - Rencana Pengembangan Kelembagaan IRI Intermediate Results Indicator LAPAN Lembaga Penerbangan dan Antariksa Nasional (National Institute of Aeronautics and Space) LIPI Lembaga Ilmu Pengetahuan Indonesia (Indonesian Institute of Sciences) LPNK Lembaga Pemerintah Non Kementerian (Non-ministerial State Institutes) MoF Ministry of Finance MoRTHE Ministry of Research, Technology and Higher Education MTR Mid-Term Review PDO Project Development Objective PMO Project Management Office Ph.D. Doctor of Philosophy Pusdatin MoRTHE’s Center for Data and Information R&D Research and Development RIRIN Rencana Induk Riset dan Inovasi Nasional – the National Research and Innovation Master Plan RISTEK The Ministry of Science and Technology – the previous name of the Ministry of SBK Research, Technology, and higher Education Standar Biaya Keluaran (output-based unit cost) S&T Science and Technology SINas Indonesia’s National Innovation System STI Science, Technology and Innovation STP Science and Technology Park TRL Technology Readiness Level TTO Technology Transfer Office Regional Vice President: Victoria Kwakwa Country Director: Rodrigo A. Chaves Senior Global Practice Director: Amit Dar (acting) Practice Manager/Manager: Harry Anthony Patrinos Task Team Leader: Ratna Kesuma Page 2 of 33 REPUBLIC OF INDONESIA RISET PROJECT CONTENTS A. SUMMARY 5 B. PROJECT STATUS 6 C. PROPOSED CHANGES 7 D. APPRAISAL SUMMARY 14 ANNEX 1: RESULTS FRAMEWORK 16 ANNEX 2: REVISED RESULTS FRAMEWORK 24 ANNEX 3: DESCRIPTION OF CHANGES IN RESULTS FRAMEWORK 31 Page 3 of 33 DATA SHEET Indonesia Research and Innovation in Science and Technology Project (P121842) EAST ASIA AND PACIFIC Education Report No: RES23224 Basic Information Project ID: P121842 Lending Instrument: Specific Investment Loan Regional Vice President: Victoria Kwakwa Original EA Category: Not Required (C) Country Director: Rodrigo A. Chaves Current EA Category: Not Required (C) Senior Global Practice Amit Dar Original Approval Date: 29-Mar-2013 Director: Practice Harry Anthony Patrinos Current Closing Date: 31-Dec-2020 Manager/Manager: Team Leader(s): Ratna Kesuma Borrower: Ministry of Research, Technology, and Higher Education Responsible Ministry of Research Technology and Higher Education Agency: Restructuring Type Form Type: Full Restructuring Paper Decision Authority: Restructuring Level: Level 1 Financing ( as of 19-Dec-2016 ) Key Dates Approval Effectiveness Original Revised Project Ln/Cr/TF Status Signing Date Date Date Closing Date Closing Date P121842 IBRD-82450 Effective 29-Mar-2013 12-Apr-2013 12-Jun-2013 31-Dec-2020 31-Dec-2020 Disbursements (in Millions) % Cancelle Disburse Undisbur Project Ln/Cr/TF Status Currency Original Revised Disburse d d sed d P121842 IBRD-82450 Effective USD 95.00 95.00 0.00 29.55 65.45 31 Policy Waivers Does the project depart from the CAS/CPF in content or in other significant Yes [ ] No [ X ] respects? Page 4 of 33 Does the project require any policy waiver(s)? Yes [ ] No [ X ] A. Summary of Proposed Changes The overall objective of the Project Level I restructuring is to: (i) revise the Project Development Objective (PDO); (ii) revise the results framework and associated indicators to align them with the revised PDO; (iii) modify the project components to reflect a reduced scope of project activities; (iv) introduce changes as a result of the merger of the Ministry of Research and Technology (Kementerian Riset dan Teknologi or RISTEK) and the Directorate General of Higher Education of the Ministry of Education and Culture to form the Ministry of Research Technology and Higher Education (MoRTHE), which is now overseeing project implementation; and (v) cancel a portion of the loan in the amount of US$15 million to reflect modifications to project activities. The cancellation is requested by the Government of Indonesia due to the fact that some of the allocated budget for Components 1 and 2 will not be used as envisaged. In addition, for Component 3, the costs of the fellowship program were overestimated during the design phase. During implementation, it has been determined that the fellowships program will require approximately 50 percent less funding than originally estimated as there are fewer Ph.D. candidates than Master's degree candidates. Project financial management performance under MoRTHE has also been reviewed during the last two supervision missions with some progress noted including on scholarship related payment to degree program. It is expected that these changes will allow the project ratings to be upgraded over time. Change in Implementing Agency Yes [ X ] No [ ] Change in Project's Development Objectives Yes [ X ] No [ ] Change in Results Framework Yes [ X ] No [ ] Change in Safeguard Policies Triggered Yes [ ] No [ X ] Change of EA category Yes [ ] No [ X ] Other Changes to Safeguards Yes [ ] No [ X ] Change in Legal Covenants Yes [ ] No [ X ] Change in Loan Closing Date(s) Yes [ ] No [ X ] Cancellations Proposed Yes [ X ] No [ ] Change to Financing Plan Yes [ X ] No [ ] Change in Disbursement Arrangements Yes [ ] No [ X ] Reallocation between Disbursement Categories Yes [ X ] No [ ] Change in Disbursement Estimates Yes [ X ] No [ ] Change to Components and Cost Yes [ X ] No [ ] Change in Institutional Arrangements Yes [ X ] No [ ] Change in Financial Management Yes [ ] No [ X ] Change in Procurement Yes [ ] No [ X ] Change in Implementation Schedule Yes [ ] No [ X ] Other Change(s) Yes [ ] No [ X ] Appraisal Summary Change in Economic and Financial Analysis Yes [ X ] No [ ] Appraisal Summary Change in Technical Analysis Yes [ ] No [ X ] Appraisal Summary Change in Social Analysis Yes [ ] No [ X ] Page 5 of 33 Appraisal Summary Change in Environmental Analysis Yes [ ] No [ X ] Appraisal Summary Change in Risk Assessment Yes [ X ] No [ ] B. Project Status Progress toward the achievement of the PDO has remained "moderately unsatisfactory" since September 2015 because: (i) PDO indicators 1 and 2: "percentage of public research agencies (LPNKs) reform milestones reached" and "share of total LPNK Research and Development (R&D) revenue generated from external partners", have not witnessed any progress since the project's mid-term review (MTR) in February 2016. This is mainly because the project team was understaffed and had difficulties in collecting comparable data from each LPNK; and (ii) the current PDO statement is broad and does not reflect the actual scope of the project's activities. Overall progress has suffered from various administrative delays, such as the drawn out process of the merger of the Ministry and its subsequent reorganization, as well as implementation constraints, including frequent changes to the project management team and the overly long and cumbersome process of withdrawing project funds at the Ministry level. Overall implementation progress was upgraded from moderately unsatisfactory in April 2016 to moderately satisfactory in October 2016. The upgrade occurred because the implementation team, as agreed during the MTR, has developed a comprehensive set of activities and related action plans for each project component for the period 2016 through December 2020 which is when the project closes. There has also been a significant increase in the pace of implementation between April and October 2016. Notable achievements include: (i) completion of initial work on policy studies related to refining the concept of Indonesia's "National Innovation System - SINas"; (ii) beginning of work on the STI management information system work with the development of an e-survey software; (iii) identification of technology transfer offices (TTOs) to support the revitalization efforts of four LPNKs; (iv) carrying out of studies and immersion programs related to strengthening public research funding; (v) significant increases in awarding fellowships for degree and non-degree program recipients, which accounts for the largest share of project investment, and (vi) increased disbursements from approximately 26 percent in October 22, 2016 to approximately 36 percent by December 2016 based on commitments. Challenges: The primary challenge for project implementation has been the prolonged reorganization of the newly formed Ministry of Research, Technology, and Higher Education (MoRTHE) that took place after the new government took office in October 2014. This process led to: (i) frequent changes to the project management team during 2014-15; (ii) lack of proactive oversight from the Ministry on project implementation in 2014-15; (iii) slow implementation of activities under Components 1 and 2 leading to low disbursements, (iv) slow payment processes for departing participants of Component 3 that were exacerbated by long and cumbersome project fund withdrawal processes from the Ministry's internal finance unit; and (v) inaccuracy in the project's annual budget document coupled with the government's cuts that frequently required revisions to the budget document (again requiring lengthy processes). By the end of 2015, only 20 percent of loan proceeds had been disbursed, compared to the planned target of 55 percent. The project's yearly expenditures and implementation average rate also lagged behind forecasts with only 36 percent expensed (by 2015) against the annual budget allocated in the government's annual budget document. The specific impact of the prolonged reorganization is as follows: Component 1. This component aims to support the improvement of the innovation policy framework and performance of the public research centers or LPNKs and has achieved an annual planned activity target of only about 35 percent. Based on the limited achievement to date of Component 1 activities, achievement of the PDO Indicator #1 and the seven related intermediate level indicators (IRIs) will be a challenge for the project. Page 6 of 33 Component 2. Component 2 activities that intended to strengthen public research funding were delayed from the beginning of project implementation. By the end of 2015 approximately 22 percent of the planned activities have been implemented. These delays will make it difficult to achieve PDO Indicator #2 and the three associated IRIs. Component 3. Component 3 activities, which support one of the main aims of the project--to "develop science and technology human resource capacity" has had notable progress. Since 2013, 202 scholars have benefited from scholarships for both Master's and Ph.D. degree studies abroad (Component 3a), almost reaching the 2016 target of 210 scholars. To date, ten scholars have already graduated (nine Master's and one Ph.D. candidate). Component 3b has also provided funding to 1,059 researchers and staff to attend various non-degree programs, as compared to the cumulative beneficiary target of 850 by 2016. However, the ratio of eligible scholarship recipients (between the Ph.D. and the Master's programs) has changed from 80:20 as originally designed to 61:39. In addition, the actual unit cost for a scholar to undertake a Master's or Ph.D. program is only about 50 percent of the planned unit cost as conceived during the project design. Component 4. Lastly, activities supported under Component 4, which support project operations and management, have been lagging because of structural changes in the institutions that support the project. Because of the institutional changes, disbursements have been below original projections and reached 36 percent by end 2015 and were only 14 percent by October 2016. In addition to the delays, the period 2013 - 2015 has experienced significant weaknesses on financial management aspects, including continued delays in the payments of scholarship-related expenditures, and weaknesses in the handling of soft expenditures, such as on travel. Steps have already been taken to strengthen financial management capacity and to reimburse funds that were used for activities not eligible for project financing. C. Proposed Changes Change in Implementing Agency Please describe the change and explain the reason for change: In late 2014, the government began the merger of the Ministry of Research and Technology (Kementerian Riset dan Teknologi or RISTEK) with the Directorate General of Higher Education of the Ministry of Education and Culture to form the Ministry of Research Technology and Higher Education (MoRTHE) in mid-2015. No changes to implementation arrangements are necessary as all project components remain under the management of the relevant director generals, which are being overseen by MoRTHE. In addition, reviews conducted as part of the Bank’s implementation support missions have confirmed that the merger into MoRTHE has not changed the financial management arrangements under the project and that no additional financial management assessment is required (i.e., that the assessment conducted during appraisal remains valid). Implementing Agency Name Type Action Ministry of Research Technology and Implementing Agency New Higher Education Development Objectives/Results Project Development Objectives Original PDO To create an enabling policy environment for research and development in science and technology, to improve the public research and development institutes’ performance, and to improve science, technology and innovation human resource capacity. Change in Project's Development Objectives Page 7 of 33 Explanation The part of the original PDO that refers to 'to create an enabling policy environment for R&D in science and technology, to improve the public R&D institutes' performance' has been removed in the revised PDO. This is because the aim 'To create an enabling policy environment for R&D in science and technology' is very large and comprehensive in scope, and cannot reasonably be achieved by the project through the limited set of activities under Components 1 and 2. In addition, the objective 'to improve the public R&D institutes performance' involves policy and institutional changes that are beyond the purview of MoRTHE and that of the LPNKs. Proposed New PDO To support the government in: (1) improving STI human resource capacity; and (2) strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs). The new PDO is designed to better reflect the scope of the project's activities and also be aligned with the government's new priorities. The new PDO also highlights that the main objective of the project is to improve human resource capacity at LPNKs and at MoRTHE especially as this constitutes the majority of the loan (80 percent). The main programs under the refocused Component 1 and 2 are related to: (i) technology transfer, in terms of the development of TTOs and the design of Science and Technology Parks (STP); (ii) development of a management information system (MIS) and database for STI; (iii) support for the implementation of the government's improved output-based unit cost methodology; and (iv) helping to increase coordination among research funding agencies and to improve international networking. Change in Results Framework Explanation: A comprehensive revision to the results framework is proposed, and documented in detail in Annexes 1 to 3. The justification for the proposed changes is to ensure that: (i) the indicators are properly aligned with the revised PDO, ensuring that the chosen indicators can adequately track and report progress towards achieving the project's objective; (ii) the new results framework reflects the revised project activities and reduced scope as a result of initial project delays and the US$15 million cancellation; and (iii) results can be reported within the given project timeline. Financing Cancellations Explanation: US$15 million will be canceled as requested by the borrower to accommodate the reduction in component 1 and 2 activities due to the initial delays, and over-estimating the cost for degree programs under component 3. Cancelled Proposed Curren Current Amount Value Date of Ln/Cr/TF Status Amount (in Amount (in cy (in currency) Cancellation currency) currency) Reason for Cancellation: BORROWER'S REQUEST FOR COUNTRY REASONS IBRD-82450- 001 Disbur USD 95,000,000.00 15,000,000.00 80,000,000.00 28-Sep-2016 sing Change to Financing Plan Page 8 of 33 Explanation: The overall loan amount will be reduced from US$95 million to US$80 million, canceling US$15 million as requested by the borrower. Source(s) At Approval Current (from AUS) Proposed BORR 0.00 0.00 0.00 IBRD 95,000,000.00 95,000,000.00 80,000,000.00 Total 95,000,000.00 95,000,000.00 80,000,000.00 Reallocations Explanation: Allocations per expenditure category have been adjusted to account for the US$15 million reduction in the overall loan amount. Current Category Actual Pending Allocatio Allocatio Disbursement % Ln/Cr/TF Currency Cat.No. of Disburse Commit n n (Type Total) Expendit ments ments ure Current Proposed Current Proposed LN 8245 IBRD-824 CS TRNG 4,000,000. 689,779.6 2,500,000. USD 1 0.00 100.00 100.00 50-001 WS IOC 00 7 00 PT1 LN 8245 CS TRNG 4,000,000. 2,500,000. 2 91,269.56 0.00 100.00 100.00 WS IOC 00 00 PT2 LN 8245 SCHOLA 80,000,00 9,214,096. 68,000,00 3 0.00 100.00 100.00 RSHIP 0.00 07 0.00 PT3 LN 8245 NONCS 7,000,000. 680,093.5 7,000,000. 4 CS TNG 0.00 100.00 100.00 00 9 00 WS IOC PT4 95,000,00 10,675,23 80,000,00 Total: 0.00 0.00 8.89 0.00 Disbursement Estimates Change in Disbursement Estimates Explanation: Disbursement estimates have been adjusted to reflect: (i) initial project start-up delays; and (ii) the reduced overall loan amount as a result of the US$15 million cancellation. Fiscal Year Current (USD) Proposed (USD) Page 9 of 33 2013 5,566,000.00 4,464,285.00 2014 13,000,000.00 4,072,052.00 2015 18,468,000.00 3,500,227.00 2016 20,800,000.00 12,514,781.00 2017 18,540,000.00 23,448,655.00 2018 12,764,000.00 20,000,000.00 2019 5,184,000.00 10,000,000.00 2020 678,000.00 2,000,000.00 Total 95,000,000.00 80,000,000.00 Components Change to Components and Cost Explanation: Component 1: Improving the Innovation Policy Framework and the Performance of Public Research Centers Changes to component 1 will entail: (i) changes to the component title; and (ii) streamlining of sub-components and activities. Component title: The original title of Component 1 will be revised from 'Improving the Innovation Policy Framework and the Performance of Public Research Centers' to 'Strengthening the Innovation System and Technology Transfer at Public Research Agencies'. This is to reflect alignment with the agreed set of activities under Component 1, as agreed during the MTR. Sub-component and activity streamlining: Sub-Component 1.a. The original title of Component 1.a. will be revised from 'Improving Innovation Policy Framework' to 'Strengthening Selected Priorities of the National Innovation Strategy'. This is to reflect alignment with the agreed set of activities under Component 1.a. Given this, the activities under Sub-Components 1.a.1, 1.a.2, 1.a.3, 1.a.4, and 1.a.5 have also been refocused into the following four sets of more suitable activities, as detailed below. Activities under Sub-Components 1.a.1: 'Coordination and consensus-building by RISTEK on a national innovation strategy (SINas) and action plan' and 1.a.2: 'Technical and policy study by RISTEK on issues relevant for the development of the national innovation policy (SINas)' will now be changed into the following two new activities: (i) 'Undertaking technical and policy studies on issues relevant for the development of Indonesia's national innovation strategy'. Here, the project will continue the implementation of the technology foresight study which was combined with the study on S&T human resource analysis; and (ii) Supporting the development of STP business models related to agriculture including the development of business plans with cash flow projections and associated action plans as well as the standard operating procedures. Activities under Sub-Component 1.a.3: 'Capacity building of RISTEK on the design and implementation of the national innovation policy (SINas)' have been dropped, as capacity building is being undertaken as part of the project's Non-Degree program under Component 3.b. Page 10 of 33 Activities under Sub-Component 1.a.4: 'Development by RISTEK of a national S&T information system' have now changed to: 'Supporting the development of a national STI information system'. Pusdatin (the ministry's Data and Information Center) will implement data collection, management, and maintenance, as well as the establishment of the STI statistics database. Activities under Sub-Components 1.a.5: 'Supporting the review by RISTEK of, and proposed revisions to, the legal framework for STI' are now refocused to: 'Reviewing and proposing relevant policy measures and regulations for STI'. In addition to drafting revisions to three government decrees (to flexibly use S&T revenues generated, provide research incentives for business entities, and develop research permits for foreign research bodies), the project will also develop recommendations for a new regulation related to the obligation of the developer of any new innovation product to provide a blueprint of the said product in order to better facilitate technology transfer and adoption in Indonesia. The project will also identify the regulations needed for the operations of the TTOs to support the reform of TTO at four selected LPNKs. Sub-Component 1.b. The original title of Component 1.b. will be revised from 'Revitalization of Public Research Centers' to 'Strengthening the Technology Transfer Offices (TTO) and Public Research Centers at selected Public Research Agencies (LPNKs)'. This is to reflect alignment with the agreed set of activities under Component 1.b. Several activities under Sub-Components 1.b.1: 'RISTEK to undertake institutional assessments of selected LPNKs' research centers and to disseminate the results of such assessments' were completed in 2014. As a continuation and as agreed during the MTR, the project will commence a new activity: 'Undertaking institutional assessments of selected LPNKs and disseminate the results of such assessments to other LPNKs'. As a start, the project team will undertake an institutional assessment of LIPI and BPPT to analyze the institutional arrangements between these two research institutes. The institutional assessment will help clarify the various institutional functions because at present they appear to have overlapping duties and functions between these two institutions. Activities under Sub-Component 1.b.2: 'Supporting selected LPNKs to develop institutional development plans' are changed to: 'Supporting TTOs at selected LPNKs to develop their institutional development plans (IDPs)'. Activities under Sub-Component 1.b.3: 'Piloting reforms by selected LPNKs to implement institutional development plans developed under Component 1.b.ii of the Project' have been changed to: 'Piloting the development of selected technologies resulting from the IDPs at selected Public Research Centers under LPNKs and helping them implement a business model'. Pilot activities will be centered on the four R&D-focused LPNKs (LAPAN, BATAN, BPPT, and LIPI) that will be revitalized, that is, on their TTO units and on their Back Office (BO) units. The above-mentioned refocusing of activities has also been done as the budget allocated for various Component 1 activities (such as workshops, seminars, study visits, and consultant recruitments) for 2013 to 2015 was not utilized as projected. Thus, taking into account past project expenditures (from 2013 to 2015) combined with funding needed to finance the agreed set of project activities between 2016 to 2020, the cost envelope now needed is estimated to be US$2.5 million, instead of US$4 million as originally allocated. Therefore, the restructuring paper recommends the partial cancellation of funding for Component 1 in the amount of US$1.5 million. Component 2: Strengthening Public Research Funding Page 11 of 33 Component 2. The original title of Component 2 'Strengthening Public Research Funding' remains as well as the stated aim of the component. Sub-Component 2.1: 'Feasibility Study on the Design of a National Agency on Research Funding' and Sub-Component 2.3: 'Fostering Relations with Industry and International Research Centres' will now be combined into a new Sub-Component 2.1. 'Supporting STI collaboration and networking'. This new sub-component will support activities to: (1) Strengthen coordination of public research funding among public research agencies and research funding providers. The activity will include a study which will highlight the roles and the inter-relationships among the ministry, the LPNKs, the LPDP, the Indonesian Science Fund, universities, as well as other with research institutions including research grant donors. (2) Establish industrial and global research linkages for STI researchers. Enhanced cooperation should lead to concrete outputs, such as publications in international journals, awards, patents, etc. Sub-Component 2.4. This will become new Sub-Component 2.2, and the original title of 'Improving the Monitoring and Evaluation of Research Funding' will be revised to 'Establishing relevant research information systems'. This sub-component will continue supporting the development of a web-based monitoring and evaluation system for publicly funded research in order to facilitate on-going development of government's research policies. This system will be integrated with the STI database that is being developed by Component 1 (previously Sub-Component 1.a.4). The two teams will collaborate with Pusdatin for systems maintenance and sustainability. Sub-Component 2.2: 'Strengthening the Governance of Research Funding' has been combined with Sub-component 2.5: 'Improving the Effectiveness of S&T Program Planning and Budgeting' to become Sub-Component 2.3: 'Supporting research budget planning system'. This sub-component will: (1) Support capacity building activities that will be conducted on the incentive research program, a funding system that supports projects based on an agreed unit cost methodology, a multi-year funding cycle, and clear project outputs, and technical assistance for the measurement of the Technology Readiness Level (TRL) of research programs. (2) Help to implement research funding programs based on the output-based unit cost (currently is called SBK) and the measurement of the TRL of research results at public research agencies. For the output-based unit cost improvement, activities will: (i) calculate the SBK for each research activity in the government's S&T priorities that have not been identified in the current MoF Decree (PMK 106/PMK.02/2016) and (ii) submit these SBKs to the MoF yearly in order to refine the previous year's SBK. These activities will involve sustained consultations with the planning ministry and MoF, and will be carried out in close collaboration with the LPNKs, particularly to calculate the SBK of each research activity. These reforms are aimed at changing the existing financing regulations and corresponding MoF revised decrees in future years will need to be issued to accommodate any new changes. The budget allocated for Component 2 activities (such as workshops, seminars, study visits, immersion programs, and consultant recruitment) for 2013 to 2015 was also not utilized as planned. Past project expenditures (from 2013 to 2015) combined with funding needed to finance the agreed set of project activities between 2016 to 2020 are estimated to be US$2.5 million, instead of US$4 million as originally allocated. Therefore, the restructuring paper recommends the partial cancellation of funding for Component 2 in the amount of US$1.5 million. Component 3: Developing Science and Technology Human Resource Capacity (1) Fellowship Program: The aim of the Component 3a (Fellowship Program) remains the same, Page 12 of 33 however the total target will be modified from the original target 210 scholars to 422 scholars. This is primarily due to the fact that there are more students looking for Master's degrees than Ph.D. degree and therefore the ratio of Ph.D. program and Master's program students will change from 80:20 to 61:39 percent. The last cohort of Ph.D. scholars will depart for their studies in April 2017, while the last cohort of Master's degree scholars will depart by the end of 2017. The estimated budget needed to finance all scholarships recipients is US$55 million instead of US$67 million as originally allocated. Therefore, the partial cancellation of the Component 3 loan is US$12 million. (2) Non-Degree Professional Course and Training Program. The original targets of this program until the project closing in 2020 was 850 scholars, and this has now be revised to 1,907 scholars. From the start of project implementation to date, the project has sent a total of 1,095 researchers and staff to attend a variety of Non-Degree Programs. The project plans to send another 450 researchers for training abroad by December 2016 and another 398 scholars will attend the programs in 2017, leading to a total of 1,907 participants. The increased targets were made possible due to the following: (1) The actual unit cost of the tailor made training, off-the-shelf, and immersion programs is lower (only about one third) of the original unit costs. The actual unit cost of the visiting scholar program is actually higher (by about three times) than the original unit cost. (2) The period of actual implementation of the non-degree professional courses and training programs is actually shorter (minimum 2 weeks and maximum 3 months) than was originally planned (minimum one month and maximum 3 months). Component 4: Project Management supporting the Operation of the Steering Committee, the Technical Committee, and the Project Management Office No new activities will be conducted under this component except to intensify project implementation, especially as related to the development of the project's website and complaint handling system. Currently the communication between PMO and scholars studying abroad is taking place through emails and a WhatsApp Group. There are no changes in the budget allocated for this component. Current Component Proposed Component Current Cost Proposed Action Name Name (US$M) Cost (US$M) Improving Innovation Strengthening the Policy Framework and innovation system and 4,000,000.00 2,500,000.00 Revised Performance of Public technology transfer at Research Centers. public research agencies Strengthening Public 4,000,000.00 2,500,000.00 Revised Research Funding. Developing Science and Technology Human 80,000,000.00 68,000,000.00 Revised Resources Capacity. Project Management. 7,000,000.00 7,000,000.00 No Change Total: 95,000,000.00 80,000,000.00 Other Change(s) Change in Institutional Arrangements Explanation: Page 13 of 33 There are no changes in the institutional arrangements. However, and as previously mentioned, the merger of the Ministry of Research and Technology with the Directorate General of Higher Education of the Ministry of Education and Culture in 2015 has formed MoRTHE. For this reason, the term RISTEK (Ministry of Research and Technology) will be revised to MoRTHE (Ministry of Research, Technology, and Higher Education) in the project documents and in the amended Loan Agreement. Specifically, the following MoRTHE director general (DG) offices oversee the following components: Component 1 is under the DG of Institution, Component 2 is under DG of Research and Development, and Components 3 and 4 are under the DG of Resources for Science, Technology, and Higher Education. The Project Management Office is also under the supervision of the DG of Resources for Science, Technology, and Higher Education. Appraisal Summary Appraisal Summary Change in Economic and Financial Analysis Explanation: This restructuring aims to strengthen ways in which the project can help to support the government in: (1) improving STI human resource capacity; and (2) strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs). It is anticipated that the estimated economic and financial benefits accruing from the project as envisaged in the original project appraisal document will not be impacted by this restructuring. Appraisal Summary Change in Risk Assessment Explanation: The proposed modifications are based on implementation experience to date: (i) increasing the Institutional Capacity risk form Substantial to High due to low capacity of the implementing agency, and the long delays it has caused, which led to the government's request to cancel US$15 million; (ii) increase the Fiduciary risk from Moderate to High to acknowledge the use of funds for activities which are not eligible under the project; and (iii) lower the environment and social risk to low given that the project is a Category C and does not have any major social or environmental impacts. However, the overall risk remains Substantial. Systematic Operations Risk-Rating Tool (SORT) Risk Category Rating 1. Political and Governance Substantial 2. Macroeconomic Moderate 3. Sector Strategies and Policies Moderate 4. Technical Design of Project or Program Substantial 5. Institutional Capacity for Implementation and Sustainability High 6. Fiduciary High 7. Environment and Social Low 8. Stakeholders Moderate 9. Other OVERALL Substantial Explanation The proposed modifications are based on implementation experience to date: (i) increasing the Institutional Capacity risk form Substantial to High due to low capacity of the implementing agency, and the long delays Page 14 of 33 it has caused, which led to the government's request to cancel US$15 million; (ii) increase the Fiduciary risk from Moderate to High to acknowledge the use of funds for activities which are not eligible under the project; and (iii) lower the environment and social risk to low given that the project is a Category C and does not have any major social or environmental impacts. However, the overall risk remains Substantial. Page 15 of 33 Annex 1: Results Framework Project Research and Innovation in Science and Technology Project Project Restructuring Status: DRAFT Name: (P121842) Stage: Team Requesting Leader(s) Ratna Kesuma EACIF Created by: Ratna Kesuma on 07-Apr-2016 Unit: : Product Responsible IBRD/IDA GED02 Modified by: Ratna Kesuma on 10-Jan-2017 Line: Unit: Country: Indonesia Approval FY: 2013 Lending Region: EAST ASIA AND PACIFIC Specific Investment Loan Instrument: Project Development Objectives Original Project Development Objective: To create an enabling policy environment for research and development in science and technology, to improve the public research and development institutes’ performance, and to improve science, technology and innovation human resource capacity. Proposed Project Development Objective (from Restructuring Paper): To support the government in: (1) improving STI human resource capacity; and (2) strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs). The new PDO is designed to better reflect the scope of the project's activities and also be aligned with the government's new priorities. The new PDO also highlights that the main objective of the project is to improve human resource capacity at LPNKs and at MoRTHE especially as this constitutes the majority of the loan (80 percent). The main programs under the refocused Component 1 and 2 are related to: (i) technology transfer, in terms of the development of TTOs and the design of Science and Technology Parks (STP); (ii) development of a management information system (MIS) and database for STI; (iii) support for the implementation of the government's improved output-based unit cost methodology; and (iv) helping to increase coordination among research funding agencies and to improve international networking. Results Core sector indicators are considered: Yes Results reporting level: Project Level Page 16 of 33 Project Development Objective Indicators Status Indicator Name Core Unit of Measure Baseline Actual(Current) End Target New 1. LPNK research agency Text Value NA In progress Reforms are reforms are institutionalized institutionalized Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 Comment Marked for 1. Percentage of LPNK reform Percentage Value 0.00 0.00 100.00 Deletion milestones reached Date 13-May-2011 31-Dec-2015 31-Dec-2020 Comment New 2. Improved regulation on the Text Value NA NA Refined output-based cost standard for regulation issued the STI research funding system based on annual is issued on an annual basis review covering research priorities Date 30-Sep-2013 31-Oct-2016 31-Dec-2019 Comment Marked for 2. Share of total LPNK R&D Percentage Value 5.00 4.00 12.00 Deletion revenue generated from Date 13-May-2011 31-Dec-2015 31-Dec-2020 external partners (industry, local community, foreign grant) Comment Revised 3. Increased % of Masters and Percentage Value 0.00 2.00 100.00 PhD degree holders in LPNK Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 and MoRTHE supported by the RISET Project Comment 3. : Increased % of Masters or Percentage Value 15.00 24.00 20.00 PhD degree holders in LPNK Date 13-May-2011 31-Dec-2015 31-Dec-2020 and RISTEK Comment Page 17 of 33 New 4. Master degree holders who Percentage Value 0.00 5.00 90.00 participate in applied science Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 research programs Comment New 5. Ph.D. degree holders who Percentage Value 0.00 0.00 90.00 manage or coordinate applied Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 science research programs Comment Intermediate Results Indicators Status Indicator Name Core Unit of Measure Baseline Actual(Current) End Target Marked for 1. National innovation study Date Value Deletion completed and disseminated Date 13-May-2011 31-Dec-2015 31-Dec-2019 (Component 1) Comment New 1. National innovation strategy Text Value NA In progress Strategy action is developed (Component 1) plans are submitted for consideration to the new national research and innovation master plan Date 30-Sep-2013 31-Oct-2016 31-Dec-2019 Comment New 2. National Information System Text Value Not started In progress STI data is for STI is established and collected and integrated into the ministry's reported database system (Component 1) 30-Sep-2013 31-Oct-2016 31-Dec-2020 Date Comment Marked for 2. National Information System Number Value Page 18 of 33 Deletion for STI is established with staff Date 13-May-2011 31-Dec-2015 31-Dec-2020 appointed and fund allocated; (Component 1) Comment New 3. Existing regulations in the Text Value NA Reviews Final STI legal framework are undertaken modifications reviewed and modifications are proposed and proposed (Component 1) submitted for consideration to MoF. Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 Comment Marked for 3. Increased number of persons Number Value 0.00 604.00 850.00 Deletion which participate in training of Date 13-May-2011 31-Dec-2015 31-Dec-2020 Innovation System and Professional Course Comment (Component 1) New 1. Increased number of research Text Value NA NA Increased programs in LPNKs and number of MoRTHE focused on higher research ranks of the Technology programs of Readiness Level (TRL) higher TRL (Component 2) 30-Sep-2013 31-Oct-2016 31-Dec-2019 Date Comment Marked for 4. Completed LPNK survey and Yes/No Value No Yes Yes Deletion institutional assessment Date 13-May-2011 31-Dec-2015 31-Dec-2015 (Component 1) Comment New 2. Strengthened R&D Number Value 0.00 0.00 12.00 monitoring & evaluation Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 capacity (Component 2) Comment Page 19 of 33 Marked for 3. Minimum annual number of Number Value 0.00 0.00 12.00 Deletion annual independent evaluations Date 13-May-2011 31-Dec-2015 31-Dec-2020 performed on RISTEK-funded projects (Component 2) Comment Marked for 5. Independent Institutional Yes/No Value No No No Deletion Development Review Panels Date 13-May-2011 31-Dec-2015 31-Dec-2013 established with terms of reference prepared (Component Comment 1) Marked for 6. LPNK and RISTEK Number Value 0.00 0.00 8.00 Deletion completed Institutional Date 13-May-2011 31-Dec-2015 31-Dec-2020 Development Plan, including a longer term HR and incentives Comment scheme for productivity, reviewed by Independent Review Panels (Component 1) New 1. Number of LPNK, Number Value 0.00 202.00 422.00 MoRTHE, Research Agency Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 staff who have studied overseas as supported by RISET-Project Comment (Component3) Marked for 2. Increased number of LPNK Number Value 0.00 135.00 210.00 Deletion staff studied overseas Date 13-May-2011 31-Dec-2015 31-Dec-2020 (Component 3) Comment New 2. Number of participants in Number Value 0.00 805.00 1907.00 RISET-Project Non-Degree Date 30-Sep-2013 31-Oct-2016 31-Dec-2018 professional courses and training (Component 3) Comment Marked for 7. Percentage of institutional Percentage Value 0.00 0.00 100.00 Deletion development targets met Date 13-May-2011 31-Dec-2015 31-Dec-2020 (Component 1) Comment Page 20 of 33 Marked for 1. RISTEK Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 3. BPPT Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 2. LIPI Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 4. BATAN Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 5. BAPETEN Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 6. BIG Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 7. BSN Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 8. LAPAN Percentage Value 0.00 0.00 100.00 Deletion Sub Type Supplemental Marked for 1. Feasibility study and design Yes/No Value No No Yes Page 21 of 33 Deletion of a national research Date 13-May-2011 31-Dec-2015 31-Dec-2015 foundation completed (Component 2) Comment New Direct project beneficiaries Number Value 0.00 1319.00 2387.00 Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 Comment Percentage Percentage indicates the indicates the number of female number of beneficiaries female divided by the beneficiaries scholars departed divided by the that year. scholars departed that year. New Female beneficiaries Percentage Value 0.00 30.00 28.00 Sub Type Supplemental Marked for 1. Increased number of LPNK Number Value 0.00 250.00 280.00 Deletion staff completing domestic Date 13-May-2011 31-Dec-2015 31-Dec-2020 Master’s program (Component 3) Comment Revised 1. Project implementation and Yes/No Value No No Yes monitoring reports (include Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 financial reports) completed (Component 4) Comment The report will be issued on the first quarter of 2017. 1. Project implementation and Yes/No Value No No Yes monitoring reports (include Date 13-May-2011 31-Dec-2015 31-Dec-2020 financial reports) completed (Component 4) Comment Page 22 of 33 Marked for 2. Minimal annual share of Percentage Value 0.00 4.00 100.00 Deletion RISTEK?s institutional Date 13-May-2011 31-Dec-2015 31-Dec-2020 research funding to projects with industry Comment co-implementation (Component 2) New 2. Computerized project Yes/No Value No No Yes monitoring database is Date 30-Sep-2013 31-Oct-2016 31-Dec-2020 established, updated, and key information is placed on Comment MoRTHE project website (Component 4) Marked for 2. Computerized project Yes/No Value No No Yes Deletion monitoring database Date 13-May-2011 31-Dec-2015 31-Dec-2020 established, updated, and key information placed on RISTEK Comment project site (Component 4) Page 23 of 33 Annex 2: Revised Results Framework with cumulative target values Project Development Objective (PDO): To support the government in: (1) improving STI human resource capacity; and (2) strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs). Cumulative Target Values Responsi Description Data Base-l bility for (indicator PDO Level Results Unit of Fre-qu Source/ ine Data definition etc.) Indicators* Measure 2013 2015 2017 2019 2020 ency Methodo Core 2013 Collectio logy n Indicator 1: Reforms are LPNK research Text NA Pilot of the Reforms Reforms annual LPNK LPNK to implemented agency reforms are reforms are are are external PMO-Dit means that the institutionalized implemente implement instituti panel . reform strategies d ed onalized review Institutio for research, n1 management of R&D, human resource development, and performance-base d funding are piloted in the research agencies of four selected LPNKs (LIPI, BPPT, BATAN and LAPAN). The results are reviewed and the model is improved based on the review results. Reforms are institutionalized means that the Page 24 of 33 reform strategies for research, management of R&D, human resource development, and performance-base d funding are adopted and implemented in the research agencies of four selected LPNKs (LIPI, BPPT, BATAN and LAPAN). Indicator 2: LPNK, The funding Improved regulation Refined Refined annual LPNK MoRTH system is based on on the output-based Text NA regulation regulation and E to an agreed unit cost standard for the issued issued MoRTH PMO - cost, STI research funding based on based on E report Dit. methodology, a system issued on an annual annual Institutio multi-year funding annual basis. review review n cycle, and clear covering covering project outputs. research research The first priorities priorities regulation on the output-based cost was issued in 2016. Indicator 3: 15% means 15% Increased % of Percenta 0 1% 15% 65% 100% annual LPNK LPNK to of the targeted Master’s and Ph.D ge HR PMO-Dit total graduates of degree holders in records . HRD the LPNK and LPNK and MoRTHE MoRTHE supported by the supported by RISET Project RISET Project. Page 25 of 33 65% means 65% of the targeted total graduates of the LPNK and MoRTHE supported by RISET Project. 100% means 100% of the targeted total graduates of the LPNK and MoRTHE supported by RISET Project. Indicator 4: Master’s 80% means 80% degree holders who Percenta 0 - - - 80% 90% annual LPNK LPNK to of the total participate in applied ge HR PMO Dit. graduates science research records HRD supported by programs RISET Project in 2019 that are participating in the research programs. 90% means 90% of the total graduates in 2020. Indicator 5: 60% means 60% Ph.D. degree holders Percenta 0 - - - 60% 90% annual LPNK LPNK to of the total who manage or ge HR PMO Dit. graduates coordinate applied records HRD supported by science research RISET Project in programs 2019 that are managing or coordinating the research programs. 90% means 90% Page 26 of 33 of the total graduates in 2020. INTERMEDIATE RESULTS Unit of Responsi Description Cumulative Target Values Data Measure Base-l bility for (indicator Fre-qu Source/ ine 2020 Data definition etc.) ency Methodo Core (2013) 2013 2015 2017 2019 Collectio logy n Intermediate Result (Component 1): Intermediate Result Technical Strategy - annual PMO-Dit PMO-Dit The new national indicator One: Text NA and policy action . . research and National innovation studies are plans are Institutio Institutio innovation master strategy is developed completed submitted n n plan is being and for developed, called strategy considerat the RIRIN: action ion to the Rencana Induk plans are new natio Riset dan Inovasi developed nal Nasional by relevant research Notes: RIRIN is stakeholder and developed by s innovation Dirgen. master RISET-Pro only plan developed strategy in prioritized technology frontier in agriculture. Intermediate Result Text Not Survey Database Database is STI data is STI data annual/ Central PMO indicator Two: started designe preparatio integrated collected is bi-annu project -Pusdatin National Information d n into the and collecte al coordinat System for STI is ministry’s reported d and ion unit established and database reported record integrated into ministry’s database Page 27 of 33 system Intermediate Result annual PMO-Dit PMO-Dit The modifications indicator Three: Text NA - Reviews Reviews Modificati Final . . would pertain only Existing regulations undertake undertaken ons modific Institutio Institutio to selected and in the STI legal n discussed ations n n prioritized framework are are regulations reviewed and propose modifications d and proposed submitte d for consider ation to MoF Intermediate Result (Component 2): Intermediate Result Technology indicator One: Text NA TRL Increased annual LPNK, LPNK Readiness Level Increased number of baseline is number of and and (TRL): research programs in established research MoRTH MoRTH TLR 1 to 4: Basic LPNKs and for levels programs E E to research programs MoRTHE focused on 1-9 of higher report PMO-Dit TRL 5 to 6: higher ranks of the TRL . Applied research Technology Institutio programs Readiness Level n TRL 7 to 9: (TRL). Commercialized research programs. Intermediate Result The figures in the indicator Two: Number 0 0 0 6 10 12 annual PMO – PMO – target columns are Strengthened R&D Dit. R&D Dit. R&D the number of monitoring & independent evaluation capacity evaluations that will be performed Page 28 of 33 on ministry-funded projects. Intermediate Result (Component 3): Intermediate Result indicator One: Number 0 27 135 422 422 422 annual LPNK PMO – Number of LPNK, HR Dit. HRD MoRTHE, Research record + Agency staff who PMO have studied overseas project as supported by record RISET-Project Intermediate Result LPNK PMO – The end target is indicator Two: Number 0 139 501 1509 1907 1907 annual HR Dit. HRD 1907 participants Number of record + attending the NDP participants in PMO courses and RISET-Project’s project training in 2018 Non-Degree record professional courses and training Intermediate Result Percentage indicator Three: Number 0 146 754 2387 2387 2387 annual LPNK PMO – indicates the Direct project Percenta (29%) (29%) (28%) (28%) (28%) HR Dit. HRD number of female beneficiaries of ge record + beneficiaries which are female PMO divided by the project scholars departed record at the year Intermediate Result (Component 4): Intermediate Result indicator One: Project PMO-Dit PMO-Dit implementation and Yes/No No Yes Yes Yes Yes Yes annual . HRD . HRD monitoring reports (include financial reports) are Page 29 of 33 completed Intermediate Result indicator two: Computerized project monitoring database PMO-Dit PMO-Dit is established, Yes/No No No No Yes Yes Yes annual . HRD . HRD updated, and key information is placed on MoRTHE project website Page 30 of 33 Annex 3: Description of Changes in Results Framework a. Revision of the PDO indicators. The following summarize the revisions of the PDO indicators: a) PDO Indicator 1. The original PDO indicator ‘Percentage of LPNK reform milestone reached’ has been dropped. The new PDO indicator is ‘LPNK research agency reforms are institutionalized.’ This new indicator will help the team to focus their efforts and activities on institutionalizing the reform of selected units – TTOs being the front office of the LPNKs and research agencies being the back offices. The cumulative target values have also been adjusted to reach the target in stages (in 2017, 2019, and in 2020, respectively). b) PDO Indicator 2. The original PDO indicator: ‘Minimal annual share of RISTEK’s institutional research funding to projects with industry co-implementation’ has been revised to: ‘Improved regulation on the output-based cost standard for the STI research funding system is issued on an annual basis.’ The previous statement indicates a longer term impact which is not feasible within the time frame of the project. The new revised indicator narrows the focus on more tangible project outcomes. The target values have also been revised from percentages to two identified steps: (i) regulation is issued based on an annual review covering research priorities (2017), and (ii) regulation is issued based on an annual review covering research priorities (2019). c) PDO Indicator 3. This PDO involves a minor revision from ‘Increased % of Master or Ph.D. degree holders in LPNK and RISTEK’ to ‘Increased % of Master’s and Ph.D. degree holders in LPNKs and MoRTHE supported by the RISET project’. This has been done so as not to focus on other fellowship or human resource development programs that are external to the project. d) PDO Indicator 4. This new PDO indicator: ‘Master’s degree holders who participate in applied science research programs’ has been identified to strengthen the outcome of STI human resource capacity enhancement programs implemented by the project. By the end of the project in 2020, the cumulative target is that 90 percent of graduates are expected to reach this milestone. e) PDO Indicator 5. This new PDO indicator: ‘Ph.D. degree holders who manage or coordinate applied science research programs’ also supports the objective of improving STI human resource capacity. By the end of project in 2020, the cumulative target is that 90 percent of graduates are expected to have met this target. b. Revision of IRI Indicators. Corresponding to the above-mentioned proposed changes, some IRIs will also be revised as follows: 1) Component 1 IRIs: (i) IRI 1.1: ‘National innovation study is completed and disseminated’ has been dropped and replaced by a new indicator: ‘National innovation strategy is developed’. The technical and policy study resulting from this work will be used to support the Page 31 of 33 development of RIRIN: Rencana Induk Riset dan Inovasi Nasional – the National Research and Innovation Master Plan that has been developed by the government. (ii) IRI 1.2: ‘National Information System for STI is established with staff appointed and fund allocated’ is revised to ‘National Information System for STI is established and integrated into ministry’s database system’. This system will be part of the ministry’s Pusdatin (Center of Data and Information). (iii) IRI 1.3: ‘Increased number of persons who participate in training innovation system’ has been dropped. A new indicator has been added (IRI 3.2) in Component 3 to measure the number of persons who participate in training. (iv) IRI 1.4: ‘Completed LPNK survey and institutional assessment’ has been dropped as this assessment has been completed. The results of the assessment are being used to develop the Institutional Development Plans (IDPs) of the four research agencies that will be subject to the revitalization reforms (LAPAN, BATAN, LIPI, and BPPT). (v) IRI 1.5: ‘Independent institutional development review panels established with terms of reference prepared’ has also been dropped as panels must be appointed to review the implementation of the IDPs. (vi) IRI 1.6: ‘LPNK and RISTEK completed Institutional Development Plan, including a longer term HR and incentives scheme for productivity, reviewed by Independent Review Panels’ has been dropped. The development of the IDPs will be included in the planned pilot activities in the four selected LPNKs which conduct R&D (LAPAN, BATAN, LIPI, and BPPT) and which will be revitalized. (vii) IRI 1.7: ‘Percentage of institutional development targets met’ has been dropped as the success of the pilot activities, including institutional development, is now being measured through the new PDO Indicator 1. (viii) One indicator is added (to become the new IRI 1.3): ‘Existing regulations in the STI legal framework reviewed and modifications proposed’. This indicator is intended to examine selected regulations related to relevant policy measures for STI. 2) Component 2 IRIs: (i) The target for IRI 1: ‘Feasibility study and design of a national research fo undation completed’ has been dropped as this target has been recently achieved by the government through the establishment of the new research funding institution: the Indonesian Science Fund. A study on the roles and inter-relationships between the ministry and other research agencies to ensure coordination and collaboration will be conducted. (ii) IRI 2: This indicator will become a new IRI 2.1. ‘Minimal annual share of RISTEK’s institutional research funding to projects with industry co-implementation’ has been revised to ‘Increased number of research programs in LPNKs and MoRTHE focused on higher ranks of the Technology Readiness Level (TRL).’ The target values have also been revised from percentages to two identified steps: (i) TRL baseline is established for levels 1-9 (2017), and (ii) Increased number of research programs to produce outputs with higher TRL (2019). (iii) IRI 3: This indicator will become a new IRI 2.2. ‘Minimum annual number of annual independent evaluations performed on RISTEK-funded projects,’ will be revised to ‘Strengthened R&D monitoring & evaluation capacity’ to ensure that the improved Page 32 of 33 evaluation capacity is met. The cumulative target will be revised to achieve 6, 10, and 12 independent evaluations in 2017, 2019 and 2020, respectively. 3) Component 3 IRIs: (i) IRI 1: ‘Increased number of LPNK staff completing domestic Master’s program’ has been dropped as the project does not finance domestic Master’s degree programs. (ii) IRI 2: This indicator will become IRI 1. The indicator: ‘Increased number of LPNK staff studied overseas’ will be changed to ‘Number of LPNK, MoRTHE, and Research Agency staff who have studied overseas supported by RISET Project’. This revision measures the number of concerned staff who have benefitted from the project’s programs. (iii) IRI 2: This new indicator: ‘Number of participants in RISET-Project’s Non-Degree professional courses and training’ will enable the measurement of achievement of activities of the Non-Degree program (which was not previously included as an indicator). (iv) IRI 3: A new indicator 3 has been added: ‘Direct project beneficiaries of which are female’ which will capture the gender dimensions of the project. 4) Component 4 IRIs: There are 2 IRIs that have been maintained as follows: (i) ‘Project implementation and monitoring reports (including financial reports) are completed.’ (ii) ‘Computerized project monitoring database is established, updated, and key information is placed on RISTEK project website.’ Page 33 of 33