Document of The World Bank Report No: ICR00003734 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-48840) ON A LOAN IN THE AMOUNT OF US$13.4 MILLION TO THE REPUBLIC OF KAZAKHSTAN FOR A TECHNOLOGY COMMERCIALIZATION PROJECT June 27, 2016 Trade and Competitiveness Global Practice Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective October 13, 20071) Currency Unit = Kazakhstan Tenge (KZT) KZT 1.00 = US$0.0083 US$1.00 = KZT 121 (Exchange Rate Effective December 1, 20112) Currency Unit = KZT KZT 1.00 = US$0.0065 US$1.00 = KZT 146 (Exchange Rate Effective December 31, 20153) Currency Unit = KZT KZT 1.00 = US$0.0030 US$1.00 = KZT 329 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS ATCP Association of Technology Commercialization Professionals CPS Country Partnership Strategy CRDF Civilian Research and Development Foundation EMP Environmental Management Plan ERR Economic Rate of Return FM Financial Management FPIP Fostering Productive Innovation Project GII Global Innovation Index GLP Good Laboratory Practice GoKZ Government of Kazakhstan ICR Implementation Completion and Results Report IIDS Innovative Industrial Development Strategy IMSC International Material Science Center IP Intellectual Property ISR Implementation Status and Results Report ISCB International Science and Commercialization Board JRG Junior Research Group M&E Monitoring and Evaluation MoES Ministry of Education and Science MoF Ministry of Finance MoIT Ministry of Industry and Trade NATD National Agency for Technology Development 1 Rate at appraisal. 2 First disbursement took place in December 2011. 3 Closing date rate. NIS National Innovation System NPV Net Present Value NURIS Nazarbayev University Research and Innovation System PAD Project Appraisal Document PCT Patent and Cooperation Treaty PDO Project Development Objective PMU Project Management Unit R&D Research and Development SSG Senior Scientist Group TCO Technology Commercialization Office TCC Technology Commercialization Center TCP Technology Commercialization Project TTL Task Team Leader Regional Vice President: Cyril E Muller Country Manager: Francis Ato Brown Senior Global Practice Director: Anabel Gonzalez Practice Manager: Lisa Kaestner Project Team Leader: Karen Grigorian and Yeraly Beksultan ICR Team Leader: Asta Bareisaite KAZAKHSTAN TECHNOLOGY COMMERCIALIZATION PROJECT CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ........................................................... i 2. Key Factors Affecting Implementation and Outcomes ......................................................... 5 3. Assessment of Outcomes......................................................................................................... 11 4. Assessment of Risk to Development Outcome...................................................................... 21 5. Assessment of Bank and Borrower Performance ................................................................ 23 6. Lessons Learned ...................................................................................................................... 25 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ................... 27 Annex 1. Project Costs and Financing ...................................................................................... 28 Annex 2. Outputs by Components............................................................................................. 29 Annex 3. Economic and Financial Analysis.............................................................................. 38 Annex 4. Bank Lending and Implementation Support/Supervision Processes ..................... 52 Annex 5. Beneficiary Survey Results ........................................................................................ 54 Annex 6. Stakeholder Workshop Report and Results ............................................................. 57 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ............................. 59 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ............................... 67 Annex 9. Additional Evidence for Assessment of Outcomes................................................... 68 Annex 10: A List of Technology Commercialization Projects ................................................ 73 Annex 11. List of People Met during the ICR Mission............................................................ 80 Annex 12. List of Supporting Documents ................................................................................. 84 MAP ............................................................................................................................................. 85 A. Basic Information TECHNOLOGY Country: Kazakhstan Project Name: COMMERCIALIZATIO N PROJECT Project ID: P090695 L/C/TF Number(s): IBRD-48840 ICR Date: 06/27/2016 ICR Type: Core ICR REPUBLIC OF Lending Instrument: SIL Borrower: KAZAKHSTAN Original Total USD 13.40M Disbursed Amount: USD 11.35M Commitment: Revised Amount: USD 11.35M Environmental Category: B Implementing Agencies: Ministry of Education and Science Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 05/16/2005 Effectiveness: 12/15/2008 12/15/2008 07/25/2012 Appraisal: 11/16/2007 Restructuring(s): 12/29/2015 Approval: 01/15/2008 Mid-term Review: 11/04/2013 10/27/2013 Closing: 12/31/2012 12/31/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Project Quality at Entry Yes None at any time (Yes/No): (QEA): Problem Project at any Quality of Supervision Yes None time (Yes/No): (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 20 20 General industry and trade sector 60 60 Tertiary education 20 20 Theme Code (as % of total Bank financing) Other Private Sector Development 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Cyril E Muller Shigeo Katsu Country Director: Mariam J. Sherman Annette Dixon Practice Manager/Manager: Lisa A. Kaestner Fernando Montes-Negret Project Team Leader: Karen Grigorian Natasha Kapil ICR Team Leader: Asta Bareisaite ICR Primary Author: Asta Bareisaite F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) Project development objective and key indicators: The specific development objective of this pilot project is to demonstrate significantly improved scientific performance and commercial relevance of research performed by interdisciplinary teams of scientists selected through transparent competitive process. By using merit-based selection procedures, the project will help to rebuild, strengthen, and restructure selected segments of Kazakhstan's research and development (R&D) base. The project will also help to link this rejuvenated R&D capacity to national and international technology markets. ii The following are some of the key indicators: -Number of established and functioning Senior Scientists Groups (SSG) and Junior Researcher Groups (JRG) -Number of graduate students involved in the SSGs and JRGs -Number of visiting professors -Number of collaborative research projects involving groups and international research partners -Establishment of the International Materials Science Center (IMSC) -Number of scientists serviced by the IMSC -Number of completed technology audits -Establishment of the Technology Commercialization Office (TCO) -Reviews of laws, rules, and regulations, and recommendations on revision -Number of grants awarded for technology commercialization -Number of licenses sold with the help of the TCO The Loan Agreement uses a shorter PDO, stated as "The objective of the project is to demonstrate significantly improved scientific performance and commercial relevance of research performed by interdisciplinary teams of scientists selected through a transparent competitive process." Given that the Loan Agreement supersedes the Project Appraisal Document (PAD), the ICR will use the PDO stated in the Loan Agreement. Revised Project Development Objectives (as approved by original approving authority) The PDO remained unchanged throughout the project, however the PDO level indicators were formally revised as part of the project's level 2 restructuring in December 2015 (changes agreed after midterm review in 2013, but restructuring delayed due to bureaucratic obstacles). The ICR assessment will be made against the original indicators. (a) PDO Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Number of international publications from Senior and Junior Scientist Research Groups Indicator 1 : in peer-reviewed journals(Cumulative) Value quantitative or 0 11-17 25 85 Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Revised target. Comments (incl. % The actual value includes articles with impact factor only. Exceeded the initial target achievement) by400%. Indicator 2 : Number of Patent Cooperation Treaty Agreements approved (Cumulative) Value 0 n.a. 7 5 iii quantitative or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments (incl. % New indicator to replace indicator 3. Fell short of the target by 29% achievement) Indicator 3 : Number of Patent Applications abroad through the TCO (Cumulative) Value quantitative or 0 6-12 n.a. 16 Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Dropped. Indicator dropped during December 2015 restructuring and replaced by (incl. % Indicator 2. achievement) Indicator 4 : Number of license agreements signed (Cumulative) Value quantitative or 0 16-23 4 11 Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 New indicator. Originally set as intermediate, but moved to PDO level during Dec 2015 Comments restructuring, with revised wording from "number of licenses sold with the help of the (incl. % TCO". achievement) Fell short of initial target. Share of enterprise sector financing of R&D in Senior Scientist & Junior Research Indicator 5 : Groups (%) Value quantitative or n.a. 5-20% 5% 5% Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Revised indicator. Revised from "Share of private sector financing of SSGs and JRGs". (incl. % PMU estimate includes "in-kind" contribution. achievement) The target was achieved. Number of technology-based start-ups created under Senior Scientist & Junior Research Indicator 6 : Groups and by TCO clients having commercial sales Value quantitative or 0 n.a. 10 40 Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments New indicator measures the economic relevance of research groups and TCO clients. (incl. % Target was well exceeded by 300% achievement) iv (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Number of internationally co-authored articles by researchers in Senior and Junior Indicator 1 : Scientist Research Groups Value (quantitative 0 n.a. 22 38 or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Revised indicator. Revised from "Number of international co-authored articles per (incl. % researchers in SSGs and JRGs". achievement) Target exceeded by 73%. Number of articles in national research journals and conferences by researchers in Indicator 2 : Senior and Junior Scientist Research Groups Value (quantitative 0 n.a. 113 145 or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments New indicator. (incl. % Target exceeded by 28% achievement) Number of paid graduate students involved in the Senior Scientific and Junior Scientific Indicator 3 : Groups (annual) 16 in 2012 Value 10 in YR1, 20 in 10 in 2012, 20 35 in 2013 (quantitative 0 YR2-3, 10 in YR4 in 2013-2015 58 in 2014 or Qualitative) 61 in 2015 Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Revised target. Revised in Dec 2015 from "Number of graduate students involved in the Comments SSG and JRG research project". (incl. % achievement) Original targets exceeded every year. Number of partnerships established between local researchers and international Indicator 4 : researchers measured in terms of number of agreements Value (quantitative 0 n.a. 28 77 or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments New Indicator. (incl. % Target exceeded by 175% achievement) Rate of equipment use in national and engineering laboratories by Senior Scientific and Indicator 5 : Junior Scientific Research Group scientists (%) Value 20% in 2015 0 n.a. n.a. (quantitative (varies by year) v or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments By the end of the project, IMSC did not have a reporting system in place to estimate the (incl. % rate of equipment use in national and engineering laboratories by SSGs and JRGs, achievement) therefore there is no reliable estimate of actuals. Rate of equipment use in national and engineering laboratories and use of Senior Indicator 6 : Scientific and Junior Scientific Groups' equipment by outside users (%) Value 20% in 2015 (quantitative 0 n.a. n.a. (varies by year) or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments By the end of the project, IMSC did not have a reporting system in place to estimate the (incl. % rate of equipment use in national and engineering laboratories by SSGs and JRGs, achievement) therefore there is no reliable estimate of actuals. Indicator 7 : Number of researchers/scientists and entrepreneurs trained by the TCO Value (quantitative 0 240 420 881 or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Revised indicator. Revised from "Number of scientists and entrepreneurs trained at the (incl. % TCO". Researchers refers to SSGs or JRGs and other researchers supported by TCO. achievement) Original target exceeded by 267%. Number of partnerships established between researchers and enterprise sector measured Indicator 8 : in terms of number of agreements Value (quantitative 0 70-80 26 119 or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Revised indicator. Revised from "Number of partnerships between SSGs, JRGs, Comments research institutions and private sector". (incl. % achievement) Original target exceeded by 70%. Number of Senior Scientist Groups (SSG) and Junior Research Groups (JRG) Indicator 9 : established Value (quantitative 0 20 n.a. 33 or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Dropped. Indicator dropped during Dec 2015 restructuring. The mid-term review of the (incl. % Project has assessed it as an output indicator rather than as an outcome as it does not achievement) refer directly to the Project objectives. Number of timely submission of scientific reports, financial statements, and audit Indicator 10 : reports Value (quantitative n.a. n.a. n.a. n.a. or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 vi Comments Dropped. Indicator dropped during Dec 2015 restructuring. It was not included in PAD, (incl. % but included in the Supplemental Target nr. 2. The mid-term review of the Project has achievement) assessed it as an output rather than as an outcome Indicator 11 : Establishment of the Technology Commercialization Office Value (quantitative n.a. Established n.a. Established or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Dropped. Indicator dropped during Dec 2015 restructuring. The mid-term review of the (incl. % Project assessed it as an output indicator rather than as an outcome as it does not refer achievement) directly to the Project objectives. Indicator 12 : Rate of equipment use Value 70-90% (varies by (quantitative 0 n.a. n.a. year) or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments (incl. % Dropped. Replaced by two new indicators (Indicator 5 and Indicator 6). achievement) Indicator 13 : Number of technology audits Value (quantitative 0 5 n.a. n.a. or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Dropped. Indicator dropped during Dec 2015 restructuring. The mid-term review of the (incl. % Project assessed it as an output indicator rather than as an outcome as it does not refer achievement) directly to the project objectives. Indicator 14 : Number of reviewed laws, rules, regulations and revision of recommendation Value (quantitative 0 15-21 n.a. n.a. or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Dropped. Indicator dropped during Dec 2015 restructuring. The mid-term review of the (incl. % Project assessed it as an output indicator rather than as an outcome as it does not refer achievement) directly to the project objectives. Indicator 15 : Number of grants awarded by the TCO for technology commercialization project Value (quantitative 0 55-60 n.a. 33 or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Dropped. Indicator dropped during Dec 2015 restructuring. The mid-term review of the (incl. % Project assessed it as an output indicator rather than as an outcome as it does not refer achievement) directly to the project objectives. Indicator 16 : Establishment of an international peer reviewed journal Value (quantitative n.a. Established n.a. Established or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 vii Comments Dropped. Indicator dropped during Dec 2015 restructuring. The mid-term review of the (incl. % Project assessed it as an output indicator rather than as an outcome as it does not refer achievement) directly to the Project objectives Indicator 17 : Establishment of the International Material Science Center (IMSC) Value (quantitative n.a. Established n.a. Established or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments Dropped. Indicator dropped during Dec 2015 restructuring. The mid-term review of the (incl. % Project assessed it as an output indicator rather than as an outcome as it does not refer achievement) directly to the Project objectives Indicator 18 : Number of partnerships established between researchers and academia Value (quantitative n.a. n.a. n.a. n.a. or Qualitative) Date achieved 01/15/2008 12/31/2012 12/31/2015 12/31/2015 Comments (incl. % Dropped. Indicator dropped during Dec 2015 restructuring. achievement) G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 06/13/2008 Satisfactory Satisfactory 0.00 2 06/23/2009 Moderately Satisfactory Unsatisfactory 0.00 3 02/26/2010 Moderately Unsatisfactory Moderately Unsatisfactory 0.00 4 05/02/2010 Moderately Satisfactory Moderately Unsatisfactory 0.00 5 02/21/2011 Moderately Unsatisfactory Moderately Unsatisfactory 0.00 6 10/23/2011 Moderately Unsatisfactory Moderately Satisfactory 0.00 7 07/18/2012 Moderately Unsatisfactory Moderately Satisfactory 1.29 8 02/15/2013 Moderately Satisfactory Moderately Satisfactory 2.89 9 09/02/2013 Moderately Satisfactory Moderately Satisfactory 4.73 10 01/09/2014 Moderately Satisfactory Moderately Satisfactory 6.66 11 04/12/2014 Moderately Satisfactory Moderately Satisfactory 6.66 12 10/29/2014 Moderately Satisfactory Moderately Satisfactory 8.57 13 05/01/2015 Moderately Satisfactory Moderately Satisfactory 10.70 14 12/03/2015 Moderately Satisfactory Moderately Satisfactory 11.23 15 12/30/2015 Moderately Satisfactory Moderately Satisfactory 11.52 viii H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Restructuring Disbursed at Reason for Restructuring & Key Approved PDO Date(s) Restructuring Changes Made Change DO IP in USD millions Extension of closing date for loan 07/25/2012 MU MS 1.29 2884-KZ to December 31, 2015 Cancellation of uncommitted 12/29/2015 N MS MS 11.52 funds and updating the Results Framework I. Disbursement Profile ix 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Country context. Kazakhstan emerged from the Soviet Union with a well-educated, scientifically literate labor force. However, the way Kazakh science was organized was not conducive to linking knowledge and markets. Existing funding mechanisms and bureaucratic practices further reinforced the status quo inherited from the Soviet system. Compounding the problem was the dramatically reduced number of research personnel and the outdated poor quality of research and development (R&D) equipment and instrumentation in research institutes at the time. As a result, the output of the scientific community was below the average for comparator middle-income economies. The vast majority of Kazakhstan’s exports consisted of unprocessed raw materials, while knowledge-intensive products comprised only an insignificant share. 2. Over the years, the country made commendable progress in stabilizing its economy and carrying out structural reforms. The Government of Kazakhstan (GoKZ) recognized that generating more value added from natural resource sectors, developing new knowledge-intensive innovative activities, restoring Kazakhstan’s scientific and research capability, and linking Kazakh science more closely to domestic and international technology markets must become the cornerstones of public policy efforts to maintain long-term economic growth and increase national competitiveness in the context of an open, global, knowledge-based economy. 3. Macroeconomic and sector background of relevance to the operation’s objectives and design. In May 2003, the GoKZ adopted an Innovative Industrial Development Strategy (IIDS) which strives to encourage investment in non-oil sectors by fostering greater supply chain linkages between the oil and non-oil sectors, providing assistance to meet international quality standards, increasing the quality and economic relevance of R&D conducted in research institutes, helping research institutes obtain commercial contracts with foreign firms, and supporting a system of incubators and techno parks to facilitate technology commercialization and link Kazakh scientists to global technology markets. The resources earmarked to finance innovation and fund tertiary education and scientific research institutes were substantially increased. 4. Rationale for Bank assistance. In the fall of 2004, the Ministry of Industry and Trade (MoIT) asked the World Bank to recommend and design specific policies that could be implemented under the auspices of the IIDS. The project appraisal document (PAD) acknowledges that from a purely financial or cash flow perspective, the GoKZ had little need to borrow from the Bank. The key rationale for Bank involvement was to provide Kazakhstan with access to global expertise and best practice in developing programs with the potential to transform the country’s innovation system and to provide fiduciary oversight during implementation. The sectors of science and technology and R&D are prone to interest group capture, especially in post-transition economies; therefore, the GoKZ insisted that even those components that were financed solely by the Government should follow Bank procedures. 5. Higher-level objectives to which the project contributes. The proposed project was grounded in the key pillars of the 2004 Country Partnership Strategy (CPS) (P083068) between the GoKZ and the Bank. These include (a) improving public institutions and policies, (b) fostering competitiveness, and (c) investing in human capital. In addition, as noted above, the project was 1 to contribute directly to the implementation of the IIDS. Since its inception this project had been one of the GoKZ’s high-priority activities, receiving support from the highest-ranking officials. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 6. PDOs. The Loan Agreement states that “The objective of the project is to demonstrate significantly improved scientific performance and commercial relevance of research performed by interdisciplinary teams of scientists selected through a transparent competitive process.” The PAD further clarifies that this is a pilot project, and outlines the higher-level objectives: “By using merit- based selection procedures, the project will help to rebuild, strengthen, and restructure selected segments of Kazakhstan’s R&D base. The project will also help to link this rejuvenated R&D capacity to national and international technology markets. In line with the higher-level objectives, a longer term follow-on development objectives of the project is to create stimulus to reform the existing R&D base, If the project is successful, five to ten years in the future Kazakhstan will have developed a new competitive, interdisciplinary, problem-oriented, internationally peer-reviewed model of financing excellent, economically relevant scientific research. Kazakh science will have closer links to domestic and international private enterprises as the scientific institutes supported by the project will be encouraged to generate up to a quarter of their income from contracts with private enterprises or other competitive funding sources. Research activities will be performed in close collaboration with the international science community and on an interdisciplinary, problem- oriented basis rather than in the sealed silos that characterize the current system. And perhaps most importantly, a new generation of young, talented scientists will be attracted to conduct science in Kazakhstan at the state-of-the-art scientific laboratory supported by the project and under guidance and supervision of eminent Kazakh and international scientists.” 7. Key indicators. In the context of the prevailing inconsistencies within the PAD and the Loan Agreement,4 the Implementation Completion and Results Report (ICR) proposes to use the outcome indicators listed in the arrangements for results monitoring in annex 35):  Number of international publications in peer-reviewed journals from Senior Scientist Group (SSG) and Junior Research Group (JRG) program  Number of patent applications registered abroad through the Technology Commercialization Office (TCO)  Share of private sector financing of SSG and JRG programs 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 8. The PDO remained unchanged throughout the project; however the PDO-level indicators were formally revised as part of the project’s level 2 restructuring in December 2015. 9. Updated PDO indicators: 4 Loan Agreement Supplemental Letter Number 2 does not separate into PDO and intermediate level indicators; different versions of PDO indicators are provided within the PAD: the datasheet (page ii), the main text (section B2; page 7), Annex 3: Results Framework (page 49), and the Arrangements for results monitoring (page 51) 5 The Results Framework in annex 3 provides specific targets for each indicator and is used in ISR reporting 2  Number of international publications from Senior and Junior Scientist Research Groups in peer-reviewed journals  Number of Patent and Cooperation Treaty (PCT) Agreements approved  Number of license agreements signed  Share of enterprise sector financing of R&D in SSGs and JRGs  Number of technology-based start-ups created under SSGs and JRGs and by TCO clients having commercial sales 1.4 Main Beneficiaries 10. Primary target groups. While the PAD did not identify specific beneficiaries, the following primary target groups can be identified: (a) researchers/scientists who receive SSG and JRG grants and TCO grants for developing eligible R&D ideas; (b) scientists who benefit from better access to research materials through the International Material Science Center (IMSC); (c) entities or individuals that benefit from the TCO services; (d) a new generation of young, talented scientists, attracted to conduct science in Kazakhstan; and (e) the key stakeholders of the National Innovation System (NIS), including the Ministry of Education and Science (MoES) and other relevant authorities and ministries of Kazakhstan, which can learn from the best practices in grants management beyond the Technology Commercialization Project (TCP). It was envisaged that if the pilot works, the GoKZ could apply the best practices to other projects. 11. Indirect beneficiaries. The indirect beneficiaries are (a) Kazakhstan’s enterprise sector, which adapts and profits from the technological achievements of the project grantees; (b) end customers and broader society of Kazakhstan whose standard of living improves due to new products brought to the market; and (c) the broader scientific community who change their mindset on conducting more market-oriented research. 1.5 Original Components (as approved) Component 1: Strengthening Science Base of Kazakhstan 6 12. Identifying and developing leading scientists and talented young researchers to conduct R&D at international standards through (a) the establishment of an International Science and Commercialization Board (ISCB) and an internationally peer-reviewed journal; (b) an SSG grant program and a JRG grant program; and (c) the establishment of the IMSC. Component 2: Linking Kazakh Science to Markets 13. Provision of technical assistance for the development of expertise required to link science more closely to domestic and international markets through (a) the establishment of a TCO to provide a range of technology commercialization services, including the development of a technology commercialization strategy, and the development of targeted training programs in foreign languages, business planning, and communication; (b) undertaking a technology audit to inform policy makers and scientists about innovation assets and to compile a knowledge map 6 The PAD and Loan Agreement define the same set of activities; however, there are some insignificant differences in the way subcomponents are organized and described 3 database for the marketing of technology; (c) a review of legal and regulatory provisions, structures, and incentives related to technology commercialization, industrial innovation, venture capital and financing, and intellectual property (IP); and (d) conducting a market assessment to complete design and administration of a range of borrower-financed grant programs. Component 3: Project Management Unit 14. Program management support through the establishment of the project management unit (PMU) for project coordination, implementation, and management. 1.6 Revised Components 15. The three headline components remained unchanged during the course of the project. However, modifications were made to subcomponents: some were dropped and others changed to accommodate the changing landscape of Kazakhstan’s scientific base and to address the challenges that surfaced during implementation.7 16. Significant changes to the IMSC subcomponent. In 2012, it was acknowledged that the circumstances in which the IMSC was envisioned at the beginning of the project had changed. Kazakhstan had made significant investments into scientific infrastructure by establishing 20 modern labs.8 In the new context, the IMSC’s role was adjusted to (a) serve as a benchmark for other laboratories throughout Kazakhstan with regard to adopting international good laboratory practices (GLPs); (b) improve the access to information on research equipment in national and engineering laboratories; and (c) promote collaboration among these laboratories. The decision was made to build the IMSC using one of the existing laboratories, selected through a competitive tender process. 17. Grants under Component 2. The PAD identified five PMU-financed grant programs under Component 2. When the international consultant firm was selected in 2013, it first analyzed the existing grant programs in Kazakhstan for development and commercialization of technologies to identify gaps. Based on this analysis, the innovation grant program was redesigned to cover two types of activities (a) proof of concept and (b) industrial prototype activities. 1.7 Other significant changes 18. Closing date extension. In July 2012, the closing date for the project was extended from December 31, 2012, until December 31, 2015, to allow for implementation to catch up, following the suspension of the project and to give additional time to achieve project targets. 19. Reallocation on funds. Due to the success of the first two rounds of the SSG and JRG grants, the MoES financed an additional third round in 2013. To cover the cost, the MoES used US$10.5 million of the GoKZ’s budget allocation originally dedicated for buying/renovating the IMSC. There was also a reallocation of the PMU component to the TCO component after noncore PMU staff and relevant resources were moved from the PMU to the TCO in 2013. 7 According to consultations with the project lawyer, the changes did not require a restructuring. 8 Five national research laboratories and fifteen engineering laboratories at universities. 4 20. Devaluations 9 affected the project budget. In 2013, the GoKZ approved the revised project budget of US$64.3 million, approximately US$10.7 million less than the original budget, because of exchange rate losses. However, as a result of changes within the components, this did not affect the project design or outcomes. Because of further devaluation of the Kazakhstani tenge over 2015, US$1.8 million funds could not be utilized on time before project closing on December 31, 2015, and were cancelled through restructuring dated December 9, 2015. 21. Updated Results Framework. As part of project midterm review, the MoES agreed to update the Results Framework to improve the quality of project assessment. De facto, the revised Results Framework was in use since 2013; however, due to bureaucratic delays it was not formally approved until the level 2 restructuring in December 2015. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 22. The background analysis in the PAD was sound and satisfactory. It was centered on Kazakhstan’s need to diversify the oil-driven economy to reduce exposure to international shocks, and to build a more robust and sustainable economic growth model. In this context, the rationale for intervention was founded on two main pillars: (a) Kazakhstan’s potential to become a knowledge-based economy and (b) the existence of market and systemic failures. This approach, combining context factors and policy rationale is deemed appropriate to justify an innovation project in Kazakhstan. The rationale for the Bank’s intervention was also clearly explained. The economic analysis was based on adequate international examples, though could have been further enhanced by strengthening linkages between Kazakhstan’s specific challenges and the project components. 23. Adequacy of the GoKZ’s commitment. The link to the country’s strategic priorities and the CPS was well established. From the start, TCP activities were designed to directly contribute to the implementation of the IIDS. Given the TCP’s strong alignment with the country’s innovation policy design and the CPS, there was strong high level Government commitment from the start of the project, evidenced by substantial GoKZ cofinancing contribution to the project (82 percent of the total project budget). 24. Strengths of the project design. The project design benefited from the integration of lessons learned from innovation projects in other countries. It considered the key lesson learned that building an effective NIS would require a longer-term approach, and therefore this project appropriately proposed a gradual approach starting with the TCP as a pilot to test new ways to organize and fund scientific activities in Kazakhstan. Other relevant lessons applied include the design of a competitive selection process, the adoption of a multidisciplinary approach, and ways to increase industry-university linkages. The design was particularly appropriate for Component 1 9 During implementation of the TCP, Kazakhstan saw three major devaluations of the national currency, tenge. The first devaluation happened in February 2009, when the Kazakhstani tenge dropped by 20 percent against the U.S. dollar, from KZT 120 to 150 per U.S. dollar. The second devaluation occurred in February 2014, depreciating overnight by 20 percent from 150 to about KZT 180 per U.S. dollar. In August 2015, the National Bank moved to a floating exchange rate, and as a result, the tenge lost a third of its value by November 2015 and was devalued by more than 85 percent by the end of December 2015. 5 and Component 2 grant programs (independent selection process and relatively large monetary size of grants), and the establishment of the ISCB and the TCO. The creation of these institutions introduced a new approach in Kazakhstan to ensure competitive grant selection procedures and appropriate guidance and monitoring toward scientific and commercial targets. 25. Weaknesses in project design. The ICR identifies the following weaknesses: (a) The PDO of this pilot project has a narrow definition. In practice, the project was piloting not only the ‘interdisciplinary teams of scientists through a transparent competitive process’, but also other elements of the new ways to organize and fund applied research, such as expert technical assistance and commercialization training, as well as the provision of significantly increased grant amounts. (b) Ongoing evaluation is particularly important for pilot projects, and the project design would have benefited from introducing a separate component or subcomponent dedicated to the demonstration affect, building on some of the evaluation measures included in section C1 of the PAD, and adding explicit activities to disseminate the learnings from the pilot project to NIS stakeholders. For example, regular evaluation studies could have strengthened the demonstration affect, as well as explicit activities to showcase the results throughout project implementation. (c) Project effectiveness could have been enhanced by a stronger explanation of expected synergies between different project activities, as well as their link toward the achievement of the PDO. (d) The limited decision-making power vested in the PMU gave rise to certain implementation challenges throughout the project (to be discussed in section 2.2). The PMU’s capacity could have been strengthened by one of the following options: creating a separate legal entity for the PMU, establishing the PMU within the MoES,10 or assigning all project management functions to the TCO. 26. Assessment of risks. Key risks were thoroughly assessed in PAD section C3, with additional financial management (FM) risks assessed in annex 7 of the PAD. Overall, the risk assessment in appraisal proved to be accurate. Most of the identified risks were mitigated through prudent implementation arrangements.11 However, the mitigation measures to reduce capacity- related risks were not sufficient and materialized over the course of project implementation. 2.2 Implementation 27. Delays at the start of the project. The project was made effective in December 2008; however, the official project launch was delayed until the end of 2010, with the first disbursement in December 2011. At the time, there were significant ownership issues at lower levels of implementation on the borrower’s side. Though it was initially planned to be housed in the MoIT, the project was transferred to the MoES in 2007. Limited readiness and execution capacity at the 10 Consisting of civil servants, instead of consultants with no authority to make decisions. 11 For example, the measures to reduce the risk of corruption and bias in the selection process were consistently recognized by the stakeholders as one of the strongest features of project design during the ICR mission. 6 MoES, combined with ministerial changes,12 and the novelty of the project, all contributed to low implementation capacity on the borrower’s side at the start of the project. Consequently, the MoES persistently failed to sustain two critical loan covenants—establishing a PMU and completing a Project Operations Manual. As a result, in February 2010, the Bank proceeded with suspension of disbursements, which was lifted in March 2010. 28. 2012 was a pivotal year for project implementation. After the turnover of five PMUs, a more permanent team was finally established. The first 10 SSG and JRG groups were selected in 2011. The progress made by the research groups during the first year of the grant program and the GoKZ’s commitment to the project objectives formed the rationale for the extension of the project’s closing date, leading to the first restructuring in July 2012. The Bank team was fully engaged to ensure that the authorities of the GoKZ could demonstrate a full ownership and commitment to advancing the implementation of the project. After the arrival of a new task team leader (TTL) in 2012, a short-term road map and action plans were established, further contributing to improved implementation in the latter half of the project. A comprehensive midterm review was prepared in 2013 confirming the relevance of the project design, agreeing on the revised Results Framework, and highlighting the remaining implementation issues. 29. Delays with the IMSC under Component 1. As discussed in section 1.6, the role of the IMSC was adjusted during the course of the project. As a result, a competitive selection process had to be completed to select a laboratory to host the IMSC. Further delays occurred due to prolonged negotiations with the host university. Consequently, the IMSC was not operational until September 2013. At the time of the ICR mission in March 2016, the IMSC remained at the early stages of the activity, completing the setup of equipment and developing the online platform tools. 30. Delays with Component 2. TCO activities were delayed due to significantly prolonged negotiations between the MoES and the competitively selected international contractor. The contract was finally signed in October 2013. As a result, all activities under Component 2 were delayed and had only 2 years left for implementation. 31. Organizational arrangements. The internal approval and decision-making procedures in the MoES, combined with the MoES and Ministry of Finance (MoF) interministerial approval processes, had a substantial negative impact on implementation efficiency. The issue was particularly acute since the PMU team was entirely dependent on the MoES for all processing and approvals. Substantial delays occurred in payments to PMU staff, reimbursement of expenses, and filling vacancies for key PMU staff. For prolonged periods of time, the PMU lacked key procurement and FM personnel, resulting in payment delays to grantees and low procurement capacity. The Bank continuously urged the MoES to streamline the existing arrangements. In 2013, it was agreed to transfer the noncore PMU staff and resources to the TCO, immediately improving the PMU performance. However, despite some further measures taken to simplify organizational arrangements, significant room for improvement remained until the very end of the project.13 32. Devaluations. During the ICR mission, many grantees raised the issue of devaluations (already mentioned in section 1.7) as one of the key factors affecting their performance. The grants 12 The new minister of Education and Science was appointed in December 2007 and again in September 2010. 13 The last Implementation and Status Report (ISR) shows that some of the actions agreed to address coordination issues were never completed. 7 were issued in the local currency at the exchange rate that was fixed at the start of the project. Many of the groups were purchasing equipment from abroad. As a consequence of reduced purchasing power due to devaluation of the Kazakhstani tenge, the groups had to adjust their planned research capacity accordingly. Nonetheless, despite devaluation, the grant size was sufficient to prove significant scientific and commercial results. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 33. M&E design. The design of the M&E framework was weak on several fronts. There were significant inconsistencies in the key indicators listed throughout the PAD and the Loan Agreement. As discussed in section 1.2, the ICR will use the arrangements for results monitoring presented in annex 3 (page 51 of the PAD) to assess the original M&E design. 34. Indicators. The original M&E design included three PDO-level indicators: one to measure the scientific performance of the grantees and two to measure the commercial relevance. No indicators were included to measure the demonstration effect of this pilot project, except the intermediate indicator on training.14 The PAD did not explain why some of the indicators covered both the grantees of Component 1 and Component 2, while others focused on Component 1 only. Assessing against SMART categorization, 15 these PDO indicators were specific, measurable, attributable, and moderately realistic,16 but not time-bound, due to time lags in publications and patents. The M&E framework could have significantly benefited from additional indicators to measure the increased commercial relevance of their research. The design of intermediate indicators was particularly weak and included a high number of output indicators, as well as indicators that were vaguely defined or had little relevance to the project PDO. 35. Baseline and targets. The M&E design was built on a premise that since SSGs and JRGs did not exist before the projects, most of the baselines should be set to zero. As a result, it was not possible to compare the performance of researchers who formed SSGs and JRGs before the project to their performance during the TCP, making it difficult to measure the ‘significantly improved’ performance. Proposed targets were set in absolute terms for all grantees, making benchmarking with national or international figures also not possible. These shortcoming are particularly severe for a pilot project aiming to demonstrate the effectiveness of new ways to organize R&D efforts. 36. Data collection and evaluation. Project design envisaged a strong data collection and evaluation arrangement, in which the ISCB and TCO would report regularly on a wide set of indicators, including a yearly survey of scientists and entrepreneurs, and annual PMU reports. However, the M&E arrangements did not provide specific indications on type, scope, and possible techniques to be used in the proposed evaluation reports and it did not include data collection from external sources and non-beneficiary agents. Some information from non-grantees, as a control 14 To measure the demonstration effect of this innovation project, the M&E framework could have included indicators on specific activities to evaluate project success such as stakeholder workshops, evaluation studies, dissemination efforts, and changing perceptions of NIS stakeholders. 15 SMART: specific, measurable, attributable, realistic, and time-bound. 16 The patents indicator has limitations as an indicator for commercial relevance since most patents do not lead to commercial outputs and there are commercial outputs that do not need patents. 8 sample, could have been useful to compare the TCP impacts and better determine the counterfactual and facilitate the monitoring of the attribution of the results. 37. Revised Results Framework. In 2013, the M&E framework was significantly revised with the help of M&E and innovation experts, to come up with a unified Results Framework to replace inconsistencies in the original M&E design, strengthen the PDO indicators, and to take into account an extended project closing date. The revised Results Framework was a significant improvement; however, many of the original problems remained unresolved. For example, some of the stronger indicators of commercial relevance and demonstration effect were still lacking, a number of the indicators suffered from vague or inconsistent definitions, and some of the intermediary indicators still had weak relevance to the PDO.17 38. M&E implementation. Implementation of the M&E framework suffered from limited PMU capacity to collect and report reliable M&E data regularly. M&E implementation strengthened over time with the PMU, TCO, and the international consulting firm18 continuously collecting data from the grantees and a dedicated M&E person in the PMU performing the quality control function to check the accuracy of the collected information. However, at the time of the ICR, there were still ongoing discussions on the precise definitions and methodology to report actual results against the PDO and intermediary indicators. Some of the envisaged monitoring arrangements discussed in paragraph 36 did not materialize. 19 The Bank’s ISRs contained inconsistencies, as well as gaps in reporting in anticipation of the revised Results Framework20 and did not report end-of-the-project results in the final ISR.21 39. M&E utilization. Although the M&E capacity remained low throughout the project implementation, there are several examples where the data collected proved to be useful in making strategic decisions within and beyond the TCP: (a) redirecting resources toward commercialization objectives in the second half of the project; (b) informing decisions regarding the design of the follow-on operation; and (c) communicating internally inside the GoKZ and externally with stakeholders. In addition, the M&E targets were driving the performance of the TCO and the international consulting firm,22 and played a key role in delivering against the PDO goals. 2.4 Safeguard and Fiduciary Compliance 40. Safeguards. The key safeguard policies triggered by the project investments are related to (a) laboratory safety and the safe disposal of wastes and (b) potential environmental issues arising in connection with the rehabilitation of existing laboratory space. Hence, the project has been screened and assigned an environmental assessment Category B status and triggered Environmental Assessment (OP/BP/GP 4.01). An Environmental Management Plan (EMP) was developed and disseminated. Compliance with the EMP was satisfactory throughout the project, 17 For example, the number of graduate students involved in the research groups and the rate of equipment use on the national and engineering laboratories. 18 Civilian Research and Development Foundation (CRDF) 19 For example, the annual survey to scientists and entrepreneurs for Component 1 that was replaced by the monthly reports from the beneficiaries to the ministry. 20 For example, ISR sequence 10, 11, and 12. 21 Both ISR 14 and ISR 15 report the March 31, 2015 results. 22 Although the CRDF Key Performance Indicators were not fully aligned with the TCP M&E framework 9 with the PMU environmental specialist engaged from the beginning, providing ongoing assessments to evaluate the performance of grantees. Regular reports on the project environmental performance did not indicate any high-risk issues during the project implementation. Overall, the project environmental performance was satisfactory. 41. Procurement. From the beginning, the project was marked with procurement issues, negatively affecting the project implementation capacity. In the second ISR, procurement was rated unsatisfactory due to failure to appoint a procurement specialist as specified in the loan covenant. Over time, the procurement performance improved; however, the major issues of concern continued to be delayed payments, incomplete filing of key documents, and evaluation faults. The Bank team was closely engaged, repeatedly urging the MoES to address these issues and also encouraging the procurement staff to take additional procurement training. Despite these efforts, throughout implementation, procurement capacity remained low (at times, with no procurement specialist in place). Consequently, procurement was rated Moderately Satisfactory for most of the project duration (briefly downgraded to Moderately Unsatisfactory before the midterm review). 42. FM. Supervision missions were conducted on a regular basis to routinely monitor project FM compliance. The FM arrangements for the project, including budgeting and planning, accounting and reporting, internal controls procedures, flow of funds and external audit, were assessed as overall acceptable and adequate through the life of the project. However there were several FM issues in the early stages of the project, resulting in less than satisfactory FM ratings in 2011 and 2012.23 In addition, in the beginning of the project, there were often delays in provision of counterpart funds due to lengthy processes in the MoES. The timeliness has significantly improved over time, and the PMU FM issues were fully addressed by 2014. Since then, the project remained in full compliance with the financial conditions of the Legal Agreement, including timely submission of satisfactory Unaudited Interim Financial Reports and audited financial statements, acceptable to the Bank. Management letters did not highlight any serious internal control or accountability issues. 43. Disbursements. The disbursements of the loan funds only started in December 2011. After that, disbursement slowly picked up and there were no major issues during the implementation of the project. By midterm review, the project was fully transferred to e-disbursement that allowed accelerating disbursement. 2.5 Post-completion Operation/Next Phase 44. Follow-on operation. Building on the momentum from the pilot TCP project, the new Fostering Productive Innovation Project (FPIP), launched in early 2016, will further stimulate greater collaboration between industry, science, and education. The project expands the range of instruments to foster the NIS of Kazakhstan, by scaling up the TCP activities, and building on the pilot innovation programs introduced by the GoKZ in the recent years to make innovation a more active element of economic growth. Specifically, the project continues with funding SSGs and JRGs and also introduces PhD research and training grants, all selected on a tender basis. In addition, it introduces grants for innovation consortia to promote collaboration among existing 23 These include inadequate staffing arrangements and lack of fully operational accounting software. 10 scientific research institutes and design bureaus and scientific and engineering profile laboratories in Kazakhstan. Start-up companies are further supported through additional financial instruments and solutions suitable to different stages of start-up company development. The project also introduces a coordinating institution to reflect increased complexity of the country’s innovation system. 45. Transition arrangements. The majority of staff from the TCP PMU will transition to the PMU of the FPIP and will be responsible for the day-to-day project and grant administration. Many of the former staff of the TCO will also transfer to the new PMU. The ISCB will provide scientific guidance related to all scientific matters associated with the new project. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Rating: Substantial 46. Relevance of objectives (High). The PDO of this pilot project remains highly relevant for Kazakhstan’s economic development. The first pillar of Kazakhstan’s CPS FY12–17 is focused on improving competiveness and fostering job creation. By strengthening Kazakhstan’s R&D base, the project would help to expand non-oil sector exports and employment, and strengthen knowledge for sustained growth, both of which are outlined as areas of focus in the CPS. CPS is also aligned with the long-term Kazakhstan-2050 Strategy announced in December 2012. Improving science potential and cooperation of science and business are among the top priorities highlighted in the strategy. 47. Other initiatives by President N. Nazarbayev further support the relevance of the TCP, showing a strong commitment to economic diversification through fostering innovation and increasing the commercial relevance of the scientific research. In May 2015, President N. Nazarbayev announced the ‘Plan of the Nation’, which outlines 100 concrete steps to overcome both short-term challenges and achieve the country’s ambition of joining the top 30 developed countries by 2050. One of these steps specifically focuses on the need to reform the scholarly grant and program structure to reflect the needs of the State Program of Accelerated Industrial and Innovative Development. 48. Relevance of design (Substantial). The relevance of the project design is evidenced in several new initiatives of the GoKZ that were launched, building on ideas embedded into the TCP design (further discussed in section 3.5), as well as the follow-on operation discussed in section 2.524. The flexibility built in the project proved to be beneficial in balancing different components corresponding to the changing context. For instance, the project quickly addressed the changing needs in light of new laboratories built and adjusted the purpose of the IMSC; the scope of technology audits was significantly reduced to consider numerous audits completed by other stakeholders in the NIS. However, despite these adjustments, the overall relevance of these two subcomponents was reduced by the end of the project. Furthermore, some additional design 24 For example, the follow-on operation will continue to fund SSGs and JRGs 11 shortcomings were discussed in section 2.1. Acknowledging these shortcomings, the overall relevance of objectives, design, and implementation is rated as Substantial. 3.2 Achievement of Project Development Objectives Rating: Substantial 49. This section of the ICR will use the PDO in the Legal Agreement and the PDO level indicators listed in the Arrangements for Results Monitoring (PAD, annex 3, page 51).25 50. Consequently, project efficacy will be assessed against three PDOs of this pilot project: (a) improved the scientific performance of the groups selected through a transparent and competitive process; (b) improved commercial relevance of the groups selected through a transparent and competitive process; and (c) demonstration of these results 51. Since the level 2 restructuring, which approved the new Results Framework, did not take place until the very end of the project 26 when all the funds had already been disbursed, split evaluation would be unjustified. Furthermore, the restructuring did not alter the PDO, and the changes made to the PDO level indicators were primarily focused on making definitional adjustments to the original indicators, as well as adding additional indicators to strengthen the M&E framework. In line with the ICR guidelines, the assessment will be made against the original indicators. Revised PDO indicators will provide complementary evidence to evaluate efficacy. Furthermore, selected intermediate indicators, 27 as well as additional ICR indicators will be introduced to strengthen the argument. Objective 1: Improved scientific performance of the groups selected through transparent and competitive process. Rating for Objective 1: Substantial Indicator Target Actualb Typea PDO Indicators Number of international publications from SSGs and JRGs in peer- O: 11–17 85 O, R reviewed journals (cumulative) R: 25 Intermediate Indicators Number of international coauthored articles per researcher in the SSGs O: n.a. O, R 38 and JRGs (cumulative) R: 22 Number of articles in national research journals by researchers in the R R: 113 145 SSGs and JRGs (cumulative) Number of partnerships established between local researchers and international researchers measured in terms of the number of agreements R 28 77 (cumulative) 25 Indicators are not separated into PDO and intermediary in the Supplemental Letter Number 2. 26 Two days before the project closing date, on December 29. 27 Only those intermediate indicators that are useful in assessing the achievement of the PDO are included in this section; includes indicators measured in ISRs before restructuring (both from PAD and from the Supplemental Letter Number 2), as well as indicators from the revised Results Framework. 12 Indicator Target Actualb Typea Additional ICR Indicators National Number of international publications from SSGs and JRGs in peer- n.a. comparator: 0.25 reviewed journals (per researcher) in 2014 and 2015c 0.16 Increase of Google scholar citations for 3 top TCP senior researchers n.a. n.a. +166% between pre-TCP (2009–2011) and after TCP (2014–2016) Average impact factor of the top 12 TCP scientific publications* n.a. n.a. 3.0 Kazakhstan’s citable documents for the H-index: relative % improvement n.a. n.a. +4 in the world ranking 2013–2015** Source:* PMU; ** Global Innovation Index rating for Kazakhstan (2013: 112; 2015: 107). Only partially attributable to the TCP. Note: a. O = Original Results Framework Indicator; R = Revised Results Framework Indicator b. In the absence of reported end-of-project results in the last ISR, the ICR is using actuals received from the PMU during the ICR preparation phase. All the data has been vetted by the Bank team and the ICR team; d. For 2014–2015. This is a simple division between the total number of publications and total number of researchers. It does not consider the publications published by several coauthors in a given research group. National comparator is based on data from Scopus. 52. With 8528 international publications from SSGs and JRGs in peer-reviewed journals, 38 internationally coauthored journals, and 145 articles in national research journals, the 33 SSGs and JRGs 29 have significantly exceeded the original Results Framework targets, although the best results are concentrated in just 12 of the groups (some of the grantees with few, low impact factor or no publications stated the need for more time to get visible or ambitious results). In addition, with guidance from the ISCB, 77 partnerships were established between local researchers and international researchers on the topics related to research under the TCP that contributed to improved scientific performance of the groups. 53. Going beyond the indicators in the Results Framework, the ICR assessed that the average number of international peer-review publications per researcher has significantly exceeded the national Kazakhstan results. For the period 2014–2015, the publications per researcher stood at 0.16 at the national level, while it was 0.25 for the TCP grantees (57 percent higher). 54. Acknowledging a possible selection bias, additional ICR indicators provide some quantitative and qualitative evidence to support the causality between TCP activities and these results:30 (a) The average impact factor of the top 12 scientific publications is 3.0,31 higher than most of the pre-TCP top publications by the same researchers and (b) the number of citations of the three top TCP senior researchers (h-index32 between 15 and 20) with statistics on Google Scholar have increased by about 166 percent between the pre-TCP period 2009–2011 and the 28 This is a conservative estimate; only publications in journals with impact factor counted. 29 Component 2 grantees are not taken into consideration, as by the time they received grants for proof of concept for prototype development, they had already produced their research results outside of the TCP. 30 Robust attribution analysis is not possible due to the lack of data 31 Includes articles in reputable international journals in the research areas of TCP groups, such as Catalysis Today, Carbohydrate Polymers, Physical Review, Journal of Cosmology and Astroparticle Physics, Soft Matter, Journal of Power Sources, and Journal of Biotechnology. 32 The h-index measures both the productivity and citation impact of the publications of a scientist. It is based on the set of the scientist's most cited papers and the number of citations that they have received in other publications. 13 period 2014–2016 (more details in annex 9). Besides these numbers, qualitative interviews with researchers confirmed attribution to the TCP project, and the PMU has verified that only articles and cooperation agreements related to research performed with TCP funding were considered. 55. Further evidence of scientific performance can be expected in the future because of the time lag for scientific publications and the h-index. Despite the issue of time lag and the limitations of the M&E system, as discussed in section 2.3, at the time of the ICR, there is sufficient evidence33 to conclude that substantial results were achieved against the first developmental objective of the project, especially considering the additional ICR indicators and the quality of the articles in the internationally peer-reviewed journals. Objective 2: Improved the commercial relevance of the groups selected through transparent and competitive process Rating for Objective 2: High Indicator Target Actual Type PDO Indicators Number of patent applications registered abroad through the TCO O 6–12 16 (cumulative) Number of PCT agreements approved (cumulative) R 7 5 Share of private sector financing of SSG and JRG program (%) O 5–20% n.a. Share of enterprise sector financing of R&D in SSGs and JRGs (%) R 5 5a Number of license agreements signed (cumulative)b R 4 11 Number of technology-based start-ups created under SSGs and JRGs R 10 40 and by TCO clients having commercial sales (cumulative) Intermediate Indicators Number of partnerships established between researchers and enterprise R 26 119 sector measured in terms of number of agreements Additional ICR Indicators National Number of patent applications registered at Eurasian level (per 1,000 n.a. comparator: 18.3 researchers)* 11.2 University-industry research collaboration: relative % improvement of n.a. n.a. +3% Kazakhstan in the world ranking 2013–2015** Sources: * PMU and National Statistics. Data for national comparator in 2015 not available; assumes that 2015 repeats the same number as in 2014. Excludes the 23 percent of researchers then working in social sciences and humanities producing no patents (based on 2013 estimates by UNESCO for Kazakhstan); ** Global Innovation Index (2013: 88; 2015: 85). Only partially attributable to TCP. Note: a. PMU estimate including in-kind contributions (no underlying evidence provided); b. Before restructuring it was an intermediate indicator with target 16–23. 56. The project exceeded original PDO-level indicators to show improved commercial relevance of 65 groups funded under TCP,34 except for the number of PCT agreements approved 33 See additional evidence of results in annex 9. 34 Results from both Component 1 and Component 2 grantees are counted. 14 (5 versus target of 7). 35 Furthermore, the revised PDO level indicators provided additional evidence of improved commercial relevance. Overall, the results show impressive performance with regard to commercialization, especially given the short time frame for Component 2. The strongest indicators of improved commercial relevance of the projects are as follows:  40 of the 65 companies created under TCP reached first sales by the project closing date, totaling over KZT 900 million36  In 2011–2014,37 the average number of PCT applications approved in Kazakhstan was 20. The TCP alone had 5 PCT agreements approved in 201538  For 2014 and 2015, the number of patents applications registered per 1,000 researchers at the Eurasian level is 11.2 (in national sciences and engineering) while TCP provides a rate of 18.3 for the same scientific fields (TCP outperforms by 64 percent).  In 2014,39 only 3 industrial prototype license agreements were made in Kazakhstan. TCP-funded projects signed 11 license agreements 40 (8 sold, 3 bought), thus significantly exceeding the national performance before the project. 57. The ICR team observed varied commercialization strategies among grantees. While some groups reported very impressive sales volumes (for example, PD-00751 ‘Industrial prototypes of ear tags for sheep and goats’ reported sales of US$3.3 million), some others have produced little volume of sales or no sales to date, but they have more impressive IP results (for example, PC- 00653 ‘Zn/NiOOH static rechargeable battery’, already applying for international patents in the United States, Europe, and China). In some other cases, the commercialization strategy is largely based on trademarks and copyrights, not on patents (for example, PD-00226 ‘Wipon system for counterfeit goods detection’ and 04/2013/JRG ‘Geo-formation system to determine the parameters for ground condition and mine working’). 58. All grantees have reached the commercialization results using the scientific research or proof of concept and prototype development results funded by the TCP. Consequently, all commercialization results can be directly attributed to the TCP. Furthermore, all of the 17 beneficiaries met during the ICR mission confirmed that they could not have achieved the commercialization and scientific results without the TCP, attributing their success to the grant funding and the expert support from the ISCB, TCO, and CRDF. This message was reinforced by the results of the end of project beneficiaries’ survey conducted by the ICR team41, which also estimates 5.2 years of technological lag in the counterfactual scenario of non-TCP.  35 However, if all patent applications abroad are counted, the total number is above target (inconsistency between indicator wording and definition provided in the revised Results Framework). 36 Following the CRDF report to use the July 2015 exchange rate, at US$1=KZT 187, this equals US$4.8 million. 37 Source: National Statistics. Data for 2015 have not been released at the time of the ICR. 38 Data not available for all international patent applications for Kazakhstan. 39 Source: National Statistics. The PMU reported that only data for 2014 is currently available. During the ICR mission, stakeholders confirmed that the number of licensing deals in Kazakhstan was extremely low. 40 Full list provided in annex 9. 41 Further details provided in annex 5. 15 59. Because of the strong results against the PDO level targets, and additional ICR evidence, the ICR concludes that the project achieved high results against the second development objective of the project.42 Objective 3: Demonstration of these results Rating for Objective 3: Modest Indicator Target Actual Type Intermediate Indicators O: 240 Number of scientists and entrepreneurs trained at the TCO O,R R: 420 881 Additional ICR Indicators High level annual forums/workshops to showcase TCP n.a. n.a. 3 Roundtables and meetings with participation of media outlets n.a. n.a. 17 Number of trainings and workshops organized by TCO n.a. n.a. 27 60. As a pilot, in addition to improving the scientific and commercial relevance of the groups funded under the TCP grant programs, the TCP aimed to successfully demonstrate its results. To achieve this objective, a pilot project requires ongoing evaluation of project activities to measure their impact toward the PDO and effective dissemination of results to the stakeholders of Kazakhstan’s NIS. As discussed in section 2.3, no PDO-level indicators were included to measure against this objective. 61. Despite lack of explicit indicators to measure the achievement toward this PDO and weak evaluation processes against scientific performance and commercial relevance objectives throughout the project implementation, some measures were taken to demonstrate the results of the pilot project. Three annual high level events were held that helped showcase the results of the TCP. The midterm workshop organized in 2013 focused on a presentation of the emerging success of the project to a broad set of stakeholders (relevant government agencies, the parliament, the press, representatives of the enterprise sector, and the donor community) and raised awareness of the TCP; it also included a discussion on the issue of transforming emerging successes to accepted role models to be followed and emulated by both the enterprise sector of Kazakhstan and its research community. In addition, in 2014, the first international forum on Technology Commercialization in Kazakhstan and in 2015, the international conference ‘Creating an innovative ecosystem in Kazakhstan’ were organized, to present the successes from the TCP and continue the dialogue with the key stakeholders on strengthening the NIS in the country. 62. To further disseminate information on emerging project success, toward the end of the project, more PMU and TCO resources were dedicated to increase media coverage of the TCP project. In particular, the TCO held 17 roundtables and meetings with participation of media outlets, and published 25 monthly newsletters, 4 information catalogues, and booklets were 42 Further evidence provided in annex 9. 16 published and distributed during 2013–2015.43 Furthermore, the TCO conducted 27 training and information workshops for 881 participants (exceeding the original target of 240 by 267%), and used it as a channel to demonstrate the results of the TCP. 63. Finally, although the PDO focused on the demonstration effect and did not include replication as an explicit target, several of the TCP elements have been successfully adopted in Kazakhstan’s NIS (to be further discussed with other impacts on institutional changes in section 3.4), and there was a big appetite for the follow-up project, implying that despite lack of robust evaluation practices, the pilot project had some success in demonstrating results. 64. Overall, combining substantial performance with regard to improved scientific performance, high performance against commercial relevance of TCP grantees objective, and  modest evidence of successful demonstration effect (taking into account the lack of robust data), the project efficacy is rated as Substantial. 3.3 Efficiency Rating: Modest 65. Traditional measures of efficiency. No ex ante measures of efficiency are provided in the PAD. This can be explained by the challenge to apply economic analysis to innovation projects owing to difficulties in quantifying positive externalities related to R&D and the gap between social (economic) and private rates of return,44 where benefits are difficult to measure because of data availability and attribution issues at various layers of benefits, as illustrated by the TCP.45 66. In light of these limitations, the ICR proceeds with standard cost-benefit analysis to assess the net present value (NPV) of the project considering direct and indirect costs, and financial and economic revenues generated by the TCP. The analysis arrives at an NPV of KZT 275.3 million (US$0.84 million),46 and economic rate of return (ERR) is estimated at 23 percent47. It must be noted that due to the retrospective nature of the analysis, it does not capture the majority of economic benefits that will be realized several years after the project closure, given the time lag with which R&D expenditures affect the economy.48 67. Available efficiency comparison for Component 2. While there is no comparator program of the same scope and similar design (using the same grantee selection, financing, and capacity mechanisms) that could be used to benchmark the efficiency indicators for the whole TCP 43 Source: Summary of the TCO final report. 44 According to the analysis in the PAD, social returns to R&D exceed the return to investments in physical capital by a factor of 6 to 10 depending upon the initial level of income per capita. 45 (a) Private economic benefits of the TCP grantees; (b) private economic benefits of Kazakhstan’s enterprise sector which adapts and profits from technological achievements of the project grantees; (c) indirect private benefits to other companies; and (d) further economic benefits to society. 46 Project end exchange rate of US$1 = KZT 329. 47 Analysis is based on retrospective costs and benefits of the TCP from the perspective of Kazakhstan from 2008 to year-end 2015; further details in annex 3. 48 Goto, A., and K. Suzuki. 1989. “R&D Capital Rate of Return on R&D Investment and Spillover of R&D in Japanese Manufacturing Industries.” Review of Economics and Statistics 71 (4): 555–64. They estimated the private and social return rates in various industrial sectors in Japan, and found that not only is there a lag in the impact of R&D activities but that the lags also vary depending on the industry. 17 project, the Foundation for Small Business Support in the Science and Technology Sphere in the Russian Federation could be used as a comparator for Component 2 of the TCP. 49 ‘Sales as percentage of grant’ is the most relevant parameter along which to compare the two programs: sales compared to grant size gives a rate of 56 percent in TCP while the Russian Bortnik Fund shows a rate of 47 percent, showing a more efficient use of funds under the TCP (by 19 percent).50 68. Several implementation and design issues can be attributed to reducing the efficiency of the TCP (already discussed in section 2.2). Table 1. Implementation and Design Issues Issue Impact on Efficiency Delays at the start of Grantees had less time to prove the commercial relevance of their projects until Negative the project the end of the project. Grantees could not benefit from IMSC services during TCP implementation, Delays with the IMSC Negative which could have potentially improved their performance. Component 1 grantees did not have access to TCO services until the last 2 years of the project, affecting their commercialization performance; Delays with the TCO Negative Component 2 grantees had very limited time to prove commercial results; Regulatory review and technology audit had limited impact on the grantee performance. Organizational Due to limited PMU capacity, there were delays in procurement that impacted Negative arrangements the performance of the grantees. 69. The positive NPV of the project shows that already at the project closing time, direct project benefits have exceeded costs. The ERR of 23 percent is higher than in other World Bank Group projects and is more than double the calculated Social Discount Rate (SDR).51 In the future, the NPV and ERR are expected to further improve because of additional benefits materializing after the time lag. Furthermore, the efficiency of Component 2 (assessed in comparison to the Bortnik Fund grant program) is high. On the other hand, sensitivity analysis demonstrates that estimates of economic net benefits of the project are highly contingent on baseline assumptions and may turn negative due to relatively small changes in assumptions or data trimming for outliers. Considering the negative implementation and design issues affecting efficiency as well as sensitivity of calculated net benefits to changes in baseline assumptions, project efficiency is rated as Moderate. 49 Also known as the Bortnik Fund, hereinafter - Bortnik Fund. Comparator: 1st stage of the Bortnik Fund’s Start program (2010–2011). The two programs are similar in terms and purpose: in the Bortnik Fund, grantees are given RUB 1 million (about US$35,000 in 2010–2011) and 1 year to reach sales; it requires establishment of a legal entity or finances R&D of an existing young firm (age not more than 2 years and sales less than RUB 1 million). 50 More details available in annex 3. 51 The calculated SDR is 9 percent; the latest available analysis of Bank projects' ERRs during 1980–2004 states that the average ERR of Bank projects in this period was 19 percent, and the median was 15 percent (S. Herrera. 2005. “The Economic Rate of Return of Bank Projects”). 18 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 70. Substantial relevance and efficacy, combined with modest efficiency result in overall outcome rating of Moderately Satisfactory, in line with the last ISR,52 which assigned a Moderately Satisfactory rating to ‘project toward achievement of PDO’. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 71. Poverty and social development impacts:  Creation of new jobs with competitive salaries among scientists and support personnel employed by the groups funded under Component 1 and Component 2 grants53  Positive impact on regional development and cohesion in smaller towns such as Stepnogorsk, offering attractive conditions to recruit talent from larger cities in the country.  Involvement of graduate students in SSGs and JRGs and PhD research and training as part of Component 1 boosted social mobility among youth. Furthermore, the project helped change the mindset of younger researchers beyond direct beneficiaries of the TCP, demonstrating the potential of academic achievements in Kazakhstan.54 This could also help attract young Kazakh scientists from abroad in the future.  No gender related targets were included in the project. However, 15 out of 65 research groups funded were led by female researchers. While gender inclusion could be strengthened moving forward, the participation of female researchers serves as an encouraging example to attract more women to the field of scientific research, and boost female participation in future innovation programs.   While the project did not directly target vulnerable groups, the emergence of innovative start-ups in agriculture, health care, and access to safe drinking water have a significant impact on social development in Kazakhstan. For example, currently, the voice recognition and recording program created by one of the grants has been requested by the National Association of the Blind under the TCP innovation grant scheme. Furthermore, the General Genetics Company provides another important example. Established by scientists from the National Center for Biotechnology the company provides services for determination of an optimum individual dosage of medicinal products (Warfarin and Plavix) for the National Center for Cardiology on a commercial basis (other examples are available in annex 9). 52 Sequence 15. 53 328 unique researchers were employed by research groups funded by TCP 54 This came out strongly during the meeting with beneficiaries and other stakeholders during the ICR mission. 19 (b) Institutional Change/Strengthening 72. Changing attitudes. The results of the TCP have demonstrated the potential commercial relevance of Kazakh science. Apart from affecting direct beneficiaries, the project had an enormous positive externality by helping change attitude of both Government officials and researchers and giving a strong stimulus for institutional changes to strengthen the NIS in Kazakhstan, in line with higher-level objectives envisaged in the PAD.55 73. Spillover of best practices. Proving the success of the pilot project, the influence of many of the TCP components or concepts can be felt throughout Kazakhstan’s NIS, including but not limited to (a) the establishment of an independent agency for international peer review (National Center of Science of Technology Expertise); (b) knowledge transfer to other competitive grant programs for R&D, technology transfer, and commercialization by agencies such as National Agency of Technological Development (NATD) and Nazarbayev University, among others; and (c) the IMSC in Karaganda State University will be the first laboratory in the country to adopt international GLPs and obtain the respective accreditation, serving as a benchmark for other laboratories throughout Kazakhstan. 74. Fostering the profession of technology commercialization managers. The CRDF Global Group’s key task was to develop capacity of the local TCO staff, and the international consultants were actively involved in the process of training and supervising the TCO commercialization managers to ensure regular knowledge transfer. As a result, the TCP helped to build a strong cadre of technology commercialization specialists in Kazakhstan, majority of them trained in the United States as part of the project. The Association of Technology Commercialization Professionals (ATCP) in Kazakhstan was created to ensure that the knowledge built through TCP is not lost. The ATCP aims to further raise the awareness of technology commercialization and entrepreneurship, and support the NIS. 75. Regulatory environment. During the TCP, considerable work has been done in Kazakhstan to improve the legislation on IP and commercialization of scientific and technological creations, and TCP international experts were active participants in this process. Their proposals and recommendations were reflected in the Law ‘On amendments and addenda to certain IP regulations’, 56 Law ‘On commercialization of scientific and (or) scientific and technological creations’, and Law ‘On amendments to certain regulations of the Republic of Kazakhstan on commercialization of scientific and (or) scientific and technological creations’.57 (c) Other Unintended Outcomes and Impacts (positive or negative) 76. This project had no observable unintended outcomes or impacts. 55 This came out strongly in conversations with government officials and beneficiaries during the ICR mission, who acknowledged that there is a newfound understanding and experience in the area of IP, project management, and technology start-up creation. 56 No. 300-V ЗРК was adopted on April 7, 2015. 57 Signed by President Nursultan Nazarbayev on October 31, 2015. 20 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 77. Midterm workshop. The midterm workshop focused on (a) a presentation of the emerging success of the TCP to a broad set of stakeholders (relevant government agencies, the parliament, the press, representatives of the enterprise sector, and the donor community) and raised awareness of the TCP and (b) a discussion on the issue of transforming emerging successes to accepted role models to be followed and emulated by both the enterprise sector of Kazakhstan and its research community.58 78. Component 2 end-of-program evaluation study. The study was designed to assess the outcomes that CRDF Global, TCO managers, and entrepreneurs have accomplished under Component 2. This survey was sent to all 32 Component 2 grantees in October–November 2015, of whom 24 responded, constituting a high 75 percent response rate. The survey results reflect on participants’ satisfaction with the program,59 analyzes major knowledge gains, highlights results achieved, and reflects on impacts made on the TCO grant program participants. A section on recommendations summarizes participants’ recommendations for future programmatic improvements. The participants of the survey expressed very high satisfaction with the TCO assistance in business development; 24 respondents highlighted 42 different accomplishments that they directly attributed to the TCO grant program; the TCO grant program had an impact on participants’ professional advancement and their projects. The grantees mentioned that a short life cycle of the grant was challenging for the implementation of their projects. 79. ICR beneficiary survey. In March 2016, end-of-project beneficiary survey questionnaires were distributed to 65 TCP grantees and the IMSC to measure the TCP’s success and to collect data for an economic and financial project analysis. Thirty-two beneficiaries responded to the survey, and the results show that the TCP had a highly positive impact on the performance of the beneficiaries (88 percent ranked the TCP’s contribution to achievement of scientific and commercial goals as 4 or above [1 - no contribution, 6 - exceeded expectation]). 4. Assessment of Risk to Development Outcome Rating: Moderate 80. The project’s sustainability depends on the impact it has on the way Kazakhstan’s scientific research is organized and funded in the future. If, despite the success of the pilot project, Kazakhstan will not expand the new practices of organizing science and strengthening the link between research and markets piloted through the TCP, the achievements of the TCP will be marginal. At the time of the ICR, the probability of this occurring is being mitigated by the following factors: 58 In addition, in 2014, the first international forum on Technology Commercialization in Kazakhstan and in 2015, the international conference ‘Creating an innovative ecosystem in Kazakhstan’ were organized, to present the successes from the TCP and continue the dialogue with the key stakeholders on strengthening the NIS in the country. 59 For example, about 80 percent of respondents considered the TCO managers’ assistance as ‘extremely valuable’. 21 (a) The follow-up operation (discussed in section 2.5) has launched in 2016 and is crucial in ensuring the sustainability of the TCP. It will further scale up TCP activities, and ensure continuity by maintaining the PMU staff of the TCP project. (b) Best international practices brought by the TCP are already being adopted by key players in the NIS, such as establishment of independent agencies for international peer review, competitive grant programs for research that builds heavily on the design of the TCP grants, and strengthened TCOs in universities across Kazakhstan. (c) The GoKZ remains committed to an ambitious strategy of fostering innovation as a driver of economic growth (as discussed in section 3.1). Throughout the implementation of the TCP, the minister of Education and Science changed three times; nonetheless there was a continuous commitment to the development objectives of the TCP, proving that the goals and concepts introduced by the TCP have strong resilience to political changes. (d) The legislative framework has undergone significant reforms to provide improved conditions for commercialization results of R&D. (e) The ATCP, created under the TCP, continues its activities to promote the development of the technology commercialization system, mitigating the risk that the capacity of the local staff built through TCP would be lost following the end of the project. 81. Nonetheless, there is a risk that if low oil prices prevail, the GoKZ might face affordability constraints and reduce funding for the R&D sector. Furthermore, the TCP implementation capacity issues on the borrower’s side also expose a risk that the future R&D funding projects might be subject to similar coordination and communication issues between different stakeholders of the NIS. Finally, the depth of R&D talent with commercialization potential in Kazakhstan is still relatively untested and could prove to be lower than expected. While the probability of any of these risks materializing is relative low, it could have a significant impact on the sustainability of the TCP achievements. 82. There are also risks associated with the sustainability of the start-ups created under the TCP. Given the fragile ecosystem of the early stage entrepreneurship in Kazakhstan, there is a moderate risk that some of the companies created under the TCP will not survive without further grant funding.60 Furthermore, since the IMSC has started its operations just before the very end of the TCP project, at the time of the ICR, there is little evidence to assess the extent to which it will fulfill its mandate envisioned under the project. Nonetheless, even if these grantee-related risks materialize, they would not have a big impact on the impetus created by the TCP to reform the Kazakhstan’s NIS. 83. Taking into account the likelihood of the risks discussed and their potential impact on the operation’s development outcomes, the overall risk to development objective is assessed as Moderate. 60 This is also an inherent risk to all innovation projects in which the failure of some of the start-ups is an inevitable scenario. 22 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 84. As noted in previous sections, the project was strongly aligned with the high-priority issues in the CPS and with the government priorities. The design of the project considered the lessons learned from previous Bank projects. The project’s risk assessment, as well as the design of fiduciary and implementation arrangements, was Satisfactory. Relevant safeguards were properly assessed and addressed in the project design. The rating was downgraded to Moderately Satisfactory due to shortcomings in the definition of a sound M&E framework, discussed at length in section 2.3, and design drawbacks discussed in section 2.1. (b) Quality of Supervision Rating: Moderately Satisfactory 85. The Bank’s supervision was strongly focused on the project’s development impact. Regular supervision missions often included innovation specialists, procurement, FM, and environmental experts. Aide memoires focused the government and the PMU’s attention on resolving implementation issues by suggested adjustments to address the underperforming areas. 86. The transition to a new TTL in 2012 was very smooth and gave a fresh impetus to project implementation. The project also greatly benefited from improved local capacity, with a project co-TTL located in Astana, which enabled close interaction with the GoKZ and the PMU. In contrast, at the beginning of the project, the Bank’s capacity on the ground was weak, coinciding with the implementation delays. 87. Close Bank supervision can be credited for the successful project turnaround in 2012 (also discussed in section 2.2). After the slow project start, the Bank team closely engaged with the GoKZ to ensure the borrower’s commitment and ownership to the project objectives before reaching an agreement to extend the project’s closing date. The Bank team also worked with the senior officials and technical staff at the MoES and MoF to jointly prepare the implementation progress and agree on a road map and action plan for the project implementation until the extended closing date. In addition, the Bank team instituted a weekly videoconference/audio meeting system to follow up on progress to assist the PMU and MoES to improve processing. A comprehensive midterm review mission was carried out in 2013, and the Bank team was actively engaged until the end of the project, providing adequate guidance toward timely achievement of the PDOs. 88. Considering the importance of the Bank’s role in achieving the PDO, the Bank’s role could be rated as Satisfactory. However, the rating was downgraded to Moderately Satisfactory because of the following reasons:  The Bank could have played a more proactive role in addressing the severe implementation issues and delays at the beginning of the project, by drawing on the lessons learned from other countries that had experienced similar implementation problems. 23  The Bank team could have made further efforts to ensure that the revised Results Framework was officially approved by the Bank on time.  Shortcomings in ISR reporting of results: (a) the number of indicators monitored is not consistent across the ISRs; (b) after the new Results Framework was agreed in 2012, in anticipation of the upcoming level 2 restructuring the results for the original indicators were not updated in the ISRs until March 31, 2015; (c) the last project ISRs61 do not provide end-of-the-project results and instead repeat data as of March 31, 2015.  (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 89. Consistent with the Bank performance in ensuring quality at entry and with the quality of supervision, the overall Bank performance is rated Moderately Satisfactory. Significant weaknesses in the M&E framework design and implementation is the primary cause behind the downgrade of ratings despite otherwise strong Bank’s performance throughout the project cycle. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 90. The high ownership at the ministerial level to achieving the development objectives played a key role in the advancement of project implementation. Appropriate annual budget allocations to the MoES were made to ensure adequate fund to cover borrower’s contribution to the project. Nonetheless, the start of the project suffered severe implementation delays owing to ownership issues at the lower levels of the GoKZ (discussed in section 2.2). Furthermore, changes at the ministerial level 62 contributed to ongoing staffing changes in the PMU. All these factors contributed to the delays and led to the breach of covenants in February 2010.63 The borrower’s performance significantly improved over the course of the project with steps taken to improve cumbersome institutional arrangements and complex interministerial coordination and approval process, although some obstacles remained until the end of the project. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 91. Due to organizational and institutional coordination issues discussed in section 2.2, for prolonged periods, the PMU’s capacity for FM, procurement, environmental monitoring, and M&E functions remained relatively low. The creation of a more permanent PMU in 2011 and the transfer of the noncore PMU staff to the TCO in 2013 had significantly improved the institutional arrangements for the project management. Nonetheless, throughout the project, the operational 61 ISR 14 (archived on December 3, 2015; last ISR before the second restructuring) and ISR 15 (archived on December 30, 2015; the only ISR using the new Results Framework). 62 Minister of Education changed in 2010 and 2013. 63 Failure to hire core PMU staff and approve the Project Operations Manual. 24 PMU difficulties affected project implementation despite ongoing efforts and dedication of the PMU staff. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 92. Despite evident high commitment to achieving the development objectives from both the GoKZ and PMU and significant improvements in performance toward the end of the project, the implementation suffered from the cumbersome intra-ministerial and interministerial coordination and approval processes. As a result, the overall borrower performance is rated as Moderately Satisfactory. 6. Lessons Learned 93. The TCP shows that innovation projects are most effective when delivered as part of a broader long-term strategy to strengthen capacity within the NIS. The way the GoKZ has been engaging with the Bank on a long-term strategy for innovation, placing Bank projects at the core of the national innovation strategy, and cofinancing operations is an example to be considered by other countries. This approach helps leverage the use of public resources, increase policy coherence, and maximize the development impact. The TCP pilot approach to strengthen links between the university research and the market served as a necessary and pioneer step to address the very resilient divorce between university and industry, which affects NIS performance in most countries of the world. The follow-on operation (discussed in section 2.5) largely builds on the TCP experience and will consolidate this long-term strategy for innovation. 94. International expert advice and support are essential for successful innovation project implementation. The professional coaching provided to the groups proved to be critical in promoting a culture of innovation and linking academic researchers and markets. The TCP strategy of creating professional expert bodies (the ISCB and TCO)64 to apply international best practices at different stages of implementation (including grant selection processes and supervision of the selected groups) proved to be instrumental in achieving the PDO goals. The day-to-day coaching advice from commercialization managers (trained in the United States and continuously guided by Kazakhstan-based international experts) further strengthened the effectiveness of the technology commercialization efforts and produced outstanding results in an impressively short period of time. Furthermore, the knowledge transfer to the local staff helped ensure that the commercialization capacity remains in the country after the end of the project, enhancing project sustainability. 95. Effective innovation programs for technology commercialization need to integrate the enterprise sector from the early stages of the project, combining both supply and demand approaches. The TCP impact could have been further enhanced if industry partners had a more active role from the very early stages of the project (for example, through public-private consortia). Market goals, like integration into the technological global value chains, should be clearly defined. Lessons learned from innovation projects with a strong focus on demand-driven innovation policies should be carefully screened and adopted when appropriate. 64 Engaging international contractor CRDF Global. 25 96. Success of project interventions targeting R&D is affected by grant ownership structure and the level and flexibility of grant funding. The TCP proved that the creation of independent legal start-up entities, keeping them separate from their housing institutions, can play a big role in the success of R&D projects. This arrangement ensured the degree of decision freedom, flexibility, and ownership necessary to change mindsets among researchers and maximize operational efficiency. Providing grants with sufficient amount of financial resources also made a difference with respect to other funding schemes for innovation in the country. In particular, grants that cover all stages from the laboratory studies until entry into a market are the ones that can pay off in countries like Kazakhstan, where other funding sources for early stage start-up funding are extremely limited. Allowing some flexibility in grant budget allocation further contributed to the success of the selected research groups.65 97. The importance of institutional arrangements cannot be underestimated. In the case of the TCP, a number of implementation delays could have been avoided and project efficiency notably improved if the PMU had been granted a clearly defined decision-making authority from the start of the project. The TCP experience has shown that despite strong government commitment, complex communication and coordination arrangements between different executive branches can be detrimental to the project implementation. Arrangements to streamline these processes should be provided at the project design stage, for example, by establishing a steering committee to make decisions in a more efficient manner. Furthermore, attention needs to be paid to foster a culture open to novel approaches in the stakeholder organizations of the NIS. Coordination is also needed between different innovation initiatives promoted within the NIS.66 98. Innovation projects need to think about a strong and ‘innovative’ M&E framework. Owing to the deficiencies in the M&E framework, the TCP struggled to gauge the ability to assess the impact of its intervention, especially during the first years of implementation. This was partly due to the traditional focus on publications and patents as main innovation indicators, which have severe time-bound issues for short-term M&E needs and they are more innovation outputs than real innovation outcomes, failing to capture the full range of impacts coming from innovation interventions.67 Moreover, some commercialization efforts do not end in patents but in other IP instruments or results not related to IP. The TCP also highlighted the importance of establishing a strong and comprehensive M&E system including, if a project budget permits, an ex ante impact evaluation mechanism with control samples, where counterfactuals and attributions can be clearly assessed. A baseline survey is indispensable in this context. Another challenge is about the integration of demonstration effects in the M&E framework. Pilot projects will benefit from explicit ongoing measures to evaluate and disseminate results, to ensure that lessons learned can be useful for other innovation projects. 99. Individual case studies could be further studied for policy implications. Policy implications from the 65 projects funded by TCP can be extracted, both on the legal-regulatory 65 This is especially relevant for research projects where it is more difficult to foresee the budget needs in advance and adjustments need to be made along the way. 66 For example, the new national Law for Technology Commercialization in Kazakhstan, developed with valid contributions from the TCP, has shown the adequacy of combining the purely innovation work with the regulatory work that improve investment climate and Intellectual Property Rights framework. 67 Such impacts include the sales or contracts coming from start-ups and the spillover effects generated by new collaborative research. 26 side and on the market functioning side, for example, to monitor and enhance integration with global value chains and the technological frontier in the specific industries. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies Borrower’s ICR is included in Annex 7. (b) Cofinanciers Not applicable. (c) Other partners and stakeholders Not applicable. 27 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$, million equivalent) Appraisal Estimate Actual/Latest Estimate Percentage of Components (US$, millions) (US$, millions) Appraisal Component 1: Strengthening Science 46.97 35.70* 76 Base of Kazakhstan Component 2: Linking Kazakh Science 15.75 18.16 115.3 to Markets Component 3: Project Management Unit 9.55 8.63 90.3 Total Baseline Cost 72.27 62.49* 86.5 Price Contingencies 2.70 2.70 100 Total Project Costs 74.97 65.19 86.9 Front-end fee Project Preparation Fund 0.00 0.00 0.00 Front-end fee IBRD 0.03 0.03 100 Total Financing Required 75.00 65.22 86.9 Note: *Proposed amount at the abovementioned restructuring. (b) Financing Appraisal Actual/Latest Type of Percentage of Source of Funds Estimate Estimate Cofinancing Appraisal (US$, millions) (US$, millions) Borrower Co-financing 61.60 53.87 87.4 International Bank for Reconstruction and Loan 13.40 11.35* 84.7 Development Total 75.00 65.22 86.9 Note: *The project restructuring that was approved on December 29, 2015 involved cancellation (dated December 9, 2015) of uncommitted funds in the amount of US$1,826,818.63 equivalent from the Loan 48840-KZ, from the category no. 1. SSG Grants and JRG Grants under Part 1(b) of the project. These funds amount to about 13.6 percent of the IBRD Loan funding under the TCP. Because of the devaluation of the Kazakhstani tenge over the past year, the funds could not be utilized on time before project closing on December 31, 2015. 28 Annex 2. Outputs by Components 1. Although the PAD does not provide an explicit explanation of the underlying theory of change, the results chain can be built using the intermediary outcomes specified in the updated Results Framework. Table 2.1. TCP Results Chain 2. The sections below discuss the output by each subcomponent 68 and the link between outputs and outcomes. Component 1: Strengthening Science Base of Kazakhstan Table 2.2. Summary of Key Outputs for Component 1 Output Comments The ISCB established Completed in 2011 33 SSGs and JRGs financed Achieved; exceeded initial target of 20 61 graduate students involved in the SSG and JRG research projects Achieved in 2015 (58 in 2014, 35 in 2013, 16 in 2012) The IMSC established Completed after significant delay Internationally peer-reviewed journal established Completed after significant delay 68 Follows the structure of the PAD. There are some insignificant differences in the way subcomponents are organized in the Loan Agreement and the PAD. 29 3. This component financed three sets of activities. Subcomponent 1.1: The Establishment of an International Science and Commercialization Board and an Internationally Peer-reviewed Journal 4. Output. The ISCB was formally introduced in 2011 with the intention to bring informed neutrality of decisions on applications for TCP grants and provide academic and commercialization support to the selected groups throughout the lifetime of the project. Five eminent scientists from the countries such as the United States, Germany, Russia, Saudi Arabia, and two technology commercialization experts from Silicon Valley and Michigan (United States) were selected to form the ISCB. It was vested with the exclusive authority to determine the grants application process. The ISCB visited Kazakhstan twice a year during 2011–2015 to monitor research and commercialization of SSGs and JRGs, and provide further guidance to achieve the project goals. In addition, it played an important role in preparing the revised terms of reference and the selection process of the IMSC. 5. Contribution toward achieving the PDO. During the ICR mission, the beneficiaries, the PMU, and the government officials highlighted the establishment of a highly competent and independent ISCB as a key ingredient to the scientific and commercial success of SSGs and JRGs. It was critical in ensuring fair and transparent selection process of the grantees. The ongoing support and advice on scientific and commercial performance also played an important role in improving the results of the grantees. Subcomponent 1.2: Establishment of a Senior Scientist Group Grant Program and Junior Researcher Group Grant Program 6. Output. Grants for the SSG and JRG programs provided funding to leading Kazakh scientists and talented young researchers motivated to conduct R&D at international standards. These groups were selected on the basis of (a) their potential for conducting world class scientific research, that is, scientific merit; (b) relevance of their research for current or future scientific, economic and industrial development of Kazakhstan, including the possibility of forging research partnerships with local or foreign business firms; (c) plans for mentoring graduate and undergraduate students and to provide training fellowships aimed at nurturing the next generation of scientists; (d) their potential for and interest in conducting multidisciplinary research, and (e) their proposals for outreach activities to universities, secondary schools, and the general public. The grant funding volume amounted to US$600,000 for JRGs and US$1,500,000 for SSGs. 7. Three rounds of grant programs took place in 2011, 2012, and 2013. Out of 785 applications, the ISCB selected 33 senior and junior researcher projects (10 in 2011, 11 in 2012, and 13 in 2013). Originally, only two rounds of grants were envisaged; however, the MoES decided to launch the third round of grants due to the initial success of the first two rounds. 8. In Kazakhstan’s context, the SSG and JRG grants introduced several innovative elements:  An independent and international competitive process  30  The funding was sufficient for the entire process from the technology development concept at a laboratory to the entry into market, in the form of finished goods or services   Simplified application process   Ongoing support from international scientific and commercialization experts  9. Selected groups spanned across a wide range of scientific fields such as oil transportation, reservoir recovery, and petrochemistry; construction materials and production waste disposal; nanotechnology; agriculture, genetics, and cytology; electronics and instrument engineering; radioecology and radiochemistry; alternative energy; and medicine and pharmaceutics; and worked under supervision from the ISCB. 10. Despite some resistance from the hosting institutions, all the research groups met the project requirement to form a private legal entity that could receive project funding. This proved to be a critical element for the project success (as discussed in section 6). Both the requirement to form a legal entity and the large grant amount were highlighted as important aspects to achieve significant improvements in scientific and commercial performance. 11. Contribution toward the PDO. Testing the design and implementation of the SSG and JRG grant formed the core of the TCP project. Other elements such as the creation of the ISCB, IMSC, and TCO were designed to enhance the scientific and commercial performance of the selected groups. Therefore, successful implementation of the SSG and JRG grants played a key role in achieving the development objective of the TCP. Subcomponent 1.3: Establishment of the International Materials Science Center 12. Output. Since the beginning of the project, the circumstances in which the IMSC was envisioned had changed. The GoKZ had made significant investments in scientific infrastructure by establishing 20 modern labs (5 national research laboratories and 15 engineering laboratories in universities). Research activities of these laboratories were fairly fragmented. Considering the changed conditions, it was agreed that instead of building a new laboratory, the IMSC would be established in one of the existing laboratories selected through competitive process. The terms of reference were expanded from being (only) a center of excellence that sets a benchmark for Kazakhstani laboratories and promotes international best practice to a hub that also improves access to information on research equipment in national and engineering laboratories through an IMSC-managed information portal. This portal was envisaged to serve as a central hub of information exchange and resource sharing for the Kazakhstani scientific research community, including the ability to share, exchange, lease, and purchase laboratory equipment. 13. For the establishment of the IMSC, a tender was held among existing scientific laboratories. Out of nine tender applications, the ISCB members selected Karaganda State Technical University. After long delays due to complications with the hosting university related to the legal status of the IMSC, it was operational only by the end of 2013. 14. At the time of the ICR mission in March 2016, the IMSC had already procured all the equipment; however, it was not yet fully functional. The web portal was created, with the option 31 to share equipment across the existing labs. Collaboration agreements have been signed with 30 scientific organizations. An internationally peer-reviewed journal with a focus on material science was launched (Central-Asian Material Science Journal, which is available online). Four issues had been launched until the end of 2015. Two international research-to-practice conferences and a number of various workshops had been held. Business processes were developed in accordance with the standard of the International Organization for Standardization (ISO) and the relevant brochure was issued. At the time of the ICR mission, the IMSC was still in the process of obtaining the GLP accreditation. 15. Contribution toward PDO. Due to delays in the IMSC launch, the SSGs and JRGs did not have timely access to the IMSC equipment and did not use the services of the IMSC to access national and engineering laboratories. Less than 10 percent of all articles published in journals come from the TCP grantees. Despite the potential to have a material influence on Kazakh science in the future, the ICR is of the opinion that the establishment of the IMSC had a very limited influence on the scientific and commercial performance of the groups’ funding by the TCP. Component 2: Linking Kazakh Science to Markets Table 2.3. Key Outputs for Component 2 Output Results Completed after delay (started operations in The TCO established late 2013) Technology audit completed Completed after delay (in December 2014) Comparative review of IP regulations completed Completed after delay (in December 2015) 881 researchers/scientist and entrepreneurs trained by the TCO Exceeded target 33 grants awarded for technology commercialization Below initial target of 55–60 Number of partnerships between SSGs, JRGs, research Achieved after significant delay institutions, and private sector 16. This component financed four main sets of activities. Subcomponent 2.1: Technology Commercialization Office 17. Output. The TCO was set up in 2012, and after concluding the negotiations with the competitively selected international technology commercialization contractor, CRDF Global, the operations fully launched in September 2013. Key outputs of this subcomponent were the following: (a) International consultants worked closely to transfer skills by means of joint work on TCPs and training sessions. In addition, intensive technology commercialization training was organized for 12 TCO employees at IC2 Institute of the University of Texas, Austin (United States) (b) Commercialization support was provided for all 65 projects of the TCPs. (c) Assistance was provided to establish an alliance of technology commercialization professionals. 32 (d) Twenty-seven training and information workshops were conducted for 881 participants from the Republic of Kazakhstan, including those in Almaty and Astana, and in the regions of Kazakhstan, including Karaganda, Ust-Kamenogorsk, Pavlodar, Petropavlovsk, Kokshetau, Kostanai, Stepnogorsk, and other cities. Figure 2.1. Knowledge Sharing throughout Kazakhstan Source: PMU presentation with final results, 2015. (e) Roundtables and conferences were organized on technology audit, IP rights, legal support of commercialization of R&D results, and implementation progress of the TCO grant program. (f) The first International Forum ‘Technology Commercialization’ was organized, during which all 65 projects were presented. (g) A technological brokerage event was organized with the participation of third-party business formation and development experts. (h) Two methodological guidelines were published: one on creation, legal protection, and commercialization of results of intellectual creative activity for universities and start- ups and another one on methodological support of universities and research organizations in creating and developing technology start-ups. 18. In addition, TCO activities were accompanied by regular media coverage, resulting in 579 media reports and materials. In all, 25 monthly newsletters, 4 information catalogues, and booklets were published and distributed during 2013–2015. Information was also made available to all interested parties through the TCO corporate website, and social networks Facebook, Twitter, and VKontakte. 33 19. Contribution toward PDO. The TCO and CRDF activities had a very strong contribution in proving the commercial results of projects funded under the TCP. The beneficiaries greatly benefited from the trainings and other events organized by the TCO, as well as the ongoing support to individual projects from the commercialization managers and the CRDF. Furthermore, the regular media coverage helped disseminate the knowledge acquired through this pilot project, contributing to the demonstration effect objective. Subcomponent 2.2: Technology Audit and Forecasting 20. Output. The purpose of the technological audit was to (a) identify projects which may be potentially promising for project commercialization and (b) determine the demand for technologies among the industrial enterprises of Kazakhstan. The audit was carried out in several stages. By the time the TCO technology audit was launched, the GoKZ had completed numerous technology audits since 2006. As a result, at the first stage, the TCO held consultations with development agencies, including the NATD, to avoid duplication of efforts (November–December 2013). Next, the planned audit was discussed with the heads of the TCOs of universities and research institutes, to reach an agreement on methods and timing of the audit reporting mechanisms (December 2013– early 2014). In summer 2014, the TCO specialists were trained on the methodology of technology audit. The next stage was collection of technological proposals, analysis of the collected data and interviews to verify the primary information and formation of database of technological proposals. At the last stage of the technological audit, the TCO prepared a final report, held roundtables to present the report (December 2014), and launched the online version of the database (January 2015). The database contained 230 research projects with commercialization potential (supply side) and 42 technology requests from the enterprise sector (demand side). 21. Contribution toward PDO. The project aide memoires and ISRs provide no assessment on the role this subcomponent played in achieving the overall mission objectives. The ICR mission confirmed that the technology audit conducted by the TCO had no direct linkage to the success of research groups funded by the TCP under Component 1 and Component 2. As a result, the contribution toward the PDO is also considered negligible. Because its design and implementation was isolated from the rest of the project, its performance has not significantly affected the project— positively or negatively. Subcomponent 2.3: Policy, Legal, and Regulatory Framework 22. Output. During July–August 2015, an international tender for carrying out of a ‘Comparative analysis of the legislation of the Republic of Kazakhstan and leading foreign countries in the sphere of IP’ was announced. The analysis of legislation governing IP and technology commercialization was completed and results were presented during the TCP Closing Workshop on December 10, 2015. 23. In addition, the TCO and CRDF were active participants in the considerable work that was done in Kazakhstan to improve legislation in the field of IP and commercialization of scientific and technological creations. Notable contributions were made to the draft laws ‘On commercialization of scientific and (or) scientific and technological creations’ and ‘On amendments to certain regulations of the Republic of Kazakhstan on commercialization of scientific and (or) scientific and technological creations,’ approved by the parliament of the 34 Republic of Kazakhstan, with the corresponding laws signed by President Nursultan Nazarbayev on October 31, 2015. 24. The CRDF final report also summarizes unresolved issues and challenges in the legislation of the Republic of Kazakhstan in the field of IP and commercialization of intellectual creations and provides recommendations to address them. 25. Contribution toward PDO. The TCO and CRDF experts made a significant contribution toward improving the legislation of the Republic of Kazakhstan in the field of IP and commercialization of scientific and technological creations, which will serve the future commercialization projects in the country, including the FPIP. Their role and impact was recognized by numerous stakeholders during the ICR mission. Nonetheless, due to delays in completing the comparative analysis and the time lag until the laws came into force, this subcomponent had a limited direct impact on the project PDO. Subcomponent 2.4: Design and Implementation of Grant Programs 26. Output. In 2013, the CRDF and TCO carried out an analysis of existing grant programs in Kazakhstan for development and commercialization of technologies. Projects such as the Science Fund, the NATD, KazAgro, and the existing Bank grants were examined to determine the gaps that existed in the commercialization chain, both in general rationale (size of funding, areas of science being funded, origins of technologies, activities covered by those programs) as well as implementation (transparency of selection process, peer review, mechanism of feedback to applicant, ease of application, method of monitoring, and mentoring). 27. Based on this information, a grant program was designed to cover both proof-of-concept and industrial prototype activities. Two substantial grant programs were launched by the TCO in the summer and fall of 2014. Over 750 applications have been submitted and reviewed by the local as well international subject matter experts. Thirty-three projects were selected to receive grants ranging between KZT 9,500,000 and KZT 74,000,000 (13 ‘proof of concept’ and 20 ‘industrial prototype’ grants). During the project implementation, one of the projects had to be closed, with the remaining 32 successfully completing the grant program. In contrast to what was envisioned in the PAD, the SSGs and JRGs did not participate in the grant program under Component 2. 28. The TCO’s project selection and grant-funding mechanisms differed from other grant programs in the following ways:  The grants did not require cofunding nor is there a reimbursement of expenses.  In case the applicant’s project is not successful, the applicant is not liable to return the entire amount of the grant money. This was common practice in Kazakhstan and guaranteed that only projects that were guaranteed to succeed (or were strictly engineering improvements) would be funded. Truly innovative projects were not considered.  Applications go through a multistage review process, including both Kazakhstani and international experts. Applications are reviewed from both a technical (scientific) and 35 a commercial point of view. Technical reviews will be conducted by three independent national reviewers to diminish claims of influence or favoritism.  Electronic applications are accepted instead of physical, stamped copies.  Reviewer comments are passed on to the applicant. This ensures that applicants ‘get something back’ from the process even if they are not chosen and allows them to improve their application if they choose to resubmit in subsequent competitions.  Commercial viability elements are emphasized as project criteria from the very beginning of the process.  Project managers are assigned to each grant recipient from the outset to mentor the teams and keep the projects on track to meet their goals from a technical and (more importantly) a commercial point of view. Having the managers in continual contact with the teams is critical to offer timely support. Having the project teams ‘go it alone’ is a model for failure.  Grant funding can include expenses for IP protection as well as business trips to promote the technology (negotiations with potential business partners, visiting relevant industrial expositions, and so on) 29. All Component 2 grant recipients received strong support from the TCO and CRDF, including advice on marketing, IP, and licensing and marketing issues. Each group worked with an assigned technology commercialization manager to find potential market opportunities for the products developed under the project. A specialist for IP and licensing provided the necessary support and advice. 30. In contrast to grants under Component 1, focus was solely on reaching commercialization targets. The M&E arrangements to measure scientific achievements of SSGs and JRGs were not applied to the recipients of Component 2 grants. 31. Contribution toward PDO. The results of the groups funded under Component 2 provide additional evidence that with adequate support, Kazakhstan has a strong potential to increase the commercial relevance of research conducted by interdisciplinary, and competitively selected researchers. The pilot highlighted the importance of the multistage grant candidate selection process, enhanced project M&E process, and the importance of business mentorship in achieving successful results. Component 3: Project Management Unit 32. Output. At the start of the project, due to lack of ownership at the MoES and ministerial changes at the start of the project (new minister appointed in September 2010), there were significant delays in establishing a fully functioning PMU. The PMU suffered significant staff turnover and during 2008–2011, five PMUs have changed. Finally, in 2011 a more permanent core PMU team was established. In the early years of the project, the PMU operated under difficult circumstances—not having a properly equipped office, limited Internet access, the PMU salaries 36 not paid on a regular basis, and delayed reimbursement of expenses and business travel. For prolonged periods, the PMU lacked key procurement and FM staff. This was due to cumbersome institutional arrangements and complex interministerial coordination and approval processes. Meanwhile, PMU staff were not public servants, and therefore had no formal authority to take decisions on project implementation. 33. The PMU was responsible for day-to-day management of project implementation. It conducted functions of M&E, reporting, compliance with safeguards and fiduciary requirements, and others outlined in the Project Operations Manual. 34. To address payment delays, in 2013, noncore PMU staff and relevant resources were moved from the PMU to the TCO. The PMU director, FM specialist, procurement specialist, and accountant remained in the PMU. 35. The performance improved over time, allowing the PMU to strengthen its role in advancing the project implementation. However, until the end of the project, the approval and interministerial processes left substantial room for improvement, also affecting the effectiveness of the PMU. 36. Contribution toward PDO. Tasked with the day-to-day coordination, implementation, and management of the project, the PMU was at the center of all TCP activities; thus, its performance played an important role in achieving the development objective of the project. Because of the difficulties discussed above, the PMU implementation capacity was significantly affected, also negatively affecting the efficiency of the whole TCP despite the highly relevant improvements and professional management in the final phase of the project. 37 Annex 3. Economic and Financial Analysis Overview 1. This economic analysis follows the Bank guidance note on economic analysis of investment project financing and uses a retrospective cost-benefit analysis to assess and monetize costs and benefits of the TCP and associated externalities. In particular, it analyzes direct costs associated with implementation of the TCP and indirect costs associated with deadweight losses due to collection of taxes and possible environmental damages. On the benefits side, the analysis covers financial and economic revenues generated by innovative activities facilitated by the TCP. However, the analysis does not include positive spillovers from enhanced capacity and improved institutional environment. The analysis arrives at the NPV of KZT 275.33 million (US$0.839 million at project end; exchange rate of KZT 329 per U.S. dollar).69 Table 3.1 presents the main economic performance indicators. Table 3.1. Economic Performance Indicators of the TCP Economic (Social) Rate of Net Present Value Benefit-to-cost Ratio Return Base case KZT 275.33 million 1.19 23% 2. While the economic analysis should be tailored to specific needs of a project, the cost- effectiveness and cost-benefit analyses are outlined in the Bank’s guideline70 on economic analysis as the two main types of analysis used in assessing impact bank projects (whether ex ante or retrospective). This analysis is based on the partial cost-benefit approach for two reasons: (a) “cost- benefit analysis is generally considered the most comprehensive approach and, in many ways, the gold standard”71 and (b) the nature of the project and country context does not allow to conduct a cost-effectiveness analysis that would capture all of the main benefits and costs of the project. Furthermore, there is no comparator program of the same scope that uses comparable project selection, financing, and capacity-building mechanisms. The closest comparator is the seed grants program of the Bortnik Fund in Russia; a close study of its selection criteria and funding mechanisms revealed that a cost-effectiveness analysis against this benchmark would be relevant only to Component 2 of the TCP. In view of these, the team decided to pursue the partial cost- benefit analysis (excluding positive spillovers from enhanced capacity and improved institutional environment). 3. Also, this analysis does not include financial analysis for two reasons:  A large number of subprojects (66 research groups and the IMSC) pose difficulties in carrying out an accurate financial analysis. The fact that some research groups have dissolved adds administrative difficulty to carrying out a financial analysis. It should be noted that the TCP in general and its subprojects did not depend on agencies that operate on commercial terms and cost-recovery requirements were not set for them. 69 Exchange Rate Effective December 31, 2015. 70 “Investment Project Financing Economic Analysis Guidance Note.” OPSPQ, October 9, 2014. 71 Ibid. 38 Thus, the PMU did not systematically collect data that would help monitor financial revenue and expense streams.  A considerable share of the TCP funding went to capacity-building activities that a priori do not allow calculation of financial ratios. 4. Public rationale. Modern economies rely on the development of new technologies for economic growth and prosperity. Kazakhstan’s slowing growth rates since the liquidity crisis of 2007 have demonstrated that the country’s economic model based on commodity industries should be altered by a model based on production of new knowledge and breakthrough ideas and creative application of new knowledge. Yet, while research and creation of knowledge define an opportunity, the market in Kazakhstan fails to capitalize on this opportunity due to difficulties faced by the private sector (including foreign investors) in assessing the quality of R&D projects (assessment of technology and team risks) and conduciveness of the economic and institutional environment to R&D commercialization (market and regulatory risks). Thus, since private returns to R&D commercialization are lower than social returns, the private sector R&D investment in Kazakhstan has been below its economically optimal level, as it is in many other countries. 5. On the other hand, a mere public subsidizing of R&D costs does not solve the above problem; instead of creating right incentives to invest in R&D, heavy public intervention may actually crowd out private investment on the one hand and lead to a moral hazard issue on the researcher’s side 72 on the other hand. Instead, a well-structured intervention was needed in Kazakhstan to address the issue of information asymmetry, lack of commercialization capacity, and institutional barriers. In other words, a public intervention must be aimed to significantly improve scientific performance, demonstrate to all stakeholders of the NIS that research projects selected through transparent competitive process are commercially relevant, and finally develop an institutional environment conducive to R&D commercialization. The TCP was designed to serve this goal. Although examples of venture fund financing as an alternative to public funding can be found worldwide, existing experience in Kazakhstan shows that venture funds and industry are not eager to commit significant funds before the basic concept is demonstrated and a prototype is developed and tested. 6. Bank’s value added. The Bank’s active participation in this public intervention helped leverage the earlier analytical work conducted by the World Bank Group and use institutional knowledge accumulated during decades of work in transition economies akin to Kazakhstan. In particular, establishment of the ISCB was only possible through use of the professional network of the Bank Group team with the international scientific society and the Bank Group’s reputation that allowed attracting of acknowledged international level scientists to serve the GoKZ. The Bank also helped the government prioritize and sequence implementation of the project, communicate the rationale for the intervention across the society, and build adequate implementation capabilities. 72 Interviews with Kazakhstani NIS stakeholders revealed the issue of existence of a cohort of ‘grant-eating companies’. These entities display considerable competences to get grants but are not targeted at full-fledged business development. 39 7. Standing. The analysis is carried out from the national perspective of Kazakhstan. The parties with standing are the following: (a) Kazakh taxpayers, since reimbursement of the Bank’s loan allocated to the TCP is financed from the state budget. Kazakh taxpayers are thus affected in two ways: increased direct tax burden and deadweight loss due to distorted decisions caused by taxes. At the same time, this group benefited from increased innovative production; creation of new, high-value-added product markets in the country; and enhanced capacity for innovative activities. (b) R&D groups and laboratories, research institutes, and universities benefited from direct financial inflows and indirect externalities in the form of an innovation-enabling environment; acquaintance with new procurement, FM, and commercialization practices; and diversification of sources of financing science and technology/R&D activities. (c) The Kazakhstan MoES and subsidiary agencies incurred indirect expenditures in the form of increased workload of the personnel and/or direct expenditures to hire new employees to maintain activities in the framework of the TCP. At the same time, these and other state agencies benefited from streamlined internal approval and project financing procedures that were introduced because of the TCP PMU and the Bank’s representative office in Kazakhstan. (d) Kazakh residents dispatched to commercialization training abroad. (e) Kazakh R&D institutes and individual researchers who benefited from a number of commercialization trainings, workshops, and networking events. (f) Kazakh companies and individuals who adopted technologies and products of TCP subprojects and carried out technological upgrade. (g) Kazakh companies and individuals involved in the design and implementation of the project, who would receive benefits in the form of rents and salaries. 8. Assumptions. The economic analysis will consider costs and benefits of the TCP based on the following assumptions: (a) This analysis is not aimed at assessing the technical outcomes from TCP subprojects or the performance of scientific groups in conducting their research. The data on cost structure of subprojects and contribution of subproject outcomes to overall economic welfare have drawn significantly on beneficiary survey73and consultations with the TCP PMU. 73 Further details in annex 5. 40 (b) Calculation of NPV under the base case will use the cumulative inflation rate74 for each year from 2008 to 2015, as outlined in table 3.2. Table 3.2. Year-to-2015 Cumulative Inflation Rate Year 2008 2009 2010 2011 2012 2013 2014 2015 Rate (%) 80.2 70.5 57.0 48.8 41.3 36.5 32.2 22.6 (c) The time horizon covered by the cost-benefit analysis is the period from formal launch of the project in 2008 until its formal closure at year-end 2015. Limitation of timeline of the analysis to an 8-year period is dictated by the requirement to conduct a retrospective analysis that considers the benefits and costs incurred up until closure of the project. This significantly reduces the benefits of the project, most of which, in the case of innovation support projects, take place with some time lag after the project closure. (d) Benefits calculated in this analysis do not include taxes and goods and services purchased on competitive domestic markets (for example, furniture, cleaning services, office rent) as these are considered transfers. For example, a benefit to the state budget in the form of taxes generated by the TCP PMU and subprojects is a loss to the taxpaying party (that is, the PMU and TCP subprojects). Similarly, revenue received by third-party suppliers as a result of TCP activities (for example, a landlord renting the PMU office) are also considered transfers since they are equal to the opportunity cost of suppliers operating on a competitive market. (e) Other key assumptions made in the estimations of costs and benefits (reported later in this annex) include the share of grant amount considered as economic cost, marginal cost of public funds, the sales estimated as economic benefit, and the economic value of licensing deals. 9. The analysis calculates incremental benefits and costs of the project compared to the ‘without-project’ counterfactual. Based on results of beneficiary survey and in-depth interviews with the TCP beneficiaries and NIS stakeholders conducted by the ICR team, it is assumed that the counterfactual that has the highest probability of realization in the ‘no-project’ case is continuation of the existing science and R&D grant financing by state agencies. In this case, there are significant differences in the type and scale of support of each research group that are summarized below:  The average amount of grant financing is up to KZT 50 million (at 2016 value) per research group.   The average amount of grant financing is up to KZT 30 million (at 2016 value) per proof of concept or prototype development project.  74 The analysis uses the non-food Consumer Price Index. Alternatively, calculations using the Total Consumer Price Index arrived at a similar NPV figure (KZT 310.3 million). 41  Grant financing is carried out through universities or research institutes with no explicit requirement for establishment of a legal entity by a grantee.   The selection process in a counterfactual scenario is cumbersome and may take up to one year.   A certain group of grant programs of comparable to TCP (in terms of grant design and size) in counterfactual scenario require either (a) cofinancing from a private entity (up to 20 percent of total project cost) or (b) are designed to reimburse R&D expenditures in ex post manner. This implies that in the counterfactual case, research groups would either (a) engage in R&D activities tailored to the needs of the cofinancing investor or (b) represent the R&D arm of a large entity. In either case, these are risk-averse R&D activities devoted to incremental innovation at an enterprise with lower economic benefits to the society as a whole.   Selection and commercialization support of research groups are carried out by institutions that existed before TCP launch, and the counterfactual assumes no establishment of the ISCB, IMSC, and TCO and no involvement of international consultants.  10. Therefore, these differences between the TCP and the counterfactual translate into significant differences in cost structure and flows of benefits. In particular, it is assumed that in the counterfactual case, (a) beneficiaries are given innovation grants of smaller size; (b) application and compliance costs to beneficiaries would be higher due to strict reporting and eligible expenditure constraints; and (c) smaller size of grants, less flexible design, and absence of commercialization support would limit results of grant financing to a lower level compared to the TCP. 11. All present values presented in the analysis are in terms of 2016 Kazakhstani tenge, unless noted otherwise. Calculation of Costs and Benefits 12. The major costs and benefits of the TCP can be divided into two broad groups: (a) direct costs associated with project launch, administration, and compliance and benefits associated with increased profit of beneficiaries and (b) side effects (externalities) of the project. These are presented in table 3.3. Table 3.3. List of Impacts of the TCP Costs Group with Component Description Standing Kazakhstan taxpayers Component 1 Direct financial outflows on three rounds of grants to research groups (individuals and Component 1 Direct financial outflows on establishment of the IMSC firms) Component 1 Direct financial outflows on operations of the ISCB Component 2 Two rounds of commercialization grants Component 2 Honoraria of international consultant CRDF Global and IC2 Institute, University of Texas 42 Component 2 Consulting services on analysis and strengthening commercialization legislation Component 2 Direct financial costs of establishing and maintaining TCO: administrative expenses, salaries of TCO employees, training of core commercialization team Component 3 Direct financial outflows related to project implementation and institutional capacity building All Awareness-raising/public relations campaigns, brokerage events, and components workshops All Distortionary cost of taxation components Components 1 Environmental impact of upgrading existing laboratories and and 2 implementation of innovative projects in the scope of the TCP R&D groups and Components 1 Administrative expenses incurred by R&D groups and technological technological firms and 2 firms during application for grants, preparation of business plans and performance reports, and expenses related to compliance with formal requirements MoES, other state All Indirect expenditures in the form of increased workload of their agencies components personnel and/or direct expenditures in the form of hiring new employees to maintain activities in the scope of the TCP Benefits Kazakhstan taxpayers Components 1 Commercialization of research findings and development of innovative and 2 products and services: demonstrated through increased income of beneficiary scientific groups and increased income/cost savings of their customers Kazakhstan taxpayers Components 1 Increased access to innovative goods and services, increased and 2 employment in industries with high value added, and establishment of innovation-enabling environment Research institutes, Components 1 Knowledge production: contributions to knowledge from TCP universities, and and 2 subprojects usually measured through contributions to scientific R&D groups publications and patents/license deals Research institutes, Components 1 Research capacity: building research capacity that benefits future universities, and and 2 research activity. This includes the use of research information to R&D groups improve targeting of future research; individual and group development of research skills and research capacity; and development of the capability to use existing national or international research Research institutes, Components 1 Commercialization capacity: benefits to the NIS from training and universities, and and 2 developing a commercialization team of the TCO R&D groups MoES, other state Components 3 Informing policy and strengthening capacity of public offices: agencies accumulation of institutional knowledge that help take informed decisions by policy makers (including database of technological proposals developed in the scope of the technological audit subcomponent); strengthening linkages between various agencies involved in the project; and benefits stemming from streamlining administrative procedures in the MoES Research institutes, All Increased rents (through enhanced innovative capacity) universities, and components R&D groups Kazakhstani resident Component 1 Increased income in the form of wages in the underemployed R&D researchers market 43 Calculation of Costs 13. General and administrative costs. The current analysis considers three groups of administrative costs incurred by TCP stakeholders: (a) up-front administrative costs during the stage of design, regulatory clearance and restructuring of the project, and costs incurred due to implementation delay during 2008–2011 (state agencies); (b) one-off administrative costs during the selection process (grant program applicants); and (c) recurrent compliance costs incurred by grantees (TCP grantees). Group (a) of administrative costs comes in the form of increased workload and/or recruiting of new employees by state agencies engaged in design and launch of the project, negotiations over the IBRD Loan Agreement, and obtaining clearance from line ministries and the parliament. Available data and experience from similar projects in Kazakhstan suggest that 18,480 person-hours were spent by public offices at this stage (see table 3.4). Applying relevant nominal average wages in the period 2008–2011 and using the values in table 3.2, the present value of the estimated administrative costs of Group (a) amounts to KZT 12.5 million. As these are costs related to delays in launch of the TCP, they are not included in calculation of the ‘no-project’ counterfactual case costs. Table 3.4. Estimated Number of Public Employees Engaged and Estimated Time Spent Number of Employees Time Spent by each Employee Total Person-hours per Year (hours) per Year Spent per Year MoIT 5 400 2,000 MoES 10 400 4,000 MoF 3 80 240 ‘No-project’ 0 0 0 counterfactual 14. Calculations of the one-off administrative costs of applying for grants (Group [a] of administrative costs) and recurrent compliance costs (Group [b]) in the scope of Components 1 and 2 are based on the data in table 3.5. Table 3.5. Applicants Engaged in Selection Process and Estimated Time Spent Number of Number of Application Costs Recurrent Compliance Costs Applications Selected Projects (person-hours) (person-hours) per Year Component 1 785 33 6,120 180,960 Component 2 726 33 5,808 34,320 Counterfactual 1,511 66 14,910 269,100 15. In the first stage of selection in the scope of grant programs of the TCP, applications were sent online and did not require significant investments on the applicants’ side. It is thus assumed that each applicant spent approximately 8 person-hours to apply for a TCP grant. Yet, once they passed the selection stage, grantees incurred larger recurrent compliance costs, which, on average, resulted in costs equivalent to hiring one full-time employee throughout the project lifetime. Considering the duration of the TCP grant programs, relevant nominal monthly salaries, and values in table 3.2 the present value of these two groups of administrative costs amounts to KZT 198.3 million. 16. At the same time, application and compliance costs in the counterfactual scenario are assumed to be higher by 25 percent due to the absence of online application channels and the larger number of participants in the application and reporting procedures. In particular, application for 44 grants would require engagement of university/research institute management (as grant funds are accrued to budgets of universities/research institutes before research groups receive them). 17. Direct financial outflows. Apart from the abovementioned indirect administrative expenses, another significant group of costs are direct financial outflows in the form of research and commercialization grants and consultancy contracts, as in table 3.6. Table 3.6. Direct Financial Outflows % Counted Absolute Value Component/Subcomponent KZT, thousands toward Counted toward Economic Cost Economic Cost Component 1 grants 2,933,697 40 1,173,479 Component 2 grants 1,003,359 40 401,344 IMSC 55,683 40 22,273 ISCB 83,000 70 58,100 CRDF Global contract 176,000 70 123,200 Legal framework analysis 40,000 70 28,000 Counterfactual 747,346 40 298,938 18. Analysis of results of the beneficiary survey and interviews with grantees of Components 1 and 2 indicate that not all of these sums can be counted as economic costs. In particular, in the case of research and commercialization grants, about 40 percent75 of the grant funds was spent on purchase of equipment and materials from abroad (net of customs duties and taxes) and thus, can be counted as economic costs from Kazakhstan’s standpoint. The remaining part of the grant funds was disbursed domestically, including salaries of researchers and administrative staff. The number of those hired by research groups from abroad is negligible and can be neglected from the analysis. Thus, this analysis takes 40 percent of grant funds as economic costs, while the remaining part of the grant funds is considered either as transfers or even rents (see the Calculation of Benefits section below). 19. In all, 70 percent of the remaining direct financial outflow items (establishment of the IMSC, maintenance of the ISCB, the CRDF Global contract, and consultancy services on legal framework analysis) are considered costs due to the import of equipment and consultancy services. The remaining part of these outflows is considered as transfers in the form of customs duties, taxes, and travel and accommodation expenditures in local competitive markets. 20. Considering the above data and applying the values in table 3.2, the present value of economic costs related to direct financial outflows amounts to KZT 2.4 billion. Direct financial outflows on operations of the TCO and PMU (Component 3) are considered as transfers and are excluded from calculations of economic costs. In the counterfactual scenario, the present value of 75 Interviews with beneficiaries and consultations with the PMU revealed that, on average, about 70 percent of grant amount is spent on purchase of equipment and expendable supplies and materials for research. Beneficiary surveys revealed that about 57 percent of equipment is imported (which equals approximately 40 percent of grant amount). Considering the uniqueness of the imported equipment (that is, equipment tailored to research needs of the group and that neither have analogues produced domestically or distributing companies owned by Kazakhstan residents), funds spent on import of equipment is considered economic costs. 45 this group of costs amounts to KZT 1.06 billion, given the smaller size of grants and absence of spending on establishing and maintaining the IMSC, ISCB and TCO. 21. Negative environmental and social externalities. Research and innovation activities may also cause negative impact on the environment due to possible emergencies at R&D sites, wastes generated by laboratories, field trials of new products and substances, and so on. At the same time, regular inspections of fulfillment of environmental and social safeguards did not reveal significant negative externalities during project implementation. The amount of pollution and waste production was negligible and, when required, the TCP beneficiaries procured waste disposal services and obtained necessary emission certificates from the authorities. Thus, environmental impact of the project is considered negligible and is not quantified in this analysis. 22. A considerable negative externality of the project is the deadweight loss resulting from distorted decisions of economic agents due to collection of taxes to finance the project. Existing literature on distortionary impact of taxes is extensive and estimates of this negative externality range from US$0.025 to US$0.30 per dollar of public funds raised through taxation.76 The current analysis takes the median of this range at about 16 tiyns per tenge of tax revenue. Given these assumptions, the present value of distortionary impact of taxes amounts to KZT 939.9 million. This is compared to the counterfactual case, where the present value of such ‘cost of public funding’ amounts to KZT 870.4 million. Calculation of Benefits 23. Direct benefits. The most obvious and measurable benefit of the TCP is the results of commercialization of research findings and development of innovative products and services in the period before the project closure. The sales of research groups as of year-end 2015 amounted to KZT 908.7 million. Analysis of beneficiary survey results indicate that about 94 percent of these sales would not have been achieved in the ‘without-project’ counterfactual scenario. The current analysis considers 100 percent of this sum as economic benefit for following two reasons: (a) As TCP subprojects were aimed to produce/render innovative goods/services, sales of these research groups were made in either completely new markets or resulted in substitution of import and/or export sales. Thus, sales are considered as benefits from Kazakhstan standing. (b) Although sales represent gross revenue rather than net income of a research group, the cost of production is financed from grant funds and is already covered in calculation of economic costs of the TCP. Thus, netting revenue of research groups from their production costs will lead to double counting of costs. 76 Feldstein, Martin. 1999. “Tax Avoidance and the Deadweight Loss of the Income Tax.” Review of Economics and Statistics 81 (4): 674. Ballard, Charles L., John B. Shoven, and John Whalley. “The Welfare Cost of Distortions in the United States Tax System: A General Equilibrium Approach.” Working Paper 1043, National Bureau of Economic Research. 46 24. Given the values in table 3.2, this implies that the present value of benefits from commercialization of research activities amounts to KZT 1.16 billion compared to KZT 69.5 million in the counterfactual scenario. At the same time, it must be acknowledged that this figure does not demonstrate full benefits from increased income of beneficiary scientific groups and increased income/cost savings of their customers in the period beyond the TCP closure. 25. As reflected in the Results Framework indicators, a direct benefit of the TCP was the number of licensing deals where TCP beneficiaries act as the licensor and the number of approved PCT applications. There were 8 such licensing deals and 5 patent applications submitted in the scope of the PCT as well as 11 patent applications submitted to foreign/regional patenting offices. As this analysis is conducted from the ex post perspective, patent applications cannot be counted toward benefits of the project as they had not resulted in obtaining of actual patents as of 2016. Estimation of actual economic value of the licensing deals is hindered by the fact that these are commercial information. However, this value is approximated by the median economic value of the nearest equivalent of a licensing deal, patents in the European Union, estimated by several recent studies to be around €200,000.77 Considering the exchange rate of KZT 390.82 per euro and the values in table 3.2, the present value of this direct benefit is KZT 766.6 million. The number of licensing deals in the counterfactual scenario is taken to be equal to three, as evidenced by results of beneficiary survey and available statistical data (only three comparable licensing deals were concluded in Kazakhstan outside of the TCP in the period 2009–2014). 26. Indirect benefits. Indirect benefits of the TCP stem from the value added from capacity- building trainings, brokerage events, conferences, and workshops carried out by the TCO and devoted to technology commercialization training of the scientific community across the country. In particular, during 2013–2015, 881 individuals were issued certificates as a result of 27 workshops devoted to technology commercialization training. Typically, the returns to training are measured with respect to increase in wages and the probability of remaining in employment (in contrast to the ‘no-training’ counterfactual) though there are a range of other benefits typically listed under the category of the ‘wider benefits of training’, which relate to factors such as the individual’s well-being, the intrinsic rewards of learning, and so on. Existing literature suggests that the incremental benefit of training depends on the ‘absorptive capacity’ of the trainee and the industry and is generally nonlinear in time78—varying from 1 percent to 22 percent wage premium and up to 14 percent increase in probability of employment. The analysis also incorporates training of research group members as a benefit. For the analysis, it is assumed that an incremental benefit to 881 trainees and 328 members of research groups was a 2 percent wage premium from 2012 to year-end 2015. Considering the Statistics Agency’s data on the average monthly salaries in the R&D sector during 2012–2015 and the values in table 3.2, the present value of this group of benefits amounts to KZT 89.1 million. 27. Other unquantified principal benefits of the project come from development of innovative research and production in Kazakhstan through the following vehicles: 77 European Commission. 2006. “Study on Evaluating the Knowledge Economy, What Are Patents Actually Worth? The Value of Patents for Today’s Economy and Society.” PatVal EU Project (European Commission 2005), European Investment Bank (2013). 78 Cambridge Econometrics. 2013. “Review of the Economic Benefits of Training and Qualifications, as Shown by Research Based on Cross Sectional and Administrative Data.” Research Paper 105. 47 (a) Increased knowledge production. Contributions to knowledge from TCP subprojects usually measured through contributions to scientific publications, and accumulation of human capacity. (b) Increased research capacity. Building research capacity that benefits future research activity. This includes the use of research information to improve targeting of future research, individual and group development of research skills and research capacity, and development of the capability to use existing national or international research. (c) Establishment of strong industry-R&D linkages and developing a technology commercialization cycle. (d) Building up commercialization capacity. Benefits to the NIS from training and developing the commercialization team of the TCO. (e) Building up institutional capacity to support commercially viable R&D activities. Accumulation of institutional knowledge that help take informed decisions by policy makers (including a database of technological proposals developed in frame of the technological audit subcomponent); strengthening of linkages between various agencies involved in the project; and benefits stemming from streamlining administrative procedures in the MoES. (f) Increasing attractiveness of scientific career among the younger generation, thus contributing to stable supply of human capital to the NIS of Kazakhstan. (g) Diversifying Kazakhstani economy and increasing share in gross domestic product with high value added. (h) Strengthening institutional capacity of the NATD and innovations clusters in Astana and Almaty, stemming from lessons learned and adoption of selection, financing, and monitoring mechanisms used in the TCP. Calculation of NPV of the Project 28. Table 3.7 presents the NPV of retrospective costs and benefits of the TCP from the perspective of Kazakhstan from 2008 to year-end 2015. All benefits and costs have been taken to their present value using the values in table 3.2. According to the results shown in the table, NPV of the TCP and associated activities, compared with the counterfactual, is KZT 275.33 million. Table 3.7. Present Values of Incremental Costs and Benefits of the Project Compared to the ‘No-project’ Counterfactual Costs Value (2016, Group Description KZT, millions) Direct Financial Outflows Kazakhstan taxpayers Funding grant programs under Components 1 and 2 1,026.32 Establishment and operation of the ISCB and IMSC, and consultancy services in the scope of Component 2 322.40 48 Administrative Costs State agencies Value of administrative cost incurred by public agencies at design, 12.50 approval, and restructuring stage of the TCP (2008–2011) Applicants and project Grant application and compliance costs (Components 1 and 2) −49.6 beneficiaries Negative Externalities Kazakhstan taxpayers Negative impact on environment (Components 1 and 2) Not counted Distortionary impact of taxes (all components) 69.6 Present value of total costs 1,381.16 Benefits Value (2016, Group Description KZT, millions) Direct benefits TCP beneficiaries Results of R&D commercialization in the form of sales/service 1,088.22 contracts (Components 1 and 2) TCP beneficiaries Results of R&D commercialization in the form of licensing deals and patent applications (Components 1 and 2) 479.14 TCP beneficiaries Benefits in the form of rents and training of TCP beneficiaries 41.47 Research institutes and Value added of capacity-building activities individuals outside the 47.66 TCP Positive externalities Kazakhstan taxpayers Growth of economy from R&D investment multiplier effect Not counted Total benefits 1,656.49 Present value of net benefits 275.33 Sensitivity Analysis 29. Although the base case calculations show more than KZT 275 million of net benefit, the analysis is based on a series of estimates and assumptions that result in a certain degree of uncertainty. The analysis uses partial sensitivity analysis and break-even analysis to determine how this uncertainty affects the bottom-line results. 30. Break-even analysis. To test the viability of the project against these arguments, the analysis recalculates the present values of costs related to direct financial outflows. The baseline case assumes that 40 percent of grant funds can be considered as economic cost (which equals average share of spending on import of equipment and expendables by research groups), with the rest being considered pure transfers between economic agents in the country (as the standing in the analysis is given to residents of Kazakhstan). Table 3.8 demonstrates alternative values for this assumption and resulting changes in NPV, including the break-even results with zero net benefits. The break-even point is reached if the share of grant amount calculated as cost is increased by 11 percentage points, implying high sensitivity of the NPV to assumptions used in the analysis. Table 3.8. Sensitivity of NPV to Alternative Estimates of Direct Costs Estimated share of direct costs (as % of grant) 40 51 60 Net benefits (KZT, millions) 275 0 -237.8 49 31. Partial sensitivity analysis. Summary of sales data in table 3.9 shows that the mean of sales data is higher than the 90th percentile of the sales sums and the standard deviation significantly exceeds the mean due to presence of an outlier that significantly affects total sales figure. When this outlier is taken out of the calculations of the net benefits, the NPV drops to KZT- 450.8 million. Table 3.9. Summary of Sales Data Number of Firms Standard that Reached Total Sales 90 Percentile Mean Median Deviation Sales KZT 908.7 KZT 18.57 KZT 22.7 KZT 1.6 KZT 112.9 40 million million million million million 32. Sensitivity analysis shows that the TCP net benefits are highly sensitive to estimated values of costs and benefits of the project. 33. In summary, the retrospective cost-benefit analysis of the TCP shows positive net benefits amounting to KZT 275.33 million. At the same time, it should be noted that because of the retrospectiveness of the analysis, it does not capture (a) majority of economic benefits that will be realized several years after the project closure (because of time lags in adopting innovative technologies developed in the scope of the TCP) and (b) significant positive externalities generated by the project (links between science and industry, increased capacity of state agencies and research groups) that could not be captured by imperfect or partial result indicators (for example, scientific publications, partnerships between researchers and enterprise sector). 34. Although the economic analysis section in the TCP PAD mentions the well-known cross- country comparative study conducted by Lederman and Maloney (2004),79 which shows that the social return rate of R&D in a middle-income country such as Kazakhstan approaches 60 percent on average, this analysis did not quantify and incorporate this benefit in calculation of NPV. This is mainly due to the time lag with which R&D expenditures affect the economy and which should be counted for in the cost-benefit analysis. Goto and Suzuki (1989)80 estimated the private and social return rates in various industrial sectors in Japan and found that not only is there a lag in the impact of R&D activities but that the lags also vary depending on the industry. Specifically, they showed that the impact takes an average of two years in the case of electrical machinery, electronic and communication equipment parts, and mechanical machines. This is a substantially different period than that for drugs and medicines where the lag period exceeds five years. The authors suggest that this heterogeneity is explained by technological differences between sectors or by different employee skill levels (absorption capacity) and competition. Following a conservative approach of estimation, the current analysis does not incorporate the social return of R&D. 79 Maloney, W., and D. Lederman. 2004. “R&D and Development.” Econometric Society 2004 Latin American Meetings 216, Econometric Society. 80 Goto, A., and K. Suzuki. 1989. “R&D Capital Rate of Return on R&D Investment and Spillover of R&D in Japanese Manufacturing Industries.” Review of Economics and Statistics 71 (4): 555–64. 50 Available Efficiency Comparison for Component 2 35. While there is no comparator program of the same scope and similar design (using the same grantee selection, financing, and capacity mechanisms) that could be used to benchmark the efficiency indicators for the whole TCP, the Bortnik Fund in the Russian Federation could be used as a comparator for Component 2 of the TCP.81 ‘Sales as percentage of grant’ is the most relevant parameter along which to compare the two programs. Table 3.10. Comparison of Programs Bortnik Fund 2010–201182 TCP 2014 (Component 2) (Stage 1 of Start program) Number of grantees 33 1,937 Volume of grant financing (LCU) 1,460,000 1,632,941 % of grantees that reached sales 48 n.a. Volume of sales (LCU) 813,964.55 767,407.00 Sales per grantee (LCU) 24,665.59 396.18 Sales per grantee (US$) 139.20 13.21 Sales as % of grant 56 47 Average grant size (LCU) 44,242.4 843.0 Average grant size (US$) 249.7 28.1 Note: LCU = Local Currency Unit. 36. The analysis shows sales as percentage of grant at 56 percent in the TCP while a similar program in Russia (Bortnik Fund) shows a rate of 47 percent, indicating a more efficient use of funds under the TCP. 81 The two programs are similar in terms and purpose; in the Bortnik Fund, grantees are given RUB 1 million (about US$35,000 in 2010–2011) and 1 year to reach sales and it requires establishment of a legal entity or finances R&D of existing young firms (age not more than 2 years and sales less than RUB 1 million). 82 More recent data not available. 51 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Natasha Kapil Private Sector Development Specialist ECSPF Team Leader Hiran Herat Operations Consultant ECSPF Team Member Elvira Mami Consultant ECSPF Team Member Naushad A. Khan Procurement Advisor ECSPS Team Member Yuling Zhou Senior Procurement Specialist ECSPF Team Member Norpulat Daniyarov Financial Management Specialist ECSPS Team Member Ruxandra Maria Floroiu Environmental Engineer ECSSD Team Member Danielle Malek Counsel LEGEM Team Member Nicholay Chistyakov Senior Finance Officer LOAFC Team Member Fatiha Amar Program Assistant ECSPF Team Member Anara Akhmetova Program Assistant ECCKZ Team Member Gulmira Akshatyrova Program Assistant ECCKZ Team Member Alfred Jay Watkins Program Coordinator HDNED Team Member Natalia Agapitova Program Officer WBIGP Team Member Aslan Sarinzhipov Operations Officer ECSSD Team Member Claudio Wernli Centers of Excellence Specialist ECSPF Team Member Phillip Griffiths Science Advisory Board Specialist ECSPF Team Member P.T. Manoharan Instrumentation Specialist ECSPF Team Member Alistair Brett Commercialization Specialist ECSPF Team Member Gulmira McHale Operations Officer ETWWP Team Member Mikhail Bunchuk Information Officer ISGEG Team Member Gulmira Kalauova Consultant ECSPF Team Member Supervision/ICR Karen Grigorian Sr. Private Sector Development Specialist GTC03 Team Leader Yeraly Beksultan Private Sector Development Specialist GTC10 Team Leader Agnes I. Kiss Regional Environmental and Safeguards Advisor OPSPF Safeguards Ruxandra Maria Floroiu Lead Environmental Specialist GEN03 Safeguards Rustam Arstanov Environmental Specialist GEN03 Safeguards Anara Akhmetova Procurement Assistant ECCKZ Team Member Gulmira Akshatyrova Program Assistant ECCKZ Team Member Galina Alagardova Sr Financial Management Specialist GGO21 Financial Management Fatiha Amar Program Assistant GWA05 Team Member Nicholay Chistyakov Senior Finance Officer WFALN Team Member Norpulat Daniyarov Sr Financial Management Specialist GGODR Financial Management Hiran Herat Consultant GED06 Team Member Aliya Kim Financial Management Specialist GGO21 Financial Management Nurbek Kurmanaliev Procurement Specialist GGO03 Procurement John Otieno Ogallo Sr Financial Management Specialist OPSPF Financial Management Danielle Malek Roosa Senior Counsel LEGES Legal Yuling Zhou Lead Procurement Specialist GGO08 Procurement Asta Bareisaite Strategy and Operations Officer GTCD2 ICR Team Leader Luis Rubalcaba Sr. Private Sector Development Specialist GTC04 ICR Team Member Asset Bizhan Consultant GTC10 ICR Team Member 52 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle US$, thousands (including travel No. of Staff Weeks and consultant costs) Lending FY05 84.09 FY06 69 305.90 FY07 13.8 111.75 FY08 18.9 164.81 Total: 101.7 666.55 Supervision/ICR FY08 8.96 FY09 19.3 129.10 FY10 26.8 127.93 FY11 30.5 131.84 FY12 23.7 141.79 FY13 11.9 198.16 FY14 6.9 261.42 FY15 26.5 82.78 FY16 18 79.31 Total: 163.6 1,161.29 53 Annex 5. Beneficiary Survey Results Component 2 Evaluation Study 1. Overview. This end-of-program evaluation was designed to assess outcomes that TCO managers and technology entrepreneurs have accomplished during the course of the program. The beneficiary survey was conducted by CRDF Global in October–November 2015 to assess participants’ satisfaction with the program, analyze major knowledge gains, highlight results achieved, and reflect on impacts made on the Component 2 program participants. Participants were also requested to recommend future programmatic improvements. This survey was sent to 32 program participants, of whom 24 responded, which constitutes a high 75 percent response rate. 2. TCO assistance in business development. Participants expressed very high satisfaction with the program. All the respondents (100 percent) highlighted that building the relationship with their designated manager/experts and receiving consultations from them were extremely valuable or valuable for their business development. Almost the same number of respondents, 96 percent (23), rated similarly their ongoing communication and the opportunities to discuss business models and suggestions regarding business models through e-mail, phone/Skype, or in person. About the same number of respondents, 95 percent (22), found the input that they received on validating their business rationale and work plan extremely valuable and/or valuable and 92 percent (22) on obtaining consultations on how to put together the implementation plan. 3. Accomplishments achieved by project teams and attributed to the TCO grant program. Reflecting on the major accomplishments achieved by project teams, 24 respondents highlighted 42 different accomplishments that they directly attributed to the TCO grant program. Almost half of the respondents (11 out of 24) mentioned building a prototype as the major accomplishment. Networking and signing contracts with new contacts established as a result of the program was mentioned as a significant achievement (6). Five respondents attributed an increased professional knowledge and more effective work of their teams to the TCO grant program, while three participants identified improved production conditions as their team accomplishment attributed to the TCO. Likewise, three participants credited the TCO grant program for commercialization of their projects. Some respondents highlighted increased product recognition on the market and some others, the development of their competences as a significant accomplishment. Among other accomplishments attributed to the TCO grant program, respondents mentioned launching a new company, receiving a new grant, improving the product, as well as applying for patents. 4. The TCO grant program’s impact on participants’ professional advancement and their projects. The frequency analysis allowed identifying major trends in how participants saw major impacts that the grant program made on their professional knowledge and practice and on their technology projects. Nine out of 24 respondents reported that the advancement of their professional competences, including improvement of project management skills, knowledge of the technological processes, and building marketing skills were the major impacts of the TCO grant program. Four respondents stated that the grant project made an impact on their business development experiences, while three highlighted that the major impact was commercialization of their products, as well as increased knowledge that they gained about commercialization processes as a whole. Likewise, three participants reported a significant impact of the networking 54 opportunities on their professional growth. Other respondents indicated that owing to the available resources and improved infrastructure, they could implement the project. Other respondents highlighted attracting young professionals and researchers and improving quality of production processes as a program impact. 5. Main shortcomings of the TCO grant program. Project participants were asked to outline three major shortcomings of the TCO grant program. First and foremost, they mentioned that a short life cycle of the grant was challenging for their projects’ implementation (11 out of 24). Furthermore, 10 participants identified funding procedures as the major program shortcoming. This included delays in funding transfers, low amount of the grant, and the absence of online banking services. Furthermore, four respondents considered extensive reporting procedures as a shortcoming and mentioned that they did not have enough support from the TCO staff in this process. Some other shortcomings included unclear requirements for procurement as well as some challenges related to administering grant reporting. 6. Outcome. The findings show that the program facilitated several licensing deals and helped start-up companies generate substantial income over the program life cycle. Anecdotal evidence from survey responses concludes that the TCO supported by CRDF Global in Astana, Kazakhstan, was of great value to the grant program participants, who praised the high-quality work and professionalism of the TCO staff and its consultants. Owing to the networking opportunities organized by the TCO and CRDF Global, grantees established, overall, 352 new linkages and contacts with whom they could possible build future collaboration. Of 352 linkages built, respondents highlighted 46 new strategic partners in several countries of the Eurasia region, including one in Belarus, three in Kyrgyz Republic, three in Russia, and one in Tajikistan. Two new business partnerships were developed in Germany and in the United States. Because of the TCO grant project, 36 new business partnerships were established with companies, firms, and universities in Kazakhstan. Survey results also prove that the TCO program participants had a significant increase in revenue generated and the amount of funds raised during the course of the program. ICR Beneficiary Survey 7. In March 2016, end-of-project beneficiary survey questionnaires were distributed to 65 TCP grantees and the IMSC to measure the TCP’s success in meeting beneficiary’s needs, to collect data for the Economic and Financial Project Analysis (see annex 3). Participation was voluntary, and 32 beneficiaries responded to the survey. The survey questionnaire aimed at revealing the impact of the TCP on research and commercialization skills of research groups, comparing TCP efficiency of subcomponents in increasing sales and overall perception of the TCP by beneficiaries in the context of achieving scientific and commercialization goals. The survey also included questions related to general metrics of subprojects, indirect costs incurred by beneficiaries from participation in the TCP. 8. All of the respondents assessed the TCP as having a highly positive impact. In particular, when asked to rank the contribution of the TCP to the achievement of scientific and commercial goals on a scale from 1 (no contribution) to 6 (exceeded expectation), 88 percent of groups ranked it 4 and higher. When queried about which of the TCP support mechanisms were most valuable, 60 percent of respondents indicated grant funds, 20 percent mentioned commercialization 55 infrastructure created in the frame of the TCP, and the remaining share of respondents highlighted other mechanisms of support. 9. When queried about the main achievements of a research group owing to the TCP, 53 percent of respondents noted that the TCP enabled them to develop an industrial prototype. Around 28 percent mentioned purchase of unique equipment, and 34 percent of respondents also mentioned establishment of working relationships with existing and/or potential customers. 10. Overall, survey results demonstrate high satisfaction of the TCP beneficiaries with the support mechanisms offered in the frame of the project. 56 Annex 6. Stakeholder Workshop Report and Results From Midterm Report High-level Workshop - “Innovation Based Economy in Kazakhstan: Emerging Success of the Technology Commercialization Project” 1. Toward the end of the mission, the Bank team, in close collaboration with the government counterparts, organized a high-level workshop on November 11, 2013. The objective of this high-level workshop was two-fold. First, to present emerging success of the TCP to a broad set of stakeholders (relevant government agencies, the parliament, the press, representative of the enterprise sector and the donor community) and raise awareness of the project. Second, relying on the current project and emerging success stories within, to discuss the way forward for Kazakhstan with regard to its transition to a knowledge-based economy. 2. The event brought together key stakeholders of the NIS, including government authorities, business associations, academia, and members of parliament. The workshop consisted of two main sessions—the first in which stakeholders of the TCP reported on their achievements and the second one focused on how to move forward on the basis of those achievements. 3. The morning session of the event was chaired by Yerbol Orynbaev, Deputy Prime Minister of Kazakhstan, who presented early successes under the Technology Commercialization Project (TCP) implemented by the GoKZ with financial and advisory support from the Bank, and discussed the ways of unleashing productive innovation in Kazakhstan. 4. In her opening remarks, the Bank Country Manager for Kazakhstan, Sebnem Akkaya, mentioned that, “One cannot accomplish a transition to innovation-based economy overnight. And there is no single model of an innovation based development—each country needs to discover its own model, yet the lesson of countries (like Norway, South Korea, Finland, and Israel) which accomplished such a transition shows that a country must have successful competitively selected research teams, technology start-ups, and spin-offs in place.” 5. In the first session, the two most successful groups (selected by the MoES) of senior scientists and junior researchers presented results of commercialization of their research, along with similar presentations of the TCO and IMSC. Drawing on these specific outcomes, the second session discussed possible actions and instruments to leverage, scale up, and diffuse emerging successes. The following two projects were showcased:  KazTrek Technology LLP presented the ‘Creation of Filters and Metal Nanostructures based on Track Membranes’ project, which has developed a water filtration device to filter out physical materials, as well as bacteria and even small- size viruses. The group already has contractual arrangements with commercial organizations for distribution of the filters in several countries in Asia and Russia.  There is a genetic difference in individuals’ drug response based upon their heritage. The ‘Real-time Pharmacogenetic Test Development for Determining Individual Drug Dosage’ project was presented by the General Genetics LLP. The team of 57 scientists has successfully tested two drugs and already has customers through local hospitals. Plans are on to test additional drugs to determine the appropriate dosage for patients on an individual basis. 6. Drawing on these specific outcomes, the second session discussed the issue of transforming emerging successes to accepted role models to be followed and emulated by both the enterprise sector of Kazakhstan and its research community. This session on the way forward was a sounding board with all key stakeholders of Kazakhstan on the ‘how to’ of the transition of their country to an innovation-based economy. From this perspective, engagement of key stakeholders and the high level of presenters ascertained that the event will be an important part of the ongoing dialogue on economic growth in the country.  7. Global experience shows that for the resource-based economies the so-called technology consortia have become a key instrument. The features of such consortia include (a) exposure of domestic agents to global performance standards and practical participation of international partners; (b) long-term, multi-year, and strategic in nature; and (c) user driven, suggesting a demand and real focus of the effort. 8. In particular, the discussion focused on establishing the institution of collaboration to overcome fragmentation of Kazakhstan’s innovation system to leverage considerable talent accumulated in the country. It should be a mechanism to help discover the best scientists in the country, bring them together into a new organization, and then fund these teams. 9. Kazakhstan has made significant investments into scientific infrastructure by now, establishing 20 modern labs. Taking this fact into account, the TCP proposes establishing the IMSC, based on one of the existing laboratories, to make it a center of excellence that would serve as a world-class R&D facility setting a benchmark for Kazakhstan laboratories and promoting international best practices. 10. From the long-term perspective, the project begins to establish a commercialization cycle, which is a system of institutions and capabilities to deal with early- and later-stage venture capital, various types of grant programs to allow a technology entrepreneur to develop a project, and so on. This is a strategic long-term challenge which, as other countries’ experience suggests, demands systematic efforts and targeted interventions and support programs. 11. The TCP team leader Karen Grigorian summarized the event highlighting that, “presentations from the research groups of the TCP project show clearly that there are success stories in Kazakhstan, but we need more of such stories and, more importantly, such success stories shall become role models for everyone, and particularly for young people to follow. New role models start to create new history.” 58 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 1. “Technology Commercialization Project Implementation Results Report” was received from the MoES in March 2015. Below is the full text of the report (excluding the title page, table of content, list of abbreviations83 and attachment184) SECTION 1. PROJECT DESCRIPTION 2. The Technology Commercialization Project was implemented under Loan Agreement No. 48840KZ signed on February 2, 2008 and ratified by Law of the Republic of Kazakhstan No. 86- IV dated November 19, 2008. 3. According to the Loan Agreement the total Project amount was US$75 million (equal to US$64.3 million due to repeated episodes of depreciation of the national currency (KZT)), including the external loan funds amounting to US$13.4 million. 4. The delay in Project implementation as compared to the project implementation schedule was about three years due to the reasons connected with some organizational and legal problems arisen at the initial stage. 5. For achievement of the specified Project goals and objectives the Loan Closing Date (31.12.2012) under the Project was extended to December 31, 2015 in accordance with Minutes of Meeting with the Deputy Prime-Minister of the Republic of Kazakhstan Ye.T. Orynbayev for discussion of the Technology Commercialization Project No. 2034/и-408 dated 14.05.2012, and Letter of the MoF of the Republic of Kazakhstan to the WB Ref. No.ДГЗ-1-19987-11057 dated 15.06.2012 and Letter of the World Bank to the MoF of the Republic of Kazakhstan Ref. No.12/294 dated 27.07.2012. 6. The Project main goal is to demonstrate a substantially improved science-based approach and strengthening of commercial relevance of the research performed by scientists in various fields, for further commercialization in the market; to ensure linking the Kazakhstan’s science to the domestic economy sector, the international academic community, and global markets through establishment of a pilot technology commercialization center. 7. According to the National Priorities the Project long-term goal is to stimulate reformation of the existing R&D base. 8. If the Project is successful, in future five to ten years Kazakhstan will develop a new, competitive, interdisciplinary, problem-oriented model of financing highly appreciated by international experts and economically relevant to the high level of the academic research activities. The science in Kazakhstan will be more closely connected with domestic and foreign industrial enterprises. The scientific institutes supported under the Project will get up to a third of their income from implementation of contracts with private enterprises or other competitive sources of financing. Research and development activities will be performed in close cooperation 83 Merged with ICR list of abbreviations 84 The attachment provided a list of Component 1 projects. A full list of TCP projects (both Component 1 and Component 2 is provided in Annex 10) 59 with the international academic community and on an interdisciplinary, problem-oriented basis. And, possibly, the most important thing is that the new generation of young and talented scientists will conduct their research activities in Kazakhstan at the state-of-the-art research laboratories supported within the framework of the project under the guidance and supervision of the distinguished Kazakhstani and international scientists. 9. The Key Project Success Indicators are: (a) The number of research grants for senior scientists and junior researchers; (b) The number of applicants involved in the above grants; (c) The number of visiting professors; (d) The number of joint research projects with involvement of the above listed groups and international research partners; (e) Established International Center for Material Science; (f) The number of scientists being serviced by the above Center; (g) The number of the completed technological developments; (h) Established Technology Commercialization Office; (i) The number of revised laws, rules, regulations, and recommendations of Kazakhstan which hinder introduction of new technologies and progress in R&D; (j) The number of grants allocated for commercialization of technologies; (k) The number of licenses sold through the Technology Commercialization Center. 10. The project consists of three components: (a) Component 1. Strengthening the Scientific Base (US$36.3 million) is aimed at identification and assistance in development of leading scientists and talented young researchers in order to perform scientific research and development in accordance with the international standards through: (a) the establishment of an International Science and Commercialization Board and a scientific internationally-peer reviewed journal; (b) a Senior Scientist Group (SSG) grant program and a Junior Researcher Group (JRG) grant program; (c) the establishment of an International Center for Material Science (IMSC). 60 (b) Component 2. Linking Science to Business (US$21.8 million) means provision of technical assistance for the development of the expertise required to link science more closely to domestic and global markets through: (i) the establishment of a Technology Commercialization Office (TCO) to provide a range of technology commercialization services, including the development of a technology commercialization strategy, and the development of targeted training programs in foreign languages, business planning and collaboration; (ii) conducting a technology audit to inform strategy-developers and scientists about innovation assets and to compile an expert database for the marketing of technologies; (iii) an analysis of legal and regulatory provisions, structures and incentives related to technology commercialization, industrial innovation, venture capital and financing, and intellectual property; (iv) conducting a market assessment for the development and administration of a range of grant programmes. (c) Component 3. (US$6.2 million) Project Management includes elements of support in the establishment of the Project Management Team for Project coordination, implementation, and monitoring. The project management measures are set forth in the Project Operational Manual. SECTION 2. PROJECT IMPLEMENTATION Component 1 – Strengthening the Scientific Base 11. Due to ageing of the scientific personnel and obsolescence of the equipment, the science and technology base of Kazakhstan depreciates rapidly and, in most cases, derives income from the earlier scientific achievements rather than new scientific discoveries. Measures should be immediately taken to prevent decline, to restore the science and technology system of Kazakhstan, and to switch it to a more sustainable and modern path of development so that it could serve again as a resource for economic growth and enhancement of the state competitive ability. 12. This component provides for financing: (i) research programs of senior scientists and junior researchers (SSG and JRG) who will conduct world-class research; and (ii) International Center for Material Science equipped with new and state-of-the-art laboratory equipment. 13. As a part of the implementation of this component an International Science and Commercialization Board (ISCB) was established. It consists of five eminent scientists from the countries such as the USA, Germany, Russia, Saudi Arabia, and two technology commercialization experts from the innovation developed USA regions - Silicon Valley and Michigan. 61 SSG and JRG Grants 14. Three tenders were held for selection of scientific projects for grant funding. Over three years of the grant programme more than 700 applications were submitted. 33 sub-projects were approved for financing by ISCB members. 15. The research range includes such fields as oil transportation, reservoir recovery, and petrochemistry; construction materials and production waste disposal; nanotechnologies; agriculture, genetics and cytology; electronics and instrument engineering; radioecology and radiochemistry; alternative energy; medicine and pharmaceutics. 16. The grant funding volume amounts to US$600,000 for junior researcher groups and to US$1,500,000 for senior scientist groups. The funding is sufficient for the entire process from the technology development concept at a laboratory and up to the entry into market in the form of finished goods or services. 17. To ensure effective use of the grant funds the grant recipients-legal entities were accredited as entities engaged in scientific and technological activities thus ensuring preferential tax treatment. 18. The ISCB performed regular monitoring of the grant recipients by visiting the research sites twice a year. 19. Based on the results of monitoring of selected groups ISCB members reported a notable progress in implementation of subprojects. 20. For example, KazTrackTechnology Company established by a group of research associates from the Institute of Nuclear Physics has come to the level of small batch production of membrane filters and established partnership with a commercial company. 21. General Genetics Company established by scientists from the National Center for Biotechnologies provided services for determination of an optimum individual dosage of medicinal products (Warfarin and Plavix) for the National Center for Cardiology on a commercial basis. 22. AimLab Company established by employees of the Center for Physical and Chemical Testing Methods of the Al-Farabi Kazakh National University has launched pilot production of carbon black with the annual production volume of 200 tons. The Company already has customers interested in purchase of the finished products. 23. On the basis of the completed monitoring results written recommendations of ISCB members were given for further project commercialization strategy (English and Russian versions). According to the review of the component 1 projects, 24 projects have reached the market with the first sales of the goods/provision of services in the amount of KZT 154.7 million. 77 partnerships have been established with various commercial enterprises. In total, more than 50 articles were published in foreign journals with impact-factor and 145 articles were published in the national scientific journals and conferences. 62 International Center for Material Science 24. As a part of the project implementation an International Center for Material Science (ICMS) was established. For the establishment of the ICMS a tender was held among scientific laboratories. Out of 9 tender applications the ISCB members selected Karaganda State Technical University. The grant amount is KZT 294,954.1 thousand. 25. The ICMS activity is aimed at:  The establishment of a state-of-the-art model of laboratory center;  The development of services in the scientific area;  The development of a web-portal in the area of material science, where participants may share free hours of work at laboratory equipment, which will ensure access for Kazakhstani scientists and researchers to the equipment and maximum load thereof;  The establishment of a scientific journal to organize discussions on the relevant scientific problems, creation of a platform for exchange of opinions, expertise and ideas by scientists of Kazakhstan and the international community. 26. The PMT has promptly resolved the problem of transfer of knowledge acquired by the Committee of Science in 2008 for the Project funds for the International Center for Material Science. The building had been long in the books of the Committee of Science and was not suitable for using for the ICMS project purposes. In 2012, the building was transferred to the Institute for Combustion Problems of the Republican State Enterprise, Ministry of Science and Education of the RK. 27. The main result of the ICMS activities was establishment of a model laboratory center with specialization in Mining Metallurgy and Materials, its own web-portal, site, popular science journal, and internet community for a scientific group exchange with scientific publications, news, job opportunities, and events, namely:  Ensured access to equipment for 30 scientific organizations;  Established coordination of five national laboratories and 15 engineering laboratories;  Publication of a journal in the area of material science was launched;  The site content is added and updated upon conduction of workshops and visits of foreign scientists;  An application was sent to the National Centre for Accreditation in order to conclude an agreement for accreditation of ICMS laboratories according to Standard 17025 “Requirements to testing and calibration laboratories;  Two international research-to-practice conferences and a number of various workshops were held; 63  Business processes were developed in accordance with the ISO standard, relevant brochure was issued. Component 2 - Linking Science to Business 28. Even if Kazakhstan improves the quality of its scientific products, the grant groups will not receive the anticipated economic effect from introduction of the developments unless they are linked to the domestic and global markets of technologies. Unfortunately, too many Kazakhstani research institutes, universities and even some high-technology enterprises operate in the commercial vacuum to the significant extent. They do not know how to find investors, customers, or to determine whether they will be able to sell their innovation embodied into a certain product or to address complex technical problems and to conduct a proper research requested by domestic and foreign customers. 29. Many scientists in Kazakhstan believe mistakenly that Kazakhstan has a heavy stock of innovations which may be easily commercialized, especially, if local investors in risky ventures will have to invest capital. 30. Unfortunately, most scientists in Kazakhstan, as most scientists worldwide, do not know how to introduce developments into production, i.e. how to commercialize innovations. They have no links to global markets and capital investors in risky ventures which are necessary for successful commercialization. There are no appropriate commercialization institutions in Kazakhstan to assist to the scientist in selling their developments. There is no doubt that there are many technology parks and business incubators or institutions supporting innovations in one way or other in the country, but they operate inefficiently and do not collaborate as a consistent and effective system. Kazakhstan will never be able to fulfil its technology commercialization potential, until this status quo area remains unchanged. Therefore, the goal of this Project component is to ensure expert assessment of projects in order to link Kazakhstan’s scientific activities to the domestic and foreign market needs. Within the framework of the above specified Component 2: 31. In August 2013, the MoES of RoK and CRDF Global have concluded a Contract for provision of services connected with consulting support of the Project and scientist groups to the total amount of US$2.4 million. The taxes (corporate income tax and value added tax) were refunded out of RB cofinancing funds according to the concluded Contract to the total amount of KZT 113.6 million. 32. CRDF Global is an independent organization engaged in development of international scientific and engineering cooperation. 33. CRDF Global employees worked immediately at the TCC 85 office and stayed in Kazakhstan to the end of 2015. CRDF Global undertook an obligation to implement 7 technology commercialization deals. 85 Technology Commercialization Center, to be used simultaneously with Technology Commercialization Office 64 34. The results of CRDF Global activities are: (a) intensive technology commercialization training was organized for 12 TCC employees at IC2 Institute of the University of Texas, Austin (USA); (b) design of Component 2 Grant Program was prepared, as a part of which projects have been selected; (c) STEP contest was held at Alatau IT-park and participation of Kazakhstan projects in Technology Commercialization Reactor in Riga (Latvia) was organized; (d) participation in a number of workshops in commercialization basics in the regions of Kazakhstan, meetings with stakeholders were held; (e) on-going training for TCC staff members and transfer of skills by means of joint work on technology commercialization projects and training sessions; (f) at least 4 license deals (10 deals concluded); (g) at least 3 start-up companies with reaching the first sales level (result: 40 out of 65 projects have reached the level of sales of finished products to the total amount exceeding KZT 900 million); (h) assistance in establishing an alliance of technology commercialization professionals; (i) development of terms of reference for the analysis of the laws and regulations (Commercialization Law was adopted), development of recommendations to the draft law on intellectual property) and conduction of technical audit; (j) methodological support (guidance papers for the TCOs operating on the basis of higher education institutions and research institutes were developed); (k) compilation of TCC development plan; (l) support for all 65 projects of the Technology Commercialization Project. 35. According to Decree of the Government of the Republic of Kazakhstan No. 989 dated 30.07.2012, Technology Commercialization TCC LLP (TCC) was established in order to increase efficiency of the scientific contribution to the innovative development of the country. 36. The principal TCC activities are:  development of strategies for the selected projects within the framework of its grant programme, financial and consulting support for innovators (innovation grants);  conduction of technology audit in order to identify, structure, and analyse promising Kazakhstani technologies and to determine demand for technologies among industrial enterprises in Kazakhstan; 65  development of the regulatory base in the area of technology commercialization, analysis of laws and regulations.  Development of a system for dissemination of knowledge to involve all stakeholders in the technology commercialization process (awareness-raising activities, consultations for scientists, workshops, roundtables, coverage in media, booklets, technology brokerage). 37. In accordance with the Project Operational Manual and Environmental Management Plan (EMP), the Ministry of Education and Science of the Republic of Kazakhstan and the TCC concluded an agreement to for the amount of KZT 2.5 milliard in October 2013. 38. The agreement provides for TCC operation on the basis of the recommendations and Development Plan being developed by CRDF Global. 39. In 2013, for the purpose of optimization and accelerated implementation of the Project, the PMT transferred its own budget to the TCC: PMT staff expenses (except for the key personnel), PMT’s operating expense (including rent of the office premises, communication services, and other), and funds for a PR campaign. Results of the TCC activities: (a) Over the period of TCC operation, the first and second rounds of the grant programmes were completed, which resulted in selection of 32 projects to the total amount of KZT 1,460 thousand. (b) The TCC conducted a technological audit in order to identify projects which may be potentially promising in terms of project commercialization and to determine demand for technologies among industrial enterprises of Kazakhstan. (c) The TCC conducted 27 training and information workshops for 881 participants from the Republic of Kazakhstan, including those in Almaty and Astana, and in the regions of Kazakhstan, including Karaganda, Ust-Kamenogorsk, Pavlodar, Petropavlovsk, Kokshetau, Kostanai, Stepnogorsk, and other cities (d) The TCC is one of the participants to development of the draft law on commercialization of scientific and scientific and technological activities. (e) A technology brokerage event was held where 50 technological projects were presented and 4 technology licensing deals were concluded. 66 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not available. 67 Annex 9. Additional Evidence for Assessment of Outcomes Achievement of the Project Development Objectives: Scientific Performance Objective 1: Improved scientific performance of the groups selected through transparent and competitive process. 1. This section will provide additional arguments and strengthen evidence to establish the causality between the TCP activities and results. Arguably, the better performance of the TCP researchers against the national average86 could be explained by a selection bias if grants were awarded to researchers already performing better than the average at the beginning of the project. Unfortunately, there is no evidence to robustly test the impact of this possible selection bias due to lack of baseline data. However, the performance of some senior TCP researchers in the last years of the TCP can be assessed, as proxy variables, through the journal impact factors and through the evolution of citations compiled by Google Scholar. These are some results: (a) The 12 top publications representing scientific performance of the 12 top research groups, selecting the best publication for each group, have been published in journals with an average impact factor of 3.0, while top publications by most of the same researchers before the TCP were mostly in journals of impact factor less than 2.0. Qualitative interviews confirmed that the largest part of this increase on scientific performance has to be explained by the TCP. (b) Table 9.1 shows that some of the TCP papers are already being cited by international journals, and the number of citations from senior researchers (those with h-index greater than 15 and annual citations greater than 120) have significantly improved from 2014 to 2015.87 Table 9.1. Examples of TCP Articles with High Impact Factor and Performance of Senior Researchers Kudaibergenov, S. E., N. Nuraje, and V. V. Khutoryanskiy. Kudaibergenov, S.E., h-index = 20 Journal impact 2012. “Amphoteric Nano-, Micro-, and Macrogels, Citations 2014 = 130 factor = 4.03 Membranes, and Thin Films.” Soft Matter 8 (36): 9302– Citations 2015 = 140 Citations = 26 9321. Rate 2014/2015: +8% Bakenov, Z. Zhang, Y., Y. Zhao, Z. Bakenov, A. Konarov, and P. Chen. Journal impact h-index = 15 2014. “Preparation of Novel Network Nanostructured factor = 6.23 Citations 2014 = 193 Sulfur Composite Cathode with Enhanced Stable Cycle Citations = 7 Citations 2015 = 236 Performance.” Journal of Power Sources 270: 326-331. Rate 2014/2015: +22% Dzhunushaliev, V. Aringazin, A., V. Dzhunushaliev, V. Folomeev, B. Journal impact h-index = 18 Kleihaus, and J. Kunz. 2015. “Magnetic Fields in Mixed factor = 5.81 Citations 2014 = 122 Neutron-star-plus-wormhole Systems.” Journal of Citations = 0 Citations 2015 = 131 Cosmology and Astroparticle Physics 2015 (04): 005. Rate 2014/2015: +7% Source: Impact factors for 2014, according to the Journal of Citation Reports, Web of Science, Thomson Reuters. Citations and h-index come from Google Scholar, March 31, 2016. 86 Number of international publications from SSGs and JRGs in peer-reviewed journals (per researcher) in 2014 and 2015 87 0.16 in the national average vs 0.25 in TCP 68 (c) The number of citations of the three top TCP senior researchers with statistics on Google Scholar have increased by about 166 percent, when comparing the pre-TCP period 2009–2011 and the period 2014–2016: from 636 to 1,667 citations (data for 2016 are projects based on January–April data). Besides, their best TCP papers have very high impact factors (between 4.5 and 6.2) that will increase the h-indexes in the coming years. The figure 9.1 illustrates this likely TCP effect. Figure 9.1. Scientific Performance of Three Top Senior Researchers in the TCP Number of citations registered in Google Scholar 250 Pre‐TCP During and After TCP 364 200 150 100 50 0 2008 2009 2010 2011 2012 2013 2014 2015 2016(*) Kudaibergenovs, SE Bakenov, Z Dzhunushaliev, V. Average Source: Google Scholar; ICR analysis Note: (*) Estimates for 2016 based on the current January–April 2016 data. (d) The number of citations, many international, has increased in all cases, showing a significant TCP contribution to science, even if some data limitations apply.88 This suggests that the TCP has played a significant role in the overall increase of publications related to the h-index according to the Global Innovation Index (GII). The GII index indicates a relative increase of 4 percent of Kazakhstan scientific performance against the rest of the world (move in the ranking from position 112 in 2013 to 107 in 2015). 2. In conclusion, there is sufficient evidence to conclude that substantial results were achieved against the first developmental objective of the project, and there is substantial evidence to suggest the attribution of results to the TCP project. 88 Unfortunately, the h-index for the scientific performance of TCP at the time of ICR cannot be measured due to the time lags between publications and citations and since the production of publications was very concentrated at the end of the project, as expected. 69 Achievement of the Project Development Objectives: Examples Demonstrating Commercial Relevance Objective 2: Improved the commercial relevance of the groups selected through transparent and competitive process Table 9.2. Ten Selected TCPs Showing Commercial Relevance and Other Socioeconomic Impacts Sales (**) Project code Title/Product Main Socioeconomic Actual and Approximate End IP (*) Innovation Potential Impacts 2015 PP-00751 Industrial prototypes KZT 726,163,000 1 application for Local production of ear tags for of ear tags for sheep (US$3,280,000) national patent livestock at low cost with and goats international standards PC-00653 Concept of Zn/NiOOH No sales yet 4 international More efficient use of energy at static rechargeable patent low cost battery applications, 1 national 84/2012/SSG Technology for virus- KZT 10,200,000 1 national patent The project contributes to the free potato from (US$46,000) high-level goal of fighting extreme Microtubes poverty by decreasing price and increasing quality of socially important food items. It also generate more incomes for farmers. 053/2011/JRG Biological product for KZT 189,000 2 national patents Social applications for water water biotreatment by (US$855) purifiers helping poor and rural lipid-depleting communities microorganism PD_00226 “Wipon” system for KZT 400,000 Trademark and Fight against the black market and counterfeit goods (US$1,800) copyright for the Intellectual Property Rights detection applications in goods (for example, 30,000 bottles confiscated by tax authorities thanks to the application); application downloaded by 27,000 people PP-00726 Long-term burning KZT 26,494,000 1 national patent Production of top-loading solid solid fuel boilers and (US$118,807) fuel boilers for fuel efficiency boilers plants PP-00104 High-tech cosmetic KZT 11,704,000 1 trademark New product line having products based on (US$52,484) national therapeutic and cosmetic effects extract of Regeneresen certificate, 1 containing ribonucleic European acids certificate 04/2013/JRG Geo-formation system KZT 2,767,000 2 copyrights Specialized software to improve to determine the (US$12,408) items mining efficiency parameters for ground condition and mine working 161/2011/SSG Technology of KZT 18,300,000 1 national patent Technology for polymer flooding polymer flooding for (US$82,063) in existing oil fields enhanced oil recovery 70 Sales (**) Project code Title/Product Main Socioeconomic Actual and Approximate End IP (*) Innovation Potential Impacts 2015 157/2013/SSG Development of KZT 11,500,000 2 PCT; 2 More efficient use of energy at innovative lithium-ion (US$51,570) applications low cost battery lithium-metal abroad free Note: (*) PP/PO = Prototype grant; PC = Proof of concept grant; (**) Sales were made along different periods of time with different exchange rates. For numbers in U.S. dollars, the annual average exchange rate for 2015 rate has been used (KZT 221.73 = US$1). Table 9.3. License Deals in Frame of the TCP Company Name Number of Project ID (Established by Counterpart Bought/Sold Licenses Beneficiary) Nonexclusive License 229/2011/ SSG LLP «AspanTau LTD» LLP KazStroyInvest 1 Sold Private enterprise "Institute 161/2011/ SSG of Polymeric Materials and LLP VostokNefteGas 1 Sold Technologies" 108/2012/SSG LLP «Bio Tech TKS» Private farm "Dzhakybayev" 1 Sold PD_00593 LLP "Aparu" LLP «Ticket Pay» 1 Sold Tianjin Keyi Jiaxin Technology PC_00653 LLP "Institute of Batteries" 1 Sold Co., LTD Exclusive License West Kazakhstan Agrarian- PD_00126 LLP "InnoTechProject" 1 Bought Technical University LLP "Scientific and State enterprise "Kazakhstani PC_00047 1 Bought Production Centre "Micro" Collection of Microorganisms" PD_00635 LLP "Angstrem" (2 шт.) «SineTheta Ltd» 2 Sold PD_00635 LLP "Angstrem" (2 шт.) Mr. Turyansky 1 Bought Cession of Rights State enterprise “Kazakhstani 30/2013/JRG LLP «Master Gene» Scientific Center of Quarantine and 1 Sold Zoonotic Diseases” Total 11 71 Poverty Impacts, Gender Aspects, and Social Development Table 9.4. Additional Examples of Projects with Strong Poverty and/or Social Impact No. of the Project/Year/ Brief Name of the Project Goal of the Project Group 053/2011/JRG Technology of obtaining of The project goal is water treatment by using filtration devices filtration materials based on track based on track membranes, obtained on DC-60 accelerator. membranes 84/2012/SSG Technology of obtaining of virus- The project goal is the development and introduction of free potato from microtubers biotechnology of obtaining of virus-free elite seed potato using microtubers for the producers of agricultural goods of the Akmolinskaya oblast. 207/2012/SSG Technology of obtaining of The project aims at the development of experimental industrial interferon for immunity regulations and creation of pilot production of interferon for improvement immunity improvement of the population. 30/2013/JRG Development of the technology The project aims at the development of the technology and and organization of production of organization of production of domestic Polymerase Chain Polymerase Chain Reaction test- Reaction test systems for molecular-epidemiologic monitoring systems of plague and goat-fever in Kazakhstan and central Asian countries. PD_00434 Creation of artezunatum Creation of artezunatum—a highly effective drug for treatment of theileriosis of beef cattle and development of technological parameters of its production 72 Annex 10: A List of Technology Commercialization Projects Component 1 First Sales Amount No. of the Brief Name of the as of Year-end 2015 Project/Year/ Description of the Project Project (KZT, thousands) Group (number of deals) 72/2011/SSG Technology of using The project goal is facilitation of transportation 14.4 (1) magnetic fields for of paraffin oils through oil pipelines by way of improvement of their processing in high magnetic fields. fluidness of paraffin oils 161/2011/SSG Technology of polymer The project aims at increasing of oil stratum 18,300 (1) flooding of oil stratum recovery at the Kazakhstani oil fields by way of recovery their polymer flooding by polysaccharide - gellan. 184/2011/SSG Technology of obtaining The project goal is the production of such 4,120 (1) of nanostructured carbon-mineral materials as technical carbon for carbon-containing extraction and concentration of rare-earth materials metals, activated carbon for water treatment, and reinforcing nanofiller for technical rubbers. 229/2011/SSG Technology of obtaining The project aims at significant improvement of 1,500 (2) of superplasticizer – the performance properties of concrete upon its (SP-NSKF obtaining from cement by way of using a [superplasticizer technology of a new superplasticizer SP-NSKF naphthalene-sulpho- (superplasticizer naphthalene-sulpho-acidic acidic formaldehyde]). formaldehyde). 290/2011/SSG Technology of obtaining The project goal is the creation of highly 2,500 (1) of gas catalysts into effective catalysts of new generation for direct ethylene obtaining of ethylene from natural and liquefied petroleum gas. 023/2011/JRG Technology of obtaining The project goal is the production of starters on 320 (1) of starters of national industrial scale for local fermented milk fermented milk products products (kumiss and camel milk) with definite properties and standard quality. 036/2011/JRG Technology of The project goal is the determination of 7,327 (3) determination of individual dosing of such medicinal individual dosing of preparations as Warfarin and Plavix, by way of medical preparations using an express pharmacogenetic test. based on an express pharmacogenetic test 053/2011/JRG Technology of obtaining The project goal is water treatment by using 189 (1) of filtration materials filtration devices based on track membranes, based on track obtained on DC-60 accelerator. membranes 056/2011/JRG The system of synthesis The project goal is the creation of a software 21.7 (1) and recognition of the product for synthesis and recognition of the Kazakh speech Kazakh speech for mass distribution on a commercial basis. 73 First Sales Amount No. of the Brief Name of the as of Year-end 2015 Project/Year/ Description of the Project Project (KZT, thousands) Group (number of deals) 063/2011/JRG Technology of obtaining The project goal is improvement of wear 29.1 (1) of self-healing resistance of metal pipes by way of depositing composite coatings of self-healing composite coatings. 072/2012/JRG Technology of The project aims at the conduct of genetic 1,700 (2) molecular-genetic analysis of sheep of Yedilbay and Kazakh classification and moufflon-merino breeds, creation of genomic cytogenetic certification certificates of breeds, and the use of assessment of sheep - Yedilbaiskaya indices of their genome in genetic and selection and Kazakh moufflon- work for accelerated and quality improvement merino of highly productive breeding sheep. 84/2012/SSG Technology of obtaining The project goal is the development and 10,200 (1) of virus-free potato from introduction of biotechnology of obtaining of microtubers virus-free elite seed potato, using microtubers, for the producers of agricultural goods of the Akmolinskaya oblast. 183/2012/JRG The system of The project goal is the development of the 50 (1) recognition and analysis system of automated analysis of mobile objects of mobile objects with the formation of an archive of scenarios for further commercialization. 232/2012/JRG Improvement of The project goal is to create an environment of – (0) multifunctional robotic intellectual manipulation of the objects for arm industrial automation by way of integration of a robotic arm with sensors, industrial robot- manipulator of the latest generation, and algorithms of machine-learning. It is expected that such combination shall improve the properties of a robot-manipulator and decrease the time spent by the operators for programming robots. The researchers intend to present a robotic arm and control algorithms as a commercial product and to offer it at the market as an instrument for productivity improvement. 157/2012/JRG New methods for The project aims at the development of new 2,725 (4) uranium determination methods of uranium determination in the productive solutions with the commercialization of the obtained results in the form of recommendations, as well as in the form of creation of new products (mathematic model, methods of assessment of artificial effect, and advance training courses for the employees of uranium production sector). 108/2012/SSG Technology of obtaining The project aims at the production of an elite 1,100 (3) of highly productive planting material kok sagyz for the creation of forms of natural rubber commercial plantation and provision of natural producer rubber-producing enterprises with vegetable raw material. 74 First Sales Amount No. of the Brief Name of the as of Year-end 2015 Project/Year/ Description of the Project Project (KZT, thousands) Group (number of deals) 151/2012/SSG Technology of The project aims at commercialization of fire – (0) processing of carbon- gas technology by way of processing of containing wastes into industrial wastes into pure fire gas for metal fire gas cutting as an alternative for acetylene. 168/2012/SSG Technology of obtaining The project aims at the creation of science- – (0) of gasoline and diesel based environmentally pure technology of fuel from brown coals of brown coal processing into synthetic oil. Kazakhstan 184/2012/SSG Technology of obtaining The project aims at the development of new 300 (1) of reflecting materials light-reflecting materials for painting road and navigation signs and transport and road marking. 207/2012/SSG Technology of obtaining The project aims at the development of – (0) of interferon for experimental industrial regulations and creation immunity improvement of pilot production of interferon for immunity improvement of the population. 229/2012/SSG Technology of obtaining The project aims at the treatment of vast – (0) of ointments and bands wounds and burns by using polymer hydrogel based on medical forms ointments and bands, containing of phytopreparations phytopreparations based on vegetable raw material of Kazakhstan. 4/2013/JRG Geoinformational The project aims at the provision of highly 2,767 (4) analysis model of accurate forecast of ground condition of vast ground condition territories on the basis of remote earth sensing. 18/2013/JRG Development of The project goal is the development of а new 230.4 (1) recombinant biomimetic recombinant method of production of self- peptides organizing peptides for treatment and prevention of dental caries. 30/2013/JRG Development of the The project aims at the development of the 1,947 (10) technology and technology and organization of the production organization of of domestic Polymerase Chain Reaction test- production of systems for molecular-epidemiologic Polymerase Chain monitoring of plague and goat-fever in Reaction test-systems Kazakhstan and central Asian countries. 43/2013/SSG Development of the The project goal is the creation of an innovative – (0) system of remote system for telemonitoring and control of protective monitoring of superextended objects of strategic importance, the mainline including mainline cable communication communication structures at the national level. infrastructure 55/2013/JRG Development of The project aims at the obtaining of flexible and – (0) composite hydrogel mechanically strong hybrid organic and scrapers inorganic composite materials for their further use in the oil sector as purifying ‘scrapers’ of the interior pipeline surface. 75 First Sales Amount No. of the Brief Name of the as of Year-end 2015 Project/Year/ Description of the Project Project (KZT, thousands) Group (number of deals) 64/2013/SSG Development of the The project goal is the development of a – (0) technology of obtaining biologically active additive into formulated feed of mineral additives into and as a veterinary sorbent for agricultural formulated feed and animals on the basis of monomineral bentonites, their putting into with the obtaining of powders, with vitally production important microelements, introduced into crystalline microstructure and their production for use in agricultural complex. 80/2013/SSG Bio-coatings on the The project aims at commercialization of the – (0) basis of natural technology and production of bio-coatings polymers for treatment based on collagen, hyaluronic acid, and fibrin of ambustial and glue for treatment of ambustial and surgical surgical wounds wounds. 85/2013/SSG Low-waste technology The project aims at the obtaining of competitive 3,000 (2) and scientific and cost-effective medical cheesecloth of cotton justification of mixture with the additives of textile wastes. cheesecloth production 117/2013/SSG Commercialization of The project aims at the organization of the 605 (2) the technology of production of power-saving material - synthetic production of innovative wollastonite based on artificial wastes of a power-saving material - number of industries (phosphorus and wollastonite metallurgic industries) and corrective additives. 125/2013/SSG Development of the The project goal is the development on the basis 3,230 (3) technology of risk of a probabilistic risk model and its numerical mapping of marine biota implementation of a commercially successful affection web-system Biota Impacted by Oil Spills (BIOS) for risk mapping of marine biota affection upon emergency oil spill at the North Caspian Sea. 149/2013/SSG Development and The project aims at the development and 21,045 (1) introduction of two- introduction of a new bio-geo-technological stage bacterial chemical method of processing of wastes and cutoff grade technology of advanced ores of copper and gold-processing enterprises, waste processing with complex metal extraction. 157/2013/SSG Development of The project aims at the development of a 11,500 (1) innovative lithium-ion prototype of a lithium-ion sulfuric battery, sulfuric battery without without lithium metal, in which a cathode lithium metal contains composites with natural sulfur and has high specific energy/capacity, energy efficiency, and cycling stability. Total 94,721.85 (49) 76 Component 2 No. of the Brief Name of the Sales Amount Description of the Project Project Project (KZT, thousands) PD_00434 Creation of artezunatum Creation of artezunatum—a highly effective 5,947 (2) drug for treatment of theileriosis of beef cattle and development of technological parameters of its production PD_00226 Development of IP Development of optimal infrastructure and 400 (1) protection systems algorithms of interaction of information technology in the field of IP protection systems, with maximum adaptation for producers PD_00056 Portable mechanism for Portable mechanism for tanker cleaning – (0) tanker cleaning PD_00104 High-tech cosmetic Establishment of industrial prototype for the 11,704 (10) products production of high-tech cosmetic products, based on the ‘regenerezen’ extracts containing ribonucleic acid PD_00403 Strategy game ‘Tengrion’ The first Kazakhstani cloud strategy game – (0) ‘Tengrion’ PD_00388 Chemical treatment of Development and implementation of cost- – (0) crops and pests lair effective technologies for the chemical treatment of crops and pests lair using a hovercraft PD_00126 Lightweight and durable Technology for production of lightweight 222 (1) artificial gravel and durable artificial gravel based on processing of chert flasks for industrial, civil, and road construction PD_00021 Self-fluxing powder filler Creating a prototype of self-fluxing powder 11,000 (2) filler material based on iron PD_00025 Enhancing oil recovery Production of pilot batch of biological agent – (0) from flooded oil bed for enhancing oil recovery from flooded oil bed using organic method PC_00047 Probiotic mixture for Commercialization of the technology of – (0) prevention and treatment obtaining a probiotic mixture for prevention of bacterial vaginosis and treatment of bacterial vaginosis PC_00301 Unique medicine from The development of normative and technical – (0) domestic raw vegetable documentation for unique medicine from materials domestic raw vegetable materials that has hepatoprotective feature PC_00102 Feed additives for poultry Production technology and experimental – (0) studies of functional feed additives for poultry, based on zeolite produced at Chankanayskogo site PC_00464 Justification of GIS Justification of GIS concepts for determining 26,500 (2) concepts for determining the parameters of maintenance of mine the parameters of workings maintenance of mine workings PC_00049 Biological wastewater The development of a biological agent for 50 (1) treatment biological wastewater treatment using lipid- degrading microorganisms 77 No. of the Brief Name of the Sales Amount Description of the Project Project Project (KZT, thousands) PC_00193 Plant growth promoter Development of small-scale production of 444 (4) KH-2 (akpinol-alpha) domestic plant growth promoter KH-2 (akpinol-alpha) and testing on different kinds of plants to combat desertification PC_00120 Magnetic stimulation of Universal module for magnetic stimulation – (0) threshing products of threshing products of combine harvesters PC_00752 Electronic document Development of the first electronic document – (0) exchange platform exchange platform www.idocs.kz PC_00653 Stagnant rechargeable Justification of concept of stagnant – (0) batteries Zn/NiOOH rechargeable batteries Zn/NiOOH PD_00745 Quality of diagnosis of Using neural networks to ensure quality of – (0) fibrosis diagnosis of fibrosis accompanied by chronic hepatitis C PD_00751 Ear tags for sheep and Production of industrial prototypes of ear 726,163 (3) goats tags for sheep and goats PD_00755 Syringe-tubes for Production of industrial prototype of the – (0) veterinary use syringe-tubes for veterinary use PC_00634 Technology of lifting Development of innovative technology of 135 (1) water for hydro- lifting water for hydro-accumulating power accumulating power plants using hydro-collision installation plants powered by renewable energy from water flow PD_00709 Metallurgical cocked Design of industrial prototype of – (0) briquette metallurgical cocked briquette using domestic nucleating filler PC_00642 Energy-saving electric Energy-saving electric steam heater of – (0) steam heater of negative negative pressure pressure PD_00722 Manufacturing of Manufacturing of structures for mounting of 215 (2) structures for mounting of current protection reeds and open and current protection reeds covered conductors and open and covered conductors PD_00654 Dry powder from mare's Innovative equipment and technology of 500 (1) milk production of dry powder from mare's milk PD_00753 Stabilizer of phases and Creation of industrial prototype of a product – (0) voltages entitled ‘stabilizer of phases and voltages SPS-63-λ ± 2% - UHL4’ PD_00635 Compact new-generation Creation of industrial prototype of a compact – (0) microfocus X-ray source new-generation microfocus X-ray source PD_00705 3D printers Establishment of a small-scale production of 2,061 (9) 3D printers with Fused Deposition Modelling printing technology (print molten plastic) and Stereolithography (photolithographic printing) PC_00746 Cloud service to support Creation of a cloud service to support 910 (4) investors (IS investors and initiators in selecting the «SmartInvest») optimal algorithm of actions for planning, financing, and implementation of investment and innovation projects (entitled IS «SmartInvest») 78 No. of the Brief Name of the Sales Amount Description of the Project Project Project (KZT, thousands) PD_00726 Production of long- Production of long-burning solid-fuel boilers 26,494 (6) burning solid-fuel boilers and boiler plants and boiler plants PD_00708 Water-based biological Establishment of small-scale pilot production – (0) fire-protecting of the «SGT» water-based biological fire- impregnation protecting impregnation PD_00593 Unified system of Unified system of electronic tickets 1,219 (1) electronic tickets ‘AparuBus’ ‘AparuBus’ Total 813,964.55 (50) Grand Total 908,686.40 (99) 79 Annex 11. List of People Met during the ICR Mission Government Officials  Takir Balykbayev, Vice Minister of Education and Science  Sansyzbay Zholdasbayev, Chairman, Science Committee of the Ministry of Education and Science  Azamat Batyrkozha, Director of the Department of Innovation and Technological Development, Ministry of Investment and Development  Ayan Ramazan, Head of the Infrastructure Division, Ministry of Investment and Development  Zhanna Temirova, Head of the State Support Instruments Division, Ministry of Investment and Development  Azamat Abdullayev, Expert of the Ministry of Investment and Development World Bank Team  Ato Brown, Bank Country Manager  Rakhymzhan Assangaziyev, Senior Operations Officer, Bank Country Office  Yeraly Beksultan, Co-TTL for TCP  Nurbek Kurmanaliev, Procurement Specialist  Aliya Kim, M&E Specialist  Yuriy Kuzmyn, M&E Specialist PMU, TCO, and IMSC Staff  Daniyar Doskarayev, TCP PMU Director  Gulmira Mustafa, TCP Financial Manager  Baurzhan Zhanguzin, TCP Procurement Specialist  Zhanel Karina, TCP Safeguards Specialist  Abdilda Shamenov, President of the Alliance of Technology Commercialization Professionals, (Former) TCO Deputy Director, and Commercialization Manager 80  Murat Sartbayev, Director of Projects Department ‘Information Analytical Center’ JSC and Former TCC Specialist  Abylay Akhymbekov, TCO Commercialization Manager  Adina Mamrayeva, TCO Commercialization Manager  Abay Zhangabylov, TCO Commercialization Manager Beneficiaries SSGs (8)  157/2013/SSG: Indira Altayevna Kurmanbayeva, Head (Development of innovative lithium-ion sulfuric battery without lithium metal)  72/2011/SSG: Askar Aryngazin, Head (Technology of using magnetic fields for improvement of fluidness of paraffin oil)  168/2012/SSG: Yerbolat Yermagambet, Head (Technology of obtaining of gasoline and diesel fuel from brown coals of Kazakhstan)  168/2012/SSG: Balpanov Darkhan, Head (Technology of obtaining of virus-free potato from microtubers)  149/2013/SSG: Staff members (Development and introduction of two-stage bacterial chemical technology of advanced waste processing)  207/2012/SSG: Oleg Ten, Head (Technology of obtaining of interferon for immunity improvement)  125/2013/SSG: Edige Zakarin, Head and his team (Development of the technology of risk mapping of marine biota affection)  161/2011/ SSG: Sarkyt Kudaybergenov and his team (Technology of polymer flooding of oil stratum recovery) JRGs (3)  036/2011/JRG: Yelena Zholdybayeva, Head (Pharmacogenetic tests for individual dosage of drugs)  4/2013/JRG: Dmitriy Mozer, Head (Geoinformational analysis model of ground condition)  55/2013/JRG: Staff Members (Development of composite hydrogel scrapers) 81 Component 2 Grantees (6)  PC_00653: Zhumabay Bakenov, Head (Justification of the concept of stagnant rechargeable batteries Zn / NiOOH)  PD_00751: Zhandos Bolatbekov, Head (Production of industrial prototypes of ear tags for sheep and goats)  PD_00226: Rustam Sultanmuradov, Head (Development of optimal infrastructure and algorithms of interaction of IT in the field of intellectual property protection systems, with maximum adaptation for producers)  PD_00709: Vasilliy Kim, Head (Design of industrial prototype of metallurgical cocked briquette using domestic nucleating filler)  PD_00056: Uzak Zhapbasbayev, Project Head, and Dauren Uzakov, Head of LLP KBTU SM (Portable mechanism for tanker cleaning)  PD_00021: Zhanserik Ilmaliev, Head (Creating a prototype of self-fluxing powder filler material based on iron) IMSC  Vyacheslav Kulikov, Scientific Leader of IMSC   Dmitry Khrustalev, Head of International Relations Unit of IMSC  Other Stakeholders in Kazakhstan’s NIS NATD  Zhumatay Salimov, Deputy Chair of the Managing Board of NATD  Timur Shalabayev, Director of the Center for Development of Technology Transfer System  Olzhas Bilyalov, Deputy Director of the Center for Development of Technology Transfer System  Zhuldyz Zhumadilova, Expert of Center for Development of Technology Transfer System NURIS  Aidar Zhakupov, General Director, Nazarbayev University Research and Innovation System (NURIS)  Yerzhan Syrgaliyev, Chief Business Officer, NURIS 82  Daniyar Zhumagulov, Director, Commercialization Office of NURIS  Bolat and Akylbek - Managers of Commercialization Office of NURIS Development Institutions  Rainer Gortz, Senior Expert, German Society for International Cooperation  Thomas Minnich, Private Sector Unit Head, ADB Central Asia and Caucasus  Arman Seisebayev, Senior Investment Officer, ADB Central Asia and Caucasus Other  Murat Baimuldin, Head of Commercialization Office of Karagandy State Technical University  Kanat Shakenov, Director of Representative Office of CRDF Global 83 Annex 12. List of Supporting Documents  Final Report prepared by the CRDF Global Expert Team, December 3, 2015  Executive summary of the final report of LLP “Technology Commercialization Center” on implementation of the agreement to provide services aimed to establish the link between science and markets as part of Component 2 of the Technology Commercialization Project dated October 11, 2013 № 352 (for the period 2013–2015), 2015  Updated Results Framework with final results  Innovative Project Catalogue, 2015  Component 1 infograph  List of 85 publications in journals with impact factor  PMU presentation with final results, 2015  International Bank for Reconstruction and Development Project Appraisal Document on a Proposed Load in the Amount of US $88 Million to the Republic of Kazakhstan for a Fostering Productive Innovation Project, December 1, 2014  Address by the President of the Republic of Kazakhstan, Leader of the Nation, N.A. Nazarbayev on Strategy Kazakhstan 2050: New Political Course of the Established State, 2012  The 100 concrete steps set out by President Nursultan Nazarbayev to implement the five institutional reforms (May 20, 2015)  Letter from the Government with final comments (May 24, 2016) 84 MAP 85