Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005522 IMPLEMENTATION COMPLETION AND RESULTS REPORT IBRD 8332-MK ON A LOAN IN THE AMOUNT OF EUR 17.7 MILLION (US$24 MILLION EQUIVALENT) TO THE REPUBLIC OF NORTH MACEDONIA FOR THE SKILLS DEVELOPMENT AND INNOVATION SUPPORT PROJECT October 28, 2021 Education Global Practice Europe and Central Asia Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective April 30, 2021) Currency Unit = Euros (EUR) EUR 0.83 = US$1 US$1.44 = SDR 1 FISCAL YEAR July 1 – June 30 ABBREVIATIONS AND ACRONYMS AQA Agency for Quality Assurance CBA Cost-Benefit Analysis CPS Country Partnership Strategy CVET Center for Vocation Education and Training EIRR Economic Internal Rate of Return EMF Environmental Management Framework EMIS Education Management Information System ENQA European Association for Quality Assurance EUA European University Association EU European Union FDI Foreign Direct Investment FITD Fund for Innovation and Technological Development GOM Government of North Macedonia HEAEB Higher Education Accreditation and Evaluation Board HE Higher Education HEI Higher Education Institution HEMIS Higher Education Management Information System ICR Implementation Completion and Results Report IPA Instrument for Pre-Accession Assistance IPR Intellectual Property Regime ISR Implementation Status and Results Report IT Information Technology MSMEs Micro, Small, and Medium Enterprises MTR Midterm Review MOES Ministry of Education and Science M&E Monitoring and Evaluation NAQA National Agency for Quality Assurance NPV Net Present Value NTTO National Technology Transfer Office PAD Project Appraisal Document PDO Project Development Objective PMU Project Management Unit POM Project Operations Manual R&D Research and Development SMEs Small and Medium-Size Enterprises TVET Technical and Vocational Education and Training VET Vocational Education and Training Regional Vice President: Anna M. Bjerde Country Director: Linda Van Gelder Regional Director: Fadia M. Saadah Practice Manager: Harry Anthony Patrinos Task Team Leader: Bojana Naceva ICR Main Contributor: Saori Imaizumi TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION .............................................................. 11 II. OUTCOME .................................................................................................................... 17 A. RELEVANCE OF PDOs ............................................................................................................ 17 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 19 C. EFFICIENCY ........................................................................................................................... 29 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 32 E. OTHER OUTCOMES AND IMPACTS ......................................................................................... 32 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 35 A. KEY FACTORS DURING PREPARATION ................................................................................... 35 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 36 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 40 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 40 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 42 C. BANK PERFORMANCE ........................................................................................................... 44 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 46 V. LESSONS AND RECOMMENDATIONS ............................................................................. 47 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 50 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 67 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 68 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 69 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 74 ANNEX 6. BORROWER’S REPORT ......................................................................................... 75 ANNEX 7. SUPPORTING DOCUMENTS .................................................................................. 95 The World Bank Skills Development and Innovation Support Project (P128378) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P128378 Skills Development and Innovation Support Project Country Financing Instrument North Macedonia Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency North Macedonia Ministry of Finance, Ministry of Education and Science Project Development Objective (PDO) Original PDO The Project Development Objective (PDO) is to improve transparency of resource allocation and promote accountability in higher education, enhance the relevance of secondary technical vocational education, and support innovation capacity in Macedonia. Page 1 of 96 The World Bank Skills Development and Innovation Support Project (P128378) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 24,000,000 24,000,000 20,399,124 IBRD-83320 Total 24,000,000 24,000,000 20,399,124 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 24,000,000 24,000,000 20,399,124 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 28-Jan-2014 11-Mar-2014 25-Sep-2017 31-May-2019 30-Apr-2021 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 01-Mar-2019 8.55 Change in Results Framework Change in Loan Closing Date(s) 27-May-2020 14.68 Change in Loan Closing Date(s) Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 31-Mar-2014 Satisfactory Satisfactory 0 Page 2 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 02 24-Oct-2014 Satisfactory Satisfactory 1.39 03 10-Apr-2015 Satisfactory Moderately Satisfactory 1.39 04 05-Oct-2015 Satisfactory Moderately Satisfactory 2.17 05 07-Apr-2016 Satisfactory Moderately Satisfactory 2.87 06 18-Oct-2016 Moderately Satisfactory Moderately Unsatisfactory 3.55 07 12-Apr-2017 Moderately Satisfactory Moderately Satisfactory 4.21 08 22-Oct-2017 Moderately Satisfactory Moderately Satisfactory 4.63 09 05-May-2018 Moderately Satisfactory Moderately Satisfactory 6.38 10 04-Dec-2018 Moderately Satisfactory Moderately Satisfactory 7.37 11 12-Jun-2019 Moderately Satisfactory Moderately Satisfactory 9.39 Moderately 12 20-Dec-2019 Moderately Unsatisfactory 11.47 Unsatisfactory 13 05-Jun-2020 Moderately Satisfactory Moderately Satisfactory 14.68 Moderately 14 18-Dec-2020 Moderately Unsatisfactory 17.99 Unsatisfactory 15 29-Apr-2021 Moderately Satisfactory Moderately Satisfactory 20.34 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Public Administration - Education 19 Secondary Education 19 Tertiary Education 8 Other Education 54 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Page 3 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Private Sector Development 130 Business Enabling Environment 30 Investment and Business Climate 30 Jobs 100 Public Sector Management 31 Public Administration 31 Transparency, Accountability and Good 31 Governance Human Development and Gender 40 Education 40 Access to Education 10 Science and Technology 10 Teachers 10 Standards, Curriculum and Textbooks 10 ADM STAFF Role At Approval At ICR Regional Vice President: Laura Tuck Anna M. Bjerde Country Director: Ellen A. Goldstein Linda Van Gelder Director: Alberto Rodriguez Fadia M. Saadah Practice Manager: Andrea C. Guedes Harry Anthony Patrinos Task Team Leader(s): Bojana Naceva Bojana Naceva ICR Contributing Author: Saori Imaizumi Page 4 of 96 The World Bank Skills Development and Innovation Support Project (P128378) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country context. The Republic of North Macedonia 1 is a small, landlocked, middle-income country in the Balkan Peninsula. North Macedonia has an open economy and has been successful in maintaining macroeconomic stability, even during adverse external shocks, such as the 2008 international financial crisis and the 2012 Euro area crisis. The challenge facing the country in 2013, when this project was approved, was to transition to a higher growth trajectory by developing a more competitive and export-oriented enterprise sector. For large-scale foreign direct investment (FDI) to serve as a catalyst for sustaining export-oriented growth at higher rates, domestic firms needed to make much more substantial investments in skills and innovation, both areas identified as critical constraints to competitiveness in North Macedonia. Improving the country’s labor market performance and economic competitiveness required a more skilled and better educated labor force, as well as increased technology absorption, diffusion of knowledge, and innovation. The Government of North Macedonia (GOM) was committed to investing in quality education, innovation, and information technology, which were identified as top strategic priority areas in its work program for 2011–2015 and in the South Eastern Europe 2020 Strategy. To accomplish these objectives, the Government sought support from the World Bank to advance its reforms to achieve better quality higher and vocational education and innovation systems. 2. Sector context. The GOM had skills development and innovation activities spanning from early childhood through adult education and into research systems. At the time of project appraisal, skills development initiatives fell under the auspices of the Ministry of Education and Science (MOES), which also had the main role in science and research and development (R&D), at both the operational level and funding levels. The Government’s commitment was demonstrated in the adoption of relevant strategies, laws, and programs2 in addition to upgrading R&D laboratory facilities at the universities. 3. In the 2013 higher education sector, five public and nine private universities and five non- university private institutions existed, enrolling about 58,000 students, 85 percent of whom were attending public universities. Enrollment in tertiary education was increasing rapidly from 28 percent gross enrollment ratio in 2004 to 39 percent in 2010—both through the efforts of private providers and government interventions. While access to education had improved, enrollment remained far behind the enrollment rates in new European Union (EU) member states.3 Moreover, the increase in enrollment was not matched by an increase in resources. Consequently, the quality of higher education, including its responsiveness to labor market demands, remained a challenge. Overall system efficiency also remained low, with high dropout rates and long average times to completion: in 2010 only 38.8 percent of all university students graduated on time. Furthermore, graduates were not equipped with the necessary skills that employers were looking for. While critical data on student learning and graduate employment 1 At the time of project appraisal in 2013, the country was called the Former Yugoslav Republic Macedonia; the name was changed in 2018. 2 The Government adopted the Innovation Strategy (October 2012), Law on Innovation Activity (May 2013), Vocational Education and Training Strategy (2013), and Program for Development of Science and Research for 2013–017. 3 Gross tertiary enrollment ratios in 2012: Slovenia 87 percent, Romania 50 percent, Bulgaria 61 percent, Slovakia 56 percent, Croatia 61 percent, and North Macedonia 39 percent. Page 5 of 96 The World Bank Skills Development and Innovation Support Project (P128378) outcomes did not exist, a 2010 World Bank employer survey of the demand for skills showed that employers continued to have difficulties in finding workers with the skills required, particularly workers who possess the higher order skills needed in the newly created jobs in modern and dynamic firms. These issues were related to the systemic issues in the higher education sector, including (a) a lack of internal and external quality assurance mechanisms, (b) funding constraints in addition to the existing input- oriented funding mechanism, and (c) the absence of a centralized mechanism for supporting innovation. These issues placed constraints on enhancing the quality and relevance of education and innovative engagements between research and firms. The World Bank was providing technical assistance work on higher education quality assurance and financing reforms since 2010, which served as a basis for this project. 4. In the vocational education and training (VET) sector, North Macedonia focused on formal secondary VET under the MOES. The secondary VET system offered four-year vocational education in 14 occupations with 50 educational profiles, as well as visual art, music, and sports education, and three-year programs with 36 educational profiles. Over half of secondary education students ages 15–19 years attended VET schools. Despite the reform in secondary VET provision between 1999 and 2006, the sector still faced challenges in terms of its relevance, planning, management, quality, and internal efficiency. Early specialization of the VET students also scrutinized students’ opportunities to broaden their skills base to cope with the rapidly changing labor market demand. Moreover, the reformed four-year secondary VET, known as the technical and vocational education and training (TVET) system, failed to incorporate mechanisms for a quick response to labor market demands. A 2010 evaluation carried out to inform the development of a new strategy found that the existing secondary TVET system was not producing graduates with professional competencies needed in the labor market.4 Similarly, a 2010 World Bank employer survey of the demand for skills showed that broad vocational training and many specific higher-level cognitive and behavioral skills sought by employers, such as problem solving, initiative, and ability to organize one’s work independently, were not yet being provided by the VET sector. The Government was aware of these issues, which were reflected in the new Strategy for Vocational Education and Training.5 To support the Government, the project focused on setting the basis for transforming the VET system from the narrow occupational education to a more general and broader technical education and focus on competency-based skills relevant to the job market. 5. The innovation ecosystem of North Macedonia at the time of project appraisal was underdeveloped (virtually nonexistent), dominated by the public sector and detached from the industry. The innovation capacity of most of the domestic firms was low, as most of them were in low value added and high labor-intensive sectors using outdated technology. Human capital and financial resources for R&D and innovation for these firms were also low. The Intellectual Property Regime (IPR), while adequate in legal terms, lacked adequate institutional support system and most patents registered were of foreign origin. In most cases, the North Macedonian research sector and industry did not produce patentable research. Academia placed little priority on research, and it was predominantly scientific, rather than applied. While few examples of applied R&D, university spin-offs and centers of excellence existed, they were mostly a result of individual effort and enthusiasm, rather than an established institutional support system. Many R&D institutions lacked modern infrastructure and were not attuned to the needs of the 4 Ministry of Education and Science and European Training Foundation. 2010. Collaborative Evaluation of the Impact of the Reformed Four-Year Secondary Vocational Education. 5 Ministry of Education and Science and European Training Foundation. 2013. Strategy for Vocational Education and Training: Better Skills for a Better Tomorrow. Page 6 of 96 The World Bank Skills Development and Innovation Support Project (P128378) economy. Their R&D outputs were low in quantity and quality and there was no culture of protection and commercialization of research outputs. The private sector, scientific community, and policy makers were and remain concerned about the brain drain, particularly in the technical and engineering occupations, due to a lack of adequate employment opportunities in the country. At the time of project appraisal in 2013, the Innovation Union Scorecard 2014 assessed that the innovation performance of North Macedonia was well below the EU average.6 Relative strong weaknesses were in public-private scientific co-publications, community designs and R&D expenditures in the business sector, and community trademarks. Relative strengths were in non-R&D innovation expenditures and youth with upper secondary-level education. The assessment underscored the lack of R&D spending, academic-industry collaboration, and research commercialization. The GOM took the first steps toward strengthening the legal framework of innovation by adopting the Innovation Strategy in October 2012 and the Law on Innovation Activity in May 2013. The Law on Innovation designated the Fund for Innovation and Technological Development (FITD) as the entity to finance the Government’s interventions in innovation. The FITD was to introduce various financial instruments to support new start-ups and spin-offs, commercialization of technology, and technology transfer activities. The FITD was aimed at connecting research and the market and stimulating business innovation, while at the same time creating a source of R&D funding and jump-starting North Macedonia’s national innovation system. At the time of the appraisal, the Government was playing a critical role in stimulating economic recovery through policies that target R&D and innovation. Before project approval, the World Bank team reviewed and commented on the proposed Law on Innovation and supported in establishing the FITD, which ensured political independence and professionalism of the fund. The FITD was expected to raise awareness of the need for innovation in business growth and help fill the existing gap between R&D capabilities in the country and industry needs through converting potential technological innovation to economic development. The project focused on improving the capacity of the FITD and pilot various financial instruments and assess their impact to determine appropriate instruments for the future. 6. Rationale for World Bank assistance. The project was aligned with the Country Partnership Strategy (CPS) No.54928-MK for FY11–FY14, which was to provide selective and targeted financing and knowledge advisory services in support of faster, more inclusive, and greener economic growth. The project aimed to contribute to the first CPS outcome of ensuring faster growth by stepping up investments in education and skills. Theory of Change (Results Chain) 7. The project was not required to include a theory of change in the project design at the time of appraisal. Figure 1 describes the theory of change for the project that has been created for this Implementation Completion and Results Report (ICR), based on the Project Appraisal Document (PAD) and interviews with the World Bank project team and clients. The project had four objectives: (a) improve transparency of resource allocation in higher education, (b) promote accountability in higher education, (c) enhance the relevance of secondary technical vocational education, and (d) support innovation capacity in the Republic of North Macedonia. These goals were achieved through four distinct interventions. The cumulative impact of the project was envisioned to support the Government’s long- term strategy of nurturing more productive human capital, development of a competitive and innovative economy, and stronger integration with the EU. The project was designed to support two distinct sectors 6 European Commission Innovation Union Scoreboard 2014. Page 7 of 96 The World Bank Skills Development and Innovation Support Project (P128378) and their lead institutions, MOES and the newly formed FITD. The MOES and FITD pursued complementary and yet distinct areas of reforms. Hence, most of the project activities were designed to separately support each sector. Interventions around the National Technology Transfer Office (NTTO) and participation of the academia in innovation subprojects were envisioned as cross-sectoral. The nature of the project was systemic reforms and institutional development. Therefore, assumptions between activities to outputs in the theory of change were timely approvals and adoptions of new institutions, rulebooks, funding instruments, occupation standards, and curricula. Figure 1. Theory of Change Project Development Objectives (PDOs) 8. The Project Development Objective (PDO) was to improve transparency of resource allocation and promote accountability in higher education, enhance the relevance of secondary technical vocational education, and support innovation capacity in the Republic of North Macedonia. Key Expected Outcomes and Outcome Indicators 9. The achievement of PDO was measured by the following indicators: Page 8 of 96 The World Bank Skills Development and Innovation Support Project (P128378) • Improved transparency of resource allocation in higher education was measured by o 80% of public universities receiving financing based on a new, transparent funding model. • Improved promotion of accountability in higher education was measured by o 80% of public universities accredited, utilizing new quality assurance and accreditation measures developed in accordance with Bologna-defined European Union (EU) norms and practices. • Enhanced relevance of Secondary TVET was measured by o 30% increase in number of secondary TVET students benefiting from practical training in small and medium-size enterprises (SME) and large-sized firms. • Improved innovation capacity in North Macedonia was measured by o Share of private funding mobilized as a percentage of FITD investments in innovation activities and o Percentage of beneficiaries that sign collaborative agreements between firms and academia. Components Component 1: Improving Transparency of Higher Education (US$4 million) 10. This component has three main sets of activities, all targeted toward improving both the transparency and efficiency of the quality assurance and finance mechanisms in the higher education sector. The details of the subcomponents are summarized below. • Strengthen institutions to improve quality assurance mechanisms in higher education through (a) administrative capacity development of the Higher Education Accreditation and Evaluation Board (HEAEB)7 and other key players managing quality assurance activities, (b) funding for external institutional evaluations by foreign experts for higher education, and (c) upgrading the Education Management Information System (EMIS). • Improve transparency and accountability of resource allocation in higher education through (a) assessing and identifying appropriate funding model for the country, (b) designing and implementing a performance-based funding model, (c) identifying internal performance indicators and Results Framework for monitoring and evaluation (M&E), and (d) rolling out a new funding model. The component also conducts communication campaigns to receive support for the new funding model. • Establish a centralized National Technology Transfer Office (NTTO) to foster mission- oriented research and university-industry collaboration. This subcomponent was to provide 7Due to the change in the Law of Higher Education, which was adopted by the Macedonian Parliament in May 2018, the HEAEB evolved into establishing the National Agency for Quality Assurance (NAQA). Page 9 of 96 The World Bank Skills Development and Innovation Support Project (P128378) technical assistance for establishment and operationalization of the NTTO and would also finance the salaries of experts and the purchase of patent and market intelligence databases. Component 2: Modernization of Secondary Technical Vocational Education and Training (US$4.5 million) 11. This component has two subcomponents to build a foundation for a modern and transparent secondary TVET system, which broadens the scope of secondary TVET from the narrow occupational education to a more general and broader technical education to increase flexibility. The activities aim at delaying tracking of students into vocational pathway, increase relevance of the standards and curriculum for the labor market, and improve school-industry collaboration mechanisms through the following methods: • Improving the quality and labor market relevance of the TVET curriculum and program. The project aimed to finance (a) development of an overarching model for the transformation of the secondary TVET provision, (b) assessment and restructuring proposal of the secondary TVET network, (c) training program development and rollout for teachers and other staff, and (d) needs assessment and acquisition of equipment for school-based practical training. • Providing school grants to selected TVET school to improve the labor market relevance of the program. The project finances designing and implementation of a School Grant Program and tests the mechanism at selected TVET schools. The grant is provided to the TVET schools and firms to facilitate practical training opportunity by the industry. Component 3: Improving the Innovative Capacity of Enterprises and Collaboration with Research Organizations (US$12.94 million) 12. This component supports the efforts of the North Macedonia FITD to build its institutional capacity and pilot financial instruments to support innovation and technological development of the enterprises. In particular, the component supports the following: • Institutional capacity development of the FITD through designing the programs, strategy, operations, and procedures of the fund; training of the staff; selecting the investment committee and peer reviewers; providing mentoring and training for the fund’s beneficiary enterprises; and developing and implementing marketing and communication strategy of the fund. • Piloting four financial instruments to be delivered by the FITD, which include (a) an accelerator; (b) proof of concept innovation mini grants; (c) commercialization matching grants and loans; and (d) sector-specific grants in technology development and technology absorption projects for new or improved technology, products, and processes. The impact of these instruments was to be evaluated. Page 10 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Component 4: Project Management and Monitoring and Evaluation (US$2.5 million) 13. This component aimed to support two subcomponents to ensure effective administration and implementation of the project and set up an effective M&E system. The project was to finance the staffing and consultancy needed for a Project Management Unit (PMU) and technical assistance and training for developing and implementing M&E tools, surveys and studies, impact evaluation, and skills observatory. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDOs and Outcome Targets 14. The PDOs were not revised; however, one outcome target was revised based on the revised PDO indicator. Revised PDO Indicators 15. While the PDOs were not changed, two out of five PDO indicators and six 8 out of nineteen intermediate results indicators were revised during the first restructuring on March 1, 2019. One PDO indicator was revised to reflect the legal changes in the higher education sector. Another PDO indicator on TVET was revised to improve the accuracy of results measurement. The citizen engagement indicator was also added to the intermediate results indicators. The revised indicators and rationale for changes are summarized in table 1. In addition, the target of one of the intermediate results indicators was modified by mistake because the team started to report in March 2014 on the Implementation Status and Results Report (ISR). 8In the restructuring paper, only five intermediate results indicators were mentioned as being restructured. However, the target of one of the intermediate results indicators was also changed during the restructuring. Thus, the ICR counted the number of modified intermediate results indicators as six. Page 11 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Table 1. Revised PDO Indicators and Intermediate Indicators Before Restructuring After Restructuring Rationale for Changes PDO Indicators Original Target at PDO Indicators Revised Target at Appraisal (2013) Restructuring (2019) % of public universities 80 % of study programs of 80 Due to the new Law on Higher Education adopted by the accredited, utilizing new public universities Macedonian Parliament in May 2018, the PDO indicator quality assurance and accredited/reaccredited, needed to be changed to meet the new requirements. accreditation measures utilizing new quality developed in accordance assurance and The new law made the quality assurance and accreditation with Bologna defined EU accreditation measures process more comprehensive. Therefore, increased efforts norms and practices developed in were required to assess each study program compared to accordance with before. In addition, a replacement of the HEAEB with the AQA Bologna defined EU was required by May 2019. This change of quality assurance norms and practices, by and accreditation body made the quality assurance and the newly established accreditation process more rigorous and transparent as the Agency for Quality evaluators in the AQA included not only professors, but also Assurance (AQA) private sector representatives and students. Also, the AQA was an independent entity whereas the HEAEB operated under the MOES. The impacts of the new law on the PDO indicator include the following: (a) expanded scope of work as more rigorous assessment was required, (b) the existing universities could not be accredited again because they already had valid accreditation, and (c) institutional reaccreditation became not mandatory while study programs needed to be reaccredited every 5 years. % increase in number of 30 % of secondary TVET 40 As the student enrollment decreased each year, the original secondary TVET students students benefiting indicator did not work properly to measure the annual change. benefiting from practical from practical training in Therefore, the team changed the unit of measurement from training in SME and SME and large-sized the annual increase in the number of students benefitting large-sized firms firms from the practical training to the proportion of students benefitting from the practical training each year. Page 12 of 99 The World Bank Skills Development and Innovation Support Project (P128378) Before Restructuring After Restructuring Rationale for Changes PDO Indicators Original Target at PDO Indicators Revised Target at Appraisal (2013) Restructuring (2019) There were no baseline data at appraisal for this indicator. After the project began, using the data from 2015/2016, the baseline was calculated to be 36.3% in 2018 for the restructuring of March 2019. Because the new curriculum mandated a practical training for TVET students, the team expected an uptake of the practical training from Year 3 when the new curricula were to be developed for at least 30% of the TVET schools. Thus, the original indicator set the first target as 10% in Year 3 and increased up to 30% in the final year (Year 5). However, in Year 3 (2017), only 4.45% of updated curricula were developed and 6.8% in Year 4 (2018). Considering these low curricula development rates and decreasing TVET student enrollment rates and the number of companies that could accept students for practical training, the unit of measurement of the indicator was changed and the new end target was set as 40% for Year 5 as a final year of the project. This change did not lower the ambition of the project. It stayed almost the same. Because the baseline data became available at Year 4 (2018) as 36.3% and the final year was Year 5 (2019), the team set the Year 5 target as 40%, which was about the same 10% annual increase as the original indicator. Intermediate Results Original Target at Intermediate Results Revised Target at Rationale for Changes Indicators Appraisal (2013) Indicators Restructuring (2019) Implementation of new New funding Same as original New funding model The target of this indicator was modified by mistake when the higher education funding model approved by implemented team started to write the first ISR in March 2014. The target, model the GOM therefore, became accidentally more ambitious. Board of Quality Board of Quality Agency for Quality Agency for Quality The new Law in Higher Education stipulated formation of the Assurance and Assurance of HE Assurance for HE Assurance of HE AQA, which made the quality assurance and accreditation Accreditation achieves achieves achieves affiliate process more comprehensive, rigorous, and transparent as the Page 13 of 99 The World Bank Skills Development and Innovation Support Project (P128378) Before Restructuring After Restructuring Rationale for Changes PDO Indicators Original Target at PDO Indicators Revised Target at Appraisal (2013) Restructuring (2019) membership in the membership status achieves affiliate membership status new entity was independent from the MOES and had private European Association for in ENQA membership in ENQA in ENQA sector representatives and students as evaluators in addition Quality Assurance to professors. According to the ENQA membership criteria, a (ENQA) in Higher newly established agency should have been operational for at Education (HE) least 2 years and should have completed at least 5 review reports to become a member in addition to meeting all the eligibility criteria for the ENQA membership. While the AQA already conducted more than 5 review reports and met other criteria, the agency would not be able to meet the 2-year operational criteria due to the project implementation period. Thus, the target was changed to the ‘affiliate membership status’. Because the establishment of the AQA itself was harder than achieving the membership through the existing agency, the change in target did not seem to change the ambition of the project. Intellectual property- Deleted These two indicators were to be achieved only after the NTTO related cases handled for was established and became operational. However, due to an industry as measured by absence of ownership of the NTTO, even the preparatory number of patent activity for launching the NTTO was conducted, and the NTTO landscape analysis (PA), was not established by the time of restructuring. While the patents filed (P) and NTTO was to be established within the university, during the technology in-licensing midterm review (MTR), the FITD proposed to host the NTTO (importation of within the FITD. However, after the director of the FITD technology) changed, the issue of the NTTO was not passed on to the new Domestic inventions by Deleted director well enough to continue the discussion. Thus, at the Research Training time of restructuring, it was still unclear who would be the Institutions (RTIs) owner of the NTTO, and it was unlikely that this indicator was commercialized as to be achieved before the project closing date. The only measured by number of indicator that could be achieved related to the NTTO was the patents filed(P), establishment of the NTTO and the MOES was committed to licenses(L), and contract leave this output indicator. research (C) Page 14 of 99 The World Bank Skills Development and Innovation Support Project (P128378) Before Restructuring After Restructuring Rationale for Changes PDO Indicators Original Target at PDO Indicators Revised Target at Appraisal (2013) Restructuring (2019) Share of updated TVET Deleted Because all secondary TVET school programs were updated, secondary school there was no need to distinguish whether the TVET secondary programs targeting key school programs targeting key sectors of the local economy sectors of the local were updated or not. Therefore, the team deleted this economy indicator. Number of beneficiary 30 Same as original 55 The target was revised to be higher. However, the firms to introduce restructuring paper did not specifically mention this change new/improved products and the reason for a change in the main text. The interview with the team member revealed that it was most likely a mistake. n.a. n.a. (Added) 60 The indicator was added as a corporate requirement citizen Percentage of students engagement indicator. The satisfaction survey aimed to assess that participated in the students’ satisfaction regarding their engagement during the Grant Program satisfied grant preparation. with their engagement during grant preparation Page 15 of 99 The World Bank Skills Development and Innovation Support Project (P128378) Revised Components 16. None of the project components were revised during the project. Other Changes 17. Project extensions. The project closing date was extended twice for a cumulative addition of 23 months (until April 30, 2021) to allow for the completion of most of the project activities. During the first restructuring on March 1, 2019, following the MTR in September 2017, additional 13 months of extension of the project closing date was requested. The project end date was extended from May 31, 2019, to June 30, 2020. The extension aimed at completing the following activities: (a) finalization of the procurement of the equipment for the TVET schools, (b) support in establishing and initial capacity building to the new National Agency for Quality Assurance (NAQA) for Higher Education and introducing the new model for higher education financing, (c) design and operationalization of the Higher Education Management Information System (HEMIS), (d) establishment of the NTTO, (e) full implementation of the FITD’s innovation funding instruments, and (f) strengthening of the citizen engagement aspects of the project. 18. The second restructuring resulted in a 10-month extension of the project closing date, from June 30, 2020, to April 30, 2021. The second extension was approved in May 27, 2020, and aimed at achieving the following activities: (a) completion of the implementation of the third round of the TVET school-industry collaboration Grant Program interrupted due to the closure of schools during the COVID- 19 pandemic; (b) capacity building for the staff of the newly established NAQA for Higher Education to use the new quality assurance measures harmonized with Bologna-defined norms as the establishment of the AQA was not the original scope of the project; and (c) full utilization of the allocated funds for innovation instruments at the FITD, which was delayed due to a broken supply chain caused by the pandemic. 19. Fund reallocation. During the second restructuring, there was a reallocation of funds. The reallocation of EUR 0.14 million was requested in November 2020 by the Ministry of Finance to shift the funding from Category 2 School Grants to Category 1 Goods, Works, non-consulting services, consultants’ services, Training, and Operating Costs. After implementing three rounds of call for proposals for the secondary TVET Grant Program, which resulted in awarding 41 grants in total, there was an unallocated amount of EUR 0.14 million. This fund was reallocated to procure equipment for the chemistry and textile sectors to contribute to the improvement of practical training in TVET secondary education. Rationale for Changes and Their Implication on the Original Theory of Change 20. All the changes discussed above have been formally processed through restructuring. Two extensions provided enough time for the project to achieve or exceed various targets of the indicators. 21. A rationale for the first restructuring was a delay caused by the political crisis and a long period of political transitions within the MOES. Since the project began, there was a new Minister of Education and Science almost every year. Furthermore, there were a series of student protests between January and April 2015 against the amendments to the Law of Higher Education. In 2018, a new law of Higher Education was adopted, which caused some changes in implementation arrangements in the higher education sector. This law stipulated establishment of an AQA, responsible for accreditation and institutional evaluation of the higher education institutions (HEIs) as well as a National Council for Higher Page 16 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Education and Research (National Council), which has a mandate to propose a rulebook on the measures and criteria for the financing of higher education. The AQA replaced the HEAEB, which was operating under the MOES. The AQA was an independent agency which made the accreditation and quality assurance process more comprehensive and rigorous by expanding the evaluators from only professors to private sector representatives and students. These events affected two major activities in higher education: (a) adoption of the secondary legislation for quality assurance measures and (b) the new funding model for the higher education sector. Also, political transitions delayed the implementation of other activities. Restructuring the PDO and intermediate results indicators was necessary to reflect the legal changes in higher education, change the unit of measurement to properly measure the progress being made, and adjust the activities according to the project progress. The inclusion of a citizen engagement indicator also contributed to increase the understanding of the satisfaction level of students through project intervention. The only change that had some implication on the original theory of change was the deletion of output indicators related to the establishment of the NTTO. 22. A rationale for the second restructuring was the implementation delay caused by the COVID-19 pandemic. Due to the pandemic, TVET schools, including practical training programs, were closed between March and June 2020. From September 2020 to June 2021, TVET schools resumed with a blended teaching and learning pedagogy. Practical training was mandatory during this period. However, small firms did not have the capacity to accept students; thus, the Government worked with large companies to provide additional practical training for students with safety protocols in place. The impact of the pandemic on the FITD includes delayed royalty payment, disbursement, and implementation of technology extension instruments due to a broken supply chain. To make up for this delay, the reallocation of funds and extension of the closing date by 10 months were implemented during the second restructuring. These changes had no implication on the theory of change. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating Rating: High 23. Relevance of PDO is High. The objectives of the Skills Development and Innovation Support Project were relevant at the time of project appraisal and remained equally relevant at the time of its completion. At the time of project appraisal, the project was developed within the overarching framework of the GOM’s work program for 2011–2015 and South Eastern Europe 2020 strategy. The project objectives were aligned with the CPS for FY11–FY14, which proposed to provide selective and targeted financing and knowledge advisory services in support of faster, more inclusive, and greener economic growth. The project was designed to contribute to the first CPS outcome of ensuring faster growth by stepping up investments in education and skills. During the second cycle of the CPS for FY15–FY18, the project was still relevant. This CPS focused on ‘growth and competitiveness’ and ‘skills and inclusion’ as two main pillars. The project contributed directly to the CPS Outcome 2 on ‘better conditions for private investment and links to FDI’ and Outcome 5 on ‘tertiary and vocational education system more focused on skills for labor market’. Page 17 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 24. At the time of the project completion in April 2021, the project objectives were still relevant to the new Country Partnership Framework (CPF) for FY19–FY23. The project was relevant to the two focus areas of the CPF and two objectives: (a) export-led growth: improve the environment for a competitive private sector and (b) inclusive growth: expand skills and opportunities for the most vulnerable. The first focus area aims to remove some of the bottlenecks that prevent the emergence of a dynamic and competitive private sector by improving connectivity and access to markets (the first CPF objective) through activities that support ‘hard’ and ‘soft’ trade connectivity and strengthen local firms’ technological uptake, innovation, access to finance, and capacities. Through multiple funding instruments and capacity development offered by the FITD, the project was contributing to strengthening of local firm’s technological uptake, innovation, access to finance, and capacities. The second focus area supports the development of human capital and skills to boost labor productivity and encourage more inclusive labor market participation. The CPF is also aimed at improving the quality and relevance of education (the second CPF objective) and the access to and quality of social services (the third CPF objective). The project supported the improvement of the quality assurance and higher education financing reform, as well as modernization of the secondary TVET curriculum and occupational standards and increasing of school- industry collaboration. These interventions were all relevant to the second focus area of the latest CPF. 25. In addition, the project was relevant to the Government policies at the time of the project completion. First, the Education Strategy for 2018–2025 for the TVET sector sets three out of four priority areas that are relevant to the project. These relevant priority areas include (a) harmonizing VET with the labor market needs, (b) improving learning environment and quality of vocational education and training, and (c) improving capacities of human resources at the TVET schools. The same strategy sets the relevant priorities for higher education, research, and innovation for the project. The following three out of five priority areas are relevant to the project: (a) ensuring quality and effectiveness of higher education in accordance with European good practices, (b) improving management and funding for higher education, and (c) supporting research and innovation. Moreover, under the MOES and the Ministry of Economy, the EU-led Smart Specialization Strategy 9 development began in March 2018. The mapping of economic, innovative, and scientific potential in North Macedonia for the development of a Smart Specialization Strategy was finalized in 2021. This policy is a process that brings together business, sciences, and governments and identifies specialization for the country to provide focused investment and entrepreneurial support. The FITD will take a leading role in implementing this strategy for North Macedonia. 26. Based on the above analysis, the PDO relevance is rated High. Its relevance and importance only grew in the process of harmonization with European higher education area norms and practices considering the EU accession agenda, which is the country’s main goal.10 9 Smart Specialization is a place-based approach characterized by the identification of strategic areas for intervention based both on the analysis of the strengths and potential of the economy and on an Entrepreneurial Discovery Process (EDP) with wide stakeholder involvement. It is outward looking and embraces a broad view of innovation including, but certainly not limited to, technology-driven approaches, supported by effective monitoring mechanisms. 10 The Completion and Learning Review of the CPS 2015–2018 also confirmed that an enhanced and structured partnership with the EU was essential for the World Bank to contribute to North Macedonia’s EU accession process. Page 18 of 96 The World Bank Skills Development and Innovation Support Project (P128378) B. ACHIEVEMENT OF PDOs (EFFICACY) 27. The project had four overarching outcomes: (a) improve the transparency of resource allocation in higher education, (b) promote accountability in higher education, (c) enhance relevance of secondary TVET, and (d) support innovation capacity in the Republic of North Macedonia. By achieving these objectives, the project would contribute to the achievement of a longer-term higher-level objective of economic growth and competitive and innovative economy of the Republic of North Macedonia. There were no other development agencies or government projects that were supporting the quality assurance, higher education financing reform; four-year TVET curriculum reform; and the establishment and operationalization of the FITD. Thus, without this project, the outcomes would not have been achieved. Only after the FITD was established and became operational with four funding instruments with the World Bank financing, did other partners start to finance the FITD to expand its activities. Overall, the project fully achieved two out of four objectives (see Objectives 3 and 4 in table 2) at the time of project closing. Objective 1 is likely to be fully achieved as the new financing model is adopted and rolled out within two to three years after project closing. Objective 2 made strong progress toward achievement during the project, with full achievement expected with the adoption of the new accreditation and evaluation process by the end of December 2021, and rollout of the new quality assurance measures. Thus, the overall rating for the efficacy is Substantial based on the summary of ratings from table 2. Table 2. Summary of Efficacy Ratings PDOs Ratings Objective 1: Improve transparency of resource allocation in higher education Modest Objective 2: Promote accountability in higher education Substantial Objective 3: Enhance the relevance of secondary technical vocational education High Objective 4: Support innovation capacity in the Republic of North Macedonia High Overall Efficacy Substantial Assessment of Achievement of Each Objective/Outcome Objective 1: Improve transparency of resource allocation in higher education 28. Overall progress toward this objective is rated Modest. To improve transparency of resource allocation in higher education, the project implemented a reform in higher education financing to be performance based rather than inputs based for the five public universities that most students (85 percent) attended (Mother Teresa University was also founded in 2016, becoming the sixth public university).11 The project successfully contributed to improving the transparency of resource allocation in higher education through sharing the higher education financing data at technical workshops and received agreements from all the six participating public universities on the new financing model. The MOES approved the new funding model and sent the whole package of the new funding model to the National Council for Higher Education and Research for review and adoption. The MOES also shared the model with the Ministry of Finance for comments during the consultation process. Yet, actual adoption and implementation of the new financing model were not realized during the project period due to multiple 11Founded by the Law for the foundation of Mother Teresa University in Skopje. (’Official Gazette of the Republic of Macedonia’ no. 226 dated December 25, 2015), adopted by the Parliament of the Republic of Macedonia on December 24, 2015. Page 19 of 96 The World Bank Skills Development and Innovation Support Project (P128378) changes in the higher education sector, which delayed the entire process as detailed in the following paragraph. The National Council is planning to prepare and propose the higher education financing decree to the GOM in 2022 using all the materials prepared during the project such as funding model and implementation guidelines. After the Government adopts the decree, the new financing model will be implemented for all public universities. An innovative financing model such as performance contracting will be introduced gradually to the universities that are ready. This reform is one of the priorities in the Government Program for 2021/2022. Thus, the PDO will likely be achieved in two to three years after project completion once the reform is rolled out to public universities. Therefore, the rating of this PDO is Modest. Table 3 summarizes performance in achieving the targets set for this PDO. Table 3. Achievements on PDO and Intermediate Results (IR) Indicators for Objective 1: Improving Transparency of Resource Allocation in Higher Education Indicators Baseline Target Actual Level of Achievement % of public universities 0 80 0 0 receiving financing based on new, transparent funding model (PDO) Implementation of new The new New funding model approved by The new funding 50% for the higher education funding funding the GOM model was original target model (IR) model is • Having been modified by approved by the not mistake in the first ISR to be MOES and was sent 0% for the developed ‘New funding model to the National modified implemented’. This revised Council for Higher target target has been used since Education and then. The team was not Research for review aware of this until the ICR. and adoption. The Thus, the change was not GOM has not captured in the approved it yet. restructuring. 29. Technical workshops contributed to improving the transparency of resource allocation among HEIs and led to the development and agreement of the new funding model among public universities and the MOES. The project successfully designed, developed consensus, and received approval on the possible funding models from six public universities during the technical workshops. A thorough diagnostics of the tertiary education sector revealed income sources, per student funding allocation, governance, and resource mobilization of the leading universities. At the workshops, attended by both public and private universities and led by the MOES, this information was made available for all the universities, which was an eye-opening experience for the participating universities, because such data had never previously been made publicly available. According to the interview with the PMU, one of the universities thought that it was underfunded but learned that it was the most strongly funded, according to the data. The new financing model received high attention and support from the MOES and from other government officials. Moreover, universities expressed their support and commitment to fully participate in the design and implementation of the new funding model. In North Macedonia, the funding had historically been politically allocated, and the universities had been putting an effort in negotiating with the MOES rather than improving their research and education. Thus, the new funding model, which is Page 20 of 96 The World Bank Skills Development and Innovation Support Project (P128378) based on performance and is more transparent, is expected to shift the focus of the HEIs toward improving quality and research of education. 30. However, adoption and implementation of the funding model is still pending. Due to a prolonged constitution of the National Council, which is responsible for adopting bylaws on financing as stipulated in the 2018 Law on Higher Education, the model was not adopted by the National Council before the project completion in April 2021. As summarized in table 4, major changes in the higher education sector took place about four years after the project became effective, including the adoption of a new Law in Higher Education in 2018. Moreover, annual changes in the Minister of Education and Science affected the decision on the selection of proposed funding models a couple of times. With the current plan, the original PDO target is expected to be achieved within two to three years after project closing. Table 4. Milestones and Plans in the Higher Education Sector Time Line Milestones and Plans February 2018 A new Education Strategy and Action Plan 2018–2025 was adopted by the Government. May 2018 Macedonian Parliament adopted a new Law on Higher Education (Official Gazette 82/2018). June 2018 A set of nine rulebooks (Official Gazette no. 102/2018) was enacted for efficient implementation of part of the foreseen changes. April 2021 National Council for Higher Education and Research was established. December 2021 (planned) Work program and the action plan for the National Council for 2022–2025 would be finalized, which includes the production and rollout of the new financing model. During 2022 National Council will prepare and propose the Higher Education Financing Decree (estimated) to the GOM. During 2022–2023 The GOM adopts the decree, and the new financing model will be implemented in (estimated) all public universities. Objective 2: Promote accountability in higher education 31. Overall progress toward this objective is rated Substantial. The project promoted accountability in higher education through (a) the establishment of the new AQA, the Board for Accreditation, and the Board for Evaluation, (b) external evaluation of universities using new quality assurance measures, (c) the establishment of the skills observatory, and (d) tracer study for the HEIs. These interventions were almost all completed contributing to the promotion of accountability for results in higher education. Thus, the rating for this PDO is Substantial. Table 5 summarizes the performance in achieving the targets set for this PDO. Skills observatory data will be regularly updated in compliance with the national data privacy protection legislation. The PDO indicator target is also expected to be achieved within a year or two after project completion because the new accreditation and evaluation processes are being reviewed by the National Council and are expected to be adopted by the end of December 2021. Page 21 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Table 5. Achievements on PDO and Intermediate Results (IR) Indicators for Objective 2: Promote Accountability in Higher Education Indicators Baseline Target Actual Level of Achievement (%) % of study programs 0 80 0 0 accredited/reaccredited utilizing new quality assurance measures developed in accordance with Bologna-defined norms by the new Quality Assurance Agency (PDO) Number of public universities that 0 5 6 120 have completed an external evaluation utilizing new quality assurance measures developed in accordance with Bologna-defined EU norms and practices (IR) Agency for Quality Assurance and Board of Agency for Agency for Quality 100 Accreditation achieves affiliate Quality Quality Assurance achieved membership in ENQA (the European Assurance Assurance affiliate member Association for Quality Assurance in and achieves status in ENQA Higher Education) (IR) Accreditation affiliate member not a member status in ENQA of ENQA Skills Observatory established, Not in The information The activities for 80 providing information about TVET existence on the platform development of the and HE performance to the public being updated skills observatory are (IR) at least twice completed. The per year activities for the first tracer study are completed. Once the data privacy issue is resolved, the data on the platform will be updated regularly. 32. Accountability in higher education was promoted through the new AQA established in December 2019. While the establishment of the AQA was not in the original project design, the new Law on Higher Education adopted in 2018 stipulated the establishment of the AQA. The World Bank team also advocated for the establishment of the AQA and provided technical assistance to support the operationalization of the AQA. Replacing the previous accreditation body—HEAEB—with the AQA itself increased the accountability in higher education quality assurance and accreditation process. First, as an independent body, the AQA operated without an influence of the MOES whereas the HEAEB operated under the MOES. Second, the evaluation committee members were expanded from only professors to representatives from the private sector and students, which made the process of quality assurance and accreditation more accountable and transparent. Third, the AQA achieved the affiliate membership in ENQA and is working toward achieving full membership. Thus, the AQA is already aligned with the European standards of quality assurance and accreditation, which is more rigorous than before. Finally, Page 22 of 96 The World Bank Skills Development and Innovation Support Project (P128378) the AQA is providing more data on the processes and results related to quality assurance and accreditation on its websites, which makes the higher education sector more accountable and transparent. 33. External evaluations aligned with the European standards were conducted for all six public universities and promoted accountability in the HEIs. The assessment reports from the external evaluations made the HEIs more accountable for their own performance and the HEIs started using the evaluation to develop their own policies. Management and governance of the self-evaluation process of the HEIs improved in accordance with the best European practices. 34. The first nationwide tracer study was published in 2017, which increased the awareness among the HEIs on the importance of collecting and sharing labor market outcome data. Under the project, the first nationwide tracer study was conducted. The study collected and analyzed information on the placement rate of graduates, the adequacy and quality of training, and many other aspects that can help assess the market orientation of the educational and vocational training system. This kind of knowledge was relatively scarce in the country despite a high unemployment rate of 26.3 percent in 2015 and a high youth unemployment rate of 47 percent. The project provided a methodology for the HEIs to collect, analyze, and share labor market outcome data for better policy making and educational and training decision-making. 35. The accreditation and reaccreditation processes were not implemented before project completion, but it is expected to be implemented in 2022 after the adoption of the new rulebooks by the National Council by the end of 2021. To implement the new quality assurance measures, the National Council was responsible for the adoption of the secondary legislation, which governs the new quality assurance measures. However, the Parliament legally constituted the National Council only just before the project closing date, in April 2021. Once the National Council reviews and approves them, the accreditation and reaccreditation of the study programs will be conducted based on the new rulebook, which was already prepared through the project. Table 6 summarizes the expected plans for the implementation of the new accreditation and reaccreditation process and the estimated time line for achieving the PDO indicator. Table 6. Steps toward Achieving the PDO Indicator Time Line Activities April 2021 National Council for Higher Education and Research was established. June 2021 First constitutive session was held on June 10, 2021, and the president was elected at the meeting held on June 24, 2021. August 2021 The National Council has reviewed the rulebooks for accreditation and evaluation processes, harmonized with Bologna norms and practices, submitted by the AQA on August 12, 2021. The National Council has requested a few clarifications on the rulebook. October 2021 The AQA’s Board for Evaluation of higher education has submitted the reque sted clarifications. The Rulebook for Evaluation should be adopted by the National Council during the next planned meeting in November 2021. October–December The AQA’s Board for Accreditation of higher education is to submit the requested 2021 clarifications for the Rulebook for Accreditation between October and December 2021. December 2021 The Rulebook for Evaluation and the Rulebook for Accreditation to be adopted by the National Council. Page 23 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Time Line Activities January 2022 onward The AQA is to use the new rulebook for evaluation and accreditation and start accreditation/reaccreditation of the study programs. During 2022 All the study programs will be accredited/reaccredited using the new quality assurance measures developed in accordance with Bologna-defined norms by the AQA (PDO target). Objective 3: Enhance the relevance of secondary technical vocational education 36. Overall progress toward this objective is rated High. The project achieved the objective of enhancing the relevance of secondary technical vocational education through the curriculum reform and incentivizing the companies to provide practical training opportunities for the TVET students. Table 7 summarizes the performance in achieving the targets set for this PDO. The PDO and intermediate results indicators were all achieved and contributed to enhancement of the relevance of secondary technical vocational education. One intermediate results indicator did not achieve the target due to the timing of the data collection. At the time of the data collection, the only available data were from school years 2019 and 2020. The accurate result of this outcome would be available most likely in February 2022 using the latest data, which would most likely achieve the target. If the project could use the employment outcomes as a PDO indicator, the relevance could have been measured better; however, there were no available data at the time of appraisal to use this as an indicator. Table 7. Achievements on PDO and Intermediate Results (IR) Indicators for Objective 3: Enhance the Relevance of Secondary Technical Vocational Education Indicators Baseline Target Actual Level of Achievement (%) % of secondary TVET students benefiting 36.3 40 40.2 100 from practical training in SME and large- sized firms (PDO) TVET action plan that promotes general Not in existence Model adopted Model adopted 100% and broader technical education and competence- based learning adopted (IR) % of developed TVET occupational 0 100 100 100 standards by professional fields (IR) % of updated TVET curricula in line with 0 100 100 100 occupational standards (IR) % Increase in number of companies 0 30 16.5 55 providing practical training to secondary TVET students (IR) Manual and training materials for quality No Yes Yes 100 delivery of new curricula adopted (IR) Students benefiting from direct 0 21,000 21,346 102 interventions to enhance learning (IR) Percentage of students that participated 0 60 84.1% 140 in the Grant Program satisfied with their engagement during grant preparation (IR) Page 24 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 37. Mandatory practical training through the curriculum reform and a Grant Program incentivized small and medium-size enterprises (SMEs) and large-size firms to provide practical training for secondary TVET students and make the skills training relevant to the employers’ needs. Curriculum reforms for all the secondary TVET institutes, which include new competency-based curriculum, mandatory practical training, and broader technical education, improved the relevance of TVET. The grant to promote school-industry collaboration also succeeded in increasing the practical training opportunities for students to learn work-based skills and increase their employability. The number of companies providing practical training to secondary TVET students increased annually. However, during the COVID- 19 pandemic, many small companies closed and could not accept students. Yet, the CVET managed to transfer students to other big companies that already had agreements with the schools. Thus, while the number of companies providing practical training did not increase toward the end of the project when the pandemic hit, the students could still benefit from practical training by transferring their practice to big companies. 38. The project maximized the impact of the Grant Program, including strengthening relevance of the TVET program to the labor market demand. The project used impact evaluation effectively to select the most impactful measures and focus its investment. During the first round of grant provision for school year 2016/2017, five measures (see annex 7) were implemented and two schools per Grant Program were awarded. From school year 2018/2019, three measures were selected for a re-launch and implemented. According to the evaluation, the following two measures had the most impact on the increase in student performance, which was mainly driven by increases in grades in the subjects in which participating schools were performing badly, such as mathematics. The two most impactful measures were (a) setting up a school company within the TVET school through business guidance and mentoring (business mentoring relationship) and (b) placement of TVET students for practical training in the company (practical training in companies). The major impact from the Grant Program based on the stakeholder satisfaction survey in 201812 shows that the Grant Program had an impact on improving multi-faceted skills of students and teachers, increasing the confidence among the business sector about the TVET students, and strengthening the cooperation between the TVET schools and business. These positive impacts show that the Grant Program contributed to the achievement of the PDO on enhancing the relevance of secondary TVET. 39. Modernization of secondary TVET shifted the mind-set of students to go to the labor market and companies to build more partnerships with TVET schools. TVET graduates tended to continue studying after they graduated from the TVET schools and did not look for a job. For instance, a tracer study conducted in 2016 showed that for those who responded to the survey (11 percent of total TVET graduates, or 1,235), 70.8 percent of the TVET graduates were never employed after they graduated and 64 percent of them continued their study. Only 15.7 percent of the graduates had a job less than three months after graduation. However, a series of impact evaluations of the Grant Program conducted in school years 2016/2017, 2018/2019, and 2019/2020 showed some changes that happened to the graduates. Considering that only 4.45 percent of updated curricula were developed in 2017 and 6.8 percent in 2018, the impact of the curricula reform could be seen only toward the later years of the project implementation period. Therefore, the results from the third evaluation showed a different result. In this evaluation, more students reported that they intended to enter the labor market rather than pursuing a university degree. This result was completely opposite from the findings of the previous two years of the 12 Stakeholders’ satisfaction with the modernization reform of technical education in North Macedonia. Page 25 of 96 The World Bank Skills Development and Innovation Support Project (P128378) evaluation. Many students reported that they would try to find a job in the labor market. Other students reported that they will try to migrate to the EU countries, and only one out of five students reported willingness to continue with higher education. This result indicates that the reform in TVET contributed to achieving the longer-term objective of the project to increase the labor market outcomes. On the other hand, companies also became more ready to collaborate with the TVET schools. By participating in different subprojects, companies realized that by placing VET students in practical training, they lower the cost for recruiting employees and shorten the duration of the training. Several companies have already established a partnership with secondary schools in dual education. Objective 4: Support innovation capacity in the Republic of North Macedonia 40. Overall progress toward this objective is rated High. The project successfully established the FITD, which worked above and beyond to develop innovation capacity in the Republic of North Macedonia. The FITD tested four funding instruments during the project to promote entrepreneurship and innovation and collaborated with various start-up communities and private sector to create an innovation ecosystem in the country. The FITD also earned its prestige and was able to raise more funds after its operation. Table 8 summarizes the performance in achieving the targets set for this PDO. Almost all the targets were either fully achieved or exceeded. At the restructuring, the target of one of the intermediate results indicators (number of beneficiary firms to introduce new/improved products) increased from 30 to 55 by mistake. But the project exceeded this target. There are two intermediate results indicators whose results were not achieved. The first one is the establishment of the NTTO. This indicator was originally under the higher education section; however, at the time of the ICR, the indicator seemed to fit better under this objective. Thus, the ICR author included this indicator as a relevant indicator under this PDO. However, because the responsible entity for establishing the NTTO was never determined in the project, the NTTO was not established. Yet, even without the NTTO, the FITD used its funding instrument to stimulate university-industry partnerships to achieve the objective of the NTTO. The second intermediate results indicator is the number of companies established through the FITD accelerator program. The accelerator program has two phases. Since the establishment of the accelerator program, 62 firms benefitted from the pre-investment phase and 3 from the investment phase. Around 15 new firms are now benefitting from the investment program and are expected to be fully benefited by the end of 2021. Thus, the result of this indicator will be 18, which will exceed the target. Therefore, the efficacy rating for this PDO is High. Table 8. Achievements on PDO and Intermediate Results (IR) Indicators for Objective 4: Support Innovation Capacity in the Republic of North Macedonia Indicators Baseline Target Actual Level of Achievement (%) Share of private funding mobilized as a 0% 25% 49.53% 198 percentage of FITD investments in innovation activities (PDO) % of beneficiaries that sign collaborative 0 20 39 195 agreements between firms and academia (PDO) National TTO established and operational Not in NTTO Not in 0 (IR) existence operational existence Page 26 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Indicators Baseline Target Actual Level of Achievement (%) Share of resources dedicated to Fund for 0% 100% 100% 100 Innovation and Technology Development (FITD) instruments disbursed (IR) Number of companies established via the 0 15 3 20 FITD Accelerator program (IR) Number of beneficiary firms to introduce 0 55 256 465 new/improved products (IR) % of beneficiaries receiving training and 0 70 100 143 mentoring (IR) Number of beneficiary firms to introduce 0 4 11 275 new processes (IR) 41. The project successfully supported operationalization of the FITD to implement four funding instruments to improve innovative capacity of enterprises and make the FITD sustainable. The project provided technical support for the governance and operational structure of the FITD to be launched successfully. Along with the FITD’s successful communication campaigns to raise awareness of the fund, its budget rose from less than US$400,000 in its first year of operations to over US$18 million in 2021. The most significant budget increase of over 200 percent took place in 2018 and 2019 when the newly established Government decided to implement part of its Plan for Economic Growth targeting the micro, small, and medium enterprises (MSMEs) through the new and existing instruments of the FITD. Moreover, the fund also successfully mobilized private funding and exceeded the target in Year 4. As a source of private funding, the FITD requested at least 10–15 percent private co-financing for grants to innovative start-up companies, 30–40 percent for commercialization of innovation, 25 percent for accelerators, and 50 percent for technological extension. Surprisingly, private co-financing exceeded the required minimal amount, which showed the increase in innovation activity of the private sector. During the COVID-19 pandemic in 2020 and 2021, the FITD adjusted its instruments to facilitate faster adaptation and recovery from the impact of COVID-19 on the businesses. To date, the FITD has supported 642 innovative projects and over US$100 million in investments. The World Bank loan was the main source of financing of the FITD in the beginning but decreased to less than 30 percent from 2018 to 2020 and only 15 percent in 2021. This clearly shows the Government’s commitment to continuously support the development of innovative business and modernization of the existing ones through the FITD, even after the World Bank loan was fully disbursed. In addition, an impact evaluation 13 conducted separately from the project measured the effect of state aid on private enterprises’ efficiency and market competition, with an application on the Governmental program for subsidizing companies in North Macedonia under the Plan for Economic Growth 2018–2021. This plan had three pillars, one of which was innovation support, implemented through the FITD. The impact evaluation proved that the FITD state aid proved considerably effective. Due to the subsidy, recipients were able to increase their sales revenue, labor productivity, and investment in technology of non-tangible form. However, no more jobs or higher wages were created for the recipients. This result shows the positive effectiveness of the FITD. 13https://www.financethink.mk/wp-content/uploads/2021/04/Measuring-economic-effects-of-state-aid-granted-to-private- enterprises-in-North-Macedonia.pdf. Page 27 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 42. The FITD’s funding instruments and technical support for the companies made an impact to improve the innovative capacity of enterprises. The FITD worked above and beyond to improve innovation capacity of enterprises through technical assistance and mentoring. While initially the mentoring was provided by an international consultant funded through the loan, the FITD created its own mentorship platform with local and international mentors who would be suitable for different phases of development of the start-ups. The number of beneficiary firms to introduce new processes more than doubled the target (the target was 4 and the result was 11). All these achievements contributed to improving the innovative capacity of enterprises. 43. As a result of the FITD activities, the start-up ecosystem in North Macedonia is beginning to mature and is creating a long-term impact on developing an innovation economy of the country. The recent analysis of over 80 successful start-ups showed that over 60 percent of them applied to the FITD for support and 60 percent of their projects were financed. Those who were rejected received valuable information on their weaknesses and how to overcome them. While a few years ago, the focus of the start-ups was product development, now they are more focused on bringing those products to markets and developing business models that fit the market. This progress is reflected in the ranking of the Global Innovation Index as 57 out of 131 countries featured in the 2020 report, a significant jump from the 84th position in 2018, and getting close to the neighboring countries such as Serbia (53), Romania (46), Croatia (41), and Bulgaria (37). According to this report, relative to gross domestic product, North Macedonia’s performance in innovation is above expectations for its level of development. But there is still a long way to go as innovation inputs (ranked 46) outperform the innovation outputs (ranked 63). Finally, the FITD actively collaborated with start-up communities, private sector, and the Government to organize hackathons and support corporate innovation and launched a start-up council to develop a start-up ecosystem and culture. This effort led to strengthening the innovation capacity of all the stakeholders and creating an innovative economy in the long run. 44. Through the ‘Innovation Voucher’, industry-academia collaboration increased, and more commercialization of research took place. Out of 265 approved projects funded through the loan, 103 projects had industry-academia collaboration. It is evident that some professors created their own spin- off companies to commercialize the research. While early on, the agreements were between firms and professors as individual consultants, circumventing the universities, in 2020, the FITD introduced a new instrument called ‘Innovation Voucher’ where the firms must establish direct relationships with the universities and research institutions for product development or improvement, proof of concept, feasibility studies, technical specifications, prototyping, testing, technology validation, software development, and training and advisory services. Through this instrument, the FITD co-financed up to US$10,000 or 80 percent of the cost of the service provided to 34 companies by the universities. This additional financing from the FITD increased the revenues of universities that they can spend on improving their labs, providing additional career improvement opportunities for academic and research staff, and paying the students and professors involved in delivering those services to the companies. This change of practice paved the way for ‘institutionalizing’ the application of research and commercialization of innovation in the universities and establishment of the NTTO in the future. The NTTO, whose original aim was to convert the results of research into competitive products and processes and steer research toward industry needs, unfortunately, was not established due to a lack of ownerships from the HEIs. There was also a proposal to institute the NTTO within the FITD, but during the transition of the leadership in the FITD, this agenda was not succeeded well and the proposal was not materialized. Page 28 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 45. While the NTTO was not established, the FITD initiated another instrument to strengthen university-industry partnership. In 2021, the FITD expanded its cooperation with academia by publishing a call for establishing digital fabrication laboratories (Fab Labs) at the HEIs and scientific institutions. The call of establishing Fab Labs aims to provide young people with the spatial, material, and technical conditions for developing innovative products, services, processes, activities, or methodologies that offer practical and socially useful solutions to certain issues or problems in priority areas of science and technology. At the same time, these laboratories will be a place for cooperation between the private sector, higher education and scientific institutions, enterprises, and civil society organizations. The maximum funding per Fab Lab is US$85,000 and requires an additional 10 percent cost-share from the institution. Justification of Overall Efficacy Rating Rating: Substantial 46. The overall efficacy rating is Substantial. Among four PDOs, achievement level of each PDO was rated as follows: (a) Modest for improving transparency of resource allocation in higher education, (b) Substantial for promoting accountability in higher education, (c) High for enhancing the relevance of secondary technical vocational education, and (d) High for supporting innovation capacity in the Republic of North Macedonia. The likelihood of achieving the remaining two objectives seems considerable because there is a sound legal framework stipulating the introduction of new quality assurance mechanisms aligned with the EU best practices and introduction of the transparent financing model. In addition, with the establishment of the AQA there is an appropriate institutional home for enforcement of quality assurance in the higher education sector. Moreover, the project achieved the long-term goal especially on developing innovative economy of the country through the FITD. Thus, based on these results, the overall efficacy rating is Substantial. C. EFFICIENCY Assessment of Efficiency and Rating Rating: Substantial 47. The ICR rates efficiency of the project as Substantial because of positive economic returns and efficiencies gained at completion. The ICR economic analysis, aligning with the analysis at appraisal, reviews the program outcomes and estimates the economic returns and efficiencies produced by the components on reforming the four-year TVET and innovation. Due to a lack of data and partially achieved outcomes for the higher education component, economic analysis of higher education component is not included. The economic internal rate of return (EIRR) of the project is within the range of EIRR calculated in other similar TVET projects in the cost-benefit analysis (CBA). Moreover, the project achieved positive outcomes in efficiency through investments in innovation and an innovation ecosystem, which is difficult to measure to measure with the CBA, but significant impact was shown from the outcome indicators. There were also some efficiency gains in higher education through external evaluation and introduction of transparent quality assurance mechanism. Furthermore, the project was implemented by experienced staff members with low turnover and highly efficient in management, reallocation, and execution of budget. Considering all these elements, this result is what would be expected in the sector. Therefore, the efficiency rating of the project is Substantial. Annex 4 presents the findings of the economic analysis in Page 29 of 96 The World Bank Skills Development and Innovation Support Project (P128378) more detail. Efficiency includes both economic analysis and aspects of implementation efficiency as described below: (a) Improved efficiencies through external evaluation for the Higher Education Institutions (HEI) and establishment of the new national Agency for Quality Assurance (AQA) 48. While the higher education financing reform was not achieved, through external evaluation and establishment of the new AQA, some efficiency gain was achieved in higher education sector. The new AQA made the quality assurance process and relevant data transparent through its new website developed by the project. With a capacity development training provided to the AQA and AQA’s achievement of the affiliate member status of the ENQA, the efficiency was improved in quality.In addition, external evaluation was conducted for all six public HEIs to assess various aspects of the HEIs, including governance and institutional decision-making, internal quality assurance, teaching and learning, research, and internationalization. All the universities had common challenges, such as legal instability; low funding levels, particularly for research; and constrained autonomy levels. While it still takes time for the HEIs to take action on recommendations from the evaluation, this assessment will serve as a basis for gaining efficiency in university administration. (b) Improved efficiencies through modernizing secondary TVET 49. By providing the quality and labor market relevance of TVET programs and Grant Program to connect between TVET institutions and business communities, the project generated economic gains and efficiencies. The impact evaluation of the Grant Program 14 found the below-listed effects of Grant Programs, especially the Measure 1 program of setting up a school company within the VET school and the Measures 2 program of placing students in practical training in companies. (a) Improved student performance in grant participating schools (i) Participating in any year of the Grant Program results in increases in student performance15 by the range of 3.4 percent to 4.6 percent in the years following the implementation of the Grant program. (ii) Impact for girls is slightly larger: the grade increased by 4.3 percent for girls and 3.4 percent for boys. (b) Teachers who participated in the projects gained new knowledge and new experience through the problem-based learning program which provided teacher training for the development of more innovative pedagogies16. (c) The estimated impact would be underestimated because the grant projects are likely to generate longer-term impact after the project cycle given that the estimated effects of the Grant program persisted over the measurable years (up to 5 years) after the implementation. 14Armand, Alex, “Grant program for enhancing collaboration between schools and business community”, evaluation report (the third year) 15 The average student grade was 3.16 out of 5 in the year before the implementation of the Grant program. 16 Page 30 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 50. The CBA shows that the modernizing TVET component generated economically viable results with the net present value (NPV) ranging from Euro 6.3 million to Euro 12.2 million and EIRR of 7 to 10 percent. At appraisal, the CBA was not conducted, while the recently completed skills development projects estimated the EIRR ranging from 10 to 18 percent. This project is within the range of comparative project in terms of the EIRR. The CBA includes the sensitivity analysis with different sets of assumed impacts on intake and quality premium. The analysis assumes that the annual increase in the number of VET students ranges from 017 to 1 percent and that the quality premium ranges from 0.2 percent to 0.3 percent of monthly earnings of the TVET graduates. The parameters and assumptions used for the analysis are described in annex 4. (c) Improved efficiencies through financing of innovation activities 51. The FITD improved the efficiencies through increasing private innovation investment activities and contributed to the improvement of country’s innovation ecosystem. Most of the target outcomes related to innovation interventions are accomplished by a wide margin and the impact of the FITD on financing is positive and significant in terms of increasing total innovation investment at the firm level according to the impact evaluation study.18 • Private funding financed the innovation activities of the FITD up to 49.53 percent, which exceeds the end target of 25 percent. • About 39 percent of beneficiary firms signed collaborative agreements with academia. • A total of 256 beneficiary firms introduced new or improved products and 11 firms introduced new innovation/business processes, which surpasses the end target of 55 and 4, respectively. • Receiving the FITD awards led to an increase in the total innovation investment by 7.5 percent and investment in the FITD project by 3.8 percent19 at the firm level. 52. At the country level, compared to 2014, North Macedonia’s innovation environment improved more than the EU in various aspects, which would be partially contributed by the FITD. In 2014, the innovation index score in the country was 33 percent of the EU and, in 2021, it is 42 percent, in particular, knowledge-intensive service exports rose from 21.0 percent to 27.2 percent relative to the EU in 2014 and the sales of innovative products from 18.0 percent to 20.7 percent. In addition, other subdimensions relevant to the FITD, even though still significantly below the EU average, are beginning to show improvement: the Government’s support for business R&D sub-index increased from 0.2 percent in 2014 to 3.4 percent relative to the EU, patent applications increased from 0 in 2014 to 8.9 percent relative to the EU in 2021, and public-private co-publications increased from 19.0 percent to 30.4 percent relative to the EU. The human resource capital necessary for innovation significantly improved as foreign doctorate students increased from 24 percent compared to the EU in 2014 to 197.6 percent in 2021, population with tertiary education rose from 35.5 percent in 2014 to 76.9 percent of the EU average in 2021, and job-to- job mobility of science and technology employees rose from 23.1 percent in 2014 to 78.6 percent in 2021. 17 The actual trend shows that the number of secondary VET enrollment has been decreasing. This is discussed more in Annex 4. 18 Carneiro, Pendro, and A. Toppeta. (2021) End line Report: Fund for Innovation and Technology Development (FITD) Grant (2015–2018 calls). 19 This estimate is not statistically significant due to small sample sizes at the firm level. Page 31 of 96 The World Bank Skills Development and Innovation Support Project (P128378) (d) Efficiency in implementation 53. The project was implemented relatively efficiently. First, fiduciary members of the PMU staff already had experience with World Bank projects in the past. Thus, procurement, financial management, and environmental and social safeguards were all implemented smoothly with no issues throughout project implementation. The only inefficiency related to human resources was a frequent change of the Minister of Education and Science. The PMU and the World Bank team had to brief the new ministers, getting their buy-ins on the already-agreed reforms, which took extra time and effort. 54. Second, the project was restructured twice and extended for additional 23 months. Project implementation cost increased as a result; however, all the PDO indicators for the TVET and innovation components exceeded the target and the project disbursed up to 99.9 percent for the relevant activities. A key agent, the National Council for Higher Education and Research was established just before the project closing date to move the project closer to achieving the relevant PDO targets. Some savings from the Grant Program of the TVET were reallocated to increase the investment on the equipment purchase for the TVET schools through restructuring to efficiently use the budget. Thus, considering all the benefits, the extensions did not necessarily reduce the impact on efficiency. 55. Third, most of the activities were implemented according to the budget estimate. Even with the extension, the PMU did not have any constraints in the project management budget. Through selecting contracts with a lower bid, negotiating with the contractor, and switching the contractor from the firm to an individual, most of the contracts were made within the budget. Table 7.2 summarizes the major activities of each component with estimated and actual spending per contract. D. JUSTIFICATION OF OVERALL OUTCOME RATING 56. The overall outcome rating is Moderately Satisfactory based on the ratings of relevance (High), efficacy (Substantial), and efficiency (Substantial). While the achievement of the first PDO on improving transparency of resource allocation was Modest and the second PDO on promoting accountability in higher education was Substantial, the ICR team considers that both PDOs will most likely be achieved within one to three years after the project completion date. For both cases, all the preparation materials and training for introducing new financing models and new accreditation and evaluation processes in higher education are ready. The National Council is also in the process of adopting these reforms. Thus, once they are adopted, implementation can take place immediately to achieve the respective PDOs. Moreover, almost all the relevant activities to promote accountability in higher education were successfully implemented to achieve the PDO. Therefore, the overall rating is Moderately Satisfactory. E. OTHER OUTCOMES AND IMPACTS Gender 57. While the project identified high unemployment and low labor force participation especially among women, the project did not specifically address the gender gaps through its interventions. However, the project tracked some gender data on the implementation agencies, recipients of the innovation fund, and the number of enrollment and graduates for TVET institutions. Overall, there were more women working at the project implementation agencies as shown in table 9. Page 32 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Table 9. Gender Ratio of Staff at Implementation Agencies Implementation Agencies Male Female PMU 35.3% (6) 64.7% (11) AQA 37.5% (3) 62.5% (5) FITD 36.5% (19) 63.5% (33) 58. Among 544 recipients of the FITD innovation fund, 345 (63 percent) were companies owned/managed by men and 199 (37 percent) were owned/managed by women. This is significantly better than the gender ratio of employers in the country (26.7 percent of all employers were women in 2019 but this fell to 21.2 percent in 2020 most likely due to the pandemic) and self-employed people (24.4 percent of all self-employed people are women). This can be attributed to the higher percentage of female university graduates (58 percent) and the selection criteria that stimulated female participation in the call for proposals. 59. The proportion of female students who were enrolled and graduated from the four-year TVET institutions has been around 45 percent since 2011 and increased to about 47 percent from 2018 onward. It is hard to assume that the curriculum reform contributed to the increase in the number of female student enrollments and graduation rates, but there was a slight increase since 2018. This might be because of the national campaign to promote the new TVET education, financed by the project proceeds. Among the secondary TVET institutions that received the Grant Program, the proportion of the enrollment of the female students ranged from 0 to 78 percent and the average was 35 percent. For the number of graduates, the proportion of female students ranged from 0 to 82 percent and the average was 31 percent. Moreover, 9 out of 41 schools (22 percent) had more than 50 percent of the enrolled students who were female, and 6 out of 41 schools (15 percent) had more than 50 percent of female graduates. Institutional Strengthening 60. The project had a substantial impact on institutional development. All the components were designed to support the institutional strengthening of the MOES, HEIs, Quality Assurance Agency, Center for Vocation Education and Training (CVET), secondary TVET schools, and FITD. Key institutional strengthening activities include (a) external evaluation for the HEIs; (b) technical assistance for the quality assurance agency; (c) conceptualization of the new TVET reform, occupational standard development, curriculum reform, and training of the CVET advisers; (d) training of teachers on the new curriculum and the purchase of new tools and machines for secondary TVET; and (e) operationalization of the FITD with new financial instruments. Upgrade of the EMIS and setting up of the labor market observatory helped the MOES use more data for decision-making and policy development. 61. The Grant Program built the capacity of the secondary TVET institutions to continuously provide better educational and training programs for students. Based on focus group discussions and interviews conducted during the third round of evaluation, the Grant Program had multiple direct and indirect benefits. The direct benefits are that the school obtained a new service for their school-run business, which could generate additional income for future investments in the school infrastructure or in the establishment of other businesses. In addition, they gained the know-how received from the external company, so that the school can potentially provide services at the level offered by other competitors in the market. The school staff and the students improved their vocational skills through work-based Page 33 of 96 The World Bank Skills Development and Innovation Support Project (P128378) training, and teaching capacity was also improved. The indirect benefits included creation of a new sustainable company, which could be used by the students for practical training. In the long run, it can generate a significant influence given the difficulties of students in obtaining practical training in external companies. Improved training would also increase the confidence of the business sector in hiring students from this school. 62. The FITD raised the bar for government institutions for accessibility by introducing a fully online application process, transparency in government spending by publicly disclosing the names of the recipients, and the value and the purpose of the grant. After media pointed out the links of some grant recipient with prominent political figures, the FITD worked with Transparency International, a nonprofit organization that focuses on anti-corruption, to assess the potential conflicts of interest and exclude the applicants where it was determined as high. In response to the World Bank’s advice, the FITD introduced a grievance mechanism that allows people to submit anonymous and specified complaints and suggestions online and established a procedure for responding to them. The FITD is the only government institution that tries to measure the impact of its MSME support programs, by analyzing the effects on new employments, salary levels, and refunds to the Government budget through taxes and pension and health contributions of assisted companies. Mobilizing Private Sector Financing 63. The project mobilized private sector financing through the FITD. Of the total amount invested in innovation and technological improving activities supported by the FITD by the end of the project, 49.5 percent was co-financed by the private sector (that is, beneficiaries). Because the nature of the FITD was to become an independent fund in the future, it was designed to mobilize private sector financing for sustainability through royalties from successful beneficiaries, but it will take some time until they accumulate. Furthermore, with the World Bank’s support (a complementary technical assistance), the FITD is preparing a prefeasibility study for establishing a hybrid (public-private) MSME equity investment fund, which is expected to leverage about US$20 million from domestic and international private and institutional investors and invest them in start-ups that are ready to expand on global markets. Meanwhile, the FITD is cultivating partnerships with local banks and corporations to provide access to loans and clients for supported start-ups. Poverty Reduction and Shared Prosperity 64. The project did not specifically measure the impact on the low-income population. However, TVET is important as a tool for inclusiveness and poverty alleviation because it tends to enroll students from lower socioeconomic backgrounds, improving their employment and earnings outcomes. Through the modernization of secondary TVET, it is expected that more graduates will be able to get jobs. There was an impact on increasing the number of employees for the companies that received the FITD funds. These companies are predominantly in the hi-tech sectors that require well-educated, better-earning employees. Furthermore, adoption of new technologies and innovation in traditional sectors such as food processing, apparel, and furniture production help them survive in the market, increase worker productivity, and create additional value, thus improving the prosperity of employees in labor-intensive sectors as well. Thus, the project contributed to shared prosperity. Page 34 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Other Unintended Outcomes and Impacts 65. Through the upgrade of the EMIS, the PMU had an opportunity to go through all the data in the existing EMIS, which had discrepancies. While it took more time and effort to validate all the data discrepancies, this exercise made the education data more accurate and enabled the project team to carry out the tracer study of TVET graduates. 66. Even though the FITD did not have an explicit goal to support environment-friendly projects, a good number of its beneficiaries are contributing to greening the economy by recycling glass and apparel waste, placing new thermal insulating construction materials, using solar energy in the production process, and installing more energy-efficient and less polluting equipment. These examples pave the way for future green economic development in line with the EU priorities. 67. Although most of the Grant Program did not target specific nationalities and those who are marginalized, in 2016/2017, a special school with hearing-impaired students (Partenija Zografski) was included in the Grant Program. A total of 15 students (ages 17–19 years) were included in the program: 6 were Macedonians, 8 were Roma, and 1 was Albanian. These 15 students represent 38 percent of all students in this school. The school received a grant for Measure 2: Placement of students in practical training in companies. The students attended practical training in Energo Sistem DOO, a private company in the energy sector. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 68. The project had rather ambitious objectives, especially for the secondary TVET and innovation component, but strong commitment of the Government existed to achieve them. At the time of project preparation, the World Bank team already had an engagement with the MOES on higher education. Since 2010, the World Bank team had been providing technical assistance on quality assurance and higher education financing reforms. There had been pilot studies on cost-per-student funding options and alternative models for financing the entire higher education sector to promote efficiencies and innovation in research and teaching. Thus, the objectives set for higher education seemed realistic. On the other hand, secondary TVET had not had any reform for more than two decades and the relevant law was from 2006, which was outdated. Thus, modernization of secondary TVET was expected to be ambitious, and there was a risk of not getting commitments from the stakeholders and the Government. Likewise, the innovation component was also expected to be challenging because only the basic law was existing and the FITD had not been launched yet. There was a risk of not having enough companies that could absorb the innovation grant. 69. The simple design through an alignment of the activities could have been improved. All the project components were simply designed, incorporating the lessons learned from the neighbor countries and global evidence. The sequence of the activities was also clear. However, the establishment of the NTTO seemed to link better with the innovation outcomes rather than the higher education outcomes. The objectives of the higher education component only included improving transparency of resource allocation and promoting accountability in higher education. The establishment of the NTTO was to contribute toward academic-industry collaborations and commercialization of research. Because the Page 35 of 96 The World Bank Skills Development and Innovation Support Project (P128378) innovation component aimed at supporting the innovation capacity of North Macedonia, which included increased number of academia-industry collaborations, the establishment of the NTTO could have been better aligned with the innovation component. 70. Clear PDOs, Results Framework indicators, and appropriate plan for monitoring. The project had a clear compound PDO statement. The Results Framework indicators were also designed to measure the progress toward the achievement of the PDO appropriately. Because most of the activities started from scratch, all the baseline data were zero. One of the indicator’s baseline was to be collected after the project started. The project planned to upgrade the EMIS and introduce skills observatory to collect relevant data efficiently and accurately. Surveys and impact evaluations were planned to measure the progress and impact of the interventions appropriately. 71. Appropriate selection of stakeholders and readiness for implementation. As the project focused on the systemic reform, multiple stakeholders needed to be engaged to build consensus. At the project preparation stage, the project established multiple stakeholder groups and started engaging with them. For instance, the project established an Interministerial Council on Innovation and a working group, which comprised technical experts from the MOES, Ministry of Labor and Social Protection, Ministry of Finance, Ministry of Economy, Cabinet of the Deputy Prime Minister for Economic Affairs, and VET center for the project. The working group met regularly with the World Bank team to discuss ongoing policy development and preparation issues. This engagement helped implement the reform-related activities without oppositions. 72. The project adequately identified the risks and risk mitigation measures. The project identified appropriate risks and took risk mitigation measures during the project preparation phase. For instance, to mitigate the delay due to the upcoming election, the Project Steering Committee was to be developed rather quickly before any political change took place. Moreover, the project identified the weakness of capacity for some implementation agencies, including the FITD. Thus, the project design included a strong governance of the FITD and capacity development of the FITD staff. As a result, the FITD performed well and became a prestigious innovation fund. B. KEY FACTORS DURING IMPLEMENTATION 73. Political transitions, protests, and COVID-19 affected the progress of the project activities. The project experienced several political transitions during the project implementation period as summarized in table 10. A delay in adopting the new Law on Higher Education also affected the implementation progress. In addition, the Minister of Education and Science changed almost every year for six times since the project became effective in March 2014. Every time the minister changed, the project team had to brief the minister and receive the buy-in for the project. For instance, after the higher education financing model was agreed among stakeholders, the new minister decided to take the opposite direction of what had been decided, which delayed the progress of the project. Furthermore, there was political interference in staffing the higher education accreditation body, which delayed the establishment of the key body for accreditation. All these factors delayed the progress of the project activities. Page 36 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Table 10. Series of Events that Affected Project Implementation Timing of Events Results Events January–April A series of student protests against The protests resulted in the withdrawal of the 2015 amendments to the Law of Higher amendments and started the dialogue between Education, which stipulated the the universities (professors and students) and introduction of state exams as mandatory the MOES regarding reforms in higher external tests for university students to be education. conducted by the MOES. December Early parliamentary elections It took almost a year to establish the new 2016 coalition Government, which assumed office in May 2017. As a result, the adoption of the new Law on Higher Education got delayed. February The New Education Strategy and a new Law The project adjusted the Results Framework 2018 on Higher Education was adopted by the indicators based on the new law. Government and the Parliament. July 2020 Early parliamentary elections The new Minister of Education and Science assumed office. Early 2020 COVID-19 pandemic The pandemic affected implementation of the Grant Program and practical training for secondary TVET as the institutions closed and small and medium companies could not absorb the students. 74. Strong coordination and engagement of the PMU. The project had a strong PMU, which had a lot of experience working with other World Bank education projects in the past. Thus, there were no major issues regarding fiduciary, safeguards, and M&E aspects of the project. Moreover, the PMU played a critical role in coordinating and engaging with the stakeholders to take on reforms. The PMU coordinated with about 6,000 people to work on some 1,400 documents, which needed to be approved by the MOES. Each curriculum required the involvement of two industry experts and teachers to revise. From school year 2019/2020, the new modular competence-based curricula for all 52 qualifications were implemented in all 76 VET schools in the country. 75. Strong commitment and leadership of the Government and stakeholders. Even with annual changes of the Ministers of Education and Science and several changes in the cabinets, in addition to student protests, there were strong, sustained commitments from the Government and stakeholders to complete the reforms throughout the project. For instance, while the original scope of the project was to update TVET secondary school programs in targeted key sectors of the local economy, all secondary TVET school programs were updated through the strong commitment of the Government and stakeholders. 76. High quality technical assistance. Positive feedback was received from both higher education and secondary TVET stakeholders on the technical assistance they received. High-quality technical assistance helped shape the foundational documents for the reforms. For instance, the curriculum reform for secondary TVET required sector analysis, methodologies for developing occupational standards, and implementation plans. Experts provided relevant and good examples of other countries. At the same time, they were flexible in adjusting the contexts to North Macedonia. A collaborative method was taken between the VET centers, MOES, and experts in terms of how to adopt and contextualize the reform. Page 37 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Because the law for TVET was from 2006, it took some time to harmonize and align the terms and concepts to be modernized. Skilled experts contributed to overcoming the issues. In higher education, through high- quality technical assistance, the World Bank supported the MOES to develop a new higher education financing model, establish a new AQA, and develop a new accreditation and evaluation processes. This also applies to the FITD, where the World Bank and the PMU provided high-quality experts to support the designing of instruments and mentoring of start-ups, which was crucial to get the most benefit of the invested funding. Follow-up advise was also provided in tweaking the instruments to improve performance based on lessons learned from the first implementation iterations and changing circumstances such as the COVID-19 pandemic or the maturing of the start-up ecosystem. 77. Strong M&E with impact evaluations. The project had strong M&E and invested in multiple data collection systems. The project introduced information technology (IT) systems for the labor market observatory, resolved the data discrepancies in the existing EMIS, and developed a new specification for the EMIS. Furthermore, the project used the impact evaluation strategically to identify the most impactful form of the School Grant Program for secondary TVET and focused its investment. The project also introduced a data collection culture. At the end of 2016, the VET centers developed a standardized questionnaire for data collection for all the TVET schools, which did not exist before. This helped monitor the student placement data and other key data to track the performance of schools. For the FITD, as the number of beneficiaries of the FITD grew over the last few years and exceeded the monitoring capacity of the limited FITD staff, the FITD decided to outsource the beneficiary monitoring function to independent external audit firms. This ensured timely review of the beneficiary quarterly reports and early detection and correction of irregularities, as well as better compliance of the beneficiaries with the grant terms. In the last few years, the FITD initiated its own annual performance evaluations, by collecting and analyzing business data of supported beneficiaries. The analysis showed their revenue, profit, employment growth, average salaries and collected tax, and health and pension contributions. While the analysis has room for improvement, such as disaggregation of results by different instruments and comparison with a control group (or the whole SME sector), it is a solid first step toward building a strong M&E system. 78. Parallel technical assistance from the World Bank. The World Bank team, in collaboration with the Organization for Economic Co-operation and Development (OECD), supported development of the Innovation Strategy and subsequently the Law on Innovation Activity, adopted in May 2013. The law outlines innovation financing mechanisms and establishes the FITD as the main government agency responsible for support to start-ups, innovation commercialization, and technology transfer. The World Bank team, through the Competitive Industries and Innovation Program Grant, financed capacity-building activities for the newly created entity. The FITD staff were taken on several study tours, including to (a) ‘Eleven’—an accelerator and venture capital fund in Bulgaria and one of the biggest early-stage investors in Central and Eastern Europe, (b) the Innovation Fund in Serbia, and (c) several institutions supporting innovation activities in Hungary. The fund’s staff also completed a World Bank Institute e-learning course ‘Introduction to Innovation Policy’. Since mid-2019, the World Bank’s Finance, Competitiveness, and Innovation team secured an Improving the Innovation and Competitiveness Ecosystem Technical Assistance grant from the UK Good Governance Fund that benefited predominantly the FITD (in addition to other SME support institutions). The assistance aimed to identify opportunities for improvement of the innovation instruments and institutions and recommend adjustments to better address market failures. Through this grant, a team of local and international experts advised the FITD leadership on overcoming weaknesses in its operations, expanding the geographical coverage of its beneficiaries, improving the entrepreneurial culture and understanding of innovation, strengthening the advisory support that Page 38 of 96 The World Bank Skills Development and Innovation Support Project (P128378) complemented the funding for start-ups, improving the transparency and accountability (such as publishing audit reports), improving the legal framework for innovation (Law on Alternative Investment Funds and Law on Innovation), adjusting the instruments in response to the COVID-19 crises, improving its organizational structure, introducing a grievance mechanism, surveying beneficiaries, and providing rejected applicants information on how to improve the instruments and processes. The most significant effort was preparation of a prefeasibility study for hybrid (public-private) MSME investment fund that will crowd in private capital for innovation investments and spark the venture capital industry in the country to fill in the financing gap for advanced start-ups ready to grow and scale up. 79. Strong governance system of the FITD, but vulnerable organizational structure. While the director of the fund changed thrice during the project implementation phase, a strong governance system of the FITD enabled the project to continue smoothly. In addition, the applications for the financing instruments were reviewed by external review committees comprising international experts, which ensured the quality of the investees. The World Bank team has also intervened on the operations manual and rulebooks of the FITD to relax the stringent requirements for the applicants of the fund to stimulate innovation. In addition, the FITD complied with the Environmental Management Framework (EMF). Due to this governance system and capacity developed through the project, the FITD could also become a trusted partner to receive funding from various funding agencies. The EU was also skeptical at the beginning but recognized the capacity of the FITD and is planning to add more funds (EUR 20 million) for new instruments for greening of business operations. The main challenge of the FITD now is that its activities grew too fast and outgrew its current organizational structure. To facilitate this growth, the FITD management decided to hire consultants on limited time contracts. As of December 31, 2020, the fund has only eight permanent employees and 47 service contractors, who occupied even managerial positions. While they do the work needed for the time being, the nature of the contract employees opens the door for significant staff fluctuations and lack of stability of operations in the long run. Furthermore, the civil servant’s salary system for permanent employees is not attractive for professionals with banking, business, and management skills, who are usually much better paid in the private sector. The FITD needs to significantly adjust its organizational structure and salary systems to meet the new realities. Some support for this was provided through the concurrent World Bank technical assistance, but the provided recommendations are yet to be put into practice. 80. Proactive and constant supervision and detailed reporting. The same task team leader engaged in the project from the project preparation to closing. Even with the political crisis and transitions, the task team leader had good judgement to make progress on project implementation. Whenever there were issues, the team addressed the issues on time to the management and resolved them. For instance, in 2014, there was not enough project funds available. The team addressed the issue to the Country Director, who discussed it with the Ministry of Finance and stressed the importance of having sufficient funds for the project. As a result, the Ministry of Finance increased the budget ceiling for project implementation and the implementing agencies received enough funds to implement the planned activities. The ISRs had detail descriptions of the project, which have been commended by the management. The task team leader addressed the issues on time and the two restructurings helped progress the achievement of the PDO and better use of the unallocated fund to achieve 99.9 percent disbursement at the end of the project. Starting in 2019, a concurrent technical assistance for the FITD was secured through the Innovation and Competitiveness Project. The goal of this assistance is to help the FITD improve its ongoing instruments and establish a hybrid MSME investment fund that should provide access to additional financing and know-how to start-up as they grow and enter foreign markets. Page 39 of 96 The World Bank Skills Development and Innovation Support Project (P128378) IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 81. The project was not required to have a theory of change at the time of appraisal, but the PDO was clear, and the Results Framework indicators were well designed except for a few indicators. While the project did not include a theory of change at appraisal, the theory of change was developed for the ICR, based on the PDO and interviews with the project team. Most of the Results Framework indicators were well designed to measure the achievement of the PDO and outputs leading to the outcomes. However, most of the indicators had zero baselines because the planned systematic reforms included establishment of the new institutions and mechanisms as well as introduction of the new innovation financial instruments and monitoring tools. One of the indicators was planned to collect the baseline after the project started. The project team took some risk in setting ambitious intermediate results indicators, which required multiple preparation activities to achieve the results. Thus, the achievement of the target was expected to take longer. 82. One of the subcomponents was not aligned with the achievement of the PDO; thus, the intermediate results indicator did not contribute to the achievement of the PDO-level indicator and PDO. Subcomponent 1.3 on the establishment of the NTTO was not aligned with the higher education PDOs on improving the transparency of resource allocation and promoting accountability in higher education. The establishment of the NTTO was aimed at facilitating industry-academia collaboration and commercialization of research. Thus, this component and the related indicators could have been under Component 3, which supports innovation capacity development of North Macedonia. The team probably put the NTTO under the higher education component as the responsible unit would be HEIs. However, the FITD could have also owned this activity and established the NTTO as a part of the innovation activity. In fact, during the project implementation period, the FITD developed a feasibility study for a science and technology park, which was envisioned to include the NTTO. M&E Implementation 83. The PMU had a strong capability in M&E. The PMU staff effectively and efficiently coordinated with project stakeholders to collect necessary data and report them on time. The quality of data was good. The PMU tracked the progress of the project effectively and reported issues in advance. Throughout project implementation, M&E rating has been satisfactory in all the ISRs. The PMU also successfully conducted baseline data collection for the PDO indicator for the TVET in 2018 and identified the proportion of secondary TVET students benefitting from practical training in SMEs and large companies. While it took some time to set the baseline and collection of the accurate data was difficult, the timing did not affect the measurement of the project progress. The PMU also implemented various impact evaluations and surveys to measure the impact of the project activities. 84. The project restructured the Results Framework indicators appropriately. The World Bank team adjusted the Results Framework indicators appropriately based on the changes in the Law on Higher Education and progress of the project activities. For instance, as the establishment of the NTTO was Page 40 of 96 The World Bank Skills Development and Innovation Support Project (P128378) delayed, the project team considered that the achievements of the intermediate indicators related to the NTTO beyond establishment of the office would be not realistic. 85. The M&E functions and processes will likely to be sustained after project closure. Through the strengthening of the EMIS and launch of the skills observatory, the project introduced a long-term M&E mechanism to improve the quality of education sector data. The FITD also set up its own metrics to measure the impact of its financing instruments effectively. Due to the data discrepancies that needed to be fixed and inconsistent data at the HEIs to develop a central HEMIS for the MOES, the project team had to spend extra time and money. Because of that, development of the HEMIS was not completed, but the specification of the HEMIS was developed, and the EU is planning to fund it. The project provided capacity development for the MOES and FITD on M&E to leverage the M&E data for tracking performance, adjusting implementation, and demonstrating the impact of policy interventions financed by the project. This capacity will be sustained after project closure and both the MOES and FITD will use relevant indicators that they used for the project monitoring to continue tracking the impact. M&E Utilization 86. Impact evaluations and surveys were used to course correct the project activities and measure the impact of the project activities. The project included impact evaluations for the secondary TVET Grant Program and innovation fund instruments to identify the most impactful financing mechanisms for each program. The results of the impact evaluations were well used to select the most impactful Grant Program for secondary TVET. The impact evaluation of the innovation fund is also useful for strategic decision- making for the fund after project closure. Moreover, through the student satisfaction surveys and tracer study, the project collected beneficiaries’ data to measure the impact of the project activities. The FITD initiated an annual performance review, collecting and analyzing microlevel performance data (employment, revenues, salary levels, and taxes paid) on supported companies and presenting the results in public. While the quality of the analysis still has room for improvement, the act of establishing a regular performance evaluation practice (the first among all government institutions providing financial support to companies) is definitely a step in the right direction toward instituting accountability for use of public finding. As part of the project closing, the World Bank team conducted a performance evaluation of the accelerator and the technological extension instruments based on interviews with beneficiaries and interested applicants, whose findings should inform the future revisions of these instruments. 87. M&E data were used to show the achievement of outcomes. Because most of the Results Framework indicators were outcome indicators rather than output indicators, the M&E data provided sound evidence of the positive impact achieved by the project especially for the secondary TVET and innovation components. This evidence resulted in having subsequent education projects. There are two new projects funded by the Instrument for Pre-Accession Assistance (IPA) by the EU and Swiss Agency for Development and Cooperation for innovation. The EU is in final stages of awarding a new EUR 20 million project (with additional EUR 7 million national co-financing) to the FITD for ‘Greening of Business Operations’, which will capitalize on the organizational infrastructure and proven implementation record of the FITD. The World Bank also has a follow-on competitiveness and innovation project in the CPF that will build upon the current achievements and lessons learned. Page 41 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Justification of Overall Rating of Quality of M&E 88. The quality of M&E was Substantial. The overall design, implementation, and utilization of the M&E system was suitable. Except for a few indicators, all the Results Framework indicators measured the achievement toward the PDO effectively. The project gathered data on time and used the evaluations and survey data effectively to amplify the impact of the project activities and inform the next project. While the results indicators were ambitious, the team managed to achieve most of them and tracked the progress in detail. The M&E design is rated Substantial due to moderate shortcomings in the M&E’s system design related to one of the subcomponents and related indicators. M&E implementation is rated Substantial due to moderate shortcomings in M&E implementation such as delayed baseline data collection. Finally, M&E utilization is High because the project effectively used the impact evaluation, created a sustainable M&E system, and used the project results to expand the activities further. Therefore, combining these three rating, the overall rating of the quality of M&E would be Substantial. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 89. The project had no implementation issues regarding environmental, social, and fiduciary compliance. Thus, the ISR ratings have been satisfactory throughout project implementation. A part of the reasons is the experienced PMU staff who worked on the World Bank projects in the past. 90. Environment. The project was assigned an environmental category B. The project EMF was prepared and publicly disclosed before project appraisal. The activities under Component 3 Innovation Grant Program were considered by the EMF due to their potential environmental impacts. The EMF procedures were designed to screen subprojects/grants to ensure (a) compliance with the World Bank Group exclusion list, (b) that no subprojects with significant impacts of Category A type are supported, and (b) that subprojects/grants would not necessitate involuntary land acquisition. The EMF also allowed provisions for all activities corresponding to sub-projects defined as Category B to have an Environmental Management Plan that would identify potential environmental impacts and provide adequate mitigation measures in place before approval. During project implementation, the FITD complied with the EMF, including proper screening of grant proposals; ensuring that Environmental Management Plans are of acceptable quality and duly implemented; and providing guidance and monitoring at all stages of subproject identification, evaluation, and implementation. Adequate capacity was built within the FITD PMU to advise the applicants and monitor the environmental compliance of subprojects. The overall environmental performance of the project was found to be Satisfactory. 91. Social. The project did not trigger the social safeguards. The Results Framework data were expected to capture data disaggregated by gender whenever possible/available. Although the Results Framework data did not have gender-disaggregated data, the PMU tracked the gender-disaggregated data for the grantees of the FITD funding instruments and secondary TVET student enrollment and graduation. All the innovation grant proposals were reviewed to avoid any negative social implication and/or potential gender bias. Furthermore, selection criteria were adjusted to stimulate companies founded/managed by women to participate. During project implementation, the proposal reviews were conducted carefully and there were no issues. 92. Procurement. The overall coordination and day-to-day management of procurement procedures under the project was with the PMU, established at the MOES. Procurement under the project was Page 42 of 96 The World Bank Skills Development and Innovation Support Project (P128378) governed by the applicable World Bank’s ‘Guidelines: Procurement of Goods, Works, and Non-Consultant Services under IBRD Loans and IDA Credits and Grants by the World Bank Borrowers’, dated January 2011; ‘Guidelines: Selection and Employment of Consultants under IBRD Loans and IDA Credits and Grants by the World Bank Borrowers’, dated January 2011; the provisions stipulated in the Legal Agreement; and the specific arrangements as agreed in the Procurement Plan and in the Project Operations Manual (POM). The Procurement Plan for the entire life of the project was prepared during project preparation and its final version was agreed at project negotiations. The Procurement Plan provided the basis for procurement methods and procurement review arrangements. The Procurement Plan was updated regularly in agreement with the World Bank through the duration of the project. 93. The PMU functioned well and had good capacity and track record of project and procurement management. It established and maintained good collaboration with the relevant project beneficiaries. The procurement function in the unit was implemented by a procurement specialist, who is experienced and knowledgeable, with sound understanding of the principles of the World Bank’s procurement policies and procedures. She also had experience as a procurement specialist and was involved in other projects financed by the World Bank. Complaints with regard to the procurement process were rare, and in case there were any, they were promptly addressed by the PMU in consultation with the World Bank. 94. The PMU had a leading role in quality assurance and management of the overall procurement process. The procurement notices for the tenders were published in newspapers of wide circulation and on the ministry’s website. In most cases, a reasonable competition was secured with three to five bids received, and in few cases less than that. The qualification and evaluation criteria were reflected in the bidding documents and were clear and balanced for the scope and nature of the contracts. In general, the contracts were awarded to the lowest evaluated substantially responsive bidder, which was usually the bidder who submitted the bid with the lowest price. In all cases, the documents and related specific procurement procedures reviewed were available and were duly filed, and it was easy to locate them. The overall implementation of contracts was timely and within the contract amount. 95. Financial management. The overall financial management risk for the project was Substantial before mitigation measures at project appraisal. As the FITD was not yet established at the time of project appraisal, and its capacity to implement Subcomponent 3.2 had not been yet assessed and appraised, the residual risk remained Substantial. There were effectiveness conditions to make the financial management arrangements acceptable. The effectiveness conditions included (a) a qualified and experienced financial officer to be employed by the MOES; (b) the POM, which included the financial management manual, to be prepared and adopted; (c) acceptable accounting software to be acquired and installed by the MOES for project records no later than 30 days from effectiveness; and (d) appropriate financial management arrangements to be instituted in the FITD before starting disbursement on Subcomponent 3.2. The POM was adopted by the MOES and the project became effective on March 11, 2014. The Government also swiftly established the FITD, which met all the disbursement effectiveness conditions and launched all four financial instruments in September 2015. In early 2016, due to the inadequate budget allocation for the project from the Ministry of Finance, the project did not have enough funds to implement some of the project activities, including external evaluation of the HEIs and establishment of the NTTO. Thus, the World Bank project team worked with the Country Management Unit (CMU) to send a letter to the Prime Minister to allocate project funds. This proactive action led to the allocation of project funds. Page 43 of 96 The World Bank Skills Development and Innovation Support Project (P128378) C. BANK PERFORMANCE Quality at Entry 96. The quality at entry is rated Satisfactory. (a) The project was strategically relevant with the CPS for FY11–FY14 and various Government strategies and policies. (b) Technical, financial, and economic aspects of the project were well thought out by providing technical assistance and involving multiple stakeholders to inform based on findings of the analytical work and develop consensus on the project design. Before project preparation, the World Bank team was providing technical assistance work on quality assurance in higher education and higher education financing reforms since 2010, which served as a basis for this project. Technical assistance included (i) a delivery and dissemination of the basic strategic document on higher education financing with a cost per student place calculation tool, manual, and a set of performance indicators; (ii) a regional round table with 25 representatives from the Ministry of Education/HEIs from Croatia, Kosovo, Macedonia, Romania, and Serbia to share current higher education financing structures, norms, and challenges; (iii) capacity development workshops for the newly established HEAEB; (iv) University Admission-Feasibility Study; and (v) and the Systems Approach for Better Education Results (SABER) student assessment system review. Through participation in a series of workshops on key higher education topics and technical assistance, the MOES confirmed the need for further World Bank support for the higher education sector, prioritizing the higher education financing reform and support for quality enhancement at the institutional level. (c) Regarding TVET, the EU has been implementing three complementary projects on, including (i) modernization of two- and three-yearlong VET; (ii) support for the integration of the ethnic communities in the education system; and (iii) capacity building of the center for adult education and development of programs for adult education, literacy, and completion of primary education for socially excluded persons. However, no agencies have been supporting the four-year TVET, where the majority of students went. Thus, there was an opportunity for the World Bank project to come in and complement the investment in the VET sector. The TVET stakeholders were also committed to the reform. (d) For the innovation component, legal frameworks were set up and the Law on Innovation designated the FITD as the entity to finance the Government’s innovation activities. The World Bank team has been providing advice on the Law on Innovation and the World Bank project was appropriate for supporting capacity building of the FITD. (e) In terms of the poverty, gender, and social development aspects, the project did not specifically address these issues; however, some data on gender were collected during the project implementation to measure the impact on gender. Also, the innovation grant encouraged female applicants through the grant design. The reforms envisaged in the project also did not necessarily include the aspects of financial and social support for Page 44 of 96 The World Bank Skills Development and Innovation Support Project (P128378) economically disadvantaged students. (f) Environment, social, and fiduciary aspects of the project were prepared well. The only concern was the compliance of the EMF by the grantees of the FITD. (g) Risks, including the country risk that affected the project the most, were also identified and well addressed. The World Bank team also engaged with the stakeholders by forming the working groups since the project preparation stage to mitigate any stakeholder risks. Implementation arrangements were also well prepared to avoid any risks. (h) While there was no baseline for the Results Framework indicators, the project team developed the Results Framework indicators well and included impact evaluations and surveys to measure the impact of the project activities. The baseline data collection for some of the indicators was also planned. While one of the indicators under the higher education did not link too well with the PDO for higher education, most of the indicators were measuring the project outcomes well. Quality of Supervision 97. The quality of supervision is rated Satisfactory. (a) The World Bank team played an important role in project implementation and achieving the PDO–from smooth project effectiveness (a little over a month) to two restructurings of the project. (b) Despite the multiple transitions of the Ministers of Education and student strike, the team worked closely with the MOES to inform the ministers on the project every time the new ministers took the office. The team also offered a technical support for the MOES to respond to the requests from the strike. (c) The World Bank team course corrected the project twice through restructuring and maximized the implementation period to achieve the PDO and demonstrate the development impact of the project. (d) It took a while from the MTR in September 2017 to the first restructuring in March 2019. This is because at the time of the MTR, the team was not expecting any political turmoil and the restructuring was not planned. However, in February 2018, the new Law on Higher Education was adopted, which required the team to change the Results Framework indicators. In addition, while the team thought that the establishment of the NTTO would not work due to a lack of ownership, the Government resisted in keeping the NTTO in the project. The negotiation with the Government on the NTTO took some time as well. As a result, the restructuring took place later than the MTR. (e) When the project fund was running low, the team worked with the CMU to write a letter to the Prime Minister to request for fund allocation. This proactive action prevented the project from being stagnant. Page 45 of 96 The World Bank Skills Development and Innovation Support Project (P128378) (f) The team organized missions every six months and multi-skilled team members and expert consultants were available to provide technical support and respond to any issues that the client had. The fiduciary and environmental and social safeguards colleagues were supportive and regularly participated in team missions and field visits as well. (g) The team members of the project were mostly constant throughout the project (with the exception of change of the local private sector specialist in the last 15 months of project implementation), which built trust with the client and stakeholders. Even with the change of the team member, the new team member immediately established a good relationship with the FITD and provided focused support based on her relevant experience. (h) Candor and quality of performance reporting was good and the ISRs were capturing key information in detail to inform the management well. The management teams also commended the quality of the ISRs. (i) After the closing of the project, the World Bank team is preparing another education project that builds on from the results of this project, and the same PMU will be used for the next project. The task team leader of this project will also serve as a task team leader for the following project; thus, the transition will be smooth. (j) Impact evaluations were also effectively used to select the most impactful measures for the Grant Program for the secondary TVET institutions. (k) The CPF envisions a potential follow-on project in the area of competitiveness and innovation that will expand on the achievements with the FITD to date, and client demand is high. Justification of Overall Rating of Bank Performance 98. The overall World Bank performance was Satisfactory based on the above assessment. D. RISK TO DEVELOPMENT OUTCOME 99. The risk to development outcome is Moderate from the following assessment. (a) The focus of the project was a structural and systematic reform in higher education and secondary TVET and innovation. During the project, major changes took place in the quality assurance mechanism, higher education financing, secondary TVET curricula, school- industry partnerships, and launch of an almost financially independent fund for innovation. These changes will likely sustain as the beneficiaries have already seen the benefits from the reforms and it is unlikely that another change will take place to revert to the old system. In higher education, the new AQA was established to provide quality assurance service that complied with the EU standards. The FITD has already started to receive the Government’s budget, which accounted for 85 percent of the total budget in 2021. In addition to the World Bank loan and Government financing, the FITD attracted other donor funding (EU cross- Page 46 of 96 The World Bank Skills Development and Innovation Support Project (P128378) border project with Albania, USAID20 Junior Achievement collaboration, Increasing Market Employability (IME) Programme’s support for agri-innovation call, the upcoming EU project, and so on). The fund also received high recognition and developed its prestigious position in the innovation community. Thus, the fund will most likely last and even flourish more to attract start-ups and other funders. (b) However, there are some project activities that were not completed. The completion of these activities will depend on continuous government commitment and the availability of budget. In higher education, the National Council for Higher Education and Research was established just before the project closing date. This body needs to approve the new quality assurance measures and new financing model for higher education. It will also take another year for the MOES to be able to disburse the budget under the new financing model. (c) Under the secondary TVET component, the Grant Program provided incentives for firms to collaborate with the secondary TVET schools. However, according to the impact evaluation, many firms reported that they were not attracted to providing practical work-based training to students due to the risks associated with the utilization of firm’s technologies but also due to a lack of motivation and soft skills among students. Thus, while the financial incentive helped the firm-school collaboration, for the sustainability of the program, it was recommended to improve students’ capacity, increase participation of teachers, or find alternative sources of collaborations between schools and the private sectors that would not include only short-term subsidies. At the time of project closure, however, there are positive developments to sustain the effort. For instance, the activities for engaging with the private sector in practical training of the VET students is now supported through the IPA 2 funding, Education for Employment Project in Northern Macedonia, and KfW 21 Regional Challenge Fund. (d) In addition, the MOES has created short-term and long-term measures, which foresee a possibility for the companies to establish training centers, through which they will be able to train staff in accordance with their requirements and needs. The state will also establish VET Centers, which will serve as resource centers for verification of the trainings and for involvement of the companies in the trainings, as well as for retraining of staff in a period shorter than one year. Starting from 2020, the ministry introduced scholarship support for the VET students, especially for the deficient occupations. V. LESSONS AND RECOMMENDATIONS 100. The project should balance between systematic reforms and activities that can be implemented at the institution level. The project was influenced a lot by the political transitions and strikes. While the World Bank team identified the country risks including the presidential election at the time of project appraisal, more-than-expected political transitions took place and affected implementation of the project, especially on the higher education component. Almost every year, new ministers came to office, and the project team had to explain to him/her and get the buy-in of the project, which took extra time for 20 USAID = United States Agency for International Development. 21 KfW = Kreditanstalt für Wiederaufbau. Page 47 of 96 The World Bank Skills Development and Innovation Support Project (P128378) implementation. Due to political changes, it also took longer to develop the new Law on Higher Education and establish key HEIs, such as the National Council for Higher Education and Research and AQA. All these delays affected the progress of the higher education component as the project activities required multiple government agencies’ approvals to move forward. Learning from this experience, it would be good to balance the reform-related activities and non-reform-related activities to ensure that some activities can move forward without involving the Government’s decision-making. For instance, digital transformation (for example, development of the HEMIS; digitalizing university administration; and upgrading the websites to share more institutional, research, and innovation data) at the institution level or university benchmarking could have been included to improve transparency of resource allocation and promote accountability. 101. Development of a strong governance model and flexible operation is important for sustainability of the newly developed institution. Despite challenging political situations and multiple changes in leadership during project implementation, the strong governance model and operation process of the FITD prevented any political influence from affecting the fund. Having rulebooks and written procedures, developing an organizational culture embracing the reforms and good governance values, having competent staff, and setting up a strong monitoring system for project implementation and disbursement were important for ensuring continuity of the operation of the FITD. Some of the prominent features of strong governance were using international experts for the grant proposal selection and working with Transparency International, a nonprofit organization that focuses on anti-corruption, to ensure that the applicants of the fund are not connected to the Government. While the preselection process of the companies was done by domestic experts, foreign experts hired by the Government conducted the evaluation of the proposals, limiting government influence over investment decisions. Most of the FITD employees were from the private sector and it was not structured as a regular Government fund. Follow-up analysis conducted by the concurrent World Bank technical assistance to assess the effectiveness of the Government’s instruments for improving innovation and competitiveness of firms, showed that the FITD’s beneficiaries were surprised by the objectivity of the evaluation process and lack of political influence that was typical for other government programs. The FITD also increased accessibility by introducing a fully online application process, and transparency in Government spending by publicly disclosing the names of the grant recipients, and the value and the purpose of the grant. This strong governance of the fund and the removal of the public officials from the investment decision-making process made the fund trustworthy and attracted other partners to fund. Moreover, the fund flexibly adjusted its operation based on the needs. For instance, when the in-house capacity was not enough to monitor the beneficiary implementation reports and disbursement request, the FITD outsourced this function to a renowned audit company. The fund also adjusted/improved the instrument parameters and selection criteria after each call to maximize the impact. During the COVID-19 pandemic, the fund adjusted the instruments based on the particular needs of the beneficiaries. Finally, to ensure sustainability and stable financing of the new institutions funded by the World Bank project, it is important to secure sizable funding from other sources such as the Government by the MTR of the project. 102. One of the key elements for building and strengthening an innovation ecosystem was through multi-sector collaboration. The FITD was good at developing partnerships with various organizations to develop a start-up and innovation ecosystem in different sectors. For instance, the FITD established the Start-Up Council for the first time in the country to represent the start-up community and its interests in the process of creating and adopting policies that should contribute to further growth and development of the Macedonian start-up ecosystem. The members of the council comprised both the Government and Page 48 of 96 The World Bank Skills Development and Innovation Support Project (P128378) start-up community, including START-UP Macedonia; the Chamber of Commerce for Information and Communication Technologies; and the Deputy President of the Government in charge of economic affairs, coordination of economic sectors, and investments. The council is interested in making the country a start- up and regional hub. The FITD also partnered with a large health care company, Roche Macedonia, to organize a hackathon on e-health in June 2021 (after project closure) to mobilize the digital health community. According to the FITD’s website, the fund was also working on corporate innovation by brokering partnerships between the corporate sector and start-ups. With a new call for proposal to set up digital fabrication laboratories at the HEIs and scientific research institutions, the FITD is also going to promote research and business-academia links in the country. 103. Effective use of impact evaluation helps develop an impactful program. The project used the impact evaluation effectively to assess the impact of the five financing measures used for the Grant Program for secondary TVET. Among those five measures, the following three measures were the most impactful in terms of students’ academic performance: (a) setting up of a school company within the VET school through business guidance and mentoring, (b) placement of students in practical training in companies, and (c) establishment of problem-based learning courses for enhancement of soft skills. Because the program was new, conducting a rapid impact evaluation was useful in using the rest of the fund in an impactful way by reducing the five measures into three after the evaluation. 104. Having the right expert for the technical workshop can bring substantial impact. Technical assistance for higher education financing improved the transparency in resource allocation in higher education by making the current higher education financing data available to all six public universities and participating private universities. A series of technical assistances for designing a new performance-based financing model included a review of existing laws, documents, and international good practices; assessment of funding options and design of an appropriate funding model; rolling out of the new funding model; and identifying of lessons learned from the first-year operation of the new funding model. During the workshop, the principles of a sound funding model; options of different funding models for North Macedonia, including formula-based funding combining input and output indicators, funding through performance agreements, and innovation-oriented funding through competitive grants; budget allocation; implementation guidelines, such as implementation plans for the MOES and universities; capacity-building requirements; audit requirements; and M&E were discussed and agreed upon. The consultant hired for this technical assistance delivered a high-quality product and facilitated the workshops effectively. Therefore, the universities successfully reached the consensus and committed to this reform. 105. Introduction and uptake of new innovative instruments takes time and requires charismatic leadership, promotion, and beneficiary consultations. Like the TVET curriculum reform, it takes time to develop and introduce a new initiative nationwide. The FITD was not an exception. Significant majority of funding for the FITD (75 percent) was disbursed during the last 18 months (during project extension). The initially projected five years were not sufficient (a) to design quality instruments; (b) to build institutional capacity; (c) to launch calls, build awareness of the funding opportunity and understanding of the innovation concepts, and overcome the trust issues for objective selection of recipients; and (d) for beneficiaries to implement projects (some of them were multiyear). Moreover, undertaking consultative processes and direct interactions with potential beneficiaries are necessary to craft an attractive program to address the needs of beneficiaries. Once the program is developed, having a charismatic leader and the budget for promotional activities are needed to effectively communicate the new initiative to the public. Page 49 of 96 The World Bank Skills Development and Innovation Support Project (P128378) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improving Transparency of Higher Education Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion % of public universities Percentage 0.00 80.00 0.00 receiving financing based on new, transparent funding 28-Mar-2014 31-May-2019 21-Apr-2021 model. Comments (achievements against targets): The original target was not achieved at the project completion. The draft higher education funding model is ready, but the adoption is pending due to the delayed constitution of the National Council for Higher Education and Research. The National Council is expected to review the draft model and adopt it. Since the model was not adopted during the ICR period, it will probably take some time for the new financing model to be adopted. After the adoption of the funding model, it takes another year for the MOES to start disbursing the funds based on the new funding model to achieve the original target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion % of study programs Percentage 0.00 80.00 0.00 accredited /reaccredited utilizing new quality 21-Jan-2019 30-Jun-2020 21-Apr-2021 Page 50 of 96 The World Bank Skills Development and Innovation Support Project (P128378) assurance measures developed in accordance with Bologna-defined norms by the new Quality Assurance Agency Comments (achievements against targets): The original indicator was "% of public universities accredited, utilizing new quality assurance and accreditation measures developed in accordance with Bologna-defined EU norms and practices," which was revised during the restructuring of March 2019. Due to the new Law on Higher Education adopted by the Macedonian Parliament in May 2018, the Higher Education Accreditation and Evaluation Board (HEAEB) needed to be replaced with Agency for Quality Assurance (AQA), which was established in December 2019. Therefore, under the new law, the accreditation agency changed from the HEAEB, which was under the Ministry of Education and Science to the AQA, which was an independent body. The new target had higher quality standards for the accreditation, which had a comprehensive accreditation process, including the labor market relevance of the academic programs. Originally, the scope of accreditation was the institutional and program accreditation, the revised scope was for the program accreditation only. However, due to the more rigorous accreditation process with multi-sectoral assessment body, which includes private sector, it took more efforts for the accreditation and reaccreditation. The project did not achieve the target of revised indicator at the time of the project completion. However, the draft secondary legislation to govern the new quality assurance measures was developed. If the newly constituted National Council for Higher Education and Research can adopt the secondary legislation, the AQA will proceed with the accreditation process in accordance with the new legislation and potentially meet the target in a few months. Objective/Outcome: Modernization of Secondary Education Vocational Education and Training Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion % of secondary TVET Percentage 0.00 30.00 40.00 40.20 students benefiting from practical training in SME and 28-Mar-2014 30-Jun-2020 01-Mar-2019 21-Apr-2021 large-sized firms. Page 51 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Comments (achievements against targets): The original indicator was "% increase in number of secondary TVET students benefiting from practical training in SME and large -sized firms," which was revised during the restructuring of March 2019. The original target was 30%. Due to the variation in total number of students enrolled in TVET every year, the project could not compare the data accurately if the original indicator was to be used. Thus, the indicator was revised to calculate the results appropriately. The baseline was calculated using the 2015/2016 student enrollment data for TVET and those who are benefitting from the practical training in SME and large-sized firms. The percentage of students benefitting from the practical training was calculated as 36.5%. The new target was set based on this baseline. Since the availability of the practical training in SME and large-sized firms was limited and the enrollment of TVET students were declining, the target of 40% was realistic. At the time of the project completion, the project exceeded the target of the revised indicator. Objective/Outcome: Improving the Innovative Capacity of Enterprises and Collaboration with Research Organization Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Share of private funding Percentage 0.00 25.00 49.53 mobilized as a percentage of FITD investments in 28-Mar-2014 31-May-2019 21-Apr-2021 innovation activities. Comments (achievements against targets): The project exceeded the original target at the time of the project completion. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 52 of 96 The World Bank Skills Development and Innovation Support Project (P128378) % of beneficiaries that sign Percentage 0.00 20.00 39.00 collaborative agreements between firms and academia. 28-Mar-2014 31-May-2019 21-Apr-2021 Comments (achievements against targets): The project exceeded the original target at the time of the project completion. A.2 Intermediate Results Indicators Component: Improving Transparency of Higher Education Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Implementation of new Text The new funding New funding model The possible funding higher education funding model is not approved by the GOM models have been model. developed presented to the loped government and public universities. In accordance with the new law for the Higher Education a National Council on Higher Education will adopt the bylaws on financing. The Council has not been formed yet Page 53 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 28-Mar-2014 31-May-2019 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project partially met the original target. The new funding model was approved by the Ministry of Education and Science, but not by the National Council for Higher Education and Research. Due to the delayed constitution of the National Council, the project was still waiting for the National Council to review and adopt the funding model in order to implement the new model. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of public universities Number 0.00 5.00 6.00 that have completed an external evaluation utilizing 28-Mar-2014 31-May-2019 21-Apr-2021 new quality assurance measures developed in accordance with Bologna- defined EU norms and practices. Comments (achievements against targets): At the time of the project completion, the project exceeded the original target since one additional public university was established during the project implementation period, and the project could conduct the external evaluation for the newly established public university. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 54 of 96 The World Bank Skills Development and Innovation Support Project (P128378) National TTO established and Text Not in existence NTTO Operational No operational 28-Mar-2014 31-May-2019 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project did not achieve the target as the NTTO was not established. FITD has developed a feasibility study for a Science and Technology Park which is envisioned to include the NTTO. However, the NTTO was not established due to a lack of champion in the government. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Agency for Quality Assurance Text Board of Quality Board of Quality Agency for Quality Agency for Quality and Accreditation achieves Assurance and Assurance and Assurance achieves Assurance achieved affiliate membership in Accreditation NOT a Accresitation achieves affiliate member affiliate member ENQA (the European member of ENQA full member status in status in ENQA status in ENQA Association for Quality ENQA Assurance in Higher Education) 28-Mar-2014 30-Jun-2020 30-Apr-2021 21-Apr-2021 Comments (achievements against targets): The original indicator was "Board of Quality Assurance and Accreditation achieves membership in the European Association for Quality Assurance (ENQA) in Higher Education (HE)," which was revised during the restructuring of March 2019. While the target was changed from achieving the membership status to achieving the affiliate membership status for ENQA in higher education, the establishment of the AQA itself was an achievement which was not included in the original project scope. Due to the new law of higher education, the project had to establish the AQA. The newly established agency, however, needed at least 2 years of operational experience and completion of at least 5 review reports to become a member in addition to meeting all the other eligibility criteria. Therefore, while the AQA was ready for the full membership status, the restriction on the 2-year operation made it difficult to achieve Page 55 of 96 The World Bank Skills Development and Innovation Support Project (P128378) the membership status during the project period as the AQA was established in May 2019, and the project ended on April 30, 2021. Thus, the project set the right target during the restructuring. At the time of the project completion, the project successfully met the target of achieving the affiliate membership of the ENQA and subsequently preparing for the full membership status. Component: Modernization of Secondary Technical Vocational Education and Training Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion TVET action plan that Text Not in existence Model adopted by the Model adopted promotes general and Minister of Education broader technical education and Science and competence- based learning adopted 10-Sep-2013 30-Jun-2016 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project achieved the original target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion % of developed TVET Percentage 0.00 100.00 100.00 occupational standards by professional fields 10-Sep-2013 31-May-2019 21-Apr-2021 Page 56 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Comments (achievements against targets): At the time of the project completion, the project achieved the original target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion % of updated TVET curricula Percentage 0.00 100.00 100.00 in line with occupational standards 10-Sep-2013 31-Jan-2019 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project achieved the original target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion % Increase in number of Percentage 0.00 30.00 16.50 companies providing practical training to 15-Jan-2015 31-May-2019 21-Apr-2021 secondary TVET students. Comments (achievements against targets): At the time of the project completion, the project did not achieve the project target based on the latest data available. Page 57 of 96 The World Bank Skills Development and Innovation Support Project (P128378) The latest number of companies providing practical training to secondary TVET students was from the school year 2019/20. It is expected that this number significantly increased during the following school year. The update, however, will be available in October 2021 and the target would most likely be achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Manual and training Yes/No No Y Yes materials for quality delivery of new curricula adopted 10-Sep-2013 31-May-2018 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project achieved the original target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Students benefiting from Number 0.00 21000.00 21,346.00 direct interventions to enhance learning 28-Mar-2014 30-Jun-2020 21-Apr-2021 Students benefiting from Number 0.00 10000.00 10,609.00 direct interventions to enhance learning - Female Comments (achievements against targets): At the time of the project completion, the project exceeded the original target. Page 58 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Component: Improving the innovative capacity of enterprises and collaboration with research organizations Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Share of resources dedicated Percentage 0.00 100.00 100.00 to Fund for Innovation and Technology Development 10-Sep-2013 31-May-2019 21-Apr-2021 (FITD) instruments disbursed Comments (achievements against targets): At the time of the project completion, the project achieved the original target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of companies Number 0.00 15.00 3.00 established via the FITD Accelerator program 10-Sep-2013 31-May-2019 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project did not meet the target. The Accelerator program has two phases: (i) Pre-investment Acceleration program; and (ii) Investment program. Since the establishment of the accelerator program, 62 firms benefited from the pre-investment phase, and 3 from the second phase. Around 15 new firms are now benefiting from the investment program and are expected to fully benefit by the end of 2021. This indicator is on track to be achieved, though beyond the project lifecycle. Page 59 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of beneficiary firms Number 0.00 30.00 55.00 256.00 to introduce new/improved products 10-Sep-2013 31-May-2019 30-Jun-2020 21-Apr-2021 Comments (achievements against targets): The target of the indicator was revised to be more ambitious during the restructuring of March 2019. The change in the target was not expected and this change took place by accident. However, at the time of the project completion, the project exceeded both original and revised target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion % of beneficiaries receiving Percentage 0.00 70.00 100.00 training and mentoring 28-Mar-2014 31-May-2019 09-Dec-2020 Comments (achievements against targets): At the time of the project completion, the project exceeded the original target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of beneficiary firms Number 0.00 4.00 11.00 Page 60 of 96 The World Bank Skills Development and Innovation Support Project (P128378) to introduce new processes 10-Sep-2013 31-May-2019 09-Dec-2020 Comments (achievements against targets): At the time of the project completion, the project exceeded the original target. Component: Project Management and Monitoring and Evaluation Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Evaluations for Innovation Text Not in existence Evaluation reports The follow-up survey grants designed, conducted completed and results is finished. Results will and reports reported disseminated be disseminated by April 30, 2021 10-Sep-2013 31-May-2019 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project met the original target. The report was disseminated and made available to all interested parties, including the Ministry of Education and Science, FITD, and the World Bank. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Skills Observatory Text Not in existence The info on platform The activities for established, providing being updated at least development of the information about TVET and twice per year Skills Observatory are Page 61 of 96 The World Bank Skills Development and Innovation Support Project (P128378) HE performance to the public finished. The activities for the first tracer study are finished. 10-Sep-2013 31-May-2019 21-Apr-2021 Comments (achievements against targets): At the time of the project completion, the project did not meet the original target. The data for students were gathered from EMIS and from several universities, and based on these data reports were generated. However, at the request of the Personal Data Protection Agency, due to lack of legal basis for gathering these data, student data were deleted from the Skills Observatory System. In order to provide a legal basis for gathering data for the purposes of the Skills Observatory, changes have been made in the Law on secondary education. Additional changes in the Law for higher education need to be done in order to provide a legal basis for gathering data. Therefore, the MOES will adjust the observatory accordingly. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of students that Percentage 0.00 60.00 84.10 participated in the Grant Program satisfied with their 13-Feb-2019 30-Jun-2020 21-Apr-2021 engagement during grant preparation Comments (achievements against targets): This indicator was added as the citizen engagement indicator became mandatory from the World Bank during the project implementation. Therefore, the project added this indicator during the restructuring of March 2019. At the time of the project completion, the project exceeded the original target set at the time of the restructuring. Page 62 of 96 The World Bank Skills Development and Innovation Support Project (P128378) The feedback from the students was obtained through the survey, conducted in June 2019. The survey showed that 83% of the students that participated in the Grant Program were satisfied with their engagement during the grant program preparation. The new survey was not conducted after 2019 due to the COVID-19. Page 63 of 96 The World Bank Skills Development and Innovation Support Project (P128378) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Improve transparency of resource allocation in higher education 1. % of public universities receiving financing based on new, transparent Outcome Indicators funding mode Intermediate Results Indicators 1. Implementation of new higher education funding model Key Outputs by Component 1. New higher education funding model is developed and adopted (linked to the achievement of the Objective/Outcome 1) Objective/Outcome 2: Promote accountability in higher education 1. % of study programs accredited/reaccredited utilizing new quality assurance Outcome Indicators measures developed in accordance with Bologna-defined norms by the new Quality Assurance Agency 1. Number of public universities that have completed an external evaluation utilizing new quality assurance measures developed in accordance with Bologna-defined EU norms and practices Intermediate Results Indicators 2. NTTO established and operational 3. Agency for Quality Assurance and Accreditation achieves affiliate membership in ENQA (the European Association for Quality Assurance in Higher Education) 1. New quality assurance measures are developed and adopted. 2. Training of the Higher Education Accreditation and Evaluation Board (HEAEB) and other key players of managing quality assurance activities was conducted. Key Outputs by Component 3. External institutional evaluation of public institutions is concluded. (linked to the achievement of the Objective/Outcome 1) 4. HEMIS is upgraded. 5. NTTO is designed and established. 6. Agency for Quality Assurance and Accreditation is established. Page 64 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Objective/Outcome 2: Enhance the relevance of secondary technical vocational education 1. % of secondary TVET students benefiting from practical training in SME and Outcome Indicators large sized firms. 1. The TVET action plan that promotes general and broader technical education and competence- based learning adopted 2. % of TVET occupational standards developed by professional fields 3. % of updated TVET curricula in line with occupational standards Intermediate Results Indicators 4. % Increase in number of companies providing practical training to secondary TVET students 5. Manual and training materials for quality delivery of new curricula adopted 6. Students benefitting from direct interventions to enhance learning (by gender) 1. The TVET action plan that promotes general and broader technical education and competence-based learning developed 2. Occupational standards and competency-based curriculum are developed 3. Efficiency assessment of the secondary TVET school network and Key Outputs by Component development of a proposal for restructuring the network are completed (linked to the achievement of the Objective/Outcome 2) 4. Training manual and materials are developed for quality delivery of new curricula 5. A need analysis and acquisition of equipment for school-based practical training is conducted 6. School Grant Program designed and implemented Objective/Outcome 3: Support innovation capacity in the Republic of North Macedonia Enhance the relevance of secondary technical vocational education 1. Share of private funding mobilized as a percentage of FITD investments in innovation activities Outcome Indicators 2. % of beneficiaries that sign collaborative agreements between firms and academia Page 65 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 1. Share of resources dedicated to Fund for Innovation and Technology Development (FITD) instruments disbursed 2. Number of companies ‘accelerated’ via the FITD Accelerator program Intermediate Results Indicators 3. # of beneficiary firms to introduce new/improved products 4. # of beneficiary firms to introduce new processes 5. % of beneficiaries receiving training and mentoring 1. FITD operational manual is completed. Key Outputs by Component 2. Staffing and training of the FITD staff are completed. (linked to the achievement of the Objective/Outcome 2) 3. Selection of investment committee and peer reviewers is completed. 4. Four financial instruments are designed and approved for the FITD. Page 66 of 96 The World Bank Skills Development and Innovation Support Project (P128378) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Bojana Naceva Task Team Leader(s) Antonia G. Viyachka Procurement Specialist(s) Anneliese Viorela Voinea Financial Management Specialist Gulana Enar Hajiyeva Social Specialist Bekim Imeri Social Specialist Supervision/ICR Bojana Naceva Task Team Leader(s) Antonia G. Viyachka Procurement Specialist(s) Tural Jamalov Financial Management Specialist Anne N. Ranasinghe Procurement Team Mohammad Ilyas Butt Procurement Team Jasminka Sopova Program Assistant Gulana Enar Hajiyeva Environmental Specialist Marta Helena Reis de Assis Team Member Bekim Imeri Social Specialist Patrick A. Biribonwa Program Assistant Roberta Malee Bassett Team Member Tatjana Markovska Team Member Page 67 of 96 The World Bank Skills Development and Innovation Support Project (P128378) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY13 35.223 135,998.55 FY14 35.601 95,537.21 FY15 15.530 20,049.22 FY16 9.970 12,208.60 FY17 1.800 2,437.26 Total 98.12 266,230.84 Supervision/ICR FY15 9.285 37,022.64 FY16 14.380 67,536.01 FY17 24.125 67,264.15 FY18 21.480 96,371.50 FY19 25.818 81,492.87 FY20 19.509 50,971.03 Total 114.60 400,658.20 ANNEX 3. PROJECT COST BY COMPONENT Components Amount at Actual at Project Percentage Approval Closing of (US$, millions) (US$, millions) Approval Improving Transparency of Higher Education 4.00 0.98 25 Modernization of Secondary Technical Vocational 4.50 5.00 111 Education and Training Improving the Innovative Capacity of Enterprises 12.94 10.52 81 and Collaboration with Research Organizations Project Management and Monitoring and 2.50 2.81 112 Evaluation Total 23.94 19.34 81 Page 68 of 96 The World Bank Skills Development and Innovation Support Project (P128378) ANNEX 4. EFFICIENCY ANALYSIS 1. Project interventions generated healthy economic gains and efficiencies. The PAD presented the analysis of the economic benefits regarding technical vocational and tertiary education and innovation. The ICR analysis, aligning with the PAD analysis, reviews the program outcomes and estimates the economic returns and efficiencies produced by the components on reforming technical vocational education and innovation. Improved efficiencies through modernizing secondary TVET 2. At appraisal, the rationale for supporting technical and tertiary education was based on the reported high returns to tertiary education22 and that improving the quality of TVET and higher education would yield long-term benefits by increasing labor market returns and more demand for TVET. 3. At completion, based on the concrete outcomes measured through studies (impact evaluation study and stakeholder satisfaction report) and assumptions on long-term impacts of the program on the labor market outcomes of beneficiaries (VET students), the CBA is conducted for Component 2: Modernization of Secondary Technical Vocational Education and Training. Measured Outcomes 4. Component 2 generated the following outcomes during the project period: Quality and Labor Market Relevance of TVET Provision • The National Expert Team, comprising 650 members from the economy and education sectors, developed 1,182 new modular curricula for both general and vocational subjects for all 52 qualifications within all 14 sectors. • All secondary VET schools adopted the newly developed curricula. • From school year 2019/2020, the new modular competence-based curricula for all 52 qualifications have been implemented in all 76 VET schools in the country • According to the stakeholder satisfaction survey 23 conducted in 2018, most of the respondents agreed on the significant contribution of the newly developed syllabi to meeting the labor market needs. Grant Program • Total of 232 applications were submitted to the grant program, and 41 were awarded with financing. 22World Bank. 2013. Employment and Job Creation in FYR Macedonia: Labor Market Assessment 2007–2011. 23Republic of Macedonia, MOES. Survey Report on Stakeholders’ Satisfaction with the Modernization Reform of the Technical Education in Macedonia Supported through the Skills Development and Innovation Support Project. Page 69 of 96 The World Bank Skills Development and Innovation Support Project (P128378) • Total of 3,635 students and 619 teachers were trained in their respective qualifications and acquired vocational, socioeconomic, and soft skills (including entrepreneurship skills) during the implementation of the Grant Program. • Eight school companies within the VET schools have been established through business guidance and mentoring. • Impact evaluation of the Grant Program24 found the below-listed effects of Grant Programs, especially the Measure 1 program of setting up a school company within the VET school and the Measures 2 program of placing students in practical training in companies: (a) Improved student performance in grant participating schools o Participating in any year of the Grant Program results in increases in student performance25 by the range of 3.4 percent to 4.6 percent. o Impact for girls is slightly larger: the grade increased by 4.3 percent for girls and 3.4 percent for boys. o The effect of Grant Program Measure 1 is the largest among the three grant measures: Grant Measure 1 increased student performance by 6.4 percent and Grant Measure 2 by 3.2 percent. o The observed increase in the average grade is driven by all subjects, especially subjects that are directly covered by the Grant Program. The program increased the performance of students in technical skills by 3.2 percent, verbal skills by 3.1 percent, and quantitative skills by 1.9 percent. o Especially, vocational training led to significant increases: improvement by 1.7 percent in technical skills, 3.5 percent in verbal skills, and 3.1 percent in quantitative skills. (b) Teachers who participated in the projects gained new knowledge and new experience through the problem-based learning program which provided teacher training for the development of more innovative pedagogies. (c) The estimated impact would be underestimated because the grant projects are likely to generate longer-term impact after the project cycle given that the estimated effects of the Grant program persisted over the measurable years (up to 5 years) after the implementation. Cost-Benefit Analysis 5. The economic analysis estimates the economic returns of the investment made through the project. The analysis estimates the NPV of the expected economic benefit and the cost of direct project beneficiaries. The CBA is conducted using the recent trends in secondary VET enrollments and returns on TVET measured through labor market outcomes of TVET students (Labor Force Survey 2017). Although positive results are recorded in the reports mentioned earlier, it is not straightforward to convert the outcomes to economic returns; hence, the analysis assumes that improved labor market relevance of the 24 Armand, Alex. Grant Program for Enhancing Collaboration between Schools and Business Community. Evaluation Report (the third year). 25 25 The average student grades was 3.16 out of 5 in the year before the implementation of the Grant program Page 70 of 96 The World Bank Skills Development and Innovation Support Project (P128378) VET program and the positive effects of the Grant Program on student performance will (a) increase the intake of TVET training among secondary students for the next five years and (b) produce quality premium reflected in the wage of beneficiaries. 6. The CBA shows that the modernizing TVET component is economically viable with the NPV ranging from Euro 6.3 million to Euro 12.2 million with EIRR of 7 to 10 percent (table 4.1). At appraisal, the CBA was not conducted, while recently completed projects in the TVET sector estimated the EIRR ranging from 10 to 18 percent. 26 The EIRR of the project overlaps the range of EIRR from other comparative projects. The CBA includes the sensitivity analysis with different sets of assumed impacts on intake and quality premium. The analysis assumes that the annual increase in the number of VET students ranges from 027 to 1 percent and that the quality premium ranges from 0.2 percent to 0.3 percent of monthly earnings of the TVET graduates. The parameters used for the analysis are described below. Table 4.1. NPV and EIRR Increase in intake: Low Increase in Intake: High (0% annual increase) (1% annual increase) Quality premium: Low (0.2%) NPV: 6.3 million NPV: 8.4 million EIRR: 7% EIRR: 8% Quality premium: High (0.3%) NPV: 10 million NPV: 12.2 million EIRR: 9% EIRR: 10% Secondary VET Trend and Parameter Description Used for the CBA 7. Over the past decade, the total enrollments in secondary VET schools have been declining. In 2011/2021, a total of 12,732 grade 1 students were enrolled in the secondary VET schools and in 2019/2020, the number decreased to 10,878, which is equivalent to 1.7 percent annual decrease in intake. In the meantime, the percentage of female students has been increasing in the recent five years, and in 2019/2020 it reached 46 percent. Central Europe and the Baltic countries have a similar trend as North Macedonia, (World Bank Databank, State statistical office in North Macedonia). 26 10 percent in the Sri Lanka Skills Development Project (ICRR0022195), 18 percent in the Bangladesh Skills Enhancement Project (ICRR0022000), and 18.5 percent in the Ghana Skills and Technology Development Project (ICRR0020874). 27 The actual trend shows that the number of secondary VET enrollments has been decreasing. This is discussed more in the next section. Page 71 of 96 The World Bank Skills Development and Innovation Support Project (P128378) 8. Based on the past trend up to 2019/2020, without any market-relevant quality assurance mechanisms in place, this trend would likely continue in the future. However, this current trend up to 2020/2021 does not reflect the outcomes of this component because the new curricula were applied since 2019/2020 and the ongoing pandemic is likely to have influenced enrollment across the education cycle. 9. In terms of parameters, the monthly average earnings of the TVET graduates are taken from the statistical office (2021). Earnings difference is assumed to be 3 percent based on the literature (Montenegro and Patrinos 2014) 28 . The inflation rate and long-term interest rates are used for the discount rate. Table 4.2Parameters for CBA Parameters Value Source/Note Average monthly earnings of TVET graduates 11,948 Value in Macedonian Denar. Statistical office, released in 2021 Earnings difference between secondary and 3% Montenegro and Patrinos (2014), secondary returns to secondary VET education: 1.7, tertiary: 7.2; one additional year of schooling: 5.7; assumption: 2% (low) to 3% (high) Inflation rate 1.2% World Bank Long-term interest rate 1.75% Northern Macedonia Bank Improved efficiencies through financing of innovation activities 10. At appraisal, public interventions and reforms in innovation were rationalized by possible acceleration of positive cross-firm and cross-sectoral spillovers from the seed innovation funding from the public sector. Furthermore, it was necessary to continue building the innovation environment through legal and institutional foundation by developing financing instruments and institutions to facilitate private sector innovation. Given the difficulty to measure the economic and social benefits of innovation due to large externalities, the long-term nature of innovation leading to concrete results, and hard-to-quantify research outputs, the analysis was complemented relaying on the international evidence as to the effect of the innovation fund (economic returns of the private R&D reaching 25 percent29) as well as successful cases with similar interventions in Croatia and Serbia. 11. The efficiency analysis at completion provides the achievement of the FITD and how the innovation environment has improved in the country over the project period, which can be partially contributed to the financing interventions in innovation. 12. Most of the target outcomes are accomplished by a wide margin. • Private funding financed the innovation activities of the FITD up to 49.53 percent, which exceeds the end target of 25 percent. • About 39 percent of beneficiary firms signed collaborative agreements with academia. 28 Montenegro C. and Patrinos H. (2014) “Comparable estimates of returns to schooling around the world”, Policy research working paper 7020, World Bank 29 Sveikauskas, L. 2007. “R&D and Productivity Growth: A Review of the Literature.” Bureau of Labor Statistics Working Paper 408. Page 72 of 96 The World Bank Skills Development and Innovation Support Project (P128378) • A total of 256 beneficiary firms introduced new or improved products and 11 firms introduced new innovation/business process, which surpasses the end target of 55 and 4, respectively. 13. Firm-level impact of the FITD on financing of innovation projects is positive and significant according to the impact evaluation study: 30 receiving the FITD awards led to an increase in the total innovation investment by 7.5 percent and investment in the FITD project by 3.8 percent31 at the firm level. 14. At the country level, compared to 2014, North Macedonia’s innovation environment improved more than the EU in various aspects. In 2014, the innovation index score in the country was 33 percent of the EU overall score and in 2021 it reached 47 percent.32 Among the 32 subdimensions, a half of them were improved relative to the EU. They are population with tertiary education, international scientific co- publications, most cited publications, foreign doctorate students, government support for business R&D, R&D expenditure in the business sector, employed information and communication technology specialists, public-private co-publications, job-to-job mobility of Human Resources in Science and Technology, the Patent Cooperation Treaty patent applications, employment in innovative enterprises, medium- and high-tech goods exports, knowledge-intensive services exports, sales of innovative products, resource productivity, and environment-related technologies. All three subdimensions of the sales impacts improved, which are closely related to the CPF’s key objectives, export-led growth. The medium- and high-tech goods export improved from 78.0 percent to 119.3 percent relative to the EU in 2014, knowledge-intensive service exports increased from 21.0 percent to 27.2 percent, and the sales of innovative products increased from 18.0 percent to 20.7 percent. In addition, other subdimensions relevant to the FITD improved: the Government’s support for business R&D sub-index increased to 3.4 percent relative to the EU from 0.2 percent in 2014 and public-private co-publications increased from 19.0 percent to 30.4 percent relative to the EU. Figure 4.1. Innovation Index in North Macedonia Relative to the EU 100 90 80 70 60 45 45 47 50 41 41 40 33 33 33 30 20 10 0 2014 2015 2016 2017 2018 2019 2020 2021 Source: European Innovation Scoreboard, 2014-2021 30 Carneiro, Pendro and A. Toppeta. Endline Report: Fund for Innovation and Technology Development (FITD) grant (2015 –2018 calls). 31 The estimate is not statistically significant due to small sample sizes at the firm level. 32 European Innovation Scoreboard 2021. Page 73 of 96 The World Bank Skills Development and Innovation Support Project (P128378) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Page 74 of 96 The World Bank Skills Development and Innovation Support Project (P128378) ANNEX 6. BORROWER’S REPORT Description of the project story line, including the operation’s context, rationale, and relevance of objectives during preparation and at completion33 The planning activities for the Project took place in 2012/2013 and in those years North Macedonia (Former Yugoslav Republic of Macedonia at that time), was very successful in maintaining macroeconomic stability, even throughout the economic crisis in the previous years. Nevertheless, the Government clearly detected the challenges facing the country on the road to transition to a higher growth trajectory by developing a more competitive and export-oriented enterprise sector. Furthermore, the Government concurred with the statements of the Macedonia Modular Competitiveness Assessment performed by the World Bank in 2011, where it was undoubtedly concluded that although large-scale FDI can serve as a catalyst for sustained, export-oriented growth at higher rates, the country must make much more substantial investments in skills and innovation. Since improving the country’s labor market performance and economic competitiveness requires a more skilled and better educated labor force, as well as increased technology absorption, diffusion of knowledge and innovation, it was obvious that the disconnect between the products of the education system and the private sector needs had to be addressed, especially as companies complained about the quality and availability of skills despite high unemployment. At the same time, the government was aware that the regulatory, institutional and financial environment had to be strengthened to further promote innovation at the firm level and improve commercial significance of its academic science and technology assets. The Higher Education sector faced serious challenges on many levels. While critical data on student learning and graduate employment outcomes did not exist, a 2010 World Bank employer survey of the demand for skills showed that employers continue to find it difficult to find workers with the skills required, particularly workers who possess the higher order of skills needed in the newly created jobs in modern and dynamic firms. The enhancement of quality and relevance of the higher education sector were further impeded by the lack of internal and external quality assurance mechanisms. The newly established Board for Higher Education Accreditation and Evaluation was focused only on the accreditation of institutions and study programs and the next logical step was to start planning for, and execute an external evaluation of higher education institutions, and subsequent linking of evaluation outcomes with institutional accreditation. That is why it was very important to introduce a component in the project, aimed at supporting the implementation of quality assurance mechanisms that are modern, fair, and participatory as well as aligned with the European Standards and Guidelines for Quality Assurance 33 This report was prepared by the SDISP project management team, April 2021. Page 75 of 96 The World Bank Skills Development and Innovation Support Project (P128378) In the area of Secondary Vocational Education , the country than offered four-year vocational education in 14 occupations with 50 educational profiles, as well as visual art, music and sports education, and three- year programs with 36 educational profiles. Over half of secondary education students aged 15-19 attended VET schools. During the preparatory activities, special attention was given to the need to align the efforts with the activities of local institutions and international organizations active in the field. At that time, the EU activities were aimed at three main areas of intervention: Support and modernization of the education and training system in the 2and 3y long VET ; With the activities of this Twinning project headed by the Center for Vocational Education and Training, the process of reforming the 2 and 3 year vocational education started. The activities were focused on two main components "Preparation of standards for professional qualifications and reform of curricula for two and three years of education" and "Training of trainers". Support for the integration of ethnic communities in the education system This was another Twinning project of the Office for Development and Promotion of Education in the languages of the members of the ethnic communities. Within the project, 2 components were implemented: Introduction of intercultural elements in the education system and strengthening the capacities to support the integration of the ethnic communities. Support for capacity building of the Center for Adult Education and development of programs for adult education and programs for literacy and completion of primary education for excluded persons In this Twinning project three components were realized: Capacity building and training of the staff of the Center for Adult Education, Development of programs for adult education, and Development of programs for literacy and completion of primary education for socially excluded persons. As one can see, the “bulk” of the VET sector, the 4 year TVET where most of the students could be found, was not addressed by any of the main players, domestic or international . The fact that the TVET system was predominantly supply-driven, rather than market-oriented, using existing programs that were dated and using existing teachers and facilities, rather than adapting to changes in the market and in the demographic situation, asked for complete overhaul of the TVET, in order to be able to answer the needs of the 21st century society and industry. Other problems of the TVET system was the fact that it was characterized by early diversification at the age of 14, excessive specialization, obsolete program content, and lack of collaboration among schools and employers. An academic consensus that early vocational tracking may harm students and that diversification should start after students have acquired strong basic skills was present in the country. Early diversification and excessive specialization have led to a narrow skill base in graduates and particularly to the lack of foundation skills needed for lifelong learning demanded by modern economies. Finally, the lack of investment in secondary TVET has left schools with obsolete and worn-out facilities and equipment. Teachers and instructors lacked opportunities for in-service training and are consequently Page 76 of 96 The World Bank Skills Development and Innovation Support Project (P128378) unaware of new good practices and more effective teaching methods. The need for school equipment modernization was recognized as one of the areas of intervention by the project. The Innovation System in the country was negligent, dominated by the public sector and detached from industry, and in need of significant strengthening in order to be able to support the country’s long term vision to become a knowledge-based economy. In general, the innovative capacity of Macedonian firms was low both in terms of human capital and financial resources for R&D and innovation. Brain drain particularly in the technical and engineering occupations was and still remains a major concern, not only for the private sector, but also for the scientific community and policy makers. Certain legislative prerequisites for addressing the deficiencies were made since the Government of N. Macedonia took the first steps towards strengthening the legal framework of innovation by adopting the Innovation Strategy in October 2012 and the Law on Innovation Activity in May 2013. The Law on Innovation designated the Fund for Innovation and Technological Development (FITD) as the entity to finance the Government’s interventions in the area of innovation. The FITD was supposed to introduce financial instruments to support new start-ups and spin-offs, commercialization of technology and technology transfer activities thus connecting research and the market, while at the same time creating a source of R&D funding and jumpstarting Macedonia’s National Innovation System. Various instruments have been designed to create a foundation for the overall innovation commercialization ecosystem with emphasis on improved exports through enhanced competitiveness. Although the project is ending, the relevance of the objectives is of immense importance to the Government of the RNM, which remains committed to further enhancements of the newly established HE quality assurance system and innovation ecosystem. As to Higher Education (HE), key milestones have been achieved, with the formation of the new Agency for Quality Assurance (AQA) as an independent legal body responsible for the quality of HEI, which should further implement new accreditation and evaluation standards and criteria as well as its development strategy and implementation plan for 2021-2023 in order to achieve full membership in ENQA. The National Council for Higher Education and Research has been officially formed on 21th of April, 2021 by the Parliament of the RNM (published in the Official Gazette no.88/2021) responsible to adopt new funding model decree as well as new quality assurance standards and criteria.This reform is one of the high priorities in the Government Program for 2021/22. Furthermore, MoES has initiated development of a new integrated educational management information system for collecting and processing the data included in the educational process of the Republic of North Macedonia, with the assistance of the EU Delegation in North Macedonia. In designing the system the MoES mostly relied on studies, assessments and designs that were produced within the project activities related to designing financing formulas, assessment of HEI information systems and the efforts related to the design of the Skills observatory. The system should simplify the work processes in the MoES and other institutions related to the management of education in primary and secondary schools and higher education institutions (HEIs). The proposed system should be a completely new solution that will accelerate the digital transformation of the institutions involved, and increase their efficiency and effectiveness. Page 77 of 96 The World Bank Skills Development and Innovation Support Project (P128378) The Government has recognized the important role of the Fund for Innovation and Technology Development (FITD) as economy’s main vehicle and will continue supporting FITD financial instruments to innovative firms and researchers, through the state budget. Besides the startup community support, FITD and MoES will continue to strengthen the capacity of human capital in innovation by support mechanisms for researchers such as the Challenge for Young Researchers competition as well as support for school-aged children to participate in international STEM competitions. FITD and MoES joint activities will continue with financial support promoting business-academia collaboration in the form of innovation vouchers and new instrument for competitive co-operative grants which is in the consultation process with relevant stakeholders, and is planned to be piloted by the end of 2021. North Macedonia has made progress to achieve its national STI objectives and improve its overall STI policy framework by the support of the WB SDISP. Assessment of the outcome of the operation against the agreed objectives; with a focus on providing evidence of the achievement of the operation’s objectives along with the contribution of the supported activities and outputs to the project’s development outcomes Component 1: Improving Transparency of Higher Education This component has three main sets of activities, all targeted toward improving both the transparency and efficiency of the quality assurance and finance mechanisms in the higher education sector: (i) institutional strengthening and stronger quality assurance in higher education;(ii) reforming the higher education financing model; and (iii) establishing a centralized office to foster mission-oriented research and university-industry collaboration. (i) providing training for improving the administrative capacity of the Higher Education Accreditation and Evaluation Board (HEAEB) and other key players in charge of managing quality assurance activities. SDISP supported the establishment of a sound quality assurance system of the higher education (HE) sector and partially achieved the PDO due to the long period of political transition that resulted with regulatory status quo and the time needed to implement the provisions related to the quality assurance prescribed in the new Law on Higher Education, once it was voted . The new Law on HE (Official Gazette of the RM no.82/2018) that came at the later stage of the project and redefined the role of the key players in the area of HE, stipulates establishment of the Agency for quality in higher education as independent legal entity, (this partially complies with standard 3.3 of ESG on independence, one of the requirements for achieving full membership in the European Association for Quality Assurance (ENQA) and the European Quality Assurance Register (EQAR)), responsible for accreditation and institutional evaluation of the HE Institutions, as well as establishment of the National Council as a new independent body of experts, responsible for HE, science and research. The process of the establishment of new Agency has been finalized in the Q1 of 2020. SDISP supported the improvement of the administrative capacities of the Agency for quality in higher education, the MoE’s and other key actors in managing quality assurance activities. The administrative capacity of the Agency has been strengthened with four professionals provided by the Project that represent the core administrative capacity of the Agency (communication and program officer, IT officer, economist and legal officer). Page 78 of 96 The World Bank Skills Development and Innovation Support Project (P128378) The project funded technical assistance for meeting the requirements for membership in the ENQA and the EQAR with the main goal of establishing an internationally recognized Quality Assurance (QA) body in North Macedonia. Through this reform process, a set of QA documents was prepared including strategy for institutional development, recommendation for new organizational structure and implementation plan with short, medium, and long-term activities and goals. Development of the Agency website as well as populating the web site with relevant data (http://akvo.mk) has been supported under the project as well, and all accreditation decisions are available for the public and stakeholders on this web site. This is one milestone in the Agency efforts to increase the transparency of its work. The management of the Agency, Board members & stakeholder community benefited trough the technical support provided by the international and national consultants during the consultation and participative process of development of the main strategic documents and legal framework of the Agency. The Agency has become an affiliate member of ENQA on 17 September 2020, which is the first and important step towards full membership in this association. (ii) funding of external evaluation by foreign experts of N. Macedonia’s higher education sector As part of the quality assurance initiatives, to provide Macedonian HEIs with institutional evaluation experiences that follow the best European practice, one cycle of coordinated external evaluations of six public HEIs was carried out. The overall objective was to provide an assessment of the current situation in the national higher education sector, including areas for improvement and investment, and recommendations for priority interventions. The entire process was completed by an independent service of the European University Association (EUA), i.e. the Institutional Evaluation Program (IEP), which is listed on the European Quality Assurance Register for Higher Education (EQAR) and is a full member of the European Association for Quality Assurance in Higher Education (ENQA). The external evaluation took place between April 2016 and May 2018 and six institutions were evaluated in the framework of this exercise: St. Clement of Ohrid University, State University of Tetovo, St. Apostle Paul University of Informatics Sciences and Technologies, South East European University, Ss. Cyril and Methodius University and Goce Delchev University. Each evaluation was carried out by a team of experts from across Europe and resulted in an institutional report presenting areas of strength and recommendations for further development (reports are available at: https://www.iep-qaa.org/reports- publications.html?utm_source=webpage&utm_medium=News&utm_name=News-webpage-26-04-2018) and one system review report (https://www.iep- qaa.org/downloads/publications/iep_mk_report_final.pdf) which highlighted the shared issues and challenges, facilitating a dialogue among all key actors and stakeholders, including the government authorities. The reports provided a set of recommendations for improving the identified deficiencies, both at institutional and at national level, which served as valuable input for further development and improvement of institutions, as well as the overall system. The main findings of the system review report were discussed with approximately 30 representatives from the evaluated institutions and national authorities during a post-evaluation workshop in Skopje on 19 April 2018, organized by the Project. Discussions focused mainly on challenges related to learning and teaching and research, and also highlighted the importance of cooperation and exchange of good practice between universities within the country and in the wider region. This particular process has significantly contributed to the improvement of the HEIs’ management and governance of the entire self-evaluation process ,from selecting the members of the self-evaluation group and ideas on how to involve the university community in the process, gathering initial information to Page 79 of 96 The World Bank Skills Development and Innovation Support Project (P128378) collecting feedback on the draft self-evaluation report, as well as to the development of new , internal quality assurance procedures. The measures taken on a system level can clearly be seen in the strategic and legislative documents adopted in the years that followed the evaluation. Crucial changes related to: • establishing an independent Agency for quality assurance of higher education, • developing a set of QA documents for accreditation and evaluation processes, harmonized with Bologna norms and practices, • developing of new Law on science & research activity, as well as well as 30% increasing in the national budget for science and research in 2021 compared to 2020 • enhancement of the collaboration between academia & business sector by introducing innovational vouchers in partnership with the Fund for innovation and technology development (first call published on 01.10.2020) The Innovation Vouchers have been piloted on 01.10.2020, and 71 applications have been received prior to closure, 34 applications have been approved for awarding an Innovation Voucher. https://fitr.mk/javen-povik-za-dodeluvane-finansiski-sredstva- preku-inovaciski-vauceri/) These reports continue to be a valuable foundation for further development, both for the MoES as a body responsible for the entire HE system, as well as for HEIs themselves in the development of their own policies. The entire process also provided all participants, particularly HEIs, with valuable practice in self- evaluation procedures in line with the best European practice. (iii) Upgrading of the education management and information system, including the development and implementation of the central data base In order to support the process of improving the quality of higher education, the project provided technical assistance in the definition of the business processes in order to design a central database for higher education (Education Management Information System for higher education - HEMIS) as a tool for monitoring the operation and outputs of the tertiary education. The design of the system covers all aspects of tertiary education including student affairs, academic programs, human resources management, budgeting and financial management, and infrastructure management. The design also includes a module for accreditation and evaluation processes for enhancement the work of the Agency for quality in HE. This activity resulted in a production of a document that is comprised of assessment of all related higher education legislation, as well as assessments of the information systems (IS) of the majority accredited HEIs in order to establish their (IS) compatibility and adequacy having in mind the modern data needs. These analyses showed that in order to ensure greater efficiency in the daily operation of HEIs and to improve the process of data exchange between the Ministry of Education and Science and HEIs, it is necessary to establish a comprehensive and data rich information system that will contain the functionalities needed for the central database for higher education as well as all functionalities needed for the information systems at HEI level. In other words, in order to produce an overarching software that would integrate the software’s of the Universities for the needs of data analysis at Ministry level, the Project had to first design the software’s Page 80 of 96 The World Bank Skills Development and Innovation Support Project (P128378) that would be used at the University levels, upon realization of the fact that they are lacking quality and substance or area even entirely non existing in some case. This was done in a time consuming process, with the participation of each end every HEI in order to include everyone’s specific need in the design. The working group meetings with HEI representatives provided an opportunity for sharing ideas and individual experiences as well as establishing a mutual cooperation and trust between HEI & MoES. The contract for the development of comprehensive software solution was not awarded due to insufficient funding to cover the proposed cost. However the technical specification is produced and represents a solid base for the development of integrated educational management information system within the MoES, funded by EU delegation. The MoES staff, ministerial agencies, university management and stakeholder community benefited trough the technical support provided by the international consultants during the inclusive process (organized workshops, interviews, work meetings) of development of the key documents for Higher Education Management Information System (HEMIS). In addition, at the university level, the Project provided recommendations for upgrading of existing HEI information systems in terms of improvement data collection processes in order to provide evidence-based analyzes that will optimize daily operations and strengthen the governance of HEIs. In addition, the HEI leadership and governing bodies will use analyzes for strategic planning, quality assurance of learning and teaching processes, research activities, international mobility of students and university staff as well as for efficient allocation of the human and financial resources in realization of the HEI mission. Sub-component 1.2: Higher Education Financing Reforms. The HE financing reform is crucial for modernization of Macedonian higher education system, as the analyses of the current financing model pointed to its serious deficiencies. The system is in need of a sustainable funding strategy that would build on global good practices, in order to design and put in place an effective resource allocation model adapted to the reality of North Macedonia and address the specific challenges faced by its HEIs, promoting flexibility and innovation throughout the higher education system. The Project supported technical assistance for design of a performance-based funding model promoting transparency and efficiency in the resource allocation mechanisms. This reform process included preparation of a comprehensive analysis of the performance of NM higher education funding system, with recommendations on how to improve it, the proposal of a new funding model aimed at achieving financial sustainability, and a detailed roadmap to guide its implementation, including transitional arrangements for the Ministry of Finance, the Ministry of Education and Science, and the universities. The proposed new funding model could be translated into legislation only after the formation of the National Council for Higher Education and Research, since the Law on Higher Education (Official Gazette of the RM no.82/2018) provides legal basis for establishment of the National Council as a new independent body of experts, responsible for higher education, science & research, authorized to prepare and propose the HE Financing Decree to the Government. This high authority body has been officially formed in April, 2021 by the Parliament of the RNM). After the enactment of the new Law on HE, this was initially delayed due to postponement of the parliamentary elections. The work on the new financing model has received support and commitment by universities to fully participate in designing the new funding model, and they have also benefited from the technical support provided by the international consultant during the consultation process of Page 81 of 96 The World Bank Skills Development and Innovation Support Project (P128378) organized workshops, interviews, working meetings discussing the draft proposals and finalizing the new financing model. This reform is one of the priorities in the Government Program for 2021 and the MoES will make sure that the newly formed National Council makes use of the documents produced under the project that deal with the matter of financing. Sub-component 1.3: Development of a National Technology Transfer Office (NTTO). Project supported the technical assistance in analysis and needs assessment for establishment of a National Technology Transfer Office, development of the legal framework and NTTO core strategic documents, organizational structure and qualification for the NTTO team (Legal Expert responsible for IPR and legal issues, Business Developer, Finance Expert, PR and Marketing Manager, Office Administrator and IT Expert). In 2016, Law on Innovation Activity was amended to grant the Fund for Innovation and Technological Development with a legal mandate to be one of the founders, as Government has decided to incorporate NTTO within the Science & Technology Park, planned to be developed in the following years. Modernization of Secondary Technical Vocational Education and Training The activities within this component for development and design of the concept for secondary TVET resulted with finalization of numerous systematic and program documents. According to the new Methodology for development of the occupational standards and based on the 18 trainings organized for over 190 members of the National Expert Teams (NETs), 152 occupational standards have been developed and adopted by Ministry of Labor and Social Policy. Based on this Methodology, 52 qualification standards and syllabi for each of them have been developed and adopted. The development of these standards has been conducted by the NETs involving around 220 representatives from the business and education sector and by organizing 16 (two-day) trainings for all members from the NETs. More than fifteen workshops were organized for writing the course aims, learning outcomes and assessment criteria according to the adopted Template for modular designed education program for the national expert teams, established for development of the curricula for both general and vocational modules/subjects in coordination with the advisors from VET Centre and BDE. The established NETs including around 650 members from the economy and education sector have developed all 100% (a total number of 1182 out of 1182) new modular curricula for both general and vocational subjects for 1 st, 2nd , 3rd and 4th year for all 52 qualifications within all 14 sectors (out of which 1094 curricula is for vocational subjects under the VET Centre responsibility and 88 for general subjects under the Burro for Development of Education (BDE) responsibility). From the school year 2019/2020 the new modular competence based curricula for all 52 qualifications have been implementing in all 76 VET schools in the country. For the purpose of smooth implementation of the new concept in the TVET secondary schools, the VET Center and BDE supported by the international team for technical support have trained 50 master trainers (VET Centre and BDE advisors and secondary school teachers) who trained around 2400 vocational and general education teachers, managers and professional associates from all secondary TVET schools in Page 82 of 96 The World Bank Skills Development and Innovation Support Project (P128378) North Macedonia. Training was conducted throughout the project starting from the train of trainers and pilot schools in June 2017 and September 2017 respectively, up to the final round of peer to peer sessions ending in May 2019. Additional training has been delivered from 20th – 30th August 2019 to 1500 teachers to train them for the start of the new school year 2019/2020 where the new competency based modular curricula was implemented in all secondary vocational schools across the country. Based on the collected data from the questionnaires sent to all 74 secondary VET schools in North Macedonia as well as focus group meetings, the Report for assessment of the VET schools network, a cost- benefit analysis and proposal for network rationalization have been prepared. In 2018, survey for the Stakeholders’ satisfaction with the modernization reform of the technical education in North Macedonia was conducted. The total number of stakeholders who were included in the survey was 556. The study finds that the stakeholders are highly satisfied with the quality of the performed activities for the modernization of the TVET. Furthermore, the stakeholders’ satisfaction from the quality of the developed methodologies for implementation of the various stages of the modernization reform is at high levels, varying from moderate satisfaction to high satisfaction. The stakeholders’ satisfaction from the support provided by the consultants hired under the Project is also on a high level and varies from high satisfaction to moderate satisfaction depending on the stages. The role of the Project Management Unit was also evaluated on a high level especially for coordination of the activities of the working groups and proper communication of the activities among the stakeholders. Without hesitation, one can say that several decades after the last reform of the VET system, for the first time, students have the chance to study VET under programs with clear site of the end results that each program needs to accomplish , programs that are not only produced to fulfill the needs of the industry, but written with outmost participation of the industry professions. As part of the input provided in order to enhance the quality of VET in the country, basic and vocational equipment for five sectors (Agriculture, fishery and veterinarian sector; Construction and geodesy sector; Graphics sector; Economics, law and trade sector; and Hospitality and tourism sector) has been procured and delivered. ➢ Grants for TVET schools for school-industry collaboration This subcomponent financed activities aimed at designing and implementing a Grant program to support TVET component and provision of school Grants to selected TVET schools to finance activities to improve the labor market relevance of the vocational education and training including involving industries into the practical training delivery. The Operational Manual for Implementing the Grant program for enhancing collaboration between schools and business community was designed in 2016, anticipating 5 Grant measures, aimed at improving the practical training of students and their employability skills. Page 83 of 96 The World Bank Skills Development and Innovation Support Project (P128378) After the first round of implementation in 2016, the design of the Grant program was simplified and reduced from 5 to 3 grant measures in accordance with the recommendation of Impact evaluation of the program: Grant measure 1: Setting up a school company within the VET school through business guidance and mentoring (business mentoring relationship); Grant measure 2: Placement of students in practical training in companies; Grant measure 3: Establishment of problem-based learning courses for enhancement of soft skills. These 2 grant measures were excluded in the second 2 years of the implementation of the Grant program: 1. Involvement of students in the In-house training courses at employers’ place 2. Establishment of the summer camps learning programs within the schools The reason why these 2 Grant measures were excluded in the next 2 years of the implementation of the Grant Program was to simplify the configuration of the program as it was recommended in the Grant Program Impact Evaluation Report. There was a lot of overlapping that makes the utilization of different Grant programs burdensome. Therefore, we simplified the design of the program in 3 grant types targeting on work based vocational training and soft-skills training Total of three Calls for application for Grant program school-industry collaboration were announced in 2016, 2018 and 2019 and total of 232 applications were received. 2016/2017 received awarded Grant measure 1:Setting up a school company within the VET school through business guidance and mentoring (business mentoring relationship) 17 2 Grant measure 2: Involvement of students in the In-house training courses at employers place 23 2 Grant measure 3: Placement of students in practical training in companies; 29 2 Grant measure 4: Establishment of problem based learning courses of a small scale 15 2 Grant measure 5: Establishment of the summer camps learning programs within the schools 26 2 TOTAL 110 10 2018/2019 received awarded Grant measure 1: Setting up a school company within the VET school through business guidance and mentoring (business mentoring relationship) 16 3 Grant measure 2: Placement of students in practical training in companies 12 3 Grant measure 3: Establishment of problem based learning courses for enhacement of soft skills 30 16 TOTAL 58 22 2019/2020 received awarded Grant measure 1: Setting up a school company within the VET school through business guidance and mentoring (business mentoring relationship) 12 3 Grant measure 2: Placement of students in practical training in companies 15 3 Grant measure 3: Establishment of problem based learning courses for enhacement of soft skills 37 3 TOTAL 64 9 Page 84 of 96 The World Bank Skills Development and Innovation Support Project (P128378) School year received awarded 2016/2017 110 10 2018/2019 58 23 2019/2020 64 9 TOTAL 232 41 Total of 3635 students and 619 teachers were trained in their respective qualifications and acquired vocational, socio-economic and soft skills (including entrepreneurship skills) during the Grant program implementation. Eight school companies within the VET schools have been established through business guidance and mentoring. The established school companies were more focused on the provision of practical training for the students and teachers but few of them also generated and reported an income in the first year and were assessed as a sustainable success story by the Performance audit team. The Impact Evaluation report shows that the Grant Program overall generated a positive impact among students enrolled in the participating schools in terms of student performance, including improvement of students both vocational and soft skills. The average grade went up as a result of the Grant program, with larger impacts in schools that implemented earlier projects. The student grades were increased in all subjects including the ones more specific to TVET schools such as technical subjects and for both male and female students, with slightly larger effects for female students. Grant measures 1 and 2 generated the largest improvements in student performance The goals set for each of the Grant measures were completely fulfilled. Not only did the collaboration between the VET schools and the business community improve through the business guidance and mentoring, it also helped in preparing the students for the world-of-work and made them more competent on the labor market. It helped them gain vocational, entrepreneur skills and especially soft skills as demanded by the companies. There were several cases where the students were employed by the VET school partner companies after their graduation. An excellent example of the Grant program’s impact on increased confidence of the business sector was hiring well-trained students, by establishing strong relations with the VET schools that would provide opportunities for the companies to find employees who are more “job ready”, thereby helping to reduce their recruitment and training costs. Additionally, the teachers also benefited from the trainings, especially those involved in running the VET companies, as they were given a rare opportunity to receive first-hand experience of being part of a work environment and being exposed to the practical use of the methods, tools, and technologies taught in the classroom. The established school companies under the Grant measure 1 also helped improve the image of the VET schools in general, in different regions of the country, and increased the percentage of enrolled students, especially female students. Few qualifications that were “unattractive” for a long period due to the lack of practical training have now become more attractive to students. The income that was generated by the school companies was re-invested into improving the school’s infrastructure, as well as purchasing equipment for improving the quality of the practical training in the respective sectors. Some of the Page 85 of 96 The World Bank Skills Development and Innovation Support Project (P128378) municipalities have recognized the importance and benefits of the school companies and helped VET schools in connecting with potential clients. This support was further strengthened, as the municipalities became the schools’ clients by utilizing the services that their companies provided. The placement of students in the practical training in companies within the Grant measure 2 was an equally valuable opportunity for the VET students to gain knowledge about the actual process in the world of business, from the very beginning of their educational process. In some regions, the Grant program provided a first-time opportunity for VET schools to place their students in a companies, allowing them to gain practical training in accordance with their syllabus over the course of the school year. Furthermore, several partner companies employed the trainees after graduation, once again reducing the company’s recruitment and training costs and giving them the unique opportunity to train their future employees in line with the company’s needs and demands. One of the greatest challenges for every VET school is to produce students who are not just well trained, but with developed soft skills, equally valued and demanded by the business sector. The problem-based learning courses under the Grant measure 3 helped both the students and teachers develop or enhance soft skills and finalized the process of training within the Grant program, helping each student join the labor market - fully enabled and prepared to respond to the obligations of the workplace. In essence, the great interest in these courses throughout the entire Grant program implementation goes to prove the fact that VET education needs of soft skill training was met by the implementation of the Grant measure 3. The participation of the VET teachers in the soft skills courses allowed enhancement of their soft skills and enabled the sustainability of this measure. Component 3: Improving the Innovative Capacity of Enterprises and Collaboration with Research Organizations This component is focused on supporting the efforts of the Fund for Innovation and Technological Development (FITD) to build the institutional capacity to stimulate innovative activities, and piloting financial instruments to support innovation and technological development in the enterprise sector. It is organized around two sub-components. Sub-component 3.1: FITD capacity building. This sub-component is financing activities required to build up institutional competence of the FITD, in particular, providing support for: (i) planning and designing the programs, strategy, operations and procedures; (ii) training of FITD staff; (iii) selection of the Investment Committee and peer reviewers; (iv) mentoring and training for FITD beneficiary enterprises; and (v) marketing and communications strategy. Sub-component 3.2: Pilot of Financial Instruments to be delivered by FITD. Once the FITD is established, it would provide specific funding instruments for each stage of companies’ life cycle to foster innovation. These instruments include: (i) an accelerator; (ii) proof of concept innovation mini grants, including but not limited to IP protection, and business plan preparation for initial capital mobilization; (iii) commercialization matching grants and loans; and (iv) sector specific grants in technology development and technology absorption projects for new or improved technologies, products and processes. It is anticipated that the pilot instruments will be introduced in a phased approach starting with the accelerator and proof of concept grants. Page 86 of 96 The World Bank Skills Development and Innovation Support Project (P128378) OBJECTIVES: 1. 100 % Share of resources dedicated to Fund for Innovation and Technology Development (FITD) instruments disbursed 2. 15 companies ‘accelerated’ via the FITD Accelerator program 3. 55 of beneficiary firms to introduce new/improved products 4. 4 of beneficiary firms to introduce new processes 5. 70% of beneficiaries receiving training and mentoring OVERALL PROGRESS WITH RESPECT TO OBJECTIVES: The Fund for Innovation and Technology Development (hereinafter: the Fund) develops and performs services and allocates financial resources according to the Law on Innovation, Innovation Strategy of the Republic of North Macedonia 2012-2020, bylaws and work programs of the Fund. The mission of the Fund is to encourage and support innovation activities in micro, small and medium- size enterprises (MSMEs) in order to achieve more dynamic technological development based on knowledge transfer, development research and innovation that contribute to job creation, and to economic growth and development, while simultaneously improving the business environment for the development of competitive capabilities of companies. Priorities and objectives of the Fund are: improved access to financial support for innovation and technological development and promoting and encouraging innovation activities in the Republic of North Macedonia. Instruments developed and financed via the World bank’s loan: • Co-financed Grants for Newly Established Enterprises: Start-up and Spin-off • Co-financed Grants for Commercialization of Innovations • Co-financed Grants for Technology Extension • Co-financed Grants for the Establishment, Operations and Investments of Business and Technological Accelerators Instruments developed and financed under the Plan for economic growth (pillar 3): • Co-financed grants for technology development • Co-financed grants for improvement of innovation and • Co-financed Grants for professional development and practice of newly employed young people • Co-financed grants for technology development for overcoming the consequences of COVID19 • Innovation vouchers and • Co-financed grants for technology development for accelerated economic growth Page 87 of 96 The World Bank Skills Development and Innovation Support Project (P128378) The instrument “Co-financed grants for technology development for overcoming the consequences of COVID19” was developed in y2020 as a result of the crisis caused by the Covid19 pandemic and was part of the 3rd government package for recovery while the instrument “Co-financed grants for technology development for accelerated economic growth” was developed in January 2021 and is part of the government 5th package for support of companies in accelerating their growth related with the consequences from the crisis caused by the Covid19 pandemic As of 31 December 2020, the Fund has 8 permanent employees and 47 with service contracts. The Fund implements the Component 3 of the Skills Development and Innovation Support Project. This component is focused on supporting the efforts of the Fund to build the institutional capacity to stimulate innovative activities, and piloting financial instruments to support innovation and technological development in the private sector. It is organized around two sub-components: • Fund’s capacity building - planning and designing of programs, strategy, operations and procedures; training of Fund’s staff; selection of the Investment Committee and peer reviewers; mentoring and training for Fund’s beneficiary enterprises; marketing and communications strategy and • Pilot of Financial Instruments to be delivered by the Fund – once established, the Fund provides specific funding instruments for each stage of companies’ life cycle to foster innovation The project applications which are supported by the Fund are selected on public open calls and are evaluated and implemented in accordance with the procedures and requirements set forth in the POM (Project Operational Manual), the IGOM (Rulebook on Management of the Support Instruments of the Fund for Innovations and Technology Development), the Procurement and Consultants Guidelines, the Anti-Corruption Guidelines, the Environmental Management Framework (EMF) and any applicable Environmental Management Procedures. The project proposals are evaluated by Committee for approval of investments consisting of 5 international experts in investments and innovation. As of March 31st, 2021, the FITD portfolio consists of 544 supported project proposals with total volume of portfolio of 74,477,014.8 eur where the beneficiaries are financing 43% of the total investment and FITD provides 57% or: Total volume of portfolio 74,477,014.80 € FITD co-financing 42,137,331.63 € Beneficiary's co-financing 32,339,683.17 € From the total number of supported projects, 185 are financed via the World bank loan, 281 are from the Plan for economic growth (pillar 3) and 78 are from the call for “Co-financed grants for commercialization of innovation” where the budget is combined (WB loan and FITD): FITD portfolio per financing programme (in number): Plan for economic growth (51.65%) 281 51.65% World bank SDISP (34.01%) 185 34.01% WB SDISP/PEG (14.34%) 78 14.34% Total 544 100.00% Page 88 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Or volume of support per support programme: FITD portfolio per financing programme (in volume) Plan for economic growth (50.11%) 21,114,742.79 € 50.11% WB SDISP/PEG (30.39%) 12,806,865.34 € 30.39% World bank SDISP (19.50%) 8,215,723.50 € 19.50% Total 42,137,331.63 € 100.00% The structure of the beneficiaries per support instrument is as follows: Start-up & Spin-off (30.51%) 166 30.51% Improvement of innovation (19.67%) 107 19.67% Commercialization of innovation (16.73%) 91 16.73% Technology development COVID19 (16.18%) 88 16.18% Technology development (15.81%) 86 15.81% Technology extension (0.55%) 3 0.55% Business Accelerator (0.55%) 3 0.55% Total 544 100.00% And the invested volume per support instrument is as follows: Improvement of innovation (33.22%) 13,999,378.54 € 33.22% Commercialization of innovation (32.97%) 13,893,673.64 € 32.97% Technology development (13.26%) 5,587,625.45 € 13.26% Start-up & Spin-off (10.89%) 4,588,533.99 € 10.89% Technology development COVID19 (3.63%) 1,527,738.79 € 3.63% Business accelerator (3.43%) 1,443,495.00 € 3.43% Technology extension (2.60%) 1,096,886.22 € 2.60% Total 42,137,331.63 € 100.00% The portfolio per size of the companies that are supported is the following: Micro (48.16%) 262 Small (43.57%) 237 Medium (6.99%) 38 Large (1,29%) 7 Total 544 Total portfolio per industry is: IT (29.00%) 158 Engineering & Technology (15.8%) 86 Health & Medical research (7.7%) 42 Marketing & Sales (6,4%) 35 Energy & energetic resources (5.1%) 28 Textile & Leather (4.8%) 26 Construction (4.8%) 26 Education (4.4%) 24 Page 89 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Agriculture (4.2%) 23 Creative industry (3.9%) 21 Food processing (3.7%) 20 Hospitality (3.1%) 17 Ecology (2.6%) 14 Biochemistry & Biomedicine (1.5%) 8 Furniture & Wood processing (1.3%) 7 Economy (0.9%) 5 Transport (0.7%) 4 Total 544 FITD portfolio according to the number of companies where women are as owners and/or managers: Analitics : Woman as owners/managers companies with women owners/managers 199 37% companies with men owners/managers 345 63% Total 544 100% In the table below we present how was the dynamics of development of FITD calls throughout the years in number of applications received, supported projects and budgets per call: FITD growth in number of applications and number of supported projects: Number of Number of Call Approval rate projects applied supported projects Call I (2015) 76 16 21.05% Call II (2015) (2 instruments) 82 16 19.51% Call III (2016) 46 4 8.70% Call IV (2017) (2 instruments) 59 15 25.42% PEG (2018) (4 instruments) 244 131 53.69% Call I Accelerator (2018) 13 3 23.08% Call I Tech. extension (2018) 8 3 37.50% Call V (2019) (Startapuvaj1 2018) 186 44 23.66% Call VI (2019) (commercialization 2019) 487 78 16.02% Call II (2019) Tech. developement 121 62 51.24% Startapuvaj 2 2019 235 84 35.74% Tech-COVID 19 171 88 51.46% Total 1728 544 31.48% Page 90 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Or on yearly basis: 532 482 451 207 158 181 105 46 59 32 15 4 2015 2016 2017 2018 2019 2020 Received projects Supported projects Poly. (Received projects) Poly. (Supported projects) Component 4: Project Management and Monitoring and Evaluation Project Management Unit (PMU) reporting to a Project Director was responsible for all the day-to-day project implementation activities, as well as procurement, disbursement and accounting functions. ➢ Evaluation of the impact of FITD instruments The data collection for baseline survey for the needs of the evaluation of the instruments for innovation support of the Fund for Innovation and Technology was completed in June 2019. Total of 179 companies from the sample size were interviewed. According to the analyzed data for the baseline survey on average the companies were founded around 2009 and the majority is located in Skopje. 92 percent of all the firms interviewed perceive the innovation activity they engage in to be novel at the national level, while only 39 of all the firms interviewed perceive it to be novel at the global world scale. According to the evaluation outcomes, the instruments have a positive effect on the number of employees that the firm hires. The data collection for the follow up survey has finished in September 2020 and the follow up report was submitted at the end of April 2021.According to the report, the estimates are fairly imprecise, but it does not rule out that the program has positive impacts on firms’ innovation and productivity. Subsequent calls to the ones that were included in the survey were probably more successful in achieving this objective and produced larger gains that could potentially be measured. Another possibility is that there are groups of firms for which innovation was effectively boosted even if this did not happen generally across grantees. There is a possibility in the future to expand the available data with new applicants from additional calls for FITD grants, so this can increase the sample sizes and would help in improving the precision of the estimates. ➢ Skills observatory The lack of information on the postgraduate performance of secondary and HE students and their employability, has been determined as one of the challenges that need to be addressed in order to make Page 91 of 96 The World Bank Skills Development and Innovation Support Project (P128378) this information available to the government officials. Skills Observatory aims at supporting the decision- making process with evidence – based information, in the context of educational and labor market policies. Based on these, the Skills Observatory has been developed as a web-based system that enable the collection of electronic information from secondary schools, higher education institutions and employment service agency and its presentation in the form of statistical reports. The main aim of the Skills Observatory is to provide information on existing supply of training programs at secondary and academic levels, their performance, their distribution across the national territory, their curricular details, placement of graduates in the labor market, in order to comparison between training/educational institutions, planning of new programs and adjustment of curricula to labor market demands. In addition, the Skills Observatory is a new source of information useful for future students and parents in selecting their future school/university as well as the occupational field which has the most promising employment prospect. The Skills Observatory has been established by the project, and MoES staff were trained for further maintenance and upgrading of the system to ensure its sustainability. ➢ Tracer study Within the component focused on developing a Skills Observatory, the first nationwide Tracer study of graduates of Technical Vocational Education and Training (TVET) and Higher Education (HE) was conducted. The study was implemented in partnership with European Training Foundation (ETF). The first study covered 2014-2015 cohorts of graduates of TVET and HE. The aim of the study was to increase existing knowledge about the skills that graduates of TVET schools and HE institutions have to offer, and to learn about the experiences of graduates in their transition from education to the world of work. It collected and analyzed information about whether the graduates managed to find jobs, including time taken to find first job after graduation, types of jobs, starting salaries, job satisfaction and career progression, as well as the relevance of formal education. The analyses have been used for a national monitoring system to assess the market orientation of the educational and vocational training system. Also, the results reported in the prepared evaluation reports are very useful for various stakeholders, including education and vocational training institutions that are interested in such feedback from their former students to improve their study programmes, and to show new applicants how their graduates have successfully managed the transition into employment. The main benefit for educational institutions is that the conducted study showed which economic sectors absorb the largest number of their students, information that was not previously available, and which allows them to developed new educational programs in relation to the demands of employers. Also, the evaluation report provides information for the potential students that are interested in which elements of the study programmes have positive effects on employment outcomes and earnings. Additional activities also involved training of local teams as an essential part of the conduction of the Tracer study. The training covered key phases and steps of the tracer study: questionnaire development; organization of the field work; methods to achieve a high response rate; data entry and data editing; coding; and data analysis. These activities contributed to building the national knowledge and capacity to organize and implement similar studies and analysis in a sustainable manner. Page 92 of 96 The World Bank Skills Development and Innovation Support Project (P128378) This entire process, therefore, resulted in the development of methodological standards, data collection instruments and expert guidance for future tracer studies. ➢ Citizens’ engagement in the preparation of the Grant Program for enhancing the collaboration between the schools and business community In June 2019, data collection for baseline survey on Citizens’ engagement in the preparation of the Grant Program for enhancing the collaboration between the schools and business community was conducted. Total numbers of 233 students were included in the survey. According to the survey, 83% of the students that participated in the Grant Program are satisfied with their engagement during grant application preparation. The follow up survey on Citizens’ engagement in the preparation of the Grant Program for enhancing the collaboration between the schools and business community was conducted in December 2019.Total number of 226 students were included in the survey. According to the follow up survey, 84.1% of the students that participate in the Grant Program are satisfied with their engagement during grant preparation. This indicator shows slight increase and the conclusion is that there is a progress in the student’s engagement in the grant preparation if we compare its value of 82.8% that was established as a baseline indicator during the baseline survey round in 2019. The study reveals very high satisfaction rates. Vast majority of the students reported that they are satisfied with their engagement in: preparation of idea for the practical training (94%) that is almost on the same level as established during the baseline survey (95%). There is a slight decrease of the satisfaction rates between the two rounds of survey for the satisfaction rates of the students with their engagement in the selection of the topics for practical training from 87% to 81% but this decrease can be explained by the limited possibility of the students to influence the selection of topics especially in the projects financed under the Grant measure 2 since those are vastly predetermined by the VET Center. There is an increase of the satisfaction rates among students for their engagement in the selection of methods for implementation of the practical training from 90% to 96% in two rounds of survey. Moreover, 84.1% of students that participated in the Grant Program are satisfied with their engagement during grant preparation – that represents the progress compared to the baseline survey. Assessment of the key factors and events pertaining to the Bank, borrower, co-financiers, other partners, and the external environment during preparation and implementation, that affected performance and outcomes The SDISP project became effective in early 2014.The project was being implemented by the Ministry of Education and Science, which is the main player in the education eco-system from primary to higher education and research and the main regulator of all processes from financing of primary and secondary education (although decentralized, the money is being transferred from central to local Government) to providing budget allocations for research. Close cooperation between the Project Implementing Unit and the structures of the Ministry and the daily coordination of activities in order to minimize duplication and to utilize resources to its fullest should be one of the main characteristics of every project that aims towards attaining the goals for which it was designed. Page 93 of 96 The World Bank Skills Development and Innovation Support Project (P128378) On the other hand, this might be a double-edged sward, since in the case of frequent change of Ministers and rotation in the management structure in the Ministry, valuable time is lost in acquainting the new management with the project goals, past activities and getting approval of the best ways to move forward in order to fulfill the plan. In the case of the SDISP the above happened on several occasions during the lifetime of the project, a total of six Ministers were replaced. The MoES received new Ministers in June 2014, April 2016, June 2017, June 2018 and August 2020. The change of the Ministers does not pain the full picture of the impediment it poses to project implementation since each change is a result of an election cycle characterize by political uncertainty and lack of incentive to make decisions that take political will and strength, as well as certain level of consensus in the society. Such was the case with the new HE financing model which had to be “re-sold” to several Ministers of Education as well as Finance and also several University rectors since during this time Rectors we elected and re-elected also. Besides frequent changes in the governance of MoES, the main obstacles which have affected the performance of two major activities, establishment of a sound quality assurance system and a transparent model for financing of the higher education (HE) sector were the long period of political transition, and the time needed to develop new Law on Higher Education, and implement provisions related to the quality assurance and HE finance. The Law on HE has been adopted in May 2018, and the establishment of an Agency for Quality Assurance and constitution of the two Boards have been finalized in March, 2020. The National Council responsible for HE, science and research with mandate to propose a Decree for HE financing was established at the end of April, 2021, by the Parliament of the RNM. Government and Parliament of the RNM have been challenged with additional risks due to the Covid 19 pandemic and their priorities have been changed since they have to solve serious health, economic and social issues as well as to adjust the education process and implement online and blended mode of education in the country for the first time. Due to the Covid 19 crisis, Government of the RNM has made few decisions related to the higher education, and to accreditation process of study programs. The deadlines for the validation of the accreditation, i.e., the re-accreditation of the existing study programs of all three cycles of higher education, including the decisions for commencement for these study programs, whose validity ceased in the academic year 2019/2020, continue to be valid until the end of academic year 2020/2021. The frequent election cycles interfere with the process of procurement also, since the Macedonian law to a large extent prohibits Government expenditures in the period preceding the elections. Despite the great support that the project had from the Government as well as form the WB staff, the above-mentioned fact managed to slow down many of the activities of the project and even bring some to a complete stall for prolonged period of time. Page 94 of 96 The World Bank Skills Development and Innovation Support Project (P128378) ANNEX 7. SUPPORTING DOCUMENTS Supporting Information for the Main Document Table 7.1 Grant Program Measures No. Measure Selected for Scaling Up 1 Setting up of a school company within the VET school through business guidance Yes and mentoring (business mentoring relationship) 2 Involvement of students in the in-house training courses at employers’ place No– 3 Placement of students in practical training in companies Yes 4 Establishment of problem-based learning courses of a small scale No 5 Establishment of the summer camps learning programs within the schools Yes Table 7.2. Estimate and Actual Spending of Major Contracts by Component Major Contracts Estimate (EUR) Actual (EUR) Note Component 1 External evaluation of 300,000 199,750 Contract negotiation with the firm to set the price universities lower. Development and 150,000 n.a. Initial estimate of the budget was too low to implement implementation of the activity. Thus, the project provided a design and HEMIS specification of the system instead. Assessment of funding 250,000 133,000 The contractor was switched from the firm to an model options and individual. preparation of funding model and formula for higher education Component 2 Development and 1,077,000 2,200,000 This activity was merged with the training for school design of the concept teachers and managers to support the reformed for secondary TVET curricula (initial budget for the training was €300,000). Procurement of 2,000,000 1,302,913 Original estimate was not accurate, thus, even though equipment for TVET the original budget was not available due to the overrun schools of the curriculum reform activity, this activity was successfully implemented. Equipment was supplied to 29 schools for 6 TVET sectors. Grant Program 649,000 442,102 As the grant was transferred to schools according to (509,000 after actual spending, there were some savings. In addition, restructuring of some contracts were terminated, and few final training project - May activities were not conducted due to COVID-19 27, 2020) restrictions. Thus, the program was successfully implemented within the budget. Component 3 FITD financial 9,159,750 9,118,570 The program was successfully implemented, and the instruments Government will continue to support the FITD and its investment. Resident adviser to the 100,000 213,560 Due to project extension, the cost overran. Also, the FITD activities of the FITD intensified toward the end of the project. Page 95 of 96 The World Bank Skills Development and Innovation Support Project (P128378) Bibliography on the Evidence Supporting Assessment of the Project Outcomes 1. Project-relevant websites/information (a) FITD: https://fitr.mk/en/ (b) ENQA affiliate status of the Agency for Quality Higher Education of the Republic of North Macedonia: https://www.enqa.eu/membership-database/heaeb-higher-education- accreditation-and-evaluation-board/ (c) TVET Reform: (d) Occupational standards: http://csoo.edu.mk/pocetna/standardi/standardi-na-zanimanja/ (e) Qualification standards: http://csoo.edu.mk/pocetna/standardi/standardi-na-kvalifikacii/ 2. Policies, strategies, laws, and studies (a) Carneiro, Pendro, and A. Toppeta. Endline Report: Fund for Innovation and Technology Development (FITD) grant (2015–2018 calls). (b) Country Partnership Framework for FY19–FY23. (c) Country Partnership Strategy (CPS) for FY11– FY14.https://documents.worldbank.org/en/publication/documents- reports/documentdetail/666721468270322446/macedonia-former-yugoslav-republic-of- country-partnership-strategy-for-the-period-fy11-fy14. (d) Demand for Skills in FYR Macedonia, June 2010. (e) European Commission. Innovation Union Scorecard 2014 https://op.europa.eu/en/publication-detail/-/publication/d1cb48d3-4861-41fe-a26d- 09850d32487b#. (f) European Innovation Scoreboard 2021. (g) Innovation Strategy of the Republic of North Macedonia for 2012–2020, October 2012.https://www.researchgate.net/publication/282734334_Innovation_Strategy_of_the_R epublic_of_Macedonia_for_2012-2020 (h) Law on Innovation Activity May 2013. (i) Ministry of Education and Science and European Training Foundation, Collaborative Evaluation of the Impact of the Reformed Four-Year Secondary Vocational Education, 2010. (j) Ministry of Education and Science and European Training Foundation, Strategy for Vocational Education and Training: Better Skills for a Better Tomorrow, 2013. (k) Measuring Economic Effects of State Aid Granted to Private Enterprises in North Macedonia: The Case of the Governmental Plan for Economic Growth, April 2021. (l) Program for Development of Science and Research for 2013–2017. (m) South Eastern Europe 2020 Strategy. (n) Sveikauskas, L. 2007. “R&D and Productivity Growth: A Review of the Literature.” Bureau of Labor Statistics Working Paper 408. (o) The Republic of Macedonia Education Strategy for 2018–2025 and Action Plan. (p) Vocational Education and Training Strategy 2013. (q) Work Program for 2011–2015. Page 96 of 96