Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) Report Number: ICRR0023004 1. Project Data Project ID Project Name P152104 Serbia Competitiveness and Jobs Country Practice Area(Lead) Serbia Finance, Competitiveness and Innovation L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-85280 30-Jun-2019 96,096,066.58 Bank Approval Date Closing Date (Actual) 16-Sep-2015 30-Jun-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 94,890,766.11 0.00 Actual 96,096,066.58 0.00 Prepared by Reviewed by ICR Review Coordinator Group Burcin Pamuksuz Ihsan Kaler Hurcan Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (p.6) and the Project Appraisal Document (p.4) the project development objective (PDO) was “to improve the effectiveness and coordination of selected public programs to alleviate constraints to competitiveness and job creation, including investment and export promotion, innovation, active labor market programs, labor intermediation, and activation of social assistance beneficiaries”. In August 2018, a restructuring changed the project objective as “to improve the effectiveness and coordination of selected public programs to alleviate constraints to competitiveness and job creation, including Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) investment and export promotion, innovation, active labor market programs and labor intermediation” (Restructuring Paper, p.8). The mid-term review revealed that measures related to the activation of social assistance beneficiaries under Component 4 were no longer seen as priority by the Government of Serbia. Hence, the PDO was revised by deleting “activation of social assistance beneficiaries” from its formulation, and the relevant Disbursement Linked Indicator (DLI) was dropped. Although the scope of the project was narrowed and project commitment remained unchanged, a split rating will not be necessary because the revision did not have a material impact on the project objective, which was still achievable through the implementation of other activities. Note: The achievement of this project's development objective in Afghanistan is assessed following the agreed OPCS methodology that recognizes all outcomes as they were prior to August 15, 2021. This approach applies to all projects closed prior to that date. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 10-May-2018 c. Will a split evaluation be undertaken? No d. Components The project had two parts with multiple components. A: Implementation of Eligible Expenditures Programs Subject to Results-Based Financing: This part of the project was to reimburse “Eligible Expenditure Programs” (EEPs) which was composed of salaries of Ministry of Economy (MoE), Research & Development Institutes and National Employment Service (NES), subject to the achievement of “Disbursement-Linked Indicators” (DLIs). With this lending instrument, project was to alleviate constraints to competitiveness and job creation through increasing effectiveness and coordination of selected government programs grouped in four components: Component 1: Program for Policy Planning, Monitoring and Coordination: (EUR 3.58 million estimate at appraisal, EUR 2.68 million at restructuring, EUR 2.68 million actual) The entity covered under this component was the Public Policy Secretariat (PPS). This component was to support the implementation of the Borrower's program for policy planning, monitoring and coordination (PPMC) through developing and piloting an inter-ministerial PPMC System including semi-annual progress reports and annual performance review reports with inputs from each participating ministry in order to support competitiveness and jobs reforms. Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) Component 2: Program for Investment and Export Promotion: (EUR 19.69 million estimate at appraisal, and restructuring, EUR 19.69 million actual) The entities covered under this component were the Ministry of Economy (MoE) and the Development Agency of Serbia (RAS). This component was to support the implementation of the Borrower's programs for investment and export promotion through:  developing and adopting a strategic framework and overall action plan for investment and export promotion, including restructuring the MoE investment and export promotion agencies; and  improving the Borrower's investment and export promotion programs and services. Component 3: Program for Innovation: (EUR 31.32 million estimate at appraisal, EUR 35.40 at restructuring, EUR 34.76 million actual) The entities covered under this component were the Ministry of Education, Science and Technology Development (MoESTD), and the Innovation Fund (IF). This component was to support the implementation of the Borrower's program for innovation through:  supporting the operations of the Innovation Fund and its Matching Grants Program;  supporting the Technology Transfer Facility ("TTF") service lines at the Innovation Fund; and  undertaking strategic planning for institutional reform of the public research and development sector. Component 4: Program for Labor Reform: (EUR 30.43 million estimate at appraisal, EUR 26.85 at restructuring, EUR 23.07 million actual) The entities covered under this component were Ministry of Labor (MoLEVSA) and National Employment Service (NES). This component was to support implementation of the Borrower's programs for labor through:  enhancing the effectiveness of the NES labor intermediation services for employers and the unemployed;  improving the effectiveness of the Borrower's active labor market programs; and  facilitating the transition of social assistance beneficiaries into formal jobs (this item was cancelled at the first restructuring). B. Project Management and Technical Assistance (EUR 4.25 million estimate at appraisal, EUR 4.54 million actual incl. project management costs). Management and implementation of Components 1, 2, and 3 were to be supported through technical assistance activities, the provision of goods, consultants' services, non-consulting services, training and covering operating costs. Project Implementation Unit (PIU) in PPS was to provide support for the PPS, MoE, and MoESTD for the realization of the activities. Technical assistance activities, through the provision of consultants’ services under Component 4, were to be carried out by a separate “mini-PIU established within MoLEVSA (this unit was dissolved later during the implementation as it was considered ineffective). TA activities foreseen for each component was to complement and/or support the achievement of the DLIs defined for each such component. Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) Revised Parts/Components: In line with the revision of the PDO in the first restructuring, activity of facilitating the transition of social assistance beneficiaries into formal jobs under Component 4 was cancelled. The activation of social assistance beneficiaries under Component 4 was no longer seen as priority by the Government of Serbia. Hence, the PDO was revised by deleting “activation of social assistance beneficiaries” from its formulation, and the relevant Disbursement Linked Indicator (DLI) was dropped. In addition, Component 3 was revised which enabled a feasibility study for and set of the Science Fund (SF). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The type of financing instrument agreed upon by the Serbian government and the World Bank was an Investment Project Financing (IPF) operation with a number of DLIs. Under agreed structure, the WB reimbursed the Serbian government with “eligible expenses” rather than funding and procuring for new goods or services, which were negotiated ex ante as government employees’ salaries. Project Cost: The project cost estimated at appraisal was US$393.00 million (total of IBRD loan and Borrower’s contribution). The actual cost at closing was US$851.00 million. According to information provided by the project team the difference occurred mainly because the project was extended for two years, and since the eligible expenditures were mostly regular salaries of government employees, this was increased “mechanically”. Also, original project cost was an estimate (again mostly of salaries) based on a single year – actual expenditures varied / were higher. Finally, change in the EUR/USD exchange rate, also had an impact on the increase. Financing: At appraisal, the IBRD Loan was estimated at US$100.00 million. The project disbursed US$96.1million at closing. For the financing of TA activities in addition to the IBRD loan (EUR 4.55 million) funding from two trust funds (Competitive Industries and Innovation Program -CIIP Trust Fund and Good Governance Fund -GGF) and cross-support from two other projects (Serbia Research, Innovation, and Technology Transfer Project- SRITTP and the Regional Jobs ASA Project) were used; total funds disbursed for TA reaching EUR 6.88 İncluding project management costs. Borrower’s Contribution: At appraisal, borrower’s contribution was estimated to be USD 293.00, and actual contribution was USD 755.33 million at project closing. Restructurings: There were two level 2 restructurings.  First Restructuring (May 10, 2018): The PDO was revised (see section 2.a Objectives above) and targets of some indicators were revised up. The project had a slow implementation progress mainly because of the accelerated election cycles (parliamentary and presidential) which had a negative impact on project decision-making. Implementation slowed down during election campaigns and post-election government negotiations. In order to compensate for the missed implementation time, the project closing date was extended by two years from June 30, 2019 to June 30, 2021.DLI #1 was dropped as it was already completed at the time of restructuring. In line with the revision of the PDO, DLI #9 (for 2017 and 2018) was also dropped as a result of changes in government priorities and institutional structures. Loan proceeds were reallocated to Component 3. DLI# 2, #3 and #5 were revised and timelines were shifted in line with changing priorities of the government. TA to be Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) provided under Component 3 was adjusted to meet MoESTD’s need and revised in a way to cover more systematic and complementary reforms of research and innovation (i.e., development of Strategy of Smart Specialization and New Science Fund). DLI# 6 was revised and timeline for DLI# 8 was shifted. Accordingly, some of the indicators were also revised.  Second Restructuring (April 14, 2020): The restructuring introduced a contractual waiver to the Loan Agreement enabling the disbursement of the part of the funds before the DLIs were completed. Advance disbursement was necessitated to finance activities to achieve DLIs, which was in line with the Guidance on IPF with Performance Based Conditions. The government could not allocate funds already appropriated in the budget for the activities to be implemented under the project because of the onset of COVID-19 for which the funds were allocated. Dates: The project was approved on September 16, 2015 and became effective on March 31, 2016. The closing date was extended by two years from June 30, 2019 to June 30, 2021 at the first restructuring. 3. Relevance of Objectives Rationale Bank Strategy: At appraisal, the project objective was relevant with the Bank’s Country Partnership Framework (CPF) for Serbia FY16-FY20. While the CPF mentions the private sector growth and economic inclusion as the main drivers for the country’s integration into the EU, it specifies financing, innovation, connectivity and labor constrains as the key issues to be addressed on the way to integration. The project remains relevant at closing. The recent CPF for Serbia for FY22-FY26 emphasizes the need to improve productivity, boost competitiveness of the economy and create jobs. In addition, the 2020 Serbia Country Economic Memorandum (CEM) 2.0 “Serbia’s New Growth Agenda” was informed by the project. The CEM highlighted private sector development, competitiveness and innovation as the key factors. Country Context: At appraisal, the objective was well aligned with the Government of Serbia’s priorities. Country’s Public Administration Reform (PAR) strategy which was being implemented at inception, provided the background and methodologies for modern public policy management which is aligned with the project’s objective. Activities defined under Component 2, matches with Serbia’s Industrial Strategy for 2011-2020 which targeted development of a framework for investment and export promotion through identifying target sectors and developing sectoral policies. The project objective is well aligned with Serbia’s Research for Innovation Strategy (2016-2020) and 2018 Research Infrastructure Road Map. The strategy and the road map targeted increasing research and its relevance and its economic impact. The Competitiveness and Jobs (C&J) project component #3 called for stable financing of the Innovation Fund and contributed to its operational functioning through the assistance provided to the grants programs that supported innovation. Previous sector experience: The project benefitted from the lessons learned in a series of DPL operations previously implemented in Serbia on reforming SOEs. The policy dialogue between the World Bank and the Government on innovation policy which had developed two other WB projects – the Serbia Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) Innovation Project (or SIP), and the Serbia Research, SRITTP also contributed to the design of the project. Overall, the relevance of project objective is rated High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Objective 1: To improve the effectiveness and coordination of selected public programs to alleviate constraints to competitiveness and job creation, including investment and export promotion, innovation, active labor market programs and labor intermediation. Rationale Rationale: Theory of Change According to the World Bank’s 2015 Systematic Country Diagnostic for Serbia, main policy interventions were needed to boost private sector investment, increase exports and productivity, and create jobs that were seen as the major barriers for economic growth and reduction of poverty. To achieve this objective, the project was designed to address labor supply, labor demand, and governance constraints (PAD, p.3) to job creation and competition through improved effectiveness and coordination of selected public programs. This was to be achieved through a results-based financing design with nine disbursement-link indicators (DLIs) and with a technical assistance component to support the achievement of DLIs and project implementation. The project’s theory of change suggests that the improvement of the effectiveness and coordination of selected public programs would be achieved through the following 29 activities linked to nine DLIs: DLI 1: Policy planning, monitoring, and coordination system piloted (in support of competitiveness and jobs reforms) Development and piloting of an inter-ministerial policy planning, monitoring, and coordination system. DLI 2: MoE investment and export promotion programs restructured Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) The development and adoption of a strategic framework and action plan for investment and export promotion; the action plan to be implemented, monitored, and updated; and increase in the number of investment leads generated. DLI 3: Enterprise innovation supported via the Innovation Fund Matching Grants Program Allocation of at least EUR 0.7 million annually in the Republican Budget for the Innovation Fund’s operating expenses and EUR 2.7 million annually for enterprise innovation matching grants; commitment of at least 80 percent of the matching grant annual budget allocation using the Innovation Fund’s international peer review and investment committee mechanisms; evaluation of the Innovation Fund’s matching grants program; and development of a proposal for scaling up the matching grants program using government or donor funds. DLI 4: Technology transfer and commercialization facilitated via the Innovation Fund Technology Transfer Facility Design and deployment of one or more Technology Transfer Facility service lines; allocation of at least EUR 0.9 million annually in the Government’s 2017 and 2018 budgets to the Technology Transfer Facility; initiation of a minimum number of technology transfer transactions; evaluation of the Technology Transfer Facility; and preparation of a government or donor proposal for scaling-up of technology transfer activities. DLI 5: Public research sector reforms designed and adopted Adoption of a 2015-2020 R&I Strategy and its Action Plan that commits to reforming the public RDI sector; adoption of a time-bound R&I Infrastructure Roadmap and its Action Plan; RDI self-assessments and RDI performance audits by international experts; definition and adoption of RDI sector reform program, including performance- based financing schemes; and preparation and submission of a donor proposal for technical assistance to conduct sector and RDI reform implementation. DLI 6: Effectiveness of NES labor intermediation services delivered to clients (employers and unemployed) improved Adoption of an action plan by NES to enhance the quality of employer services and case management; certification of NES case workers; increasing collection of vacancies among employers; increasing the number of NES-registered unemployment cases transitioning into formal jobs; and introduction of a special registry of unemployed persons who are temporarily prevented from working. DLI 7: Case load management in NES branch offices improved Reduction of the dispersion in case load rates across NES branch offices. DLI 8: Effectiveness of ALMPs improved through statistical evidence Development and application of a new methodology to profile the unemployed; development and implementation of an action plan on monitoring, evaluation, and re-design of ALMPs; piloting of a newly designed ALMP; and engaging a training provider (service provider for labor market trainings) based on a performance-based contract. Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) DLI 9: Transition of social assistance beneficiaries into formal jobs facilitated (Dropped at the first restructuring) Enhancement of the activation of social assistance beneficiaries by designing and piloting a program to decrease work disincentives for a select group of social assistance beneficiaries. The technical assistance component of the project was to support the design and implementation of the reforms listed above through the procurement of consulting services and delivery of trainings (PAD, pp.6-13). Some of these reforms were also to be supported by a Bank-executed Competitive Industries and Innovation Program (CIIP) trust fund-financed technical assistance (PAD, pp.7-8). The reforms that were prioritized by and had strong ownership of the government were included in the project’s scope (PAD, p. 6). The analytical work of the World Bank identified these reforms as key drivers of competitiveness and jobs creation in the country. The reforms were expected to improve the efficiency of the expenditures by not requiring any additional funding. The implementation of these reforms was expected to make existing government spending more effective and improve coordination in public programs in investment and export promotion, innovation, labor market, and labor intermediation. The improvement in the effectiveness and coordination of the above-mentioned public programs was expected to fully or partially alleviate the constraints in labor supply (skills constraints through NES training programs; job search constraints through improved NES services; and labor market participation constraints through in-work benefit for social assistance recipient), labor demand (uncertainty and appropriability constraints through innovation matching grants and export promotion; where to invest constraint through investment and export promotion and R&D and technology transfer; and access to finance constraint through restructuring of MoE’s Development Fund), and governance (strategic planning and policy coordination constraint through the implementation of inter-ministerial policy planning, monitoring, and coordination system) (PAD, Table 1, p.3). The full or partial alleviation of these constraints would be expected to contribute to increased competitiveness and job creation. The critical assumptions underlying the ToC were the alignment of incentives to reform at ministerial and agency levels, and continuous provision of funding to implementing agencies (ICR, p.7). Overall, the causal pathways from inputs to outcomes were valid and direct, and the achievement of the intermediate outcomes were to be expected to contribute to the alleviation of the constraints to competitiveness and job creation. However, since the project focused on a targeted set of achievable reforms rather than attempting to address all competitiveness and job creation constraints, achievement of the project outcomes could not be fully attributed to the project’s intervention. Constraints related to competition, investment climate, and infrastructure and trade logistics were covered under other World Bank and IFC- financed projects or European Union accession reforms (PAD, p.3). Outputs and Outcomes The project supported the achievement of the DLIs through technical assistance and policy dialogue. The project delivered following outputs and outcomes: DLI 1: Policy planning, monitoring, and coordination system piloted (in support of competitiveness and jobs reforms) Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104)  Annual performance reviews of pilot policy planning, monitoring, and coordination system (to support competitiveness and jobs reforms) conducted: At project design three PPMC frameworks’ annual performance reviews were targeted. Three reviews were conducted, and the target was archived in 2018. However, the target refers only to the number of reviews but not to the impact of the frameworks on better policy making or effectiveness of government spending on competitiveness and jobs. The ICR refers to positive remarks of the former head of PPS on the impact of project on policy making and evidence-based policy making. Also, it is a positive indication about the impact of the project that the government of Serbia wanted to develop a parallel project (the “Results Based Management Reimbursable Advisory Service project) with the World Bank that continued to create a broader results framework for the entire Government. However, these are not measurable indicators. This issue is subject of weak M&E design and will be elaborated in the relevant section. DLI 2: MoE investment and export promotion programs restructured  New investor leads generated by the reformed investment promotion agency (annual): Investor leads were 20 in 2014 and the revised target for 2020 was 33. The project achieved 90 leads at closing.  Participants in export promotion programs that engage in new export activities (annual): Eighty-five firms participated at RAS’s export promotion programs at project start. The figure reached at 71 in 2020. The revised target was 200 and could not be achieved. It is mainly because of Covid-19 pandemic as pre-pandemic figures were high (188 in 2016, 350 in 2017) (ICR, p.17).  Number of documented interests by investors (annual): The number of documented interests by investors were 61 in 2014 at project start, and in mid-2020 it was down to 31 after a steady increase in between 2014 and 2020. The fall is most likely because of the Covid-19 pandemic. They peaked again to 95 at the end 2020, above the revised target (90) for the same year. The project targeted an increase by at least 10% annually, which was met (ICR, p.16).  Number of firms participating in the new Supplier Development Program: Participation exceeded expectations. Number of participants increased from 10 in 2019 to 32 by the end of 2020. The revised target was 20 firms.  MoE’s investment and export promotion programs were restructured: ICR reports adoption of sector- specific strategies and action plans for four priority sectors (wood and furniture, plastics and rubber, food processing, and machinery and equipment) formally in March 2018. RAS implemented MoE’s sector specific recommendations in multiyear plans starting with the adoption of action plan for 2017- 2019 and then action plans for 2020 ad 2021. DLI 3: Enterprise innovation supported via the Innovation Fund Matching Grants Program  New collaborations facilitated between research organizations and private sector: In 2014, IF was implementing three types of pilot marching programs. At closing 28 new collaborations or consortiums (Revised Target 25) facilitated by IF were reached.  Private financing catalyzed through IF support programs (Amount USD) IF disbursed funding for: o Mini Grants program: EUR 5.5 million funded 74 companies over 4 calls for grants in 2017, 2018, 2019, and 2020 (up to EUR 80,000 per company); o Matching Grants program: EUR 6.5 million funded 27 companies over 3 calls for grants in 2017, 2018 and 2019(up to EUR 300,000 per company); o Collaborative Grant Scheme (CGS)s program: EUR 5.1 million funded 20 companies over 2 calls for grants in 2018 and 2019; Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) o Proof-of-Concept program: EUR 1.8 million funded 93 applications over 2 cycles of funding in 2019 and 2020 (up to EUR 50,000 per application); o Innovation Voucher program: EUR 2.9 million funded 578 vouchers over 5 calls for application in 2017-2020 (up to approximately EUR 6,800 per voucher). The revised target was US$2.1 million, the project achieved a total of US$11.57 million.  Enterprises and startups financed with female owner: In 2014 only one enterprise/startup with female owner was financed. The enterprises financed with female owners were 26 by 2020 exceeding the target (Target was 10). Innovation fund activities for promotion of grants and programs did not create significant impact. Share of female ownership in calls for grants remained limited (between 13 and 21%). The share of women participating in the IF events turned were very low – on average, attendance by women was around 10%. DLI 4: Technology transfer and commercialization facilitated via the Innovation Fund Technology Transfer Facility  Technology transfer: IF TTF service lines deployed  Technology transfer: transactions initiated The TTF was established as a unit within the IF to work with universities in allowing for transfer technology and innovation to get to the market. The work of the TTF was organized across service lines. During the duration of the project, the IF established 11 service lines (baseline:0, revised target: 8 lines). The 43 transactions (a set of activities that led to a signed commercialization deal with a company) were initiated (revised target: 15) eventually led to 3 deals with private companies to commercialize the technologies prototyped.  Enterprises and startups financed through IF programs (including MMG):  Innovative products or services developed by MG beneficiary firms  Innovative products or services launched by MG beneficiary firms From 2017 to 2021, the matching grant programs financed total of 126 beneficiary enterprises and startups. The revised target was 75 (by end 2020). In the same period, 106 companies developed 233 products and services (target: 25, achieved 71, by end 2020) out of which 122 (52%) were commercialized.  Citizen Engagement: Innovation matching grant beneficiary annual survey (number of surveys): The IF conducted 3 beneficiary annual surveys (target: 3) Target was achieved. DLI 5: Public research sector reforms designed and adopted  R&I strategy and action plan and public research sector reforms designed and adopted, Science fund designed and adopted: There is no indicator for this particular action in the results framework. This action refers to DLI Through the technical assistance provided to MoESTD, R&I strategy was designed and was adopted in July 2018 by the government. In December 2018 Infrastructure Roadmap and its relevant action plans were adopted.  Science Fund programs initiated Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104)  Science Fund programs competitive grants awarded With the restructuring of the project, among other components, Component 3 was revised which enabled a feasibility study for and set of the Science Fund (SF). The fund started operations in March 2019 to provide the research community with competitive research grants and launched 5 new programs (Target:1 program). The program had awarded 85 grants (target: 20) as of December 2021. DLI 6: Effectiveness of NES labor intermediation services delivered to clients (employers and unemployed) improved  Number of employees contacted per year by the NES (field visits): As the project progressed, NES employees conducted an increasing number of field visits to registered jobseekers: almost 19,000 were reached by the end of 2020, while in 2014 the field visits were about 16,000. Target was 18,000 employees.  % Of total NES staff that is operating as certified case worker: The certification of case workers started in 2017. By project end in June 2021, almost 90% (89.33%) of the NES staff had been certified (targe was: 85%).  NES registered unemployment cases transitioning into formal job (annual): Target was 280,000 cases. Project could only achieve 218,854 cases.  Citizen engagement: Number of employers reached by client satisfaction and needs survey. Target for this indicator was 3,000 whereas the project reached only 500 employees. DLI 7: Case load management in NES branch offices improved  Standard Deviation of mean case load per branch office over all branch offices: NES relocated some of the personnel across different branches to better distribute the caseloads per employee. (ICR, p.21) and as a result their average diminished to 252 in 2020 from 381 in 2014. The revised target was 260. DLI 8: Effectiveness of ALMPs improved through statistical evidence  Gross placement rate of NES start-up support program for self-employment 6 months from completion. Gross Placement rate of NES increased from 95% in 2014 to 97.77% 2020. The revised target was 98%.  Gross placement rate of NES hiring subsidy program for new employment 6 months from completion. The rate increased from 80% in 2014 to 93.71% in 2020.The revised target was 85%.  Number of new or re-designed ALMPs in the NES: One new ALMP was introduced in NES. However, it is difficult to measure its impact due to limited implementation time as it was introduced in 2021.  NES registered female unemployment cases transitioning into formal job: End-target for this objective was 145,000 female cases. The project achieved 118,226 female cases (below base-line). The program’s impact on female was limited. Participation from female citizens to awareness programs or trainings were very low. Impact on Governance Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) The impact of the PPMC frameworks on the short-term outcomes of the project is not easily quantifiable, but anecdotal evidence supports the connection between the frameworks put in place and better policy making at the core of the Serbian government, where policies and programs became better targeted to strategic priorities, better coordinated, and performance issues were addressed (ICR, p.15).  The project increased capacity of PPS in developing Regulatory Impact Assessments (RIAs) and Policy Impact Assessments (PIAs) as well as supported PPS in of drafting the Law on the Planning System (adopted in 2018) and the Regulation on the methodology of public policy management (adopted in November 2019). Both acts regulate the process of conducting RIAs and PIAs.  In 2015, only 16 (7.92%) of the draft laws submitted for an opinion contained a complete RIA, 73 (36%) contained a partial RIA, and 44 (21%) had an insufficient RIA. In 2019, the PPS issued 198 opinions on draft laws – 22 draft laws contained a complete RIA, 52 partial, 21 insufficient, and 103 laws (66 international agreements) did not require RIA. In 2019, the percentage of compliance of the proposed regulations with the provisions of the acts was 84%. Prior to the entry into force of the acts, the PPS did not evaluate PIAs. In 2019, out of 23 PPDs submitted for opinion, the Government adopted 14 PPDs, 12 of which were in accordance with the provisions of the above-mentioned act.  Since the beginning of the project in 2015 government priorities have been streamlined from 98 in 2015 to 4 in 2021, as reflected in the respective Action Plans for the Implementation of the Government’s Program and creation of the results framework.  Recognizing the importance of developing a solid result planning and management framework introduced by the C&J project, at the request of the government, the WB later delivered a parallel project (the “Results Based Management”. Impact on Labor Demand and Supply and Job Creation  Between 2017-2021 total number of beneficiary enterprises of Mini Grants, Matching Grants and CGSs was 121 and these created a total of 844 temporary and permanent jobs, an average of 7 jobs per firm. Mini Grants created 154 jobs, Matching Grant created 317 jobs and CGS created 373 jobs. 55% of the companies that received a grant were in the Information and Communications Technology (ICT) sector (ICR, p.19).  Several reports show increased investment levels and jobs attracted in Serbia during project implementation (ICR, Annex 4, Table9). However, the ICR argues that since the available data come from different sources, it was not possible to precisely estimate the impact of RAS’ activities on investments and jobs. Despite this difficulty, in 2018, there were 48 incoming FDIs directly serviced by RAS through attraction, facilitation or both phases of the investment project, which attracted an estimated USD 3.8 billion worth of investments (85% of FDI in 2018) and created about 22,000 jobs corresponding to 83% of the new jobs in Serbia in 2018.  Design and implementation of sector-specific strategies and action plans for four priority sectors (wood and furniture, plastics and rubber, food processing, and machinery and equipment), which were formally adopted in March 2018. These plans, as well as RAS’ strategic framework and action plans, offered guidance on the overall MOE’s strategic framework and action plan for investment and export promotion. These strategic frameworks addressed one of the constraints for investors in Serbia: “Where to invest”. Sector specific data is not available from ICR which could demonstrate impact of the intervention.  Participants in export promotion programs that engage in new export activities (annual): Eighty-five firms participated at RAS’s export promotion programs at project start. The figure reached at 71 in Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) 2020. The revised target was 200 but could not be achieved. It is mainly because of Covid-19 pandemic as pre-pandemic figures were high (188 in 2016, 350 in 2017). Impact on Competitiveness: While the project activities and outputs would be expected to contribute to the improvement of competitiveness in the economy, evidence is insufficient to assess the impact of the project. Competitiveness was not adequately defined at appraisal; therefore, it was not easily measurable. However following achievement would be expected to have a positive impact on competitiveness: The project supported the Smart specialization strategy — business, innovation, and research, which was approved in early 2020. The process of transformation of the research institutes started, which consisted in the self-evaluation of 61 out of 69 institutes.  Results of a 2020 survey showed that the support through innovation vouchers appeared to have had positive impact as awardees reported positive incentives to innovation activities, stronger RDI culture in the industry, improved knowledge-based economy, and strengthening of the innovation ecosystem and environment for RDI. As the survey reported on perceived improvements, the results should be interpreted with caution (ICR, p.19).  An online survey conducted in 2020 confirmed that companies benefitted from collaborations facilitated by IF – when asked whether the funding provided by IF further strengthened or improved the quality of the cooperation with researchers, almost 60% of the respondents agreed with the statement. Moreover, 85% of awardees indicated that they continued to cooperate with the research community after the project ended (ICR, p.18).  Through the TTF, 43 transactions (a set of activities that led to a signed commercialization deal with a company) were initiated (revised target: 15) eventually led to 3 deals with private companies to commercialize the technologies prototyped. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project achieved all the DLIs except the dropped DLI9 and had a positive impact on job creation through addressing the labor supply, demand and governance constraints. This impact can be partially attributable to project’s intervention since some constraints were addressed by other World Bank and IFC-financed projects and the European Union accession reforms. However, while the project activities and outputs should be expected to improve competitiveness in the economy, evidence is insufficient to fully establish an attributable link between project’s intervention and competitiveness. Overall, the project’s efficacy in achieving the project objective is rated Substantial. Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency: In the case of Serbia competitiveness and jobs project, at appraisal neither a cost benefit analysis nor a cost- effectiveness analysis was conducted. The project’s economic analysis mainly relied on literature reviews which made above mentioned comparison impossible. The PAD argues that since the project is not supporting new expenditures and investments, with the exception of some innovation-related activities, the economic impact of the supported activities will be indirect, through improved efficiency of key public programs. It is true that the project may have generated some benefits which could not be easily measured, still an effort should have been made at appraisal to conduct an economic analysis which would cover calculation of the financial rate of return and the economic rate of return. These would layout at minimum the profitability of the project and the benefit created by the implementation of the project. At closing the project team conducted economic analysis of each component with the exception of Component 1. Within the scope of this components, although the developed frameworks and action plans increased efficiency for government it is not possible to quantify these benefits. Although the ICR reports limitations in availability and reliability of data, cost-benefit and cost-effectiveness analysis for components 2, 3, and 4 were conducted. According to ICR, several challenges exist for calculating both cost-benefit analysis and cost- effectiveness analysis. Despite these challenges and making several assumptions Component 2’s ex-post NPV is about EUR 20 million with a discount rate of 12%. EIRR is 5.8%. Cost-effectiveness analysis was made by relating RAS operating budget and TA expenses with FDI inflows. FDI inflows increased during project implementation. However, as available data on FDI inflows is not specific to project intervention it is not possible to attribute all FDI changes to the project. For Component 3, ICR partially estimates the effects of the intervention, using job creation as a proxy. The 121 firms that received funding through the 9 calls for grants created 844 temporary and permanent jobs, saw a EUR 5.4 million rise in revenues from the commercialization of new services and products, and leveraged EUR 8.7 million of private funds. A partial NPV was calculated based on job creation and the impact on GDP, as a proxy for the entire program, as the other results cannot be fully isolated or are not easily quantifiable. Assuming a discount rate of 12% (4%), the ex-post NPV of these jobs is EUR 10.3 million (EUR 15 million) and the EIRR is 1.6%. However, as the calculations only include jobs created and not all the other benefits to firms that the program likely generated, it is not possible to calculate the full net present value (ICR, p. 82). For component 4, according to ICR relevant information regarding output delivered by the project which could be attributable to costs and benefits was not available. Some limited claims had to be made in this regard: The strengthened capacity of NES to deliver services to its registered jobseekers – notwithstanding the yearly reduction of NES budget (see Table 27 below) – should then reduce the cost of service provided, all other factors being equal. In the absence of disaggregated internal budget data from NES, and statistics about the reduced unemployment time of NES registered workers, the costs savings are difficult to estimate. However, it is reasonable to expect that as the mean case load per NES branch office decreased significantly, and the number Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) of field visits conducted increased from 16,000 to almost 19,000, the cost per visit should have decreased (ICR, P.85). Administrative and Operational Efficiency: The major shortcoming related to efficiency resulted from complexity of the project design. The project design foresaw three implementing institutions (MoE, MoESTD and MoLEVSA) and one coordinating agency (PPS). This slowed down the project implementation. The complex project design also put heavy administrative burden on the World Bank project team and the PIU. The PIU had to process 82 procurement contracts, and monitor progress in DLIs against verification protocols. The creation of an additional “mini-PIU” within MoLEVSA, which was later dissolved, complicated project implementation during the early years of the project. The project identified DLIs against EEPs to create necessary incentives to the Government for proper project implementation. However, their impact remained limited as some of the DLIs were retroactive (for the first year, DLIs Nos.1,5,7,8), some were dropped (DLIs Nos. 9, and 1) and others did not have the expected impact. On top of monitoring the DLIs the teams were in charge of monitoring and reporting developments in 27 indicators defined in the project’s results framework also. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project objective is substantially relevant to the Bank strategy and the country context. The project’s efficacy in achieving the objective to alleviate the constraints to job creation and competitiveness is rated Substantial. However, the project’s efficiency in achieving the project objective is rated Modest because of significant shortcomings in administrative and operational efficiency of the project. Overall, the project outcome is rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) 7. Risk to Development Outcome At project closing, risk to development outcome of the project was moderate. Serbia with the support of previous WB projects, EU supported projects as well as the current competitiveness and jobs project introduced reforms to build capacities in agencies in charge of investment, export promotion, innovation and labor market intermediation. Special emphasize was given to increasing policy making, better coordination, and efficient use of government funds. However, continuation of the funding is critical for reforms to reach their targets and be sustainable in the above said sectors. Continued political support and consensus of public opinion will be crucial. On the other if Serbia continues its EU membership agenda and takes EU laws and regulation as well as market practices as anchors, it is more likely that the reform processes pursued, and the outcomes achieved through the project are sustained. 8. Assessment of Bank Performance a. Quality-at-Entry The project was consistent with both the Government’s development priorities and the Bank’s CPS. The type of financing agreed between the World Bank and the Government (IPF operation with a set number of DLIs) was designed to actual needs and national priorities by making spending on existing programs more effective rather than increasing Government spending beyond current levels. The project was one of the first of such operations, Nevertheless, the project benefitted from the lessons learned in a series of DPL operations previously implemented in Serbia on reforming SOEs. The project design also benefitted from ongoing dialogue on innovation policy which had developed two other WB projects – the Serbia Innovation Project (or SIP), and the Serbia Research, Innovation, and Technology Transfer Project (SRITTP). The implementation arrangements were in place but complex involving three line-ministries, four government entities, and two PIUs (until project restructuring) and complexity of working with these numerous entities with varying capacities was not sufficiently taken into consideration. Both the world bank project team and its counterparts lacked the previous experience of implementing a result-based project. M&E arrangements were complex and not adequate. The results framework included 27 indicators of achievement and 9 DLIs some of them composite and with comprehensive approval protocols, putting a big burden on implementing and monitoring parties as well as delaying realization of the project activities. The indicators, majority of the which are at activity or output level, and did not provide enough evidence whether the intervention alleviated constraints to job creation and competitiveness. Safeguards and fiduciary mitigation measures were appropriately designed and the overall quality of project preparation was satisfactory in that respect. Risks were adequately identified but mitigation measures were not sufficient, resulting in implementation delays when risks materialized. At appraisal, accost-benefit and a cost effectiveness analysis were not conducted (ICR, p.22). To assess possible impact of the project, policy dialogue and literature reviews were used. This approach made Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) comparison of with and without project cases impossible. Some impacts foreseen by the project intervention cannot be easily quantifiable. For other interventions project team could have made an attempt to conduct an ex-ante analysis. Overall, the quality at entry is rated Moderately Satisfactory because of moderate shortcomings in implementation arrangements and M&E design. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision According to the ICR, supervision inputs and processes were in place. Qualified and committed staff both on the WB, PIU and the implementing agency sides took part in supervision of the project. The project team included senior FM, procurement, and environmental specialists –experienced in supporting private sector development (PSD) reform projects. Project team participated at supervision missions in the form of site visits together with the PIU. Regular supervisions resulted in decision to restructure the project in 2018, making necessary reallocations, adjusting the timeframe and cancelling funds. However, the restructuring did not attempt to change project’s indicators which would allow them to better capture the project’s outcomes. Instead, the existing targets were adjusted and some were revised up without changing their quality in capturing results. The second restructuring in April 2020 enabled the project to respond to Covid- 19 and achieve its objectives. Despite the lack of experience on management of result- based project as well complex project design, project managed to fulfil DLIs and reached 95% disbursements. During the implementation of the project, 12 Implementation Status Reports (ISRs) and 11 Aide Memoires were prepared. The PIU prepared regular progress reports which provided input for supervision missions and to counterparts and made it possible for them to assess progress in implementation of the project. A technical expert within the PIU performed regular M&E of project results. Overall, the quality of supervision is rated Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) a. M&E Design Theory of change is sound and casual pathways from inputs to outcomes were valid and direct. However, there are some shortcomings related to indicators. Although, the M&E framework intended to capture the outputs, outcomes, and long-term impact of the project activities, the actual choice of indicators reflected a focus on outputs rather than outcomes (ICR, p.28). For example, for the first component there is no PDO indicator and only one intermediate indicator which is development of three annual reviews which were in fact conducted. The component in essence aims to improve governance in policy making and coordination. However, its indicator does not provide meaningful evidence. To put this into perspective ICR puts together some relevant figures regarding number of RIAs and PIAs conducted by PPS to reflect on improved policy making. In addition, since the project focused on a targeted set of achievable reforms rather than attempting to address all competitiveness and job creation constraints, in parallel project’s indicators remain limited in reflecting full attribution of outcomes to the project’s intervention. The objective was to alleviate constraints to job creation and competitiveness. The term “competitiveness “is rather generic and the M&E design does not specify how to assess improvements achieved in competitiveness. Indicators in the results framework provide some evidence regarding output level indicators related to innovation, research and increased funding in these areas. However, there is no M&E design element or a link which could confirm adequacy of these output indicators in capturing achievement in PDO level outcome (improvement in competitiveness). M&E design and arrangements and relevant institutions were identified clearly. According to PAD (PAD, p. 16) results monitoring is expected to take place on a semi-annual basis and be managed by an inter- ministerial working group with official representatives of the PPS and each of the line ministries. In addition, reports of Innovation Fund and NES provided input to M&E framework. M&E did not intend to make use of other M&E mechanisms located outside the project to capture impact of the project activities. b. M&E Implementation Data on results indicators were collected by line ministries and agencies, delivered to the PIU, which were then delivered to the project team (ICR, p.29). The PIU also reported additional indicators not part of the results framework and documented them in progress reports. However, as there was no dedicated M&E officer checking data collection or report aggregation, this function was fulfilled by a technical expert in the PIU (ICR, p. 33). Based on the interviews conducted with the PIU and the C&J project team, ICR reports that the PIU managed to collect and analyze M&E data in a methodologically sound manner. c. M&E Utilization M&E findings were regularly communicated to participating institutions and relevant stakeholders through PIU’s reports, supervision missions, and ISRs. These findings resulted in two project restructurings. As mentioned at the M&E design section, actual choice of indicators focused on activities and outputs rather than outcomes. Therefore, the M&E data provided evidence for activities and outputs, not outcomes. Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) The ICR makes and effort to collect evidence from national sources to reflect impact of the project activities. However, as this was not foreseen at the M&E design stage, it was not possible to collect such evidence sufficiently. Overall, the M&E quality is rated modest because of significant shortcomings in the M&E design M&E Quality Rating Modest 10. Other Issues a. Safeguards At appraisal, the project was classified as Category Bunder Environmental Assessment (OP/BP 4.01). No other safeguard policy was triggered. An Environmental and Social Management Framework (ESMF) was prepared by the borrower, which excluded activities that would have triggered any other safeguard policies. In line with the agreement between the Government of Serbia and the Bank, the project reimbursed EEPs (three groups of salaries). Such reimbursement did not result in any physical investments that could have (i) caused potential significant harm or negative impacts to the environment or (ii) triggered any resettlement safeguards. b. Fiduciary Compliance Financial Management The Financial Management arrangements were rated as satisfactory in all ISRs conducted during the project. Unqualified opinion was provided by the auditors on annual project financial statements. The WB FM specialist conducted a desk review of the semi-annual IFRs and of the annual audit. Any observations noted either by the auditors or the C&J project team during regular supervision missions were resolved without much delay (ICR p.31). Procurement Procurement activities were handled by the Project Implementation Unit established within the PPS. The line ministries and the agencies did not have to involve in procurement activities. The mini-PIU which was established in MoLEVSA was dissolved due to inefficiency. Within the scope of the project 96 procurements were planned and 82 procurements were completed. PIU completed procurements in coordination with the WB’s procurement specialist and activities were smooth. Procurement plan was modified from time to time subject to the Bank’s no objection. Although several challenges were experienced due to low capacity of implementing agencies, quality of selected consultants or low Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) procurement capacity of mini-PIU of MoLEVSA, overall, the project complied with the Borrower’s and the WB’s procurement requirements c. Unintended impacts (Positive or Negative) Implementation of C&J project contributed to Serbia’s agenda of integration with the EU. Coordination and improvement of policy making could be expected to play a role in attracting further EU funds for capacity building and technical assistance. Establishment of funds and programs to support competitiveness and creation of jobs could boost private sector development and demonstrate Serbia’s determination in pursuing reform processes that would signal for further FDI inflows into the country. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons Composite and suboptimal number of DLIs introduced at project design can complicate project management and become administratively cumbersome for the World Bank and implementing agencies. Project design involved 9 DLIs and most of them were composite (ex. DLIs regarding the innovation component). Although, detailed verification protocols were defined to assist the World Bank and implementing entities, they were difficult to manage and required intensive work. Broad consultations with all relevant counterparts in a structured manner can help addressing some of the implementation issues. As reported in the ICR, some of the counterparts were not fully on board with the proposed activities and felt that additional consultations to further narrow down the priorities were needed, creating complications during implementation. At the same time, for activities (Component #3 on Innovation) where existing dialogue was better developed, Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Serbia Competitiveness and Jobs (P152104) including through engagement from previous projects, priorities were more clearly developed and ownership from the counterparts stronger. Technical assistance provided to Borrower’s agencies at various implementation capacities can complement and improve achievements of investment operations. According to the ICR, team members acknowledged that it was key to have a TA embedded in the project, which was also funded by trust funds (about EUR 2.3 million, or almost 40% of the total TA provided). In addition, line ministries and agencies confirmed that the main benefit of the C&J project was the technical assistance received, as the capacity of the counterparts was uneven (i.e., the PPS showed high technical capacity whereas the capacity of the MOESTD was weaker). 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is candid and provides a detailed overview of the project and the issues encountered during implementation. The narrative is sufficiently evaluative rather than descriptive. The shortcomings in the M&E framework are discussed in detail. Although the M&E design does not provide sufficient evidence to support the achievements of some of the outcomes, the ICR makes a genuine effort to provide evidence outside of the M&E system. The report is consistent with the Bank guidance and draws lessons based on the experience gained during project implementation but they are mostly in the form of findings. While there is a logical linking and integration of the various parts of the report, the report does not sufficiently explain how the ratings have been reached. a. Quality of ICR Rating Substantial Page 21 of 21