84328 v2 ROMANIA REGIONAL DEVELOPMENT PROGRAM Agreement for Advisory Services on assistance to the Romanian Ministry of Regional Development and Tourism (now the Ministry of Regional Development and Public Administration) Dated November 20, 2012 Assessment of the Communication and Collaboration between the Managing Authority and Intermediate Bodies of the Regional Operational Programme and Facilitation of Proactive and Direct Support for Beneficiaries - Final Report, Component II - December 20, 2013 Project co-financed from the European Regional Development Fund through the Regional Operational Programme 2007-2013 www.inforegio.ro This report was prepared by a core team comprised of Sebastian I. Burduja, Marcel Ionescu-Heroiu, Florian Gaman, Valentina Rădoi, Roxana Neșa, Ciprian Ciobanu, Mihai Magheru, and Nadja Ohranovic. The report benefitted from thoughtful comments by Keith McLean, Cătălin Păuna, and Pedro Rodriguez. The report and corresponding missions also relied on administrative support provided by Cristina Zirimis and Adina Vințan. The team would like to thank Elisabeth Huybens for all the advice, support, and guidance provided continuously throughout the process. Special thanks go to our main counterparts in the Managing Authority of the Regional Operational Programme (Ministry of Regional Development and Public Administration), for the data they provided, their timely feedback, and the excellent collaboration. In addition, this study was made possible by the extensive support and dedication of critical stakeholders around Romania: the eight Regional Development Agencies, the Intermediate Body for Tourism, and the 150+ interviewed beneficiaries of the Regional Operational Programme (public sector institutions, private companies, and nonprofits). During a very busy period, these critical staff volunteered their time and offered extensive suggestions for improving the Regional Operational Programme, as reflected in the current report. Last but not least, the team would like to thank all those involved in designing and implementing the survey of ROP beneficiaries, which gathered 470 responses and helped substantiate many of the recommendations in this study. The partners who supported this effort include the Managing Authority of the Regional Operational Programme, the Regional Development Agencies, and the portal for information on EU Structural Funds www.fonduri-structurale.ro. The findings, interpretations, and conclusions expressed in this report do not necessarily reflect the views and position of the Executive Directors of the World Bank, the European Union, or the Government of Romania. Table of Contents List of Figures..................................................................................................................................... v List of Pictures ................................................................................................................................. vii List of Boxes..................................................................................................................................... vii List of Acronyms ............................................................................................................................. viii The Regional Operational Programme (ROP): Snapshot ...................................................................... x Executive Summary............................................................................................................................1 Introduction .................................................................................................................................... 13 I. Context ......................................................................................................................................... 15 Objective and Scope ............................................................................................................................... 19 Methodology........................................................................................................................................... 21 Audiences ................................................................................................................................................ 23 Structure of the Report ........................................................................................................................... 24 II. Theoretical Framework ................................................................................................................ 27 Organizational Performance ................................................................................................................... 27 Enabling Environment ............................................................................................................................. 30 III. Overview: ROP Beneficiaries and Types of Support Mechanisms .................................................. 33 The Demand Side: ROP Beneficiaries...................................................................................................... 33 The Supply Side: MA and IB Support Mechanisms ................................................................................. 43 IV. Horizontal Cross-Themes ............................................................................................................. 47 A. Beneficiary Perspectives on Regionalization ...................................................................................... 47 B. Human Resources ............................................................................................................................... 49 Internal Capacity ................................................................................................................................. 49 ii External Resources .............................................................................................................................. 52 C. Access to Finance ................................................................................................................................ 54 D. Electronization: Toward a Functional e-ROP ...................................................................................... 55 E. Targeted Support: Engaging Marginalized Communities and the CLLD Approach ............................. 62 Relevance of the CLLD approach for ROP 2014-2020 ......................................................................... 64 V. Pre-Application ............................................................................................................................ 73 Beneficiary Engagement in Programming .............................................................................................. 74 Pipeline Development............................................................................................................................. 77 Awareness-Raising Activities............................................................................................................... 78 Project Development .......................................................................................................................... 80 VI. Application, Evaluation, and Contracting ..................................................................................... 83 Applicant Guides ..................................................................................................................................... 84 Eligible vs. Ineligible Project Expenses .................................................................................................... 86 Helpdesk.................................................................................................................................................. 89 Training ................................................................................................................................................... 91 IB-led ................................................................................................................................................... 91 MA-led................................................................................................................................................. 92 Ongoing Communication ........................................................................................................................ 94 VII. Implementation ......................................................................................................................... 97 Public Procurement and Corrections ...................................................................................................... 97 Detailed recommendations for improving the public procurement framework.............................. 105 New dynamics in the EU public procurement framework ................................................................ 113 The Design Stage: Technical Documentation ....................................................................................... 119 Accountability mechanisms .............................................................................................................. 119 Bill of Quantities, Schedule of Rates, and Detailed/Consolidated Items .......................................... 122 The Execution Stage: Contracts, Monitoring, Reimbursements ........................................................... 123 The FIDIC standard contract ............................................................................................................. 123 iii Other key service providers and corresponding accountability mechanisms .................................. 128 Monitoring ........................................................................................................................................ 128 Reimbursements ............................................................................................................................... 129 VIII. Post-Implementation .............................................................................................................. 132 Ex-Post Monitoring and Evaluation (M&E) ........................................................................................... 132 Knowledge Exchange ............................................................................................................................ 135 IX. Roadmap of Priority Recommendations ..................................................................................... 138 Annex A: Survey of ROP Beneficiaries (April-May 2013) .................................................................. 148 Annex B: Beneficiary Interview Guide ............................................................................................. 156 Annex C: List of Interviewed Beneficiaries....................................................................................... 158 ANNEX D: Spatial Distribution of ROP Projects (contracted as of December 2012) ........................... 162 (1) Regional Level .................................................................................................................................. 162 (2) County Level .................................................................................................................................... 163 A. Total .............................................................................................................................................. 163 B. Public institutions ......................................................................................................................... 164 C. Private organizations .................................................................................................................... 165 (3) Local Level........................................................................................................................................ 166 A. Total .............................................................................................................................................. 166 B. Public bodies ................................................................................................................................. 167 C. Private organizations .................................................................................................................... 168 (4) By Key Intervention Area ................................................................................................................. 169 iv List of Figures Figure 1. Relative performance of EU structural programs in Romania (as of November 30, 2013) ......... 15 Figure 2. Results of a February 2010 survey of EU-funds beneficiaries...................................................... 18 Figure 3. Percentage of respondents who identify an obstacle as extremely/very important .................. 19 Figure 4. Framework for assessing available assistance for ROP applicants and beneficiaries.................. 22 Figure 5. Organizational performance (Kaplan's balanced scorecard vs. Moore's strategic triangle) ....... 28 Figure 6. MA/IB assistance under the three pillars of organizational/project performance ..................... 30 Figure 7. A theoretical framework for assessing ROP support mechanisms .............................................. 31 Figure 8. Financing applications submitted as % of priority area allocation (December 31, 2012) ........... 34 Figure 9. Regional split of ROP projects (by axis) ........................................................................................ 35 Figure 10. The number of completed vs. approved projects by region (December 2012) ........................ 36 Figure 11. The number of completed and approved projects by county (as of December 2012) ............. 36 Figure 12. County-level split of ROP projects (by axis) ............................................................................... 37 Figure 13. Distribution of ROP contracted funds (December 31, 2012) ..................................................... 38 Figure 14. Distribution of municipalities, towns, and communes .............................................................. 38 Figure 15. Spatial distribution of ROP projects (by number) ...................................................................... 39 Figure 16. Spatial distribution of ROP projects (by value) .......................................................................... 39 Figure 17. Beneficiary survey perceptions of ROP benefits’ level of importance ...................................... 40 Figure 18. Beneficiary survey perceptions of obstacles' level of importance ............................................ 41 Figure 19. Beneficiary perceptions of benefits/costs of ROP projects ....................................................... 42 Figure 20. Beneficiary responses to whether they would apply for future ROP financing ........................ 42 Figure 21. Romanian beneficiary perspectives are very similar to those in other Member States ........... 44 Figure 22. Typology of ROP support mechanisms ...................................................................................... 46 Figure 23. All beneficiary types support the idea of an online ratings system for service providers......... 54 Figure 24. A useful e-application platform is already in use in Romania (UEFISCDI) ................................. 58 Figure 25. An overwhelming majority of beneficiaries welcome the introduction of e-documentation... 61 Figure 26. Coverage of LAG during 2011-2012 timeframe ......................................................................... 67 v Figure 27. WB CDD projects from 2000 to 2013 ......................................................................................... 71 Figure 28. Balancing absorption and impact .............................................................................................. 73 Figure 29. The current structure for 2014-2020 programming of EU structural funds in Romania ........... 76 Figure 30. Satisfaction with RDA support during the application phase is generally high ......................... 84 Figure 31. Beneficiary perceptions of RDA/MA helpdesk support (# of ROP projects implemented) ....... 90 Figure 32. Level of satisfaction with MA-led sessions by beneficiary level of experience ......................... 94 Figure 33. Example of online tracking system (USCIS)................................................................................ 96 Figure 34. Beneficiary perceptions of the public procurement framework as an obstacle ....................... 98 Figure 35. Corrections by type of irregularity detected ........................................................................... 100 Figure 36. Corrections as % of total budget of projects under implementation (Oct. 2012) ................... 101 Figure 37. Time limits for key procurement steps .................................................................................... 113 Figure 38. Beneficiary perceptions of requiring FIDIC contract forms for ROP-financed projects ........... 125 Figure 39. Selecting the right type of contract (i.e., FIDIC color) is relatively simple ............................... 126 Figure 40. Beneficiary satisfaction with visits by officers from RDAs (left) and the MA (right) ............... 129 vi List of Pictures Picture 1. The ROP includes a very diverse portfolio of beneficiaries and projects ................................... 17 Picture 2. The heavy reliance on hard copies has generated serious storage problems ........................... 57 Picture 3. Proper early planning ensures a smoother implementation later on ........................................ 80 Picture 4. Even in the most modern classrooms, some of the old chairs were kept ................................. 87 Picture 5. Complex projects require contractors with special competencies (not always the lowest price offer) ......................................................................................................................................................... 117 Picture 6. A good technical design is essential for a high-quality execution phase ................................. 121 Picture 7. An oversized boiler creates budgetary pressures for the school and local authorities in Livada .................................................................................................................................................................. 134 List of Boxes Box 1. The cross-cutting theme “equality of opportunity” was ineffective in supporting Roma-focused projects .......................................................................................................... Error! Bookmark not defined. Box 2. An alternative system for determining eligible and ineligible expenditures ..... Error! Bookmark not defined. Box 3. The 5% total cap is inadequate ........................................................... Error! Bookmark not defined. Box 4. The Spanish equivalent of the Eligible Expenditures Order is much more flexible ......................... 89 Box 5. The unintended consequences of rigid rules ................................................................................. 133 Box 6. The positive power of knowledge sharing is beyond doubt .......................................................... 137 vii List of Acronyms ACSI Authority for the Coordination of Structural Instruments BOQ Bill of Quantities CCRD Consultative Committee on Regional Development CCTC Consultative Committee on Territorial Cohesion CDD Community Driven Development CLLD Community-Led Local Development CSF Common Strategic Framework DCFA Directorate for Coordinating Field Activities EARDF European Agricultural Fund for Rural Development EC European Commission EEO Eligible Expenses Order EMFF European Maritime and Fisheries Fund EU European Union ERDF European Regional Development Fund ESF European Social Fund FC Financing Contract FDI Framework Document for Implementation FIDIC Fédération Internationale des Ingénieurs Consultants FIFO First-in-first-out FS Feasibility Study GEO Government Emergency Ordinance GOR Government of Romania ICT Information and Communication Technology IDA Intercommunity Development Association IB Intermediate Body ITI Integrated Territorial Investment ICPA Interinstitutional Committee for Elaborating the Partnership Agreement LAG Local Action Group LDS Local Development Strategy LEADER Liaisons entre Actions de Développement de l'Economie Rurale MA Managing Authority MA-ROP Managing Authority of the Regional Operational Programme MC-ROP Monitoring Committee of the Regional Operational Programme MARD Ministry of Agriculture and Rural Development M&E Monitoring and Evaluation MEF Ministry of European Funds MEI Ministry of European Integration MO Monitoring Officer MRDPA Ministry of Regional Development and Public Administration MRDT Ministry of Regional Development and Tourism NAA National Audit Authority NARMPP National Authority for Regulating and Monitoring Public Procurement NCRD National Council for Resolving Disputes NRDP National Rural Development Program viii OSI Open Society Institute PARDF Payment Agency for Rural Development and Fishing RDA Regional Development Agency RCR Regional Committee for Elaborating the Regional Development Plan RDP Regional Development Plan ROP Regional Operational Programme RSDF Romanian Social Development Fund SIC State Inspectorate for Constructions SMIS Single Management Information System TO Thematic Objective TCC Thematic Consultative Committee UCVPP Unit for the Coordination and Verification of Public Procurement ix The Regional Operational Programme (ROP): Snapshot Overall purpose: The ROP’s overall objective consists in supporting the economic, social, territorially balanced, and sustainable development of Romanian regions, according to their specific needs and resources, focusing on urban growth poles, improving the business environment, and basic infrastructure. The Romanian regions, especially the ones lagging behind, could become more attractive places where to live, visit, invest and work. Impact: It is expected that the implementation of this programme will create better conditions for the economic and social balanced territorial development of all Romanian regions and for urban growth poles able to spread the development. The ROP’s strategic objectives are: 15,000 new jobs by the end of 2015; and reduced interregional disparities between 2007 and 2015 (measured as GDP/capita). Priority axes: 7. Support to sustainable development of urban growth poles 8. Improvement of regional and local transport infrastructure 9. Improvement of social infrastructure 10. Strengthening the regional and local business environment 11. Sustainable development and promotion of tourism 12. Technical assistance From a total value of 4.38 billion EUR, the ROP’s financial allocation by axes is as follows: Key players:  The Managing Authority (MA) is under the Ministry of Regional Development and Public Administration (MRDPA) and is responsible for the ROP’s design and implementation.  The Intermediate Bodies (IBs) consist of 8 Regional Development Agencies (RDAs) and 1 central IB- Tourism. They fulfill tasks related to the ROP’s programming and implementation, as delegated by the MA.  Beneficiaries are public, private, or non-profit organizations that submit financing applications and receive ROP funding for implementing proposed projects.  Other key stakeholders include: at the regional level, the Regional Development Council (RDC); at the national level, the Monitoring Committee of the ROP (MC-ROP), the Audit Authority, the Certification and Payment Authority, and the Ministry of European Funds. Executive Summary The Regional Operational Programme (ROP) is a top performer in terms of absorption rates, in the Romanian context, despite its complex and diverse portfolio of projects. The ROP addresses a wide- ranging set of needs at the level of each of the eight regions, covering areas like urban development, transport infrastructure, social infrastructure, business development, and tourism. Some projects are relatively small and easy to implement, while others are large, complex infrastructure works, with cumbersome technical documentation and lengthy procedures. As such, it is remarkable that, as of November 30, 2013, the ROP remains the country’s top performing operational program, particularly when it comes to payments from the EC, which stood at 44.2% of the total ROP allocation. Interview and survey feedback is overwhelmingly positive in terms of beneficiaries’ satisfaction with the ROP, particularly when compared to other operational programs (OPs). Beneficiaries welcome their collaboration with the Intermediate Bodies (IBs) and the Managing Authority (MA). There is a sense of shared partnership and a common understanding that all stakeholders involved genuinely want the program to succeed. The ROP owes much of its relative success to the fact that MA and IB staff: are closer to program applicants and beneficiaries; demonstrate enhanced understanding of what needs to be done on the ground; rely on experienced people who know how to design and implement optimal solutions, working in close partnership with beneficiary teams, many of whom they know and trust; have greater accountability for outcomes and results vis-à-vis beneficiaries and the general public; and, finally, they benefit from quicker, more effective feedback loops from beneficiaries. This is neither to say that assistance mechanisms currently available to ROP applicants and beneficiaries are sufficient and flawless, nor that a program’s performance is simply a matter of superior absorption rates. For the current report, a survey was sent out to beneficiaries in all eight regions to inquire about the main obstacles faced in accessing ROP funding and implementing projects. Of the total 470 responses, the proportion of people who characterized various obstacles as “extremely important” and “very important” varies from a staggering 81.5% for “excessive bureaucracy and documentation” and 80% for “frequent legislation changes” to only 31.7% for “poorly trained RDA staff” (see figure below). Also, over half of the respondents identified additional obstacles as “extremely important” and “very important”, in the following order: delays in project evaluation and selection (73%), delays in reimbursements (69%), lack of capital for co-financing and cash-flows (68%), inadequate public procurement framework (62%), insufficient prefinancing (56%), and poor-quality technical documentation (52%). In addition, some important issues fall beyond the immediate scope of this report, although they are worth mentioning as part of the broader context of the ROP’s implementation. For one, a better effort is needed to improve the selectivity and prioritization of ROP investments. It is not always clear that current ROP projects match local needs and are aligned with broader regional and national-level strategies, but the selection process (based on the first-in-first-out rule) has often privileged absorption instead of strategic impact. Moreover, there are concerns that local authorities will not be able to shoulder the operation and maintenance (O&M) costs related to finalized ROP projects. These issues are central to another ongoing World Bank technical assistance project on “improved project selection.” Under preparation is also the ROP’s intermediary evaluation, prepared by KMPG, which includes highly useful data on projects’ impact, in many ways offering a complementary analysis to the current work. Finally, national budget funding for regional and rural development continues to duplicate ROP resources through the newly established National Program for Local Development, although selection and implementation procedures are generally less rigorous than under the ROP. There is a need to ensure coherence between national-budget projects and EU-financed ones, which is the focus of an 1 upcoming World Bank assessment for the Ministry of Regional Development and Public Administration (MRDPA). While keeping these issues in the background, time and resource constraints require that the scope of the current work is strongly centered on two elements: (a) assessing current support mechanisms available to ROP applicants and beneficiaries; and (b) making recommendations for improvement. The first guiding principle of the current analysis remains: “first, do no harm.” The ROP components that are functional and efficient in supporting applicants and beneficiaries should be strengthened to leverage accumulated institutional knowledge and to minimize disruptions – a high level of predictability is valuable, in of itself, as a way to provide a stable environment with clear rules for applicants and beneficiaries. Equally important, it should be clear that this report is about how to assist beneficiaries, taking as a given their current capacity. This is not to say that there is not much to do on beneficiaries’ end to boost their ability to design and implement ROP projects. But the key topic of this analysis remains the support available to ROP applicants and beneficiaries. Beneficiary survey perceptions of obstacles' level of importance Excessive bureacracy & documentation Delays in project evaluation & selection Delays in reimbursements Lack of capital for co-financing & cash flow Frequent legislative changes Inadequate public procurement framework Insufficient prefinancing Low quality technical documentation Lack of high-quality consulting services Lack of transparency Low quality of applicant guides Poorly trained AM staff Poorly trained RDA staff 0 50 100 150 200 250 300 350 400 450 500 Extremely important Very important Relatively important Hardly important Unimportant N/A While there is a long, well-known list of issues related to EU-funded projects’ implementation, the good news is that most beneficiaries continue to believe in the net positive value of the program. 87% of respondents to the survey answered that the costs involved are lower than the benefits of ROP financing. As for beneficiaries’ appetite for applying for future ROP funding, 83% of the answers were 2 positive, with only 4% rejecting the idea of seeking this type of financing again – a very encouraging sign for the ROP’s attractiveness in the next period. While the perceived “benefit-cost ratio” remains greater than 1, the MA and the IBs teams should aim to increase it further, minimizing hurdles for beneficiaries, to the full extent possible. In this context, the first recommendation is to shift the paradigm to a “beneficiaries first” model, which aligns perfectly with the aim of the current report. To be sure, there is an inherent tension in the two critical functions that the ROP system must fulfill in relation to applicants and beneficiaries: monitoring and control versus partnership and support. Evidence from field interviews with beneficiaries and other key stakeholders suggests that the ROP system would benefit from balancing the current rigorous focus on control and sanctions with stronger efforts around partnership and support. Institutionally, the IBs are better placed to support applicants and beneficiaries on the ground, while control functions more naturally rest with central authorities (the MA, the Audit Authority, etc.). Providing adequate beneficiary support mechanisms at the MA and the IB level depends on two key variables for each applicant/beneficiary: applicable selection model and portfolio value. Regarding the former, dedicated funding for a project or set of projects (as in the case of growth poles) can and should be complemented by customized, close engagement by MA/IB teams, throughout the project cycle. The aim is to ensure technical assistance that is fully aligned with specific beneficiary needs and helps deliver the most impact possible. By contrast, if the selection is done on a competitive basis, support should be standardized. Typology of ROP support mechanisms Dedicated Extensive, customized support Type of selection model Efficient, standardized support Competitive Small, simple Large, complex Value (complexity) of ROP portfolio (# projects * average budget) The typology in the figure above is a basic representation of how optimal support mechanisms vary across applicants/beneficiaries and corresponding axis/priority area/selection models. The portfolio 3 value is equivalent to the total number of ROP projects multiplied by their average budget. As noted above, there are two relatively distinct groups: public-sector institutions implementing a large, complex set of projects; and public, private, or non-profit beneficiaries of relatively smaller, simpler investments. Importantly, this approach is fully aligned with the recommendations made by the EC in its October 2012 position paper regarding Romania’s preparations for 2014-2020, as follows: “Whereas tailored assistance could be directly channeled to larger beneficiaries, accompanied by sound ownership and appropriate endogenous reinforcement and restructuring, streamlined assistance should be provided to sectors sheltering a broader scope of smaller and more heterogeneous beneficiaries (tailored guidance, help desk, dissemination of standardized documents, etc.).”58 Horizontal Cross-Themes A. Regionalization Most beneficiaries are in favor of the Government’s continued efforts to push forward the plan for devolving increased responsibilities to the regional level, although some elements will not apply before the 2021-2027 programming period. First, one key advantage of a regional-focused ROP was identified as a higher level of responsiveness to the regional-level needs and priorities. Second, beneficiaries note that the MA in Bucharest is inevitably more removed from the situation on the ground, unlike RDAs, which are able to customize their assistance and intervene more quickly, as needed. Beneficiaries would welcome a regional MA as a best-of-both-worlds option: it would be close to the ground, knowledgeable of their specific situation, able to react quickly, and it would also hold the decision-making power to adopt certain needed measures and provide informed answers. Perhaps most importantly, regionalization promises to deliver truly regional projects by ensuring proper coordination of the political and administrative levels, as long as it does not replicate the current dynamics that do not encourage projects across multiple jurisdictions (e.g., cities, counties, etc.). In particular, one possibility worth considering further for 2014-2020, based on interview feedback and in conjunction with the project selection models workstream, is a “regionalization-light” option, with an ROP system adapted to different types of projects, depending on their magnitude. B. Human Resources The ROP’s success depends first and foremost on the people who make the system work – particularly those responsible for designing and implementing projects. Without their passion, engagement, and persistent involvement, even the best institutional set-up would be ineffective in reaching the ROP’s fundamental objectives. As such, the highest priority for the ROP system is to ensure low turnover of critical internal staff. First, this requires creating the conditions for adequate pay to project teams in public-sector institutions, throughout an investment’s cycle, and not only during the implementation stage. In addition to expanding the bonus system, options should be explored for increasing the salary base in line with performance. Equally important, many of the units working on EU-financed projects remain understaffed and overburdened with responsibilities related to managing large portfolios. This is particularly true of the recent period, when teams have had to both accelerate the implementation of ongoing projects to complete them before the 2015 deadline and also prepare new projects for the 2014-2020 budgeting cycle. In this context, the legal framework constraining the hiring of public servants should be reconsidered to ensure that units focused on EU-financed projects are adequately 58 “Position of the Commission Services on the development of Partnership Agreement and Programmes in Romania for the period 2014-2020,” Ref. Ares(2012)1240252 - 19/10/2012, p.37 4 staffed at all times. All this should be based on a proper system that puts the right people into the right positions, with adequate, meritocratic hiring practices that bring much needed skills from key areas (economics, civil engineering, and legal affairs). One final recommendation for leveraging internal resources is for the MA and the IBs to capitalize on the existing knowledge of experienced beneficiary staff. One possibility is to bring them together under an ROP Corps of Practitioners (ROP-CP), which would meet periodically and send suggestions to IBs, the MA, and the MC-ROP, as necessary. It is expected that the Corps could quickly become a critical component of the ROP’s continuous learning and adaptation system, and would ensure an even higher degree of the system’s responsiveness to the problems identified in the field. Moreover, members of the ROP-CP could also be selected to become trainers of staff involved in EU-funded projects. At the same time, it is unrealistic to require ROP beneficiaries to complete all tasks internally – in fact, outsourcing certain services is both encouraged and viable, given than such expenses are typically eligible to be covered by EU funds. The key challenge in managing external resources is that the market is not fully mature or transparent, so selecting the right consulting company is a daunting task for most beneficiaries of ROP funds. Solutions to increase the market’s transparency include: (a) publishing the names of consulting firms who prepare and/or manage projects and (b) enabling user-feedback mechanisms by allowing beneficiaries to post comments about their experiences with various service providers.. In addition, contracts with external resources should be strengthened to protect the interests of applicants and beneficiaries and hold consultants accountable for the quality of their work, contributing to, among other things, the reduction of errors in filling out financing applications and reimbursement requests. C. Access to Finance Adequate access to finance throughout all stages of a project is of paramount concern for interviewed beneficiaries, across all sectors (public, private, and non-profit). Field research observations suggest that several banks in Romania, including some of the major ones, have specific products targeting the needs of project beneficiaries. Recently, the market for such instruments has grown in size and complexity, but many beneficiaries remain unaware of available opportunities, while, for their part, a number of banks continue to regard beneficiaries of structural funds with high skepticism. As lending is expected to resume along with the recovering economy, the main anticipated need is to “connect” beneficiaries to financing institutions and representative associations (e.g., the Romanian Association of Banks). This is a role that the MA and the IBs can easily play, acting as a bridge between the demand and supply of financing needs for ROP projects Banks could also play a role in assessing the viability of specific proposals (e.g., as part of the cost-benefit analysis), based on their accumulated experience and knowledge of market dynamics. D. Electronization There are several important advantages of an e-based system: speed, cost savings, and accuracy. Recognizing this, the EC has promoted e-cohesion strongly, proposing that by December 31, 2014 beneficiaries should be provided with a way to communicate only electronically with MAs and IBs. The Authority for Coordinating Structural Instruments (ACSI) has launched a process for developing the mySMIS application, which would serve as an interface with beneficiaries, complementing the current SMIS backend database. What remains to be determined is the quality of the final product and its implementation, but in any case it is recommended to leave ample time for consultations, beta testing, and piloting. While the need for avoiding duplication is paramount, if the mySMIS launch will be delayed much further, the MA-ROP should strongly consider developing its own electronic system, even a simplified one to cover the basic needs of the program. 5 The full project cycle should become e-based. As a first step, the introduction of electronic forms for the application phase would reduce bureaucracy and associated costs, and help prevent beneficiaries from making mistakes (e.g., by only allowing certain answers), ensuring smoother completion of projects further down the line. The evaluation process could also take place on an easily accessible electronic platform. Under this setup, there would be no need to pay the extra costs of transporting evaluators to the RDA headquarters for the desk evaluation. Evaluators could perform a review of submitted documents remotely and then a single on-the-spot verification, so a larger proportion of the budget could be dedicated to covering actual fees and ensuring higher-quality evaluations. During the implementation phase, all required exchanges of information, including reimbursement requests, progress reports, and supporting documentation, would be submitted and verified electronically. This would avoid duplication (e.g., submitting the public procurement file with every reimbursement request). Importantly, an e-based ROP does not mean that all in-person interactions would be eliminated. Above all, verification missions in the field will be needed to ensure that documents submitted electronically are exact copies of originals – under this system, beneficiaries would be required to keep all original paperwork in an easily accessible location and make it available whenever needed by auditors. At the same time, not all MA-IB-beneficiary interactions can happen electronically. In-person trainings, for instance, have proved much more useful than online seminars. The same goes for field visits by evaluators to verify feasibility studies against the reality on the ground. E. Targeted Support: Engaging Marginalized Communities A key priority for Romania’s development in 2014-2020, supported by the EC, other international donors, and the Romanian Government, is the integration of marginalized communities and the promotion of inclusive growth. In particular, Roma integration has to be a primary focus of any efforts aimed at sustaining Romania’s development, taking into account the community’s strong potential to make a contribution to the country’s absorption of EU funds and long-term growth. Despite all this, ROP projects targeting Roma and other marginalized communities are few and remain mostly concentrated on rehabilitating infrastructure like social centers. Without downplaying the importance of these initiatives at the local level, the reality is that the ROP has not generated substantial impact on critical issues like social integration, labor market participation, and school enrolment. To address the current situation, there is a need to improve the ROP’s programming and support mechanisms to deliver more impact on marginalized communities in urban areas. This will depend on adopting targeted selection criteria and encouraging beneficiaries to engage Roma and other marginalized communities through their ROP projects. Also, MA/IB teams should develop targeted support for members of marginalized communities to apply as potential beneficiaries of ROP funds. Along the same lines of bottom-up participation, the MA-ROP and the IBs should work together to evaluate whether the Community-Led Local Development (CLLD) instrument is appropriate for the ROP 2014-2020. The CLLD holds great promise for a program like the ROP, which is well positioned to take advantage of its closer presence to beneficiaries on the ground. More generally, CLLD approaches are expected to increase the effectiveness and efficiency of projects and service delivery, and contribute to social inclusion through empowerment. 6 Phase-Specific Support Phase I: Pre-Application In the pre-application phase, assistance to potential applicants should target two key areas: programming and pipeline development (i.e., awareness-raising and project development support). On programming, it is recommended that working groups of beneficiaries stay engaged throughout the entire process, which should aim to become increasingly transparent and participatory. Beneficiary preferences should also be reflected in the program’s implementation framework (e.g., specific procedures, definition of eligible vs. ineligible expenditures, etc.). It is also recommended that those decision-makers responsible for drafting the PA, the ROP, and related documents experiment with online consultation tools. As for project development efforts, it is recommended that the IBs take a more proactive stance in working with potential beneficiaries to define and operationalize viable, impactful ideas. Additional best practices include appointing a liaison person and improving pre- application training sessions to ensure the submission of error-free, high-quality financing applications that contribute to a smoother implementation further down the road. Phase II: Application, Evaluation, and Contracting Five areas are most relevant for providing enhanced assistance to ROP applicants at this point in the process: applicant guides, eligible expenditures, helpdesk activities, training, and ongoing communication. A. Applicant Guides Beneficiaries generally appreciate the guides as comprehensive, useful, and sufficient for understanding the process of filling out and submitting a financing application, but there are also two main areas of improvement. First, repeated changes should be minimized, particularly on finalized guides, to maintain predictability. This would avoid indirectly penalizing proactive beneficiaries, the “early-risers,” who are forced to make adjustments and spend additional resources to align their proposals to the requirements from the updated guides. Second, two key elements should be added to the guides to improve their transparency: the potential timeline (minimum and maximum duration) of each step in the process (including the contracting phase), based on the MA-IB audit trails (“pistele de audit”); and a full section on “beneficiary rights and obligations,” as included in the financing contract. Even though the latter is an annex to the applicant guide, to avoid any confusion among applicants, it is recommended to maintain the same content for both documents referenced here. B. Eligible vs. Ineligible Expenses For the ROP, the list of eligible expenses is defined through a common order (“the Eligible Expenses Order” – EEO) by the Minister of Regional Development and Public Administration and the Minister of Finance.59 Interviewed ROP beneficiaries urge decision-makers to either (A) improve the Eligible Expenses Order (EEO) based on their feedback; (B) eliminate the EEO and rely on applicable national- level legislation instead; or (C) define only ineligible expenditures. Given the accumulated experience in the current period, the ROP 2014-2020 would need to take into account extensive inputs from beneficiaries regarding the revision and expansion of EEOs (option A). At the same time, it is doubtful that one can regulate every single project-specific instance, so options B and C appear more feasible. What is more, the current EEO establishes a maximum value of 5% for expenses related to project design and technical assistance (Article 3), which has contributed to the low quality of technical 59 See “Monitorul Oficial al României,” No. 327, from May 11, 2011 7 documentation and construction work. International benchmarking suggests that this artificial cap should be raised to at least approximately 10% of the project’s value. One consequence of inadequate budgeting relates to the performance of construction site supervisors, who often serve in this role for multiple contracts at once, even in different localities. In some cases, they endorse progress and inspection reports without actually verifying the situation on site, which later turns against the beneficiary in the form of unanticipated, ineligible additional expenses. Increasing the cap to a reasonable level would help facilitate high-quality services by both technical designers and construction site supervisors, leading to better, smoother implementation. C. Helpdesk Each RDA has established a functioning helpdesk that offers assistance to interested ROP applicants, with positive results to date. For their part, IB-level helpdesks are able to act as a first point-of-contact for applicants who need guidance around how to fill out an application and how to obtain the required supporting documentation, functions that the MA has delegated to RDAs. The closer this assistance gets to beneficiaries, the better it can address their needs in a timely manner, which suggests that most of the helpdesk activities should remain concentrated at the IB level. Sure enough, feedback from interviews confirms this hypothesis: most beneficiaries contact their local RDA representative first, in the county or regional office, and only turn to the MA staff under special circumstances, when they feel the need to reach someone in the Ministry. The helpdesk function should be maintained and strengthened going forward. D. Training RDAs and the MA organize post-contracting training sessions with beneficiaries who have signed contracts or have started project implementation. The importance of these sessions for ensuring smooth project completion is reflected in both survey results and interview feedback from field visits. Going forward, the MA and the IBs should continue to organize this training and improve its delivery, keeping in mind the need for: effective timing and coordination; increased frequency; and enhanced customization for different types of beneficiaries. E. Ongoing Communication A recurring theme in survey responses and interviews is the lack of transparency and predictability of the evaluation and contracting process. Beneficiaries understand procedural steps and selection models, but they have less visibility into the timeline and progress of their application throughout the process and particularly once their documentation is sent to the MA-ROP. A solution with benefits on all sides would be to increase the transparency of the flow of documents, developing a simple online tracking system. Applicants would be able to access the platform to determine where their application is at any one point in time, how long the process is expected to take, and who the main point of contact is at each step in the process. This could increase the timeline’s predictability and transparency, mitigate concerns among beneficiaries, and free up capacity in the MA and RDAs helpdesks, which could quickly refer applicants to the online system. It would also help improve accountability up and down the process chain within the MA and the IBs, ensuring visibility around bottlenecks and capacity constraints. Phase III: Implementation 8 A. Public Procurement By far the greatest concern expressed by ROP beneficiaries in field interviews centers on issues related to public procurement and associated corrections. Over 70% of public beneficiaries rate the public procurement framework as an “extremely” or “very important” obstacle. Financial corrections related to public procurement procedures are also widespread among local authorities. The key issue is that Romanian legislation on public procurement procedures related to EU-financed projects failed to incorporate until 2011, 4.5 years into the ROP’s implementation, several key elements (e.g., clear definitions of restrictive criteria, applicable corrections, etc.). The total sum imposed in penalties by December 2012 had exceeded EUR 300 million. By far the most common problem is related to “illegal qualifying criteria” – these are also referred to as restrictive/disproportionate criteria and often result from requesting a particular level or type of experience from the bidders. Beneficiaries characterized the vast majority of these errors as purely non-intentional or wrongfully imposed. 80% of sanctioned beneficiaries challenged the corrections and 41% have sought remedy through legal court action, often arguing that retroactive application of law is unconstitutional. Many of the initial decisions from Appellate Courts are favorable to beneficiaries, though most final definitive verdicts by the Supreme Court are only expected in 2014 or 2015. The key issue of public procurement is the suboptimal, inefficient ex-ante and ex-post verification system, with a chronic lack of accountability of all stakeholders involved, except for beneficiaries, who are left to bear the entire burden of potential mistakes. Interviewed beneficiaries call attention to the urgent need of publishing an official collection of cases and their related corrections. Maximum transparency is required in order to help beneficiaries avoid making the same mistakes in the future. A complementary measure is to put together lists of acceptable selection criteria. Equally important, there need to be clear-cut standards for which level of sanction applies in a particular case, with consistent application of these rules regardless of which body makes the proposal for corrections. Penalties should be applied only as a last resort and after all verifications have been carried out by specialized auditors with the right set of skills (including technical expertise). Moreover, as far as control institutions are concerned, they should take responsibility for their recommendations and share the costs of potential corrections if they failed to prevent beneficiaries from going forward with a faulty public procurement procedure, or have reimbursed them despite obvious irregularities. B. The Design Stage: Technical Documentation Beneficiaries often mention the low quality of technical documentation as the leading cause of implementation delays and ineligible expenditures. A total of 48% of interviewed subjects consider this factor to be an extremely/very important obstacle against effective absorption of ROP funds. Simply put, beneficiaries are currently exposed in their relationship with both project designers and contractors: the latter seek to point out faults in the technical project, hiking the agreed price set through the public procurement process; the former are reluctant to agree to changes of the project and put the blame on the workers in the field for failing to apply the technical documentation. This eventually leads to significant ineligible expenditures when there are differences between the technical project (TP) and the situation on the ground during implementation. EU-level regulations do not accept such discrepancies as “unpredictable.” The leading causes for this situation revolve around lack of capacity and inadequate accountability mechanisms. For their part, beneficiaries rarely have the available resources to properly check projects submitted by designers. Technical experts are often selected based on the lowest price criterion (a broader issue of the current public procurement framework). The ROP system tried to counteract these weaknesses by including a review of technical projects during the evaluation stage. This has not always worked, as it puts more pressure on beneficiaries to finish studies before the six-month deadline and, 9 also, the technical project evaluators mainly perform an “administrative” check on the documentation to see if all supporting documents are included, without having the capacity to perform in-depth checks on technical solutions, the lists of quantities, etc. MA and IB teams of experts should help beneficiaries avoid discrepancies between the technical documentation and the situation on the ground by endorsing standard contract forms. These would be part of the public procurement documentation and help ensure that qualified, professional bidders enter the competition. Best practices would include the following: a clear stipulation of the designer’s responsibility to deliver a high-quality output; a complete timeline for every deliverable, including scheduled payments, through the implementation stage; a clear requirement that all property rights over the technical documentation belong to the beneficiary; and a direct relationship with project verifiers, who should be contracted and paid by beneficiaries, and should be accountable for approving deliverables that do not abide by strict standards. Another solution for reducing friction among designers, beneficiaries, and construction firms during the implementation phase is to allow design-and- build contracts (and eliminate the requirement of submitting the technical project during the evaluation stage). The winning bidder takes on the responsibility of developing the technical project and implementing it, so any discrepancies must be solved without incurring additional costs for the beneficiary, unless particular construction works are truly unpredictable, in which case they would qualify as eligible. C. The Execution Stage: Contracts, Monitoring, and Reimbursements In particular, the FIDIC standard contract forms would help beneficiaries manage complex projects better, while protecting the interests of all stakeholders involved. The main benefits of the FIDIC contracts revolve around a balanced sharing of risks between the beneficiary and the contractor and a greater degree of familiarity of international investors with the specific conditions and procedures mandated by these contracts. Reintroducing FIDIC in the Romanian legislation (primarily the Red Book with design separate from execution and the Yellow Book for design-and-build contracts) promises important benefits, as long as the risks (e.g., legislative misalignment, lack of capacity) are properly addressed. More broadly, stronger contracts are needed vis-à-vis the whole set of service providers that play a role during the implementation stage: construction supervisors, auditors, and project managers. For instance, construction supervisors should be required to stay on-site full time, and penalized accordingly if they breach this provision. For their part, auditors currently play a limited role, with low accountability: they certify expenditures by checking accounting documents, but do not typically go beyond desk assessments. Instead, they should evaluate reimbursement requests, including those procedures that have led to particular expenses. The same principles of stronger accountability should also apply to project managers, who must be held accountable for mistakes resulting from their own work (e.g., errors in submitting reimbursement requests). For all such actors, increased accountability should be complemented by a requirement to have professional insurance. This would facilitate the process of recuperating at least some of the financial corrections generated by mistakes that make it passed these filters and would also help “clean” the market in the long run. Further research is needed to determine how this model could work in practice in Romania. As for monitoring and reimbursements, most of the room for improvement lies with measures aimed at procedural simplification and greater decentralization of responsibilities at the regional IB level. It is important to acknowledge the government’s recent efforts (e.g., GEO 27/2013) aimed at short -circuiting reimbursement processes and enable beneficiaries to maintain proper cash flow. Interviewed public- sector beneficiaries were enthusiastic about this measure, while private companies and NGOs expressed 10 their hope that they would become eligible for the alternative mechanism soon. Such initiatives should be replicated going forward. Phase IV: Post-Implementation There are two priority areas for improved beneficiary assistance in the post-implementation phase: ex-post monitoring and evaluation (M&E) for durability; and enhanced knowledge exchange. Beneficiaries note the lack of technical knowledge of some monitoring experts and their overly rigid interpretation of rules. Fundamentally, this has to do with the understanding of what qualifies as “substantial modification” in article 57 of CE 1083/2006 – flagging an issue as such generates a special procedure that can result in financial corrections. In practice, there are multiple situations where beneficiaries avoid any adjustments, even though good reasons may exist to perform them. A more clear definition of the ROP’s interpretation of Article 57 in EC 1083/2006 would be very helpful to beneficiaries, along with a procedure for allowing changes to projects, when justified by special circumstances. Clarifications are also needed when it comes to acceptable project-level indicators and procedures used in assessing them after the completion of an ROP investment. First, on the choice of indicators, applicants should be encouraged to select from a pre-defined list aligned with program- and axis-level targets, which would be easy to implement particularly through an e-application form. At a minimum, chosen indicators should fit the “SMART” set of criteria: specific, measurable, attainable, relevant, and time-oriented. Second, interviewed beneficiaries have also requested a detailed methodology applicable to how project-level indicators are monitored and evaluated throughout the implementation and post- implementation phases. Third, for projects failing to reach their targets, if a potential penalty is deemed necessary, it should be, at most, equivalent to the difference between the set target (100%) and the level achieved in practice. Still, imposing such penalties will depend only on the MA’s decision. Last but not least, the ROP’s positive features can and should be replicated during the next programming period, which depends fundamentally on leveraging the resources that already exist in the system. This is certainly true of the program’s “graduates” – i.e., beneficiaries who have successfully completed ROP-financed projects. Given available data from SMIS and other RDA-level databases, the MA and the IBs would have the possibility to select the top-performing projects and beneficiaries, seeking to draw lessons learned and best practices. There are multiple ways for leveraging program graduates and facilitating knowledge exchange. For instance, one simple possibility is to add to the final progress report and to ex-post monitoring reports submitted by beneficiaries to the IBs a few questions asking for suggestions for future beneficiaries. Another option is to host in-person workshops with successful ROP graduates acting as trainers and mentors for new beneficiaries, within the proposed Corps of Practitioners. In addition, graduates could disseminate their experiences through a range of information and communication technology (ICT) solutions for knowledge management and sharing. Top Recommendations The figure below summarizes the top challenges and corresponding recommendations. This is not an exhaustive list and, due to space constraints, leaves out some of the rich details of the report’s main sections. In particular, the last chapter of the current work presents a full implementation roadmap, including a comprehensive list of priorities, with a recommended timeline, resources required, and potential risks. 11 CHALLENGES RECOMMENDATIONS There are still significant  Internal: Improve the compensation system (higher salary capacity constraints on base, bonuses covering the entire cycle) and focused trainings beneficiaries’ ability to  External: Promote standard contract forms to boost providers’ manage internal and accountability and guarantee high-quality services. Enable external resources user-feedback mechanisms Project implementation  Clarify and complete the public procurement framework, in line continues to be plagued with upcoming EU-level regulations for 2014-2020 by issues related to  Move away from the “lowest price” as sole selection criterion public procurement and  Adopt the red and yellow FIDIC and build beneficiary capacity to poor quality technical manage these types of contracts documentation Bureaucracy continues  Develop an e-ROP for the entire project cycle, allowing to be excessive and an beneficiaries to submit all documentation electronically electronic interface with  Implement automated verifications to speed up processing beneficiaries is missing  Develop a simple, easy-to-use online tracking platform for beneficiaries to check the status of all their requests Knowledge sharing  Establish an ROP Corps of Practitioners bringing together the remains relatively weak most experienced beneficiaries and enabling the dissemination and best practices are of best practices through institutionalized channels not effectively  Leverage the knowhow of beneficiaries who successfully disseminated complete ROP projects (i.e., close the feedback loop) Marginalized  Define targeted selection criteria to encourage ROP-financed communities lack investments in marginalized communities targeted ROP  Develop the capacity of marginalized groups to design and interventions & support implement ROP projects (e.g., through the CLLD instrument) 12 Introduction 1. The Government of Romania (GOR) aims to enhance the preparation and implementation of projects financed from structural instruments by the European Union (EU). In January 2012, the GOR and the World Bank signed a Memorandum of Understanding on Partnership and Support in the Implementation of the EU Structural and Cohesion Funds in Romania and Modernization of Public Administration. As a result, five projects were agreed with the then Ministry of Regional Development and Tourism (MRDT) – now the Ministry of Regional Development and Public Administration (MRDPA): (1) Romania's urban development and spatial planning strategy; (2) growth poles policy review; (3) implementation of the Regional Operational Program (ROP) assessment, including a functional review of the communication and collaboration between the ROP’s Managing Authority (MA) and Intermediate Bodies (IBs) and the facilitation of proactive and direct support for program beneficiaries; (4) assistance for the identification of ROP project selection models; and (5) the elaboration of integration strategies for poor areas and disadvantaged communities in Romania. The findings across all projects are meant to be complementary in nature and contribute directly to the design and implementation of the ROP in Romania, particularly for the 2014-2020 programming period. 2. In particular, the functional review of the Regional Operational Programme’s implementation (project #3 above) is fundamental to improving EU structural funds’ absorption rate, efficiency, and impact in Romania.60 While the analysis focuses on the ROP, many findings are applicable across multiple operational programs, with the hope of contributing to the country’s improved overall results in accessing EU funding for the 2014-2020 period. This is particularly important in light of the fact that Romania remains as of November 2013, by most measures, the weakest performer among all Member States in terms of EU funds absorption for the 2007-2013 budgeting cycle. 3. This report focuses on the second component under the functional review of the ROP: the facilitation of proactive and direct support for ROP beneficiaries. It reviews currently available assistance mechanisms and makes recommendations for improving them. This engagement is absolutely critical for reaching the objective of improving Romania’s performance in utilizing available EU funds, for the following key reason: beneficiaries are the main driver of any EU operational program, including the ROP. In other words, a well-functioning ROP depends on knowledgeable and effective beneficiaries with adequate capacity to design projects, submit them for evaluation, and complete their implementation, while following all relevant regulations and reaching set targets. In fact, in the EC’s October 2012 Position Paper on the development of the Partnership Agreement and programs in Romania for 2014- 2020, the need to improve support to beneficiaries of structural funds is a key highlight. 61 The EC calls for further efforts to improve the capacity of beneficiaries in delivering projects, while reducing the administrative costs and burden of controls, in line with the proportionality principle. In this context, considering the task at hand and the ROP’s complexities throughout a project’s cycle, support mechanisms for applicants and beneficiaries are a vital component of a successful ROP and initiatives aimed at improving them are timely and welcomed. 60 The first component of the current analysis is the “Assessment of the Communication and Collaboration between the MA and the IBs,” which is covered by the other report (Component 1) of the current project. 61 “Position of the Commission Services on the development of Partnership Agreement and Programmes in Romania for the period 2014-2020,” Ref. Ares(2012)1240252 - 19/10/2012 13 4. The main method of this functional analysis focuses on assistance mechanisms at every stage of a potential ROP project, from the perspective of potential and actual program beneficiaries. The main project phases are the following: pre-application, application, implementation, and post- implementation. As noted in the methodology section below, each of these stages includes specific challenges: for example, if project design and awareness-raising activities are critical during pre- application, the quality of technical documentation and financing capabilities, among other things, are particularly important during a project’s implementation. Equally important, each stage of an ROP project has a specific target group for support mechanisms, from potential applicants, to actual and eligible applicants, beneficiaries, and so-called “graduates” (i.e., beneficiaries who have successfully implemented one or more ROP projects). A few major themes span the entire project cycle, including the availability and quality of human capital and the need for greater simplification of processes and procedures, including through the introduction of e-services. 5. Observations included in the current report are directly related, and complementary, to findings included in the “ROP 2.0: The MA-IB Collaboration and Communication” report. Fundamentally, it is impossible to approach any part of the MA-IB-beneficiary triad without looking at the system as a whole. At a practical level, any obstacle faced by beneficiaries generates proportional pressure on the MA-IB relationship; by the same token, a functional MA-IB collaboration is needed for providing adequate assistance to program applicants and beneficiaries. 6. This report draws on results from a series of working visits to the eight development regions in Romania, as well as from discussions with key stakeholders within the Ministry of Regional Development and Public Administration (MRDPA). Additional qualitative and quantitative insights have resulted from a May 2013 survey of 470 ROP beneficiaries from across the country. The report also includes, where relevant, applicable insights from the World Bank’s 2011 Functional Review of the Ministry of Regional Development and Tourism (MRDT), which recently became the MRDPA – compared to the MRDT, the new structure has added public administration functions, while tourism is now under a separate ministry. 7. Finally, it is important to recognize that the current report’s scope remains focused on applicant and beneficiary support mechanisms, although there are several broader issues that affect the ROP system. These are addressed by other ongoing or future World Bank studies that are separate from, but build on, the current assessment. As noted in the executive summary, these important aspects (albeit tangential to the current report) include: selection and prioritization of projects based on local needs and existing territorial and sectoral strategies; ROP selection models balancing absorption and strategic impact; more precise estimates on operation and maintenance (O&M) costs related to completed ROP investments; and harmonizing procedures between projects financed by the EU and from the state budget by the MRDPA. At the same time, the ROP’s intermediary evaluation, carried out by KPMG, will be finalized in the near future. The preliminary version of that document already includes a range of very useful data, particularly with respect to the estimated impact of ROP-financed projects, the efficiency of expenditures relative to results achieved, and the functioning of the data collection system. As such, there is a welcomed complementarity between this current work and the ROP’s intermediary evaluation. 14 I. Context 8. The Regional Operational Programme (ROP) is a top performer in terms of absorption rates, in the Romanian context, despite its complexity and diverse portfolio of projects. The EU has dedicated EUR 3.7 billion to the Regional Operational Program (ROP), which amounts to 19% of the total structural funds available through the National Strategic Reference Framework (NSRF) between 2007 and 2013, making the ROP the third largest program after transportation (23.7%) and environment (23.5%). Because of its broad focus on regional development issues, the ROP addresses a wide-ranging set of needs at the level of each region, from urban development to transport infrastructure, social infrastructure, business development, and tourism. Accordingly, some projects are relatively small and easy to implement (e.g., purchasing equipment for a polystyrene producer), while others are large, complex infrastructure projects with cumbersome technical documentation and lengthy procedures. As such, it is remarkable that, as of November 30, 2013, the ROP remains the country’s top performing operational program, particularly when it comes to payments from the EC to Romania, which stand at 44.2% of the total available ROP allocation. Based on available quantitative data and qualitative interview data, it is likely that the ROP will face few issues in reaching close to 100% absorption by the end of 2015, so the risk of ROP funding recommitment remains relatively low. Figure 16. Relative performance of EU structural programs in Romania (as of November 30, 2013) 120% 118% 110% 103% 100% 85% 82% 80% 78% 80% 60% 48% 45% 46% 44% 40% 40% 33% 30% 27% 22% 22% 21% 19% 18% 18% 20% 0% ROP Environment Transport Competitiveness Administrative Human resources Technical capacity assistance % contracted of total ROP funds available % paid to beneficiaries % paid by EC to Romania Source: Ministry of European Funds (www.fonduri-ue.ro) 9. This report argues, much like the evaluation of the MA-IB collaboration, that the ROP’s performance to date is no coincidence: it is Romania’s most decentralized operational pr ogram and has benefitted from a core group of experienced people in the MA, IBs, and major beneficiaries. While substantial responsibilities remain with the central MA, the ROP has leveraged the presence of the eight Regional Development Agencies (RDAs) – one for each NUTS II region in the country – by appointing them as Intermediate Bodies. In simple terms, it is the program with the shortest distance between beneficiaries and Intermediate Bodies. Along with the recognized professionalism and competence of RDA staff, regional IBs are close to on-the-ground challenges and can respond quicker and better to emerging beneficiary needs. Even if they do not always have all the tools necessary to address the full 15 range of issues and rely on MA decisions, IBs have developed close working relations with many beneficiaries in their respective regions, earning their trust and respect. At the same time, the ROP has built on the experience of specialists at all levels, from key beneficiaries to IB and MA staff, many of whom have been working together for a long time. There are many examples of long-standing, personal relations, particularly between project teams at the level of public authorities (city halls and county councils), RDA staff, and MA senior management. The benefits of low turnover at these critical levels are evident in how people work together and respond to emerging challenges in implementation. 10. Interview and survey feedback is overwhelmingly positive in terms of beneficiaries’ satisfaction with the ROP, particularly when compared to other operational programs (OPs). Beneficiaries welcome their collaboration with the RDAs, as their main point of contact – as expected given the purpose of IBs – but also mention generally productive, positive relations with the MA. There is a sense of shared partnership and understanding that all stakeholders involved genuinely want the program to succeed. Five factors help explain why the ROP is in many ways the best performing Romanian OP and how it delivers superior assistance to its beneficiaries, as is to be expected given that it is the most decentralized structural program in the country. From this perspective, the ROP provides many best practice examples for other Romanian OPs. Specifically, MA and IB staff:  Are closer to program applicants and beneficiaries;  Demonstrate enhanced understanding of what needs to be done on the ground – i.e., possible solutions to address regional and local-level needs and challenges at the beneficiary level;  Engage experienced people who know how to design and implement the optimal solutions, working in close partnership with beneficiary teams, many of whom they know and trust;  Have greater accountability for outcomes and results vis-à-vis beneficiaries and the general public because, in the words of one interviewed beneficiary, “people can hold them accountable right here and right now, at the local level”;  And, finally, they benefit from quicker, more effective feedback loops from beneficiaries who are affected by the decisions they take and actively voice their issues, helping to correct potential issues as they appear. 11. This is not to say that the assistance mechanisms currently available to ROP beneficiaries are sufficient and flawless. In fact, even though the ROP is a top-performing operational program compared to average absorption rates for Romania, its results remain lower than benchmark programs in other Member States, as of November 2013. At the time of the functional review, in 2010-2011, the ROP still fared better than Bulgaria’s average absorption rates, yet by February 2013 Romania’s neighbor had reached 31.5% in payments received from the EU, still higher than the ROP at the end of May 2013.62 As another case in point, in Slovakia, by February 2013, the Regional Operational Programme had certified 60.59% of the total allocation available.63 Now it is critical to understand and address any remaining gaps in the Romanian ROP’s implementation in order to increase absorption rates through 2013 and especially to prepare the ground for better performance during the 2014-2020 budgeting period. As noted above, this report argues that delivering more direct and proactive assistance to beneficiaries is fundamental to achieving these objectives. 62 See http://insideurope.eu/taxonomy/term/186 63 Ibid. 16 Picture 1. The ROP includes a very diverse portfolio of beneficiaries and projects 17 12. To be sure, applicants and beneficiaries have drawn attention in the past to a few key obstacles that hinder the implementation of EU-financed projects. For example, a February 2010 survey of 637 stakeholders (beneficiaries, MA representatives, consulting companies, and banks), implemented by the fonduri-structurale.ro portal, identified excessive bureaucracy and long evaluation and reimbursement times as the most important factors responsible for slowing down the absorption of EU funds. Importantly, this particular survey targeted the entire population of EU-funds beneficiaries, not just those served by the ROP specifically. As noted in the report on the “ROP 2.0: The MA-IB Collaboration and Communication” (Component I of this project), some of the elements cited in the table below fall beyond the immediate control of MA and IB staff (e.g., “the lack of co -financing and banks’ reticence in lending”) and will require broader solutions at the national level; other factors, like the “excessive bureaucracy” and the “rigid, unclear applicant guides,” can be addressed through faster, easier, and more direct interventions within the ROP system. Figure 17. Results of a February 2010 survey of EU-funds beneficiaries What affects the absorption of EU funds? (February, 2010) 9. Excessive Bureaucracy 33,59% 10. Long evaluation and reimbursement times 16,22% 11. Lack of co-financing and banks’ reticence in lending (given that finished constructions and 16,22% purchased equipment cannot be mortgaged) 12. Rigid and unclear applicant guides, with last-minute corrections 9,65% 13. Poorly trained staff in the MA and in the Intermediate Bodies, and poor communication with 7,72% applicants/beneficiaries 14. Poor promotion of funding sources 6,95% 15. Lack of quality of consultancy services 6,56% 16. Lack of transparency 3,09% Source: fonduri-structurale.ro 13. For the current report, a survey was sent out to beneficiaries in all eight regions to inquire about the main obstacles faced in accessing ROP funding and implementing projects. The total sample included 470 responses. Where necessary, the sample was adjusted to exclude those answers that had selected the “I do not know / I cannot evaluate” option – typically, a very small segment of the surveyed population. The proportion of people who characterized various obstacles as “extremely important” and “very important” varies from a staggering 81.5% for “excessive bureaucracy and documentation” to only 31.7% for “poorly trained RDA staff.” The figure below summarizes this information and implicitly ranks obstacles according to their perceived importance. Compared to 2010 data, it is notable that some issues remain prevalent at the top of the list: excessive bureaucracy (81.5%), frequent legislative changes (80%), and delays in project evaluation and selection (73%). Also, over half of respondents identified additional obstacles as “extremely important” and “very important”, in the following order: delays in reimbursements (69%), lack of capital for co-financing and cash-flows (68%), inadequate public procurement framework (62%), insufficient prefinancing (56%), and poor-quality technical documentation (52%). Other significant obstacles included “lack of high-quality consulting services” (47%) and “lack of transparency” (44%), and low quality of applicant guides (40%). The least pressing obstacles mentioned were related to poorly trained MA and IB staff (37.5% and 31.7%, respectively). 14. Subsequent sections of this report delve deeper into these data and their corresponding implications, but it is worth noting that room for improvement exists with respect to the entire set of obstacles listed above. The good news is that a few factors that depend directly on the MA and the IB, including their staff and the quality of applicant guides, are at the bottom of the list of beneficiary 18 concerns. At the same time, to have more than two-thirds of respondents point out bureaucracy, legislative changes, and procedural delays (in evaluation, selection, and reimbursements) as major obstacles is reason for concern, particularly because these are recurring themes from the 2010 survey, suggesting that the situation has not improved markedly in the meantime. In this light, observations included in this report appear timely and, once again, confirm that improving the ROP’s performance for 2014-2020 is closely tied to adopting better support mechanisms for applicants and beneficiaries. Figure 18. Percentage of respondents who identify an obstacle as extremely/very important 64 100% 90% 81.5% Excessive bureacracy & documentation 79.8% 80% Frequent legislative changes 73.4% 69.4% Delays in project evaluation & selection 70% 67.9% Delays in reimbursements 62.1% 60% Lack of capital for co-financing & cash flow 56.0% 52.1% Inadequate public procurement 50% 47.1% Insufficient prefinancing 44.2% Low quality of technical documentation 40.3% 40% 37.5% Lack of high-quality consulting services 31.7% Lack of transparency 30% Low quality of applicant guides Poorly trained AM staff 20% Poorly trained RDA staff 10% 0% Objective and Scope 15. Based on the agreed terms of the advisory services, this report’s main objective is to improve the performance of the ROP system by enhancing currently used methods and instruments for providing direct support and assistance to ROP applicants and beneficiaries. The term “assistance” is interpreted to include all actions designed to help beneficiaries access ROP funding more quickly and easily. As such, the scope includes, but is not limited to, traditional assistance tools (e.g., helpdesks, guides, financing instruments, etc.). It also covers some procedural simplifications as well, although that remains the main focus of the report on the MA-IB relationship. Indeed, it is important to keep in mind that the three elements of the ROP “triangle” (MA-IB-beneficiaries) are deeply interrelated and the overall efficiency of the ROP depends on them, individually and jointly. 16. It must be clear from the start that this report recognizes that there are capacity issues at the level of beneficiaries around the country, but the main focus point remains centered on available assistance mechanisms. It is beyond doubt that these two elements – beneficiary capacity and support systems – go hand in hand, and it is equally certain that there are issues in the field caused by some beneficiaries’ lack of required competencies. But the key question remains: given the current situation of a typical ROP beneficiary, what are those optimal mechanisms for supporting her throughout the process of preparing and implementing a successful project? 64 These data exclude the small proportion of “I do not know / I cannot evaluate” answers. 19 17. Thus, the scope of the current work is therefore centered on two elements: (a) assessing current assistance available to ROP applicants and beneficiaries; and (b) making recommendations for improvement. The previous report associated the ROP system to a network of “pipes,” with water coming in (as project proposals from beneficiaries) and going out (as implemented investments by beneficiaries). For this report, the predominant focus is still on ensuring a steady, efficient flow between all stakeholders involved, which involves making sure that all requests received from program applicants and beneficiaries are efficiently and proactively addressed by the MA and the IBs. 18. In other words, the current work remains primarily about properly handling quantity/volume, but adds an important nuance around the quality of projects in the pipeline. Specifically, one sub- section will assess solutions for helping beneficiaries conceive higher-quality proposals from the beginning, leading up to both smoother implementation with fewer delays and higher impact upon project completion. Simply put, this report is about helping beneficiaries design good, impactful projects, enter them into the ROP’s “pipes,” and push them through the system as efficiently as possible. What is left beyond the scope of this assignment is how the system filters through different types of projects, which is the focus of two complementary, ongoing World Bank projects for the MRDPA: namely, “Competitive Cities,” which provides insights into the kinds of investments that are most conducive to productivity growth and development at the local, regional, and national levels; and the “Technical Assistance for the Identification of Project Selection Models,” which provides recommendations for how to improve the evaluation of strategic, impactful projects. 19. To further define this report’s scope, it is important to note from the beginning that the optimal type of assistance will vary with the type of beneficiary targeted. While the exact details of the ROP 2014-2020 are very much in flux and fervently debated, the current report seeks to provide guidance for all possible project and beneficiary types. As a subsequent chapter on profiles of ROP beneficiaries explains in detail, and taking two extreme examples, it is one thing to assist a major local/county/regional body implementing large projects worth millions of Euros through a dedicated funding stream (allocated ex-ante), and it is another thing to provide guidance to an applicant with a single application for a small ROP investment, attributed competitively. Both are vital, but the key difference is between customized support and standardized assistance. In any case, the scope of the current report is not limited to a particular type of project, beneficiary, or selection model. Rather, it is designed to stand on its own in terms of the recommendations put forth and their applicability. 20. Equally important, this report takes place in the context of the Romanian Government’s ongoing “regionalization” effort, which involves the introduction of an administrative layer between the county and the national levels. Nonetheless, the findings and recommendations of the current work remain valid regardless of the final institutional set-up of Romania’s administrative system and specifically the MA-IB relationship in the ROP 2014-2020. When it comes to assisting beneficiaries, most of the tools perform best when they are delivered as close to the ground as possible – i.e., at the local/regional level, either by an MA or an IB. In some cases, where a national-level organization can bring value add as an aggregator of information and knowledge from all regions, some beneficiary assistance can be kept at the level of a national MA within the MRDPA. The regionalization subsection in chapter IV goes into further detail about beneficiary perspectives on regionalization. 21. On a final note, the main focus areas of this assessment derive from the priorities and needs expressed by beneficiaries through field interviews and survey responses. As such, themes like public procurement and technical documentation receive considerably more attention than other issues. By 20 the same token, for those areas where no significant observations were formulated by beneficiaries, the report finds it reasonable to assume that the ROP is performing adequately and no major improvements are needed at this point in time. Methodology 22. As in the “ROP 2.0: The MA-IB Collaboration and Communication,” the first guiding principle of the current analysis remains: “first do no harm.” Interview and research data reflect that, in part owing to the improved assistance delivered to applicants and beneficiaries, the ROP is a high performer compared to other similar programs and in the context of Romania’s average absorption of EU funds. The components that are functional and efficient in supporting applicants and beneficiaries should be strengthened to leverage accumulated institutional knowledge and to minimize disruptions to the ROP system. More broadly, a high level of predictability is valuable, in of itself, as a way to provide a stable environment with clear rules for potential beneficiaries who are applying for ROP financing and implementing projects. 23. It follows that any efforts targeted at improving the ROP and the assistance delivered to applicants and beneficiaries should build on the program’s current strengths and address remaining gaps. Within its defined scope, the current assignment involves several steps in assessing the ROP’s implementation: first, identifying assistance mechanisms that work well and should be continued; second, determining areas for improvement and particularly the root causes of potential issues; and third, making recommendations to address current obstacles, while maintaining and strengthening the components of the system that have performed satisfactorily. 24. The methodology for the current functional analysis is straightforward, placing the potential and actual beneficiaries at center stage. As argued repeatedly, if the ROP system is to be fully optimized, its “clients” (i.e., the public, non-governmental, and private applicants and beneficiaries) must be at its core. As such, this report first lays out a theoretical framework for understanding applicants’ and beneficiaries’ organizational performance, as well as the external environment in which they operate. From this starting point, the bulk of the analysis focuses on evaluating assistance tools and procedures at each stage of a project’s lifecycle. Indeed, based on numerous conversations with program beneficiaries, it is clear that the assistance they require is quite specific to the stage of their engagements: “wide reach, wide focus” during pre-application (i.e., large audience, relatively basic information required); “medium reach, medium focus” for the application period (i.e., fewer actual applicants than potential applicants, with needs that are relatively basic, but increasingly project- specific); “narrow reach, narrow focus” during implementation (i.e., small audience, customized guidance for each beneficiary); and finally “wide reach, narrow focus” during the post-implementation phase (i.e., sharing project-specific lessons learned with a large audience). 25. As the figure below suggests, each phase in the ROP’s implementation process comes with its own challenges and targets different audiences (from a large pool of potential applicants initially to a limited number of eligible, approved beneficiaries). During the pre-application period, the focus is on distributing the message to wide audiences and possibly working with potential applicants to help them refine project ideas and increase the quality of their proposals. Later on, during the application phase, the analysis concentrates on the level of transparency of the process and existing accountability mechanisms. In simple terms, the main question is around the applicants’ ability to understand every step of the application process, estimate its duration, and know who is responsible for any potential 21 issues or delays. The implementation stage comes with its own beneficiary support mechanisms to address particular challenges – some of them specific to each project, others of a more systemic nature (e.g., public procurement legislation, technical documentation, etc.). Finally, once a project is completed successfully, support mechanisms refocus on identifying and capturing the lessons learned and best practices that have emerged to ensure a smoother process for future applicants. Figure 19. Framework for assessing available assistance for ROP applicants and beneficiaries 26. Within this overall framework, observations and conclusions draw from four primary sources, as is typical for functional reviews: in-depth desk review, field research, interviews, and surveys. First, where available, the report relies on existing analyses, including the World Bank 2010-2011 Functional Review of the MRDT (now the MRDPA). Second and most importantly, the project team organized visits of up to two weeks in each of Romania’s eight development regions and covering the large majority of counties in Romania. The team interviewed 50+ staff of Regional Development Agencies and 150+ program beneficiaries, including local public authorities (city and county-level councils), non-profits, and private-sector beneficiaries.65 In selecting beneficiaries, the team chose some interviewees directly based on independent desk research, while others were chosen based on recommendations received from the RDAs. The sample included a diverse set of beneficiaries and priority areas, targeting both successful and less fruitful projects. At multiple points during the process of developing the current report, the team reached out to counterparts in the MA, the RDAs, and beneficiary organizations, holding workshops specifically designed to test and refine intermediary and final findings. 27. Additionally, as mentioned earlier, for the specific purpose of this project, the team has designed a survey of ROP beneficiaries to gather quantitative and qualitative insights into the 65 See Annex C 22 program’s current strengths, weaknesses, and potential solutions for 2014-2020.66 This instrument was implemented with the help of the Managing Authority (MA), the Regional Development Agencies (RDAs), and the fonduri-structurale.ro portal, ensuring a wide reach of 470 respondents from across the country. This is a reasonably representative sample, albeit one that is not randomly selected and may be slightly biased toward more active beneficiaries who seek to provide feedback into the process and make their voices heard. This is likely why, for instance, public-sector respondents are slightly over- represented, at 42% of the sample (compared to 38% in the December 2012 total ROP beneficiary population), while private companies are at 52% (compared to 59%), with the rest corresponding to NGOs and other types of organizations. In terms of regional split, the sample of respondents is relatively well distributed, with the exception of the South Muntenia region where the response rate (equivalent to 6% of the sample) was lower than expected based on the relative proportion of beneficiaries from this part of the country. This in of itself is not a major issue, as most responses do not appear to vary depending on region, which confirms the hypothesis that beneficiaries face similar problems, regardless of where they come from within Romania. All in all, the survey is not meant as a tool with scientific precision, but it serves its intended purpose very well – that is, to provide a snapshot of beneficiaries across the country, including the obstacles they face, their satisfaction with current support mechanisms, and their initial reactions to several proposed improvements. 28. Beyond the MA-IB-beneficiaries triangle, the team met with a range of other actors relevant for the ROP system’s performance and beneficiary support mechanisms. These include: representatives of the National Audit Authority (NAA); senior managers of the Authority for the Coordination of Structural Instruments (ACSI); prominent consulting firms; and local and national financial institutions. Last but not least, the project team has engaged – and will continue to engage – Ministry counterparts to assess the accuracy of findings and viability of recommendations in order to fine-tune this analysis and facilitate its full-scale implementation. Audiences 29. The main audiences of the current report are senior managers from the Ministry of Regional Development and Public Administration (MRDPA), as well as officials from the European Commission (EC) responsible for the ROP’s oversight. For this reason, the analysis includes, whenever possible, specific observations and recommendations, including a roadmap with proposed solutions for improving beneficiary assistance (see Chapter IX). Other key stakeholders who may benefit from this assessment include IB managers and staff, as the main providers of the ROP’s interface with beneficiaries (under the program’s current setup), as well as MA and IB teams from other operational programs. A broad distribution and discussion of the findings is desirable, particularly among ROP beneficiaries, who volunteered their time and contributions for the extensive research behind this report’s findings. They may find use in proactively incorporating into their projects some of the ideas presented here, or adapting them to fit their particular needs and priorities. 30. Above all, this analysis will produce positive results if it is accepted, endorsed, and “owned” by MRDPA leadership and staff and by decision-makers across the Romanian Government. As noted repeatedly throughout, some measures will depend on the MA/IB teams directly, while others will require the cooperation and perseverant efforts of wider coalitions of stakeholders. The hope is that these key actors will continue to discuss the report’s assessment and internalize recommended actions, 66 See Annex A 23 pushing for their full-fledged implementation. To the extent that lessons from the Romanian ROP apply to other EU Member States, and in many cases the challenges faced by beneficiaries are indeed similar, the application of this report’s observations and recommendations to a broader context would be welcomed and encouraged. Structure of the Report 31. In what follows, Chapter II summarizes the theoretical framework of the current study. This broader picture includes two primary components: an organization’s performance (endogenous factor) – specifically, its mission, customers, and operational capacity – and its environment (exogenous factor). For the ROP specifically, nurturing an enabling environment for applicant and beneficiary organizations depends on three guiding principles: access, accountability, and predictability. This section is meant as a foundation for the subsequent functional review of ROP support mechanisms at each stage of the project cycle. 32. Chapter III provides a more in-depth profile of ROP applicants as the key target of the program’s assistance mechanisms. This section starts with a brief review of EC-level provisions on applicants/beneficiaries of structural funds. It further presents some key statistics on the pool of ROP beneficiaries, as reflected by data available from the MA as of December 2012. This information is helpful in customizing the MA’s and IBs’ assistance efforts, which need to consider the type of beneficiaries (public vs. private / non-profit), their level of experience with structural funding, the scale of ROP projects in their portfolio, etc. 33. Subsequent chapters (IV-VIII) are dedicated to assessing current assistance mechanisms and formulating recommendations for their improvement, for each of the following phases: pre- application, application, implementation, and post-implementation. Multiple types of support are reviewed under each category:  Horizontal Cross-Themes (affecting entire project cycle):  Regionalization: Beneficiary perspectives on regionalization and decentralization  Human resources: Current challenges and recommendations for strengthening the system’s most critical component  Access to finance: Assistance in securing pre-financing and co-financing  Electronization: Benefits of an e-ROP and recommendations for development  Targeted support for marginalized communities: Necessity of designing special tools for engaging marginalized groups as project applicants/beneficiaries and end beneficiaries (including a brief assessment of the future CLLD approach)  Pre-application:  Programming: Beneficiary participation in defining the ROP’s priorities and procedures  Awareness-raising: Information regarding the availability of financing (and related procedures) for a particular axis/major intervention area  Pipeline development: Guidance on developing a high-quality, financially viable proposal (i.e., zero-error, effective impact, etc.)  Application, evaluation, and contracting: 24  Applicant guides: Provision of clear, stable application and evaluation guidelines, and progression of project evaluation and contracting based on a predictable timeline;  Eligible vs. ineligible expenses: Guidance on better defining eligible expenditures  Helpdesk: On-demand assistance in response to beneficiary inquiries  Training: Development of human resources in beneficiary organizations (IB and MA-led)  Ongoing communication: Communication and interaction on a constant, open basis with MA and IB staff, including a strong focus on transparency and close communication;  Implementation:  Public procurement: Assessment of current challenges, including financial corrections, and their primary causes and remedies  Technical documentation: Standard contract forms and enhanced accountability of key service providers  Execution phase: Summary of best practices in managing relations with contractors and service providers (FIDIC contract form, stronger provisions, professional insurance, technical and financial audit)  Monitoring: Direct assistance through monitoring visits and related procedures  Reimbursements: Evaluation of reimbursement delays, related challenges, and solutions  Post-implementation:  Ex-post M&E: Monitoring and evaluation of post-completion outcomes and indicators based on clear, transparent, and fair standards;  Knowledge sharing: Sharing of knowledge between the program “graduates” (i.e., beneficiaries who have completed ROP projects successfully) and new applicants and beneficiaries in the pipeline. 34. The last chapter presents a full action-oriented roadmap with specific recommendations. The roadmap includes list of recommendations, each with a proposed timeline, estimates on needed resources, as well as a summary of key risks. The hope is that this will serve as a tool for tracking the progress made against the recommendations put forth by this report. This is meant to be complementary to the roadmap included in the Component 1 report on the MA-IB collaboration and communication. 25 II. Theoretical Framework 35. The ROP’s future depends, above all, on a paradigm shift that puts beneficiaries in the center of the entire system, with the MA and IBs providing proactive assistance to support successful project completion. Most interviewees recognize that the partnership between beneficiaries, MA, and IB staff is essential and viable, given that they all share the common goal of effectively absorbing structural funds for the benefit of Romania’s development – locally, regionally, and nationally. Indeed, the MA and the IBs should ensure that all rules and procedures are followed, but also make every effort possible to reduce the burden on ROP beneficiaries. There is an obvious tension between controls and verification tasks, on the one hand, and support and capacity building functions, on the other hand. Under the status quo, the former can sometimes completely overshadow the latter, as noted in numerous interviews with beneficiaries, who felt like they faced too many controls and sanctions and not enough proactive support that would enable them to avoid making mistakes in the first place. The general perception was that MA and IB staff with monitoring and support functions are well-intentioned and genuinely willing to help, but have little say vis-à-vis the numerous auditing and verification bodies that, as one interviewee put it, “appear to hunt down every single error, from placing the stamp too high to putting the comma in the wrong place.”67 36. The perceived disproportionate focus of some institutions on excessive controls and sanctions has contributed to beneficiaries’ distrust of the entire ROP system, discouragement, and reduced appetite for accessing future EU funding. To be sure, control functions are essential and should not be downplayed under any circumstances. Procedures for verifying projects should be transparent and fair, and mistakes should be sanctioned proportional to their consequences. That said, auditors should not treat all beneficiaries as potential lawbreakers, but rather help prevent their mistakes, together with the available support mechanisms (e.g., helpdesk, monitoring officers, intermediary audit reports, etc.). If the current situation continues, some beneficiaries have warned that the risk for low absorption in 2014-2020 will increase, as fewer and fewer public institutions will be willing to incur the risks involved in implementing ROP-financed projects. By contrast, the paradigm shift toward “ex-ante prevention before ex-post punishment” and “beneficiaries first” would significantly improve the functioning of the entire ROP system. Organizational Performance 37. The starting point for improving current support for ROP applicants and beneficiaries is to understand the broader task that each of them faces when designing and implementing a project. In other words, taking a step back, the success of a particular ROP project is fundamentally tied to the beneficiary’s organizational performance, as reflected by a set of established metrics. The literature on performance management has evolved greatly over the past few decades, refining the initial focus on pure financial indicators. In particular, two prominent models are helpful to understand the task faced by private and public/non-profit organizations: the first was developed by Harvard Business School professor Robert Kaplan, who designed the balanced scorecard, primarily for measuring the performance of private companies; the second was promoted by Harvard Kennedy School professor Mark Moore, who adapted Kaplan’s theory to the public and non-profit sector through the public value strategic triangle. 67 Interview with ROP public-sector beneficiary, North-West Region 27 38. Kaplan’s balanced scorecard model argues that high-performing private firms should attend to the following four priority areas, in the following order: financial, customer, operational, and learning and growth. The first is, quite simply, about profitability delivered to shareholders. The second refers to serving customer needs and achieving customer satisfaction. The third is about ensuring the functioning of those key business processes required to accomplish the firm’s objectives. Finally, learning and growing refer to a company’s ability to change and improve continuously to respond to emerging market trends and pressures. 39. As for public and non-profit institutions, Moore’s strategic triangle evaluates performance along three key drivers: social mission, legitimacy and support, and operational capabilities. Moore argues that an organization’s performance is first measured against its defined social mission as the ultimate value that it seeks to produce for society (e.g., completing a public infrastructure project, improving social conditions, etc.). The triangle’s second point is called “legitimacy and support,” encompassing those key stakeholders who validate an organization’s activities – specifically, the end beneficiaries of a project; and the actors who finance a particular project. Without winning over these constituencies, an institution may have a great mission in theory but lack the practical support for implementing the projects it needs to reach its objectives. Finally, “operational capacity” is the triangle’s third point and includes: available internal resources (e.g., human, IT, etc.); and external support that an organization can enlist (e.g., consultants, technical engineers, etc.). As with Kaplan’s model, the three points of the triangle are deeply interrelated and mutually dependent, and correspond roughly to the same categories, as suggested by the color coding in the figure below: social mission is the equivalent of financial profitability; legitimacy and support are similar to the customer perspective; and, last but not least, operational capacity corresponds to internal business processes and learning and growth. Figure 20. Organizational performance (Kaplan's balanced scorecard vs. Moore's strategic triangle) 68 Balanced scorecard for private companies Strategic triangle for public & non-profit organizations Social Mission Legitimacy Vision & Strategy & Support Operational Capacity 40. As applicable to the current report, the two models described above can be reconciled by using a framework based on three key pillars: mission, customers, and capacity. These apply both to the organization as a whole and to the specific projects for which it may seek ROP financing. For each of 68 http://www.hks.harvard.edu/hauser/PDF_XLS/workingpapers/workingpaper_18.pdf and http://www.hbs.edu/faculty/Publication%20Files/10-074.pdf 28 the three pillars, there are specific characteristics that the MA/IB support system should look for and strengthen in ROP projects (i.e., impact, legitimacy, and operationalization/viability). 1. Mission: ROP-financed projects are, for any organization, potential means to achieve a set mission and corresponding goals. The reason for undertaking a particular investment with ROP funding is the desire to make progress toward reaching a particular goal, whether it is higher profitability for a firm or better public services for a local public administration. To give a few examples from the current field research: for a private printing company, acquiring new equipment is aimed at boosting profitability and shareholder revenue; for a non-profit health and retirement center, a planned expansion is meant to serve more patients; for a town hall, a renovated museum means higher quality of life for citizens in that community. In this view, ROP projects should be impactful – i.e., make a direct, clear contribution to an organization’s mission and create value for relevant stakeholders or for society as a whole. 2. Customer: ROP-financed investments should be aligned with customer needs and priorities. The term “customer,” while most applicable to private-sector settings, is increasingly common for non-profit and public entities, which also need to respond to the demands of particular constituencies – most notably, (a) citizens, as end beneficiaries; and (b) donors, as the providers of resources for accomplishing set goals. From this perspective, ROP-funded projects should be legitimate – i.e., reflect the demand and inputs from end beneficiaries of investments and remain consistent with the objectives and standards drawn by funders, including the European Commission and the Romanian Government. 3. Capacity: Every ROP applicant and beneficiary has a particular set of resources and competencies, which are critical throughout a project’s cycle. Indeed, an investment could fit an organization’s mission and promise to deliver substantial value to its customers, but it will never come to fruition if there is no capacity to design, implement, and sustain it. This is not to say that all required resources should be in-house; in fact, there are very few examples of beneficiaries who implement projects fully on their own, without outsourcing some tasks (e.g., technical documentation, project management, etc.). In any case, from the perspective of available capacity, both internal and external, ROP-financed projects should be operational – i.e., beneficiaries should be able to ensure efficient and effective completion. 41. This overall framework helps differentiate among multiple types of assistance that the MA and IBs can offer ROP applicants and beneficiaries along the three dimensions: mission, customer, and capacity. On the mission component, assistance takes the form of advisory support in the pre- application period to help applicants design impactful projects. There should also be adequate filters in the evaluation stage, making sure that projects with weak value-add propositions do not receive financing. Taking the customer perspective, assistance in ensuring compliance with donor requirements is already well built into the program through such instruments like the administrative and eligibility verification. More interestingly, the MA and the IBs could design instruments for helping beneficiaries connect better to their customers and seek their feedback in designing and prioritizing projects (e.g., more participatory processes involving citizens at all levels in helping a local authority decide on the list of ROP-financed proposals; access to databases of customer needs for private companies in a certain industry/region, etc.). Special care should be devoted to involving members of marginalized groups, who may face special obstacles in voicing their preferences, concerns, etc. Finally, but most importantly, actions to strengthen operational capacity are needed to support beneficiaries in navigating through the complex requirements of the ROP and corresponding Romanian legislation, particularly on issues like public procurement and contract forms with service providers. The table below provides a few examples for assistance mechanisms under each of the three pillars. 29 Figure 21. MA/IB assistance under the three pillars of organizational/project performance Objectives of MA/IB interventions Examples of potential MA/IB interventions 1 • Increase project impact on the organization’s • Work with applicants to conceive and/or Mission: performance improve projects in the pre-application phase How does the project contribute • Deliver greater value-add for key • Filter out low-impact projects during to the organization’s goals? stakeholders (e.g., shareholders, citizens, evaluation stage through proper selection end beneficiaries of services) criteria, independent evaluators, etc. 2 • Ensure compliance with “donor ” standards, • Uphold strict eligibility standards to ensure Customer: requirements, and preferences that financed projects respond to ROP Is the project aligned with ROP objectives requirements? And with end • Involve end beneficiaries in decisions related beneficiary needs & priorities? to ROP projects to ensure alignment with • Provide the setting for beneficiaries to engage their needs their customers in prioritizing, designing, and implementing ROP-financed projects 3 Capacity: • Help beneficiaries bolster internal resources • Develop a complete implementation guide needed throughout the project cycle (e.g., covering the most common mishaps Is the organization able to successfully complete the staff capacity and skills) • Propose standard contract forms for some project, from start to finish? • Assist beneficiaries in enlisting proper services (technical documentation, auditing, external providers when in-house expertise project management, etc.) is unavailable Enabling Environment 42. The performance of ROP applicants and beneficiaries, and the corresponding assistance at the MA and IB levels, should take into account the fact that organizations do not operate in a void. Indeed, they function in a complex environment that can greatly hinder or support their efforts to design and implement successful ROP projects. There are three guiding principles of fostering an “enabling environment”: access, accountability, and predictability. Any effort to improve the support available to ROP applicants and beneficiaries should aim to foster these three positive characteristics, either directly – if the measures required are under the direct control of the MA/IBs – or indirectly, by working with other decision-makers to push forward the necessary changes. Subsequent sections of this report provide specific examples of assistance that clearly fit within one or more of these three categories. For now, it is important to define them and understand their implications at a general level: A. Access: An optimal enabling environment provides ROP applicants and beneficiaries with easy and equitable access to all the knowledge required for successful project completion. This includes, for instance, information about financing opportunities available, how to properly fill out application forms, how to identify potential external resources when needed (e.g., co- financing partners, consulting service providers, etc.), how to draft a strong contract, and how to exchange information and best practices with other ROP beneficiaries. Many of these elements are lacking or remain insufficient under the status quo. Knowledge is power, which is even more applicable to the complex, ever-evolving system of structural funds. MA and IB assistance should strengthen beneficiaries’ access to information whenever possible. B. Accountability: Each actor involved in the ROP system should be held accountable for its actions. For instance, if a particular public procurement procedure passes through the filters of one or 30 more public institutions, they should be jointly responsible for any related corrections. As pointed out repeatedly in interviews, under the current system the entire burden rests with project beneficiaries, which is perceived as unfair and discouraging of future applications for funding. Accountability should also apply to the entire chain of stakeholders that play a role in pushing a project from the design phase through implementation, including: consultants, project evaluators, technical project designers and verifiers, engineers, auditors, etc. Any fault that does not belong to beneficiaries should be rightfully apportioned to the responsible parties. C. Predictability: A predictable environment requires transparent and stable procedures and regulations. The number of legislative changes in some areas (e.g., public procurement) is staggering. Instruments by MA-ROP have surpassed 100 since the beginning of the programming period. Beneficiaries have been surprised by adjustments in, among others, the level of available pre-financing, eligible vs. non-eligible expenditures, extended timelines for the completion of certain stages in the application process, ex-post financial penalties resulting from a legislative void in national-level public procurement legislation, etc. Ensuring a high degree of predictability is of great help to all ROP beneficiaries, allowing for better preparation, fewer errors, more reliable financial forecasts, and, simply put, a higher level of comfort in accessing ROP funding. 43. In sum, this section argues that effective ROP assistance for applicants and beneficiaries depends on two essential factors: enabling environment (exogenous) and organizational performance (endogenous). MA and IB support should concentrate on strengthening both, directly if an intervention is within immediate reach, or indirectly, through collaboration with other stakeholders. The overall framework represented below is therefore a useful test of whether a particular action is helpful for ROP beneficiaries, i.e., whether it is directed at one or more of the elements represented. Building on this theoretical framework, the rest of this report takes on a functional perspective on the analysis of ROP support mechanisms, going through each phase in a project’s cycle to assess the status quo and make recommendations for improvement. Figure 22. A theoretical framework for assessing ROP support mechanisms Successful ROP project Pre-application  application  implementation  post-implementation High-performing organization project performance (ROP applicant/beneficiary) Organizational / 1 2 3 Mission Customer Capacity A Guiding principles of the enabling environment Access B Accountability C Predictability 31 III. Overview: ROP Beneficiaries and Types of Support Mechanisms 44. EC Regulation No. 1083/2006, which lays out the governing rules of several structural instruments, including the ERDF, defines a beneficiary as “an operator, body or firm, whether public or private, responsible for initiating or initiating and implementing operations.” This document includes substantial sections on mechanisms to monitor and verify beneficiary expenditures, but also leaves open the possibility of dedicating funds for training and preparation activities to assist applicants seeking EU funding. More recently, at the EU level, there is increasing recognition of the key role played by beneficiaries in the successful absorption of structural funds, along with the necessity of improved mechanisms for helping beneficiaries navigate through a complex system. This is the demand side of the equation – applicants and beneficiaries who need and/or request support for initiating, developing, and completing their ROP projects successfully. 45. On the supply side, the main actors who can answer this call are the MA and the IBs. There are also other state bodies related to the ROP system that provide support functions for applicants and beneficiaries, most notably the Authority for Coordinating Structural Instruments (ACSI). The universe of relevant institutions that influence a beneficiary’s performance is much wider, however, as shown in the “ROP 2.0: The MA-IB Collaboration and Communication” (Component I of this project). Prominent examples include, in no particular order: the National Audit Authority (NAA), the National Authority for Regulating and Monitoring Public Procurement (NARMPP), the National Council for Resolving Disputes (NCRD), the Unit for the Coordination and Verification of Public Procurement (UCVPP), the State Inspectorate for Constructions (SIC), etc. Additionally, there are a many private-sector consulting firms, financial institutions, and non-profit organizations that provide assistance to the teams involved in ROP projects. It is beyond the scope of this report to cover all support mechanisms potentially available to ROP applicants and beneficiaries, but the key insight to note is that the MA-IB-beneficiaries triad does not function in a vacuum and its success depends on the broader enabling environment. The Demand Side: ROP Beneficiaries 46. The Romanian ROP 2007-2013 identifies project applicants and beneficiaries as responsible for two key actions: submitting the application for funding and carrying out the implementation of the project. The same document defines the institutional infrastructure in which beneficiaries operate: specifically, their two main counterparts are the Intermediate Body corresponding to their region and the national-level Managing Authority. There is no formal, institutionalized link from beneficiaries to the Monitoring Committee of the Regional Operational Programme (MC-ROP), but major public beneficiaries (e.g., County Presidents) have participated to MC-ROP sessions as representatives of each of the eight regions. As such, assistance efforts targeting applicants and beneficiaries are necessarily tied to the functions fulfilled by the MA and the IBs, although bodies like MC-ROP can bring to the attention of other decision-makers those broader issues that lie beyond the immediate control of the ROP system and need to be addressed at the government level (e.g., increasing accountability of technical designers and project verifiers; improving access to co-financing, etc.). 47. The ROP includes a diverse portfolio of projects and an equally heterogeneous set of applicants and beneficiaries, each with specific needs throughout a project’s cycle. The most relevant distinction is between public beneficiaries, on the one hand, and private and non-profit ones, on the other. Typically, the former design and implement more complex projects, from road construction to 33 rehabilitation of hospitals, schools, and historical monuments. The latter tend to have smaller projects, on average, including many simpler operations (i.e., lacking major construction components) – for example, purchasing of equipment, running advertising campaigns, etc. Based on the type of beneficiary and the type of project, the applicable legislation varies, particularly when it comes to public procurement procedures, as well as for different construction-related requirements (i.e., the technical documentation for a road needs to be much more elaborate than for a typical private-sector business development project). Ideally, effective assistance mechanisms should be as customized as possible to the needs of each beneficiary and each project. Throughout this report, some proposed measures will apply to all ROP projects, while others will only be relevant for public or for private beneficiaries, respectively. Whenever needed, proper segmentation of findings and recommendations, according to beneficiary type, is included. 48. The following insights are meant to provide a snapshot of the ROP’s diverse beneficiary pool and its main characteristics, as reflected by data available at the end of December 2012. Five years after the launch of the program, there were 8,221 financing application submitted into the system for a total value of 12.66 billion Euros – the equivalent of 206% of the total allocation from the EC (7.68 billion Euros from the European Regional Development Fund - ERDF). For all ROP axes, with the exception of technical assistance, the demand for funds exceeded the available budget. For priority areas like business infrastructure, education, roads, microenterprises, and tourism infrastructure, the difference was a factor of 2 to 1, which suggests that the program was reasonably well advertised and perceived as attractive by many applicants. The ability to shift some of funds across axes, for instance from industrial sites where initial applicants were few (largely because of unclear or ineligible status of property rights), has helped address the demand more flexibly, although there are still numerous beneficiaries who note that they have prepared expensive projects only to see them fail through the evaluation, often because they fell behind simpler applications that made the cut based on the “first-in-first-out rule.”69 Figure 23. Financing applications submitted as % of priority area allocation (December 31, 2012) 350% 326% Business infrastructure 294% Educational infrastructure 300% 262% Road infrastructure 251% 243% Microenterprises 250% 221% Tourism infrastructure 200% 191% Social infrastructure 166% Cultural heritage 143% 150% 133% 131% Health infrastructure 103% Urban development 100% Industrial sites 60% Tourism promotion 50% Emergency situations Technical assistance 0% Source: MA-ROP 69 This issue is the focus of an ongoing World Bank project on upgrading the ROP’s selection models for 2014 -2020. 34 49. Of the 8,221 requests received, 3,346 were approved and contracted, of which only around a quarter had been finalized by the end of December 2012. Of the total budget for all contracted ROP projects, 79% is managed by public beneficiaries, 16.7% by private-sector firms, and 4.3% by non-profit organizations. The picture looks somewhat different when measured against the absolute number of projects: close to 38% are under public authorities, approximately 59% under the for-profit sector, and only 3.5% under NGOs. At a basic level, this confirms the observations above in that most ROP beneficiaries of contracted projects are private-sector companies, but the bulk of the funding goes toward public authorities, which are in charge of larger, more difficult projects. In fact, when looking at completed investments as of December 2012, the value of an average ROP project corresponding to a public-sector beneficiary is close to 10 million RON, while for a private-sector beneficiary this figure is nearly 9 times lower, at only 1.1 million RON. Smaller projects are typically easier to implement: as evidence, of the total number of completed projects, 84% were implemented by private beneficiaries and only about 12% by public ones; by the same token, of the total value of projects still left to be finalized, 81% is under public authorities, with only 16% under private companies. As of December 2012, the average value of completed projects totaled 3.37 million RON, while the average value for approved projects, in various stages of implementation, was more than double (7.32 million RON). Figure 24. Regional split of ROP projects (by axis) Data source: MA-ROP 50. In terms of the regional split, December 2012 data reveal differences based on beneficiaries’ location: some regions started early and pulled ahead in finished projects, while others are bound to 35 catch up based on approved contracts in the pipeline.70 For instance, although regions have different allocations, North-East (NE) leads in terms of completed investments, with 221 projects, compared to only 45 for Bucharest-Ilfov (BI), but the former has 366 projects to be implemented by the end of the programming period, versus 282 for NE. When adding data for both completed and approved projects, the regions’ total budget mirrors the set allocation that was agreed in the beginning of the programming period, which was inverse proportional to the level of GDP per capita in each region. As such, compared to their richer counterparts, poorer regions received a bigger share of the total ROP funds available. There is also substantial variance in terms of the number and value of ROP projects contracted for a particular beneficiary: for example, the city of Craiova has a portfolio of 16 projects (many of them still under implementation as of December 2012) worth 625,063,984.6 RON. At the other end of the spectrum, there is one private-sector beneficiary who has a single project with a total eligible budget of less than 79,000 RON. Similarly, counties like Iași and Timiș have completed a lot more projects than Ialomița, Giurgiu, or Mehedinți. This diversity in the number of projects under implementation is also evident in the data from the survey corresponding to the current report: of the 470 respondents, 53% are implementing only 1 project, while 17% have portfolios of at least five projects. Figure 25. The number of completed vs. approved projects by region (December 2012) Data source: MA-ROP Figure 26. The number of completed and approved projects by county (as of December 2012) 200 180 160 140 120 100 80 60 40 20 0 BISTRITA-NASAUD CARAS-SEVERIN MARAMURES COVASNA GIURGIU DÂMBOVITA NEAMT CALARASI PRAHOVA VRANCEA SATU MARE IALOMITA BACAU SALAJ SUCEAVA CONSTANTA VALCEA BUZAU IASI BOTOSANI VASLUI HUNEDOARA ARAD BUCURESTI BRAILA MEHEDINTI ILFOV ARGES ALBA GALATI TIMIS DOLJ MURES HARGHITA GORJ BRASOV BIHOR TELEORMAN SIBIU OLT CLUJ TULCEA Completed Approved Data source: MA-ROP 70 For additional illustrations of the spatial distribution of ROP projects, please see Annex D. 36 Figure 27. County-level split of ROP projects (by axis) Data source: MA-ROP 51. Data also show that most of the ROP funds are channeled toward urban areas and larger cities in the country, as evident from comparing the maps reproduced below (figures 13, 14, 15, and 16). This is, in part, by design. Overlaps with other operational programs are meant to be minimal, and there is a special instrument dedicated to rural areas – the National Rural Development Programme. At the same time, axis 1 of the ROP attracts roughly a third of the total allocation and is entirely dedicated to urban development, including cities of different rankings: seven growth poles, 13 development poles, and urban centers (other cities over 10,000 inhabitants).71 In fact, one of the ex-ante evaluation recommendations was to promote a more balanced development in Romania by better using cities as “motors for socio-economic development in their region.” Most of the other axes have also targeted interventions at larger urban communities (e.g., schools, hospitals, social centers, business infrastructure) and connections between those areas through regional and local transport infrastructure (axis 2). The one exception is axis 5 for tourism infrastructure and promotion, where some beneficiaries come from rural areas and small towns and resorts. 71 See Government Decision No. 998/2008 37 Figure 28. Distribution of ROP contracted funds (December 31, 2012) Source: ROP Implementation Report (2012) Figure 29. Distribution of municipalities, towns, and communes Source: “Competitive Cities,” World Bank Report (2012) 38 Figure 30. Spatial distribution of ROP projects (by number) Data source: MA-ROP Figure 31. Spatial distribution of ROP projects (by value) Data source: MA-ROP 39 52. The straightforward implication is that the vast majority of beneficiaries of ROP funding come from urban areas of the country, which follows the logic of sustainable development going hand in hand with urbanization. As argued in the “Competitive Cities” report, growth in Romania – and elsewhere – is primarily a factor of individual productivity, which increases with the level of urbanization: if an area is more urban, it is also denser in resources (human, financial, knowhow, etc., with a greater potential for economies of scale). In the same vein, the ROP has privileged public, private, and nonprofit applicants in urban areas and larger cities because: (a) that is where the potential for development is highest, including from a stronger, more vibrant private sector; and (b) that is where beneficiaries themselves have the required capacity to design, co-finance, and implement ROP projects. This further means that ROP beneficiaries are relatively easier to reach, have more resources to travel and seek information by interacting with the IBs and the MA, and are reasonably well prepared in terms of available equipment, including for a potential switch to an e-based ROP in the near future. 53. The survey conducted as part of this project helps shed further light on beneficiary characteristics, particularly in terms of what they perceive as the main pros and cons of accessing ROP funding. In general, the top two perceived benefits are that ROP funding is non-reimbursable (an impressive 94% of respondents see this as an extremely or very important benefit) and it constitutes an additional source of capital for investments (84% of the sample selecting it as an extremely or very important benefit). Next on the list with respect to the same statistic, expressed as percentage of respondents who selected “extremely important” or “very important,” are ROP funds as an opportunity for: professional development (65%), organizational capacity improvement (57%), and enhancement of team motivation (50%). At the same time, 41% of the sampled beneficiaries see the ROP funding as extremely/very important in terms of facilitating networking and more active membership in professional groups. The picture looks favorable across the board. The relatively high proportions of people identifying each possible value proposition as an important one suggests that most beneficiaries see beyond the immediate financial impact of ROP funding – although that remains at the top of their list – and also appreciate opportunities for improving organizational performance and incentivizing staff working on ROP projects. This confirms observations from field research: the vast majority of ROP beneficiaries are truly passionate about what they do, genuinely appreciate their participation in the program, and see it both as a personal and as a professional challenge. Figure 32. Beneficiary survey perceptions of ROP benefits’ level of importance 100% 90% 80% 70% 60% N/A Unimportant 50% Hardly important 40% Relatively important Very important 30% Extremely important 20% 10% 0% Non-reimbursable Investment Professional Organizational Team Networking funding capital development capacity motivation improvement 40 54. At the same time, beneficiaries also provide key insights regarding the main obstacles slowing down the absorption of ROP funding. Even for a high-performing structural program for the Romanian context, nearly all interviewed beneficiaries had some grievances regarding how the system is currently functioning. The qualitative feedback provided in interviews and through the online survey is particularly interesting and subsequent sections of this report discuss it at length. For now, the figure below presents a quantitative, detailed picture of beneficiary perceptions of obstacles and their relative importance, suggesting, once again, that the toughest challenges have a lot more to do with system- wide issues than with the skills of MA and RDA staff. It follows that proper assistance mechanisms should target precisely what beneficiaries perceive as the most pressing obstacles: excessive bureaucracy (including procedural delays); lack of adequate financial resources (co-financing, cash flow, pre-financing); unpredictability (legislative changes, inadequate public procurement); and poor-quality service providers (technical designers, consultants). As for how these perceptions differ between different beneficiaries, there seems to be little variation, on average, between regions or between types of organizations (public, private, and non-profit), but this is not to say that all projects face the same challenges. In this case, averages can be misleading and it is much more useful to investigate the specific set of obstacles that beneficiaries face depending on their level of experience with structural funds, available capital, in-house vs. outsources expertise, etc. Figure 33. Beneficiary survey perceptions of obstacles' level of importance Excessive bureacracy & documentation Delays in project evaluation & selection Delays in reimbursements Lack of capital for co-financing & cash flow Frequent legislative changes Inadequate public procurement framework Insufficient prefinancing Low quality technical documentation Lack of high-quality consulting services Lack of transparency Low quality of applicant guides Poorly trained AM staff Poorly trained RDA staff 0 50 100 150 200 250 300 350 400 450 500 Extremely important Very important Relatively important Hardly important Unimportant N/A 41 55. In weighing the pros and cons related to accessing ROP funds, most beneficiaries remain convinced of the net positive value of the program. The two final survey questions addressed precisely this point in asking: (a) whether the costs of implementing ROP projects exceed the benefits; and (b) whether beneficiaries would still seek to access ROP funds in 2014-2020 based on their experience with the 2007-2013 budgeting period. On the former inquiry, 87% of respondents answered that the costs involved are lower than the benefits of ROP financing. As for beneficiaries’ appetite for applying for future ROP funding, 83% of the answers were positive, with only 4% rejecting the idea of seeking this type of financing again – a very encouraging sign for the ROP’s attractiveness in the next period. But given quantitative and qualitative interview feedback around the enduring difficulties of accessing ROP funding, the fundamental conclusion remains that better support mechanisms for applicants and beneficiaries are still much needed. While the perceived “benefit-cost ratio” remains greater than 1, the MA and the IBs teams should aim to increase it further, minimizing hurdles for beneficiaries to the full extent possible. Now the ROP has the opportunity to capitalize on the superior results to date and on beneficiaries’ accumulated goodwill, and deliver even stronger performance for 2014-2020. Figure 34. Beneficiary perceptions of benefits/costs of ROP projects 13% Benefits exceed costs Costs exceed benefits 87% Figure 35. Beneficiary responses to whether they would apply for future ROP financing 4% 13% Yes No I don't know yet 83% 42 56. In sum, the profile of ROP applicants and beneficiaries appears segmented across several dimensions: type of organization (public, private, and non-profit), level of experience with ROP funding, and location. This brief assessment seeks to lay the foundation for better understanding the task at hand for MA and IB teams. When it comes to providing successful support mechanisms, the fundamental questions remain: Who is the target group? What are beneficiary-specific needs (i.e., where do they require the most help)? And how can they be fulfilled most effectively? Beyond some common basic elements (e.g., mostly urban location), the pool of ROP beneficiaries appears diverse, with a wide-ranging set of needs. The Supply Side: MA and IB Support Mechanisms 57. Before reviewing the main types of beneficiary support, it is important to note the inherent tension in the two critical functions that the system of structural funds must fulfill in relation to applicants and beneficiaries: control and sanctions versus partnership and support. On the one hand, it is important to protect the financial interests of the EU and ensure that funds are channeled fairly and transparently, upholding the principle of “equal treatment” toward all beneficiaries. On the other hand, beneficiaries require customized, proactive assistance for accessing EU funds and abiding by all applicable regulations, at the national and supra-national levels. There needs to be a balance between these two functions: if the former takes priority, there is reduced appetite for implementing EU-funded projects; if the latter always prevails, controls might become too lenient and the risks of funds misuse increases. Importantly, fulfilling one function should not come at the expense of the other. There are legal and institutional ways for ensuring adequate separation of control and support mechanisms. 58. Evidence from field interviews with beneficiaries and other key stakeholders suggests that the ROP system would benefit from balancing the current rigorous focus on control with stronger efforts around partnership. This is the implication of what this report, along with the concurrent assessment of the MA-IB relationship, calls a “paradigm shift” toward a “beneficiaries first” mentality. As argued repeatedly, absorbing structural funds efficiently, effectively, and in full accordance with domestic and Community regulations, depends first and foremost on ensuring that beneficiaries are successful throughout the project cycle. On the positive side, the fundamentals look right: interviewed staff in the MA and the IBs recognize, without exception, the need to nurture strong partnerships with beneficiaries to achieve the shared goal of sustainable and inclusive development. IBs, in particular, are well placed to deliver close support to applicants and beneficiaries on the ground. 59. But other factors in the broader environment are not always fully aligned with this vision, which can lead to excessive auditing, further adding to the bureaucracy, diminishing creativity and innovation, and increasing duplication and inefficiencies. Going back to Ken Miller’s metaphor about government work as a system of pipes, one of the main causes for a disproportionate focus on control and sanctions originates when “the big boss comes along and says, […] never let it happen again.”72 People (and institutions) that get blamed tend to react by creating additional checks, tracking systems, verifications, penalties, reporting, and the whole range of instruments that keep twisting the pipes, blocking or delaying those critical services and support mechanisms from reaching customers in time. The other byproduct of this cycle is fear of responsibility: nobody wants to take the blame when something goes wrong and, with more layers of complicated processes, nobody needs to take the blame because (a) it is hard to identify exactly who is responsible; and (b) it is much easier to find fault with the 72 Ken Miller, “Extreme Government Makeover.” Also see the report on the MA-IB Relationship (Component I). 43 generic system, which leads to even more controls, processes, and bureaucracy. In parallel, as risk adversity rises, innovation becomes increasingly rare.73 The system’s overall performance decreases in a downward spiral with potentially devastating circumstances for all parties involved and, above all, for the system’s customers. To be clear, the ROP is not at that critical point, but the research conducted for this report suggests that some elements of the story above exist and may aggravate, if left unaddressed. 60. As implied before, these observations do not only apply to the ROP, or only to Romania: putting beneficiaries first and controlling excessive auditing are increasingly recognized as much needed shifts in the broader EU-level approach. A 2011 study by Carlos Mendez and John Bachtler, two leading experts on structural funds, argues persuasively that the proliferation in EU Cohesion policy auditing “has taken a distinctly compliance-oriented form, in contrast with the performance audit model adopted in other countries, […] and has in fact undermined the strategic performance of the policy and is even threatening its sustainability.”74 The two authors describe the compliance and performance- based auditing systems: the former focuses on detecting fraud and errors and performs verifications of monitoring and control mechanisms; the latter is more similar to evaluations of how a certain program is performing, based on quantifiable indicators. Figure 36. Romanian beneficiary perspectives are very similar to those in other Member States 61. More recently, the EU has brought the issue of simplification and beneficiary assistance to the top of the agenda for 2014-2020. One very positive signal is the adoption of Regulation No. 996/2012, which sets the principles and governing framework of the EU budget. This was aimed specifically at 73 Victor Thomson, “Bureaucracy and Innovation,” Administrative Scientist Quarterly, Vol. 10, No. 1 74 Carlos Mendez and John Bachtler, “Administrative Reform and Unintended Consequences: An Assessment of the EU Cohesion Policy Audit Explosion,” Journal of European Public Policy, 18:5, 746 44 enabling beneficiaries to implement EU-funded project more easily by “cutting red tape, speeding up procedures, and shifting the focus from paperwork to performance.”75 For its part, the Romanian Government is moving in the same direction, demonstrating increased recognition of an approach that encourages and actively helps beneficiaries succeed in absorbing structural funds. One of the latest examples is Government Emergency Ordinance 27/2013, which has simplified the mechanism for processing disbursement requests, promising to reduce delays and lower the need for beneficiaries to provide “bridge” financing to pay for contracted providers. Further details are provided in a subsequent section on financing needs. 62. In this broader context, the types of ROP beneficiary support mechanisms have also evolved and improved over the duration of the 2007-2013 programming period. Romania has followed in the footsteps of other Member States, setting up a range of useful resources for applicants and beneficiaries: specialized web portals; in-person presentations, workshops, and caravans; written materials (e.g., brochures, magazines); and helpdesks (“information bureaus”), mostly in charge of providing assistance over the phone or email and point inquiries in the right direction. In addition, IBs often provide more customized, on-demand support for beneficiaries with significant portfolios of projects under administration, through monitoring visits and ad-hoc interactions. 63. The support mechanisms provided by the MA and the IBs should depend on two key variables for each applicant/beneficiary: selection model and portfolio value/complexity. Regarding the former, funding dedicated ex-ante to a particular beneficiary for a project or set of projects (as in the case of growth poles for the current programming period) can and should be complemented by customized, close engagement by MA/IB teams, throughout the project cycle. The aim is to ensure technical assistance that is fully aligned with specific beneficiary needs and helps deliver the most impact possible. By contrast, if the selection of beneficiaries is done on a competitive basis, support should be standardized, ensuring equal, solid opportunities for all. 64. The portfolio value is equivalent to the total number of ROP projects multiplied by their average budget. In this context, project value is considered to be a proxy for complexity, although it is certainly an imperfect measure. There are some ROP projects that are small, but technically complex, while some large investments can be relatively simple to implement. On the average, however, size and complexity typically go hand in hand. As evident from the section describing the profile of ROP beneficiaries, there are two relatively distinct groups: public-sector institutions implementing a large, complex portfolio of projects; and public, private, or non-profit beneficiaries of relatively smaller, simpler investments. There is value in serving both types of “customers,” starting from an understanding that the customization of assistance mechanisms is much needed. At one level, customized assistance means that beneficiary needs are better served. At another level, MA/IB staff will use their time and resources more efficiently. Importantly, this is not to say that the principle of equal treatment is to be breached, but it is a simple recognition that one size does not fit all when it comes to addressing applicant and beneficiary needs. At the end of the day, the quality of the assistance provided should be the same for all beneficiaries, regardless of the optimal instrument that the MA/IB should deploy to that end. 65. The typology in the figure below is a simplified representation of how optimal support mechanisms vary according to a particular applicant/beneficiary and the targeted axis/priority area 75 See, for instance, the justification for adopting this regulation: http://europa.eu/rapid/press-release_MEMO-12- 795_en.htm 45 and its corresponding selection model. Of course, this does not capture all the nuances of different beneficiary segments – including applicants from marginalized communities, who will face special challenges in accessing information about ROP financing available, implementation challenges, etc., and hence will require more customized assistance, even for a relatively small project. Nor does this imply that a large beneficiary who needs more customized support does not also need standard documentation. All in all, the two-variable model presented here is sufficient and, importantly, derives directly from with the field research and the survey conducted as part of this study. Figure 37. Typology of ROP support mechanisms Dedicated Extensive, customized support Type of selection model Efficient, standardized support Competitive Small, simple Large, complex Value (complexity) of ROP portfolio (# projects * average budget) 66. At the same time, this approach is fully aligned with the recommendations made by the EC in its October 2012 position paper regarding Romania’s preparations for 2014-2020. At one level, the Commission called for improved assistance to strengthen beneficiaries’ capacity, as follows: “Whereas tailored assistance could be directly channeled to larger beneficiaries, accompanied by sound ownership and appropriate endogenous reinforcement and restructuring, streamlined assistance should be provided to sectors sheltering a broader scope of smaller and more heterogeneous beneficiaries (tailored guidance, help desk, dissemination of standardized documents, etc.).”76 At the same time, the EC also advocated for lighter administrative burden on beneficiaries, ensuring “proportionality of the applicable requirements with the capacity of beneficiaries and the risks at stake.”77 Both topics are related to providing enhanced support for ROP applicants and beneficiaries, and the next chapters expand on these issues and put forth a set of specific recommendations for improving beneficiary support through the various stages of a project cycle. 76 “Position of the Commission Services on the development of Partnership Agreement and Programmes in Romania for the period 2014-2020,” Ref. Ares(2012)1240252 - 19/10/2012, p.37 77 Ibid. 46 IV. Horizontal Cross-Themes 67. There are five major topics that shape the overall ability of ROP beneficiaries to design and implement successful projects: regionalization, human resources, access to finance, electronization, and targeted support for marginalized communities. The reason for evaluating these upfront is that they cannot be treated separately under specific phases of the project cycle – i.e., included in subsequent chapters of the current report. Rather, these factors need to be understood as horizontal cross-themes that have the power to influence the overall success of applicants and beneficiaries in completing impactful ROP investments. A. Beneficiary Perspectives on Regionalization 68. The Romanian Government is continuing to push forward the plan for devolving increased responsibilities at the regional level. This is a timely endeavor in preparation for the 2014-2020 programming cycle, as it could significantly speed up the absorption of EU funds and enhance their impact. In the course of the field research for this report, numerous beneficiaries were asked about their overall perspectives regarding the prospects of introducing administrative responsibilities at the regional level. Most beneficiaries were in favor of this possibility, although they also expressed concern about whether there is sufficient time to switch to the new system before the start of the 2014-2020 programming period. This section summarizes the input received from interviewees on the expected benefits and risks of regionalization, noting that recommending a particular model is beyond the scope of the current assignment. 69. From the beneficiaries’ perspective, one key advantage of a regional -focused ROP was identified as a higher level of responsiveness and customization for the regional level. Nearly all interviewees noted that the current regions in Romania – and this holds for any regional split, regardless of which counties would be included in the various regions – have very different needs and available opportunities. This conclusion results also from the World Bank report on Competitive Cities, which notes that leading and lagging areas within a country require different types of investments: for example, leading regions (as the economy’s main engines) demand investments in quality of life and improved connectivity to surrounding areas, while lagging regions still need significant investments in basic institutions and infrastructure (schools, hospitals, etc.). But the current ROP does not allow for an adequate degree of customization to fit region-specific priorities (although the allocation of funding does vary by region, favoring the less developed ones) in that all beneficiaries across the country have the same funding opportunities. Beneficiaries generally appreciate the idea of having investment programs more customized to their needs, but also recognize that some degree of central coordination is necessary to leverage cross-regional synergies and also ensure a smart, balanced use of resources available. 70. One of the more common comments made during interviews by beneficiaries is that the MA in Bucharest is inevitably removed from the situation on the ground, unlike RDAs that are able to customize their assistance and intervene more quickly, as needed. As argued earlier, this decentralization – albeit limited in an absolute sense, but quite substantial relative to other operational programs – has been essential to the ROP’s success to date. Beneficiaries share this view and would welcome a regional MA as a best-of-both-worlds option: it would be close to the ground, knowledgeable of their specific situation and able to react quickly, and it would also hold the decision-making power to 47 adopt certain needed measures and provide informed answers. Currently, this latter part is missing from the RDAs’ mandate and they often need to defer a response to the MA. If the new system is properly implemented and regional MAs had sufficient capacity and expertise, beneficiaries would probably wait less for approvals and other procedural required steps. Equally important, improved assistance would also come as a by-product of redistributing some of the pressure from a single, central authority to eight regional ones that would face, simply put, an eighth of the current workload burden at the MA level. The perception that all power belongs to Bucharest, echoed in a substantial number of interviews around the country, would hopefully be replaced with a sense of shared mission at the regional level and, in addition, a sense of healthy competition may develop among regions within Romania with respect to EU funds absorption and impactful investments. Last but not least, decentralization would build on accumulated experience at the regional level from both pre-accession and the 2007-2013 budgeting period, meaning that the need for centralized oversight is lower when RDAs and beneficiaries already have a deep knowledge of how the system should work. 71. Perhaps most importantly, regionalization promises to deliver truly regional projects by ensuring proper coordination among political and administrative levels, unlike the current system whereby cross-jurisdictional projects are almost entirely lacking. Under the current centralized system, with no layer between the national government and the county level, it is very hard to design and implement integrated projects at the regional level. Beneficiaries are the first to recognize these limitations. In some instances, it is true that some partnerships among local authorities have indeed materialized (for instance, through the Intercommunity Development Associations – IDAs – and their purchasing of equipment for emergency interventions). In other instances, some level of coordination was achieved through the Regional Committee for Strategic Evaluation and Coordination (RCSEC), with some regions faring better than others in that respect. Even in such instances, projects were still fundamentally county or city-specific. Very few attempts were made to design complex projects between counties and across multiple ROP axes (let alone multiple operational programs) and, when this happened, complex projects involving multiple counties took longer to draft and submit and ended up losing against simpler projects that took advantage of the first-in-first-out (FIFO) rule (e.g., projects for axis 2 – transport infrastructure, in the West Region). In the context of much greater EU focus on instruments like Integrated Territorial Investment (ITI), regionalization appears to be the optimal solution for facilitating collective action among beneficiaries. 72. While most interviewees agree that regionalization is needed, in principle, they also caution that there are significant potential pitfalls related to the regionalization process. The first deals with timing: RDA representatives and ROP beneficiaries fear that a full overhaul of the system will undermine the preparation of the 2014-2020 budgeting period. These fears are aggravated by current delays at the EU level, as the final budget for the next programming cycle is yet to be voted by the European Parliament. A related theme is that the current system has started functioning properly, with most beneficiaries finally comfortable with procedures. Any radical change risks undermining the accumulated institutional knowledge and generating further errors during the next programming period. Additionally, some beneficiaries worry that the consultations around regionalization have been too focused on apparently minor issues (e.g., deciding on the lead city in each region) and not sufficiently thorough on defining the exact allocation of responsibilities between the local, county, regional, and national levels. As the process moves forward with more consultations, beneficiaries are eager to engage on increasingly detailed discussions around an optimal future setup. 73. As far as the ROP’s implementation goes for 2014-2020, one possibility worth considering further, based on interview feedback and in conjunction with the project selection models 48 workstream, is a system adapted to different types of projects, depending on their magnitude. Specifically, RDAs could become MAs in each region, but maintain their current legal status, and handle small and medium-sized regional and local-level projects. The current MA could capitalize on its accumulated experience by becoming the MA for the large strategic projects to be agreed upon by the local/regional and national-level beneficiaries. Under this system, IBs may not be needed (and are not required), which promises to speed up the evaluation and implementation of projects by essentially bringing the MA directly next to beneficiaries. Such a framework promises to exploit the advantages of triage between various projects and build on the system’s experience at both regional and central levels. 74. As argued in the report on the MA-IB relationship, the optimal solution is far from clear and simple, but arguments appear to weigh in favor of regionalization – even with a “light” version at first. The experience of other EU member states suggests that decentralized systems with regional-level legislative and executive bodies are better suited to drive a coherent regional convergence agenda and negotiate with the central government based on local priorities. The need for further decentralization appears to be an obvious solution to increasing the ROP’s efficiency and impact, but a full-swing regionalization process needs to carefully balance the costs and benefits of different setups. “Regionalization-light,” which would involve a more limited, defined set of responsibilities delegated to the regional level, can be a viable option in the short-to-medium term and would ensure that Romania embarks on the new programming period with the right foot forward, avoiding the kind of radical changes that would upset the currently functioning mechanisms. Later on, based on continuous assessments of the system’s needs and performance, the split of responsibilities among various administrative layers can be adjusted to reach an optimal, efficient setup. B. Human Resources 75. As noted elsewhere, the ROP’s success depends first and foremost on the people who are responsible for designing and implementing projects. Without their direct involvement, even the best institutional set-up would be futile in reaching the ROP’s fundamental objectives. Human resources are the critical component of the “capacity” pillar that was earlier described under the organizational performance framework, which reinforces the idea that the ROP is only as good as its people. The “ROP 2.0: The MA-IB Collaboration and Communication” makes the case that leveraging existing know-how at the MA and IB levels depends on reducing the risk of turnover to the full extent possible. The same is true of the third link in the MA-IB-beneficiaries triangle, where continued success hinges on implementing adequate policies for human capital management. After interviews with project teams in over 150 public-sector beneficiaries, it is clear that they are a most valuable resource for the system and, together with their MA and IB colleagues, deserve much of the credit for the ROP’s success to date. Internal Capacity 76. Importantly, the most active, successful public-sector beneficiaries have dedicated teams of specialists working on EU-financed projects, from concept design through implementation, demonstrating a mixed set of experience and expertise. In general, though not always, successful organizations rely on a highly experienced director heading the investment / EU projects / technical unit, a key person who maintains the institutional memory and collaborates closely with the top decision- maker. These experts usually demonstrate a profound understanding of the system’s issues, having worked on them since Romania’s pre-accession days, and have great ability to work around obstacles, often finding innovative solutions. They are very knowledgeable on a range of key issues, from legal to technical and financial aspects, and they also tend to speak up when something is not functioning 49 properly. Still, beyond their unique set of abilities, these team leaders would not be able to singlehandedly manage projects. They are usually supported by several young people who are also deeply involved in EU-financed projects and benefit from active mentorship and on-the-job learning. Thus, there is an optimal division of responsibilities between younger staff involved in day-to-day operations and experienced department heads that have the broader perspective and can effectively advise decision-makers at critical junctures in a project’s cycle. 77. On the positive side, there are signals that the current ROP system values the importance of key human resources involved in the absorption of EU funds. Interviewees at both the MA and the IBs have openly recognized the significance of having strong, long-term partnerships with counterparts in beneficiary organizations, particularly people who have a vast experience with structural funds and can figure out solutions to complex problems that arise. As noted earlier, it is a defining characteristic of the ROP system that it relies on a relatively tight group of people across all levels – from beneficiaries to the IBs and the MA – and there is alignment between these stakeholders around the common objectives of high absorption and high impact. Equally important, most interviewed staff in public-sector beneficiary organizations demonstrated an intrinsic motivation to make a difference in their community through ROP-financed projects. This holds for both experienced heads of departments, who face tremendous pressure and stress in their daily jobs, and of younger staffers, who earn very low salaries. In truth, for a public servant, working on a ROP project is usually perceived to be an exciting opportunity for professional growth and a way to generate concrete positive impact. 78. That said, monetary aspects are never unimportant, and some key personnel is reported to have left the public sector for more attractive jobs in private consulting companies. In recognition of their importance, some measures have been put in place to motivate staff working in local administration. Specifically, law no. 284/2010 has allowed for awarding bonuses of “up to 25 classes” – equivalent to 62.5% – to personnel involved in the implementation of EU-financed projects.78 While the ultimate decision on granting this salary increase is up to the local decision-makers (e.g., the Mayor/County President and the City/County Council), most public authorities have opted for this solution. This is much welcomed by interviewed staff, despite some critics who have noted that the measure introduces inequalities in the pay of public servants, affecting the performance and morale of staff not involved in project implementation. At the same time, young personnel have noted that the salary base of a new hire is extremely low – around 1,200 RON or approximately 250 EUR per month – meaning that the percentage increase does not make a tremendous difference. The situation is somewhat better for public servants with longer careers, who receive higher salaries based on seniority. 79. The government should strengthen the bonus system for public servants involved in EU- financed projects, while at the same time increasing their salary base to remain competitive compared to alternative opportunities in the private sector. Law no.284/2010 only allows bonuses for staff involved in the actual implementation of projects. Currently, there is no equivalent incentive for the preparation of project proposals, which is a critical phase that can shape the success of the entire project. It is recommended that the bonus system be extended to cover all phases of the project cycle. Equally important, if the salary base for young employees remains low, even adding the highest-end (62.5%) proportional increase will fail to come close to salaries offered by private consulting firms. Options should be explored for increasing the salary base according to performance for those critical resources involved in EU-funded projects. 78 See Monitorul Oficial nr. 877, 28.12.2010 50 80. Equally important, many of the units working on EU-financed projects remain understaffed and overburdened with responsibilities related to managing large portfolios. This is particularly true of the recent period, when teams have had to both accelerate the implementation of ongoing projects to complete them before the 2015 deadline and also prepare new projects for the 2014-2020 budgeting cycle. In parallel, many public-sector beneficiaries have also faced a high number of audit and verification missions, including by the IBs, the MA, and National Audit Authority (NAA). While sometimes necessary, these efforts further distract staff in beneficiary organizations from vital programming and operational tasks. What is more, key people involved in project implementation have expressed concern that potential financial corrections, which many projects have suffered since 2011, may eventually be imposed upon them directly. In this context, it is no surprise that many heads of unit have pointed out the lack of capacity in terms of an adequate number of personnel. The consequences are multiple-fold: the people who stay in the system are facing unsustainable time pressures and workloads, as small teams end up supervising significant project portfolios, often in excess of EUR 100 million in the case of larger cities; as long hours accumulate proportionally with increased responsibilities, staff performance and motivation diminish and the likelihood of unintentional errors increases, putting even more pressure on the entire system, including on MA and IB staff who need to address errors in due time. For all these reasons, the risk of turnover of existing staff in public-sector beneficiaries has become significant and swift measures are needed to reduce the pressures they are facing. 81. The legal framework constraining the hiring of public servants should be reconsidered to ensure that units focused on EU-financed projects are adequately staffed at all times. In particular, Government Emergency Ordinance 34/2009 imposed the seven-to-one rule whereby one new hire was accepted for every seven vacant spots. This measure was meant to reduce the public sector’s expenditures in the context of Romania’s broader efforts aimed at stronger fiscal discipline, and was part of the government’s agreement with the International Monetary Fund. In the meantime, this act has suffered several adjustments and some flexibility was introduced, including for the personnel working on EU funds absorption through Government Emergency Ordinance 32/2010. Even so, interviewed beneficiaries note that hiring remains very rigid and requires a range of special approvals, meaning that, in practice, they still cannot fully staff the units focused on EU-financed projects. Legislative changes are needed to address these hiring needs promptly and efficiently. As the report “ROP 2.0: The MA-IB Collaboration and Communication” shows, staff shortages are also noted at the MA level, so corrective legislative measures would benefit all public-sector stakeholders with a key role in the ROP’s implementation. 82. In areas where internal resources cannot ensure the needed expertise, beneficiaries should be able to define and manage relations with specialized consultants and contractors. From this point of view, local public institutions should rely on core human resources, demonstrating basic skills in economics, civil engineering, and legal affairs. These should be able to both manage the relationship with departments within the MRDPA and the RDA according to the Financing Contract, as well as the relationship with other actors – designers, project verifiers, contractors, engineers, financial and technical auditors, etc. – based on the contracts signed with each of them. 83. The proposals above would likely make a significant positive impact on Romania’s absorption capacity but, nonetheless, such measures remain outside the direct control of the MA-ROP and are also challenging to adopt, from both a political and economic standpoint. A range of laws would require amendment, in the context of persistent caution in managing Romania’s overall budge t expenditures. Also, the risks related to uncontrolled spending on new hires and additional bonuses should be carefully mitigated through close supervision, ensuring that these measures apply only as 51 needed. This is neither easy to accomplish, nor a certain solution, but the status quo alternative does not appear to be sustainable. People working on EU-financed projects acquire a set of coveted skills, easily transferable to a private-sector consultancy. The risk of turnover is particularly high for young people, who are generally more mobile in the labor market. In the absence of proper remuneration systems, all the mentorship and know-how transfers received by these employees from their department’s head, along with their accumulated experience, become lost resources for public authorities that invested in their professional development. As the more senior, experienced public servants retire from the system and the talented, up-and-coming experts keep switching to the private sector, Romania’s ability to absorb ROP and other structural funds will increasingly weaken. If what is at stake are billions of EU funds flowing into the country, there should be openness and flexibility with respect to designing the proper, performance-based systems for managing critical human resources in public-sector institutions. 84. One final recommendation under this subsection urges the MA and the IBs to capitalize on the existing knowledge of experienced practitioners at the beneficiary level. As noted earlier, these are people who have implemented EU-financed projects since the pre-accession period. They know the system very well and are generally keen to speak up when problems arise, and proactively look for solutions. One possibility is to facilitate the formation of an ROP Corps of Practitioners (ROP-CP), i.e., a forum bringing together these experienced people from around the country. The ROP-CP would meet periodically and send suggestions to the IBs, the MA, and the MC-ROP, if necessary. Depending on what is viable in a given timeframe, this committee would meet at both regional levels (say, monthly or every other month) and nationally (for instance, at least twice a year). It is expected that the Corps could quickly become a critical component of the ROP’s continuous learning and adaptation system, and would ensure an even higher degree of the system’s responsiveness to the problems identified in the field. Moreover, members of the ROP-CP could also be selected to become trainers of staff involved in EU-funded projects. Capitalizing on their experience mentoring their own teams, these expert practitioners could have a tremendous impact by teaching beneficiaries who are new to the system, across local public administration and, possibly, non-profit and private organizations as well. Beyond the obvious impact of helping others improve their performance, this initiative would also further boost the motivation of the experienced practitioners in local public administration, as an opportunity to share knowledge and strengthen the entire system. External Resources 85. While the observations above apply primarily to public sector beneficiaries, proper management of human resources is equally important for other beneficiary types. A non-profit or a private company implementing an ROP project also needs to rely on people who know the system well and can respond adequately to potential challenges. For these kinds of beneficiaries, it is much less common to have in-house expertise for EU structural funds. There is a range of outside resources that can be involved on a contractual basis, primarily consulting firms that focus on working with beneficiaries throughout the process of applying for and implementing ROP investments. The importance of these external experts cannot be understated: in numerous interviews, beneficiaries who had hired their services argued that a project can succeed or fail depending on who is selected as a consultant. In some cases, beneficiaries had had the misfortune of working with unprofessional consulting firms that were charging high fees, writing bad projects, and, in a few extreme cases, even misinforming their clients about available ROP financing opportunities. At the other end of the spectrum, there are some stellar consulting firms that have proved vital for designing new project ideas and boosting the ROP’s absorption rate. 52 86. The key challenge in managing external human resources is that the market is not fully mature or transparent, so selecting the right consulting company is a daunting task for most beneficiaries of ROP funds. The number of consulting companies specializing in this field has gone up significantly since Romania joined the EU in 2007. A few firms are emerging as reputable, dominant players in the market, and there are also many smaller companies who seek to increase their project portfolio. Some of them are quite aggressive in recruiting potential beneficiaries and determining them to apply for EU funds, which can be a positive thing as long as the company is well qualified and proposed projects are impactful. But, as noted above, this is not always the case, and beneficiaries are at risk of selecting the wrong consultant. Based on interview feedback, a common way of choosing an outside expert is through word of mouth or acquaintances, which is neither a foolproof method, nor a very efficient one. What is more, quite a few private firms (e.g., medical practices, small factories, etc.) have selected consultants without doing any research beforehand, including, for example, after a “marketing call.” In a few instances, interviewed beneficiaries were perseverant enough to replace their consultant once the project failed for the first time, but it is expected that there are many more situations of people who have been permanently dissuaded to apply for ROP financing if they had a bad first experience. 87. Solutions to increase the market’s transparency include (a) publishing the names of consulting firms who prepare and/or manage projects and (b) enabling user-feedback mechanisms by allowing beneficiaries to post comments about their experiences with various service providers. On the former, the MA already publishes the full database with basic information on beneficiary projects; it could add two columns on the consulting firms that prepared and managed the project. On the user-feedback platform, while the MA and the IBs cannot directly initiate such a project and need to remain impartial, they can work with other important players in the field to facilitate the development of an accessible review system. This would enable applicants and beneficiaries who have worked with consultants to provide reviews of their experience on a platform that would be easily accessible to all interested users. The ratings would be validated by an administrator to make sure that they are based on actual experiences. As more and more information gets entered into the system, an implicit ranking of companies emerges, as a primary resource for potential beneficiaries interested in working with a consultant for attracting ROP funds or EU structural funds more generally. Interestingly, all beneficiary types look favorably upon this solution – per the figure below – and, if deemed appropriate, the system could be expanded to include the full range of service providers. 88. An alternative solution, which has been considered by some stakeholders in Romania, is the blacklisting of companies reported to engage in unprofessional conduct (e.g., charging illegal fees). This would be a step forward compared to the status quo, but peer reviews remain, overall, a superior option because they provide much richer insights into the performance of a wide array of players in the market. Moreover, this platform would be relatively easy to develop and the MA and IB roles would be limited to starting a conversation with interested parties and, once the portal is complete, to promote is as a useful resource for ROP applicants and beneficiaries. 53 Figure 38. All beneficiary types support the idea of an online ratings system for service providers 89. In addition, contracts should be strengthened to protect the interests of applicants and beneficiaries and hold consultants accountable for the quality of their work. Currently, there are few mechanisms, if any, to impose sanctions on consultants who are at fault for issues that arise during the application and implementation phases. This is true of other service providers as well (e.g., technical project designers and verifiers, auditors, etc.) and will be further discussed under the “implementation” chapter of the current report. The solution for strengthening accountability mechanisms lies in adopting the proper contract form, with adequate safeguards for the beneficiaries of contracted services. C. Access to Finance 90. Adequate access to finance throughout all stages of a project is a paramount concern for interviewed beneficiaries, across all sectors (public, private, and non-profit). Resources are required for multiple critical tasks: to kick-start a project; to ensure co-financing, depending on the type of investment; and to secure adequate cash-flow throughout the cycle, providing a safety net in situations where reimbursement requests are delayed or may suffer adjustments based on financial corrections. It is hard to overstate the importance of having the capital to complete an ROP project – essentially, a beneficiary organization may have the best concept and the best team for implementing it and still fail if the required funds are lacking. This has been particularly evident in the post-2008 context, when the financial crisis hit Romania hard, along with other countries, reducing banks’ appetite for co-financing EU-funded projects. As several programs were pre-suspended and EC payments stopped, the budget came under increased pressures and there were inevitable, significant delays in honoring reimbursement requests. This further constrained beneficiaries’ ability to pay service providers, who became weary of continuing projects’ implementation in the absence of proper financing, putting at risk the implementation timeline for numerous projects. The situation improved as Romania’s economy slowly recovered, but this goes to show the domino effects that inadequate financing can generate at the level of a project, of a beneficiary organization, and even of an entire country. 91. On top of everything, national-level legislation has not always been particularly stable, going up from 15% to 30% in the immediate aftermath of the crisis, and then coming back to 10%, paid in two installments of 5% each. Such changes have affected beneficiaries’ financial forecasts and created multiple delicate situations in terms of ensuring adequate cash flows. This lack of predictability, aggravated by relatively unexpected decisions by the EC to pre-suspend programs, has further reduced banks’ desire to get involved in EU-financed projects, particularly as “mother banks” in foreign countries set deleveraging targets and increased risk adversity on the Romanian market. All this, combined with 54 banks’ relative lack of knowledge about how EU structural instruments work, has made it very challenging for beneficiaries to attract much needed financing. 92. Field research observations suggest that several banks in Romania, including some of the major ones, have specific products and services targeting the needs of project beneficiaries. These include the possibility to cover ineligible expenditures, provide required co-financing, and ensure cash- flow support. Some banking units have specialized staff working on EU funds, with specific skills. These experts can recommend different banking products and also analyze the project proposal to see if it is viable, working with clients to better align the value proposition of intended investments to the particular capacity constraints that they face. There have also been situations where a bank refused to grant financing for a particular project, despite the fact that an applicant had secured ROP support – for instance, if the bank believed that developing a new line of business is not optimal for the current market, it would caution beneficiaries about the risks of the investment and its longer-term costs (e.g., maintenance). 93. At the same time, many beneficiaries tell somber stories about trying to “sell” ROP -financed investments to different banks. There are cases when public beneficiaries have organized tenders for such services and no banks showed any interest whatsoever. Private-sector companies also described shopping around extensively to find a willing lender. To some degree, things have changed in recent years, as banks have begun to understand the entire process of absorbing EU funds and the specific challenges of the Romanian context. For example, one particular product warrants that no penalties shall be imposed on a beneficiary for failing to pay back a loan by the agreed date if there are objective circumstances beyond their control, including the pre-suspension of payments by the EC. The market for such products is growing in complexity and the main anticipated need for the future will be to “connect” beneficiaries to such financing opportunities. This is a role that the MA and the IBs can easily play – without taking sides or favoring one bank over another, they can provide public information on products offered and other relevant details, acting as a bridge between the demand and supply of financing needs for ROP projects. 94. Another idea that has been considered by the current government is to work more closely with banks in channeling funds from the EC to private-sector beneficiaries of projects. This would cut through the bureaucracy involved in disbursements through state institutions. Banks could also play a role in assessing the viability of specific proposals (e.g., as part of the cost-benefit analysis), advising the MA and the RDAs based on their accumulated experience and knowledge of market dynamics. At a minimum, banks should preserve the current role of providers of the “comfort letter,” to be used by applicants for the evaluation process as proof that their project will receive financial support for co- financing, from a banking institution (the introduction of this requirement in 2010 was much welcomed and replaced a system whereby applicants merely filled out a statement “at their own risk”). D. Electronization: Toward a Functional e-ROP 95. In a recent survey of 470 beneficiaries conducted as part of this report, “excessive bureaucracy and documentation” ranks first in a list of obstacles that affect the proper functioning of the ROP. This finding confirms observations from field research: virtually all interviewed beneficiaries mentioned the very high amount of documents drafted, submitted, and sometimes resubmitted in a hard-copy paper trail that runs from beneficiaries to the IB, the MA, and often back and forth through this chain, as necessary. Borrowing a metaphor from Ken Miller’s “Extreme Government Makeover,” the 55 report “ROP 2.0: The MA-IB Collaboration and Communication,” compares the current circuit flow to a system of twisted, leaky pipes. This creates delays, significant paperwork, and increases the risk of error and duplication. Beneficiaries have noted several challenges faced under the status quo:  High volume of hard-copy documents: Nearly all key exchanges between the MA, the IBs, and beneficiaries are based on hard-copy documents. This holds for the entire project documentation (e.g., proposal, annexes, amendments, etc.), as well as for all communications.  Duplication: The requirement to print multiple copies of the same document increases administrative and transportation costs. It is not uncommon for beneficiaries to send, literally, a van full of thousands and thousands of pages. This is true of the application, as well as of subsequent reimbursement requests, some of which may contain lengthy files related to public procurement procedures. On that note, interviewed beneficiaries express disbelief that it is necessary, for instance, to send the entire public procurement documentation for every single reimbursement request.  Storage: Field visits offered many opportunities to see the tangible result of excessive bureaucracy. There are rooms filled with project-related documents and often a single project will take up several shelves, particularly if it required a complex technical project. This system is not only costly, but also risky in terms of losing or misplacing documents and, because everything is in hard copy, some beneficiaries do not take the time to save an electronic copy of the documents.  Mentality: The principle enunciated by many beneficiaries and RDA representatives is the following: “if we do not see it in writing on paper, with a stamp next to it, it is not to be trusted.” This is a direct consequence of the lack of experience with electronic forms, as well as a byproduct of a system that has focused disproportionately on audits and control mechanisms. Interviewed beneficiaries have expressed concern that they need to always be insured against potential risks. As a result, they write formal addresses to the IBs and the MA and keep the entire trail of responses to be able to show, if need be, that they did everything in their power at a particular junction in a project’s implementation. This is understandable and often a good, prudent practice, but it continues to add to the volume of paperwork and to the work of all parties along the chain (beneficiary, IB, and MA staff). 96. At a basic level, the level of bureaucracy that ROP beneficiaries face is dependent on two factors: actual procedures (the what) and the channels used for going through required steps (the how). On the former, the system has proved adept at improving procedures and processes, and there are several important examples of simplification that have helped applicants and beneficiaries and also reduced the high workload faced by MA and IB personnel. For instance, the number of required field visits during implementation was reduced – for some projects, IB staff are no longer required to visit beneficiaries to verify every reimbursement request, but only at key junctures (e.g., upon project completion). The previous work on assessing the MA-IB collaboration makes several suggestions regarding the further simplification of procedures (the what) and additional suggestions around this component are also included in subsequent chapters of this report, for each stage in the project cycle. 56 Picture 2. The heavy reliance on hard copies has generated serious storage problems 97. The main focus of this subsection is the development of an e-ROP. There are several important advantages of an e-based system: speed, cost savings, and accuracy. First, everything is faster because information can move almost instantaneously between beneficiaries, IBs, and the MA, which reduces delays and contributes to the broader goal of efficient absorption of EU funds. Second, overall costs are lower because there is less need for expansive storage and physical transfer of documents. Third, an electronic system allows for more automated verifications and filters to ensure that information sent by applicants and beneficiaries is accurate and complete. This would eliminate some of the back-and-forth between the IB, the MA, and applicants/beneficiaries, and generate additional time and cost savings. The Authority for Coordinating Structural Instruments (ACSI) has recognized the potential benefits of e- based systems and has experimented with such components for instruments like the Human Resources Development Programme. Another useful example is the UEFISCDI Direct Manager (pictured below), which suggests that replicating these features would be feasible. 98. Recognizing the potential of “e-cohesion,” the EC has pushed strongly in this direction in recent years, as part of the effort to reduce the administrative burden on beneficiaries and streamline the implementation of the ERDF, the ESF, and the CF. The most prominent example is in Article 112 from the EC proposal on common provisions regulation for cohesion funds. This covers Member States responsibilities in the area of management and control systems, and mandates under Section 3: “Member States shall ensure that no later than 31 December 2014, all exchanges of information between beneficiaries and managing authorities, certifying authorities, audit authorities and intermediate bodies can be carried out solely by means of electronic data exchange systems. The systems shall facilitate interoperability with national and Union frameworks and allow for the beneficiaries to submit all information only once.”79 In April 2013, the EC published an implementing act 79 European Commission, “REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL laying down common provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund, 57 on the rules concerning “e-cohesion,” including the definition and minimum requirements of electronic data exchange, the “only once” encoding principle (the same data shall be submitted only once by a beneficiary of a particular OP), and other details on how these e-systems are expected to work.80 Figure 39. A useful e-application platform is already in use in Romania (UEFISCDI) Source: http://uefiscdi-direct.ro/main/index.php? the European Agricultural Fund for Rural Development and the European Maritime and Fisheries Fund covered by the Common Strategic Framework and laying down general provisions on the European Regional Development Fund, the European Social Fund and the Cohesion Fund and repealing Council Regulation (EC) No 1083/2006, http://ec.europa.eu/regional_policy/sources/docoffic/official/regulation/pdf/2014/proposals/regulation/general/ general_proposal_en.pdf 80 See “Fiche no. 6, Implementing act on the rules concerning electronic information exchange with beneficiaries (“e-cohesion”), Version 1 – 10 April 2013, http://ec.europa.eu/regional_policy/what/future/pdf/preparation/1_fiche6_ia_e-cohesion_2013_04_09.pdf 58 99. Importantly, this proposal would require Member States to enable beneficiaries the option of electronic exchange – this is not an obligatory provision for beneficiaries, but countries are specifically allowed to make it a firm requirement. This is in fact the path that this report recommends for the Romanian ROP; otherwise, if the “paper” option remains a possibility, changing the mentality of some beneficiaries and MA/IB staff will become extremely difficult and the new system may not gain ground. The EC proposal further clarifies the term “all exchanges of information” to refer to information requirements applicable to the beneficiary “once a grant has been awarded, [which] includes reporting on progress, declaration of expenditure and exchange of information related to management, verifications, and audits.”81 Moreover, it notes that documents available through the computer system should suffice and beneficiaries shall be requested to provide original copies only through on-the-spot verifications and ad-hoc controls (for exceptional, risk-based situations). Again, this report goes further, and recommends the implementation of e-documentation throughout the project cycle, i.e., covering the application phase as well (as the UEFISCDI Direct Manager example provided above). 100. For its part, Romania has launched a process for developing a “mySMIS” application, which would serve as an interface with beneficiaries, complementing the current SMIS backend database. MySMIS has already been contracted by ACSI using technical assistance funding, although the exact details on when it will roll out and what functionalities it will actually include are lacking. Based on interviews with RDA representatives, piloting had not begun, as of October 2013, at least not at the regional level. But the quality of the request for services used in the public procurement procedure was very good and included a comprehensive list of requirements. What remains to be determined is the quality of the actual product and its implementation, which would benefit from ample time for consultations, beta testing, and piloting. Learning from the experience of developing SMIS, it is recommended to maintain service agreements with the software developer for a longer period of time, including assistance on demand for any issues that need to be addressed. Since the finalized form of mySMIS is not known at this point in time, the observations below remain relevant and summarize the main benefits of an e-ROP at various stages of the project cycle. What is more, if the mySMIS launch will be delayed further, the MA within the MRDPA should strongly consider developing its own components of a future electronic system. 101. As a first step, the introduction of electronic forms for the application period would reduce bureaucracy and associated costs, and help prevent beneficiaries from making mistakes, ensuring smoother completion of projects further down the line. A relatively simple system can be implemented, whereby the applicants would fill out an electronic form and attach supporting documentation. All forms should be foolproof, meaning that they should have predefined sets of acceptable values and required information. For example, beneficiaries would not be able to enter project indicators that fall beyond the acceptable program-level set and they also should not be able to submit the final form until all required documents are attached. Not only would this process reduce the amount of paperwork generated, but also it would significantly reduce the time needed for eligibility and verification checks, which could be partly automated. Smart software could analyze applications received and flag potential areas of concern for RDA staff, which would need to spend even less effort on this first verification. Depending on pilot results, the principle of four eyes may also be eliminated if the system proves to be sufficiently rigorous with an automated computer check doubled by one RDA expert verifying all submitted documents. 81 Ibid. 59 102. The evaluation process should also take place on an easily accessible electronic platform. Under this new setup, there is no need to pay the extra costs of transporting evaluators to the RDA headquarters for the desk evaluation phase. An e-based ROP would mean that evaluators could analyze submitted documents remotely, so a larger proportion of the budget dedicated to covering evaluation- related costs could go toward actual fees, supporting the goal of higher-quality evaluations. All interactions between evaluators, beneficiaries, and IB staff could take place through electronic channels during the desk evaluation phase, and the evaluation software could connect all these parties quickly and securely. It would also be easy to track when documents were submitted, including initial and updated versions, reasons for any changes, as well as how long a process is taking to identify bottlenecks and take corrective measures. A fully integrated system would also mean that evaluation results can be quickly organized and communicated to applicants, and procedures for contracting those qualifying proposals could also begin in a timely manner. 103. During the implementation phase, all required exchanges of information, including reimbursement requests and supporting documentation, would be submitted and verified electronically. This would avoid duplication of tasks for beneficiaries. For instance, by submitting the public procurement documents one time, they would no longer need to upload them again with every reimbursement request. If further verifications are needed, the documents could be easily found in the system by all verifying officers who are granted access to a particular project. As in earlier stages, formal communications between beneficiaries and MA/IB staff could be e-based to save time and ensure proper records of all exchanges. 104. Importantly, an e-based ROP does not mean that all in-person interactions would be eliminated. As argued in the report on MA-IB collaboration (Component I), evaluators should conduct field missions after the “desk review” evaluation phase to verify the situation on the ground and signal to applicants any potential errors (e.g., with the technical documentation). Before the contract is signed, original paperwork should also be verified by IB personnel to ensure that it fully corresponds to electronic copies submitted during the application stage. During the implementation phase, monitoring missions – in addition to their current mandate – will also need to ensure that all documents submitted electronically (receipts, progress reports, etc.) are exact copies of originals. Under this system, beneficiaries would be required to keep all original paperwork in an easily accessible location and make it available upon request. At the same time, not all MA-IB-beneficiary interactions can happen electronically. In-person trainings, for instance, have proved much more useful than online seminars. This is to say that electronic components of the ROP should not entirely replace the current mechanisms, but complement them whenever possible. 105. While the advantages of an e-ROP are obvious, potential challenges include beneficiary capacity constraints that would limit their ability to use electronic platforms for all key project steps. A few interviewees in RDAs have expressed concern that applicants and beneficiaries would not know how to use electronic documents and e-signatures. Based on field research observations, these concerns are not founded. Even smaller beneficiaries generally have proper IT equipment, often acquired with the help of EU funds. It is also true that beneficiaries who organize public tenders according to Government Emergency Ordinance 34/2006 are already required to upload documents electronically onto the e- licitatie.ro official portal, including complex technical documentation (e.g., feasibility studies) that potential bidders need to formulate their offers. Keeping in mind the profile of ROP beneficiaries (i.e., typically residing in urban areas), as described in an earlier section, it is expected that the vast majority of applicants and beneficiaries do indeed have the technical capacity to function in an e-based ROP. In any case, a module of specific trainings would also need to be implemented to show ROP applicants and 60 beneficiaries how to use the platforms, and a special technical helpdesk would also be useful in providing continuous on-demand assistance. 106. In addition to adequate capacity, a change in mindset is also needed to ensure the success of switching to an e-based system. As mentioned earlier, there is currently a strong bias toward relying on stamped, hard copy documents. A common understanding is required at all levels (MA, IBs, and especially beneficiaries) that an electronic document endorsed with an e-signature is as valid as a hard- copy version. Given that both EU and Romanian legislation allow for the use of electronic documentation and e-signatures, there are no obvious legal barriers against the switch to an e-ROP. Habits and mentalities can be tough to change, but the alternative of maintaining the status quo is a lot worse for beneficiaries struggling through overly bureaucratic procedures. If the switch to an electronic platform is decisive, meaning that the hard-copy option is only allowed in exceptional circumstances, people within the system will gradually accept the advantages of electronic exchanges and rely increasingly on this type of communication. Figure 40. An overwhelming majority of beneficiaries welcome the introduction of e-documentation 107. The good news is that beneficiaries seem to be very open to the introduction of an e-based platform. In all field interviews, virtually every ROP beneficiary praised this idea as a way to reduce bureaucracy and procedural delays, and as a solution for reducing the high amount of paperwork circulating between beneficiaries, IBs, and the MA. Beneficiaries also expressed their willingness to work with the IBs, the MA, and ACSI to develop optimal technical solutions that are aligned with the specific needs of the ROP system. Survey results confirm these qualitative findings: over 71% of respondents consider the introduction of e-documentation to be extremely and very useful, while less than 9% think this would be of little or no use. 108. For all its benefits and the positive reactions of beneficiaries to the idea of an e-ROP, technology can be a double-edged sword if not properly implemented. This was partially the case with the SMIS platform, which has generated significant additional work at the MA and IB levels because it was not perfectly aligned from the start with the system’s needs. Indeed, the sine-qua-non condition for a successful, functional e-ROP is the quality of the software developed and the extent to which beneficiaries and MA/RDA staff can test the beta-version with sufficient time to send suggestions for improvement. 61 E. Targeted Support: Engaging Marginalized Communities and the CLLD Approach 109. One of the key priorities for Romania’s development in 2014-2020, supported by both the EC and the Romanian Government, is the integration of marginalized communities and the promotion of inclusive growth. In particular, the Roma continue to struggle with poverty, discrimination, social exclusion, and lack of access to opportunities, despite the increasing attention they have received from the EU and other international donors. At the same time, Roma integration has to be a primary focus of any efforts aimed at sustaining Romania’s development, taking into account the community’s strong potential to make a contribution to the country’s absorption of EU funds and long-term growth. The numbers speak for themselves.82 The proportion of Romania’s population of 65 years or older will rise from 17% to 28% by 2040, while the Roma share of new labor market entrants is continuously growing. Today, the Roma make up to one in five new labor market entrants.83 However, only 42% and 19% of Roma men and women, respectively, have jobs – including informal employment. The average productivity of working age Roma men in Romania is estimated to be only 20% of that of the general population, and among Roma women even lower (12%). 110. Despite all this, ROP projects targeting Roma and other marginalized communities remain mostly concentrated on rehabilitating infrastructure like social centers. Without downplaying the importance of these initiatives at the local level, the reality is that the ROP has not generated substantial impact on critical issues like social integration, labor market participation, and school enrolment. There are multiple causes for the current situation, which is not necessarily limited to Romania. As the EC notes, “EU funds (in particular the Structural Funds) could be a powerful tool to improve the socio - economic situation of disadvantaged groups, such as Roma, but too little of the €26.5 billion allocated to support Member States' efforts in the field of social inclusion for the 2007-2013 period benefits disadvantaged Roma communities.”84 111. For one, the same factors that contribute to the marginalized groups’ current conditions also tend to prevent their access to opportunities related to structural funds, either as applicants for projects or as end beneficiaries. The Open Society Institute (OSI), under its “Making the Most of EU Funds for Roma” program, summarizes the main challenges that the community faces: “weak visibility of Roma issues at the local level, resource and expertise disadvantage, and overly bureaucratic funding procedures, which block access of the most deprived to rights and opportunities.”85 At the same time, the ROP has not been effective at prioritizing and incentivizing projects targeting Roma communities, despite the fact that the programming document recognizes that social exclusion is one of the greatest challenges faced by cities across Romania, “particularly [with respect to] the Roma population, characterized by chronic poverty, low level of education, and high school dropout rates.”86 There was no priority axis or sub-axis dedicated to issues specific to marginalized communities and no evaluation criteria tied to this area. The closest integration of such factors is the crosscutting theme of “equality of opportunity.” The box below briefly reviews why this particular requirement remained ineffective, despite including it as a distinct section in financing applications. 82 For related data, see “Roma Inclusion Is Smart Economics,” World Bank Europe and Central Asia Knowledge Brief, Joost de Laat and Christian Bodewig, April 2011 83 National Roma Integration Strategy 84 “National Roma Integration Strategies: a first step in the implementation of the EU Framework,” May 2012 85 See http://www.opensocietyfoundations.org/about/programs/making-most-eu-funds-roma 86 “Regional Operational Programme,” June 2007, p. 60 62 Box 1. The cross-cutting theme “equality of opportunity” has been ineffective in promoting Roma-focused projects Equality of opportunity was one of the main cross-cutting themes of the EU 2007-2013 budgeting period. It was also adopted by the ROP, as noted in the programming document: “All areas of activity that will be implemented in the field of regional development will respect the Community regulations and provisions regarding the contribution to the horizontal objectives (sustainable development and equal opportunities).” In practice, this has translated into inserting a required section of the financing applications that prompted applicants to explain how their projects ensure equality of opportunity – including nondiscrimination on the bases of race, gender, religion, disability, and age – in the design and implementation phases, in project management and activities, and in identifying target groups. This requirement is vague and does not specifically refer to marginalized communities. It is unclear to what extent a project’s evaluation took into account the content of this section and whether superficial explanations were penalized in any way. Moreover, interviewed beneficiaries noted that it is often difficult to fill out this section for certain project types that do not have a clear-cut impact on marginalized communities or other vulnerable groups. One study by NGO Coalition for Structural Funds, “112 for Structural Funds,” cited a project evaluator: “There are specific projects where you can’t write about sustainable development and equality of opportunity. One beneficiary was rehabilitating a county road… [Under the section on equality of opportunity] they wrote that once the project is over they will ask women, elderly people, and children in the community to sweep the side of the road and cut the grass.” Obviously, this is not what was intended by the programmatic requirement for this cross-cutting theme. An alternative solution would be to have dedicated funding for issues related to marginalized communities. There should also be criteria prioritizing these kinds of projects through the evaluation stage. 112. The argument above point to the need for improving the ROP’s programming to deliver more impact on marginalized communities in urban areas. The Government’s responsiveness to this issue is evident in the draft Partnership Agreement (PA), which will guide Romania’s 2014-2020 strategy for utilizing structural funds. The PA includes a thematic objective (TO) on promoting social inclusion and combating poverty (TO 9), and another one on investing in education, skills, and lifelong learning (TO 10). Also, marginalized communities, including the Roma and other vulnerable groups (e.g., children whose parents are working abroad), are specifically mentioned throughout the document. At the same time, the PA maintains the previous trend of channeling ERDF money – through the ROP – toward social and educational infrastructure. In truth, issues targeting marginalized communities can be addressed through the whole range of projects financed under the ROP, from integrated development plans (e.g., infrastructure connecting excluded communities to the main social and economic life of a city) to private-sector development (e.g., SMEs that seek to catalyze employment opportunities for vulnerable groups). It follows that progress on such issues will depend on adopting targeted selection criteria and encouraging beneficiaries to engage Roma and other marginalized communities through their ROP projects. As the report on “Competitive Cities” argues, improving the conditions of these groups requires targeted, sustained measures for promoting development. The same goes for shaping the type of projects coming through the pipelines. 113. By the same token, MA/IB teams should develop targeted support for members of marginalized communities to apply as potential beneficiaries of ROP funds. It is expected that barriers to accessing information are higher for these groups, so efforts to provide assistance and take, say, a 63 young, inexperienced Roma non-profit from project design through implementation will require additional resources. The OSI’s model of the “project generation facility (PGF)” and complementary actions for building the capacity of applicants for structural funds are worth leveraging, as is the experience of the Romanian Social Development Fund (RSDF). There are also many other international and local non-profits that are increasing their efforts aimed at integrating marginalized groups in Romania. The MA-ROP could make a substantial contribution to the conversation by addressing the urban dimension of poverty and exclusion, along with the benefits of economic growth resulting from improved spatial integration. It is beyond the scope of this report to recommend a particular set of tools for engaging potential applicants from marginalized communities. As noted earlier, these will need to be carefully researched, planned, and customized to the Romanian context, while making sure that duplication with other OPs are avoided. One aspect worth mentioning in the context of the current report relates to the recommendation for switching to an e-based ROP. The MA should take special care that marginalized groups have access to this new system and know how to access and use it (e.g., submitting electronic applications, reimbursement requests, etc.). Simply put, targeted capacity building efforts may be needed to enable marginalized groups to take advantage of opportunities available under an e-ROP. 114. Along the lines of engaging beneficiaries from the ground up, the MA-ROP and the IBs should work together to evaluate whether the Community-Led Local Development (CLLD) instrument is appropriate for the ROP 2014-2020. Current EU regulations make this optional (except for the European Agricultural Fund for Rural Development – EAFRD). But the CLLD holds great promise for a program like the ROP, which is already more decentralized than other OPs in Romania and is well positioned to take advantage of its closer presence to beneficiaries on the ground. More generally, CLLD approaches are expected to increase the effectiveness and efficiency of projects and service delivery by using a bottom- up approach that involves active participation of communities, which not only increases the ownership and relevance of projects by identifying stakeholders’ priorities and their ability to maintain/operate investments, but also contributes to their social inclusion through empowerment. The following sub- section presents a preliminary assessment of the CLLD instrument’s potential adoption under the ROP 2014-2020. Relevance of the CLLD approach for ROP 2014-2020 114. The current analysis on the relevance of Community Led Local Development (CLLD) approach within the programmatic framework of ROP 2014-2020 relies on two key considerations: the legal and administrative provisions, and technical specificities. As for the former, the Romanian Partnership Agreement (PA) for the 2014-2020 programming period establishes several general criteria on how CLLD is envisaged to be used effectively (an overview follows below).87 As for technical specificities and their implications for future programming, the inner principles of this approach depend on a cross-sectoral and multi-actor method, which is very much based on concrete involvement of different stakeholders in the design and implementation processes. It is therefore relevant for beneficiaries, but also for the mechanisms of communication and collaboration between them and MA and IBs. 115. When carrying out this analysis it is important to take into account the key implications the CLLD approach might have on the programming of ROP 2014-2020 and the scenarios to address potential risks for effective implementation. They are reflected in the structure of this sub-section: 87 First draft sent Oct. 1, 2013, available at http://www.fonduri-ue.ro/res/filepicker_users/cd25a597fd-62/2014- 2020/acord-parteneriat/Partnership_Agreement_01.10.2013.pdf 64  Provisions of the current PA;  2007-2013 context, with a particular focus on the experience of DG AGRI (EARDF) with LEADER;  Lessons learned applicable to the Romanian CLLD; and  Critical factors for success. The CLLD approach in the Romanian Partnership Agreement 116. The effective involvement of communities and, particularly, of vulnerable groups, in consultations and the design of interventions they are subject to, constitutes the very backbone of an effective bottom-up approach. Accordingly to the PA, the CLLD is an “important tool for Romania to reduce the territorial discrepancies through the involvement of local communities.”88 The PA plans to make effective use of the CLLD approach in several OPs during the 2014-2020 cycle: i) in rural areas, through EARDF; ii) in deprived areas within urban centers, through ESF and ERDF; and iii) in coastal areas (both seaside and rivers), through EMFF and ESF. Particularly for the urban deprived areas and urban development, the PA intends to cover an indeterminate number of towns and cities from Romania with deprived areas and neighborhoods, tackling the following issues: 1) risk of poverty or exclusion, particularly of vulnerable groups; 2) people living in poverty, particularly in regions and territories facing high rates of poverty; 3) former participants in ESF-funded operations, who are to earn their qualification in social services upon leaving these operations; 4) social housing infrastructure; and 5) social and medical integrated services. 117. Despite the experience of the LEADER/CLLD approach in rural and fisheries areas, there is neither consistent practice nor evidence on what a CLLD approach in urban areas would look like in practice. Therefore, the PA only provides detailed information on how a CLLD approach will be implemented in rural and coastal areas, based on previous experience, but is still vague on how to effectively implement the same approach in urban areas. In fact, at least 13 recommendations of the PA tackle the challenges of continuing LEADER89 and FLAG,90 including some examples on potential innovations to be developed, but when referring to urban areas the PA just simply states that “CLLD will help tackle the development challenges (…) with the aim to increase social inclusion and reduce poverty among marginalized communities within Romanian cities.”91 The LEADER approach in Romania, 2007-201392 118. The LEADER approach in Romania has posed a significant challenge to the public institutions in charge of managing European funds, particularly in what concerns the design for implementation of local development activity plans by the LAGs. One of the important issues appears to be related to the bottom-up approach, a sine-qua-non condition of successful development and implementation of local strategies. Even when local stakeholders manage to gather and generate the basis of this bottom-up approach, they have to face other challenges at central level. A closer look to LEADER in the respective period indicates that, despite considerable progress during 2007-2013, Romania also registered significant delays and the first contracts on projects through LAGs were signed only in May 2013. 119. There are seven essential principles of the LEADER approach in terms of programming and implementation: i) the bottom-up approach, which implies active participation of the local population in 88 Ibid., 145 89 Liaisons entre Actions de Développement de l'Economie Rurale 90 Fisheries Local Action Groups / in areas without fisheries they are simply LAG 91 Partnership Agreement, p. 146 92 The data referenced in this section are primarily from: http://www.pndr.ro/content.aspx?item=2243&lang=RO 65 the programming process, in the decision-making process, and in the implementation of local development strategies; ii) the territorial approach, which implies the efficient use of local resources from a certain territorial area and activities developed in an integrated manner through a shared vision; iii) the partnership approach, which requires the development of effective public and private partnerships in the specific area of the LAG; iv) the networking approach, which sets up articulated links among partners; v) the integrated approach and multi-sectorial strategies; vi) the innovative character; and vii) the cooperation between the created partnerships. 120. The LEADER approach was applied to Romanian rural territories using the national definition for a rural area, plus a number of 206 small towns of up to 20,000 inhabitants . Including cities as eligible territory, LEADER ensures the necessary territorial coherence and the critical mass in terms of human, financial, and economic resources for an articulated strategy to support sustainable local development. In other words, the eligible territories are constituted of small size areas with homogenous characteristics and strong local identity, but also include sufficient critical mass of the population and the financial and economic resources to support viable local development strategies. Adjacent localities may be included and there is no constraint in terms of administrative limitations of the local administrative units – the area may include several counties or even regions. 121. As eligible beneficiaries of LEADER, the LAGs include a balanced representation of public and private stakeholders. One of the key eligibility criteria is that at least 51% of the members in LAG are private stakeholders (economic, academia, social or civil society stakeholders). This leads to a representation of up to 49% of the public stakeholders in the LAG. The main role of the LAG is to develop the Local Development Strategies (LDSs), to launch the call for proposals, and to carry out the selection of the proposals received from eligible beneficiaries. The chosen projects are then funded based on the approved LDS by the Managing Authority for National Rural Development Program. 122. The financial allocation for LEADER within the NRDP 2007-2013 was initially established, based on rural development regulations’ provisions, at 2.3% of the total allocation of 10B EUR. If this was the case in 2008, after a large number of applications were submitted, the financial allocation has been supplemented with 2.5 % in 2010, reaching a 5% of the total public allocation for the NRDP 2007-2013 (453 million EUR). As for the management and implementation structures of LEADER, the Ministry of Agriculture and Rural Development (MARD) is the MA-NRDP and is responsible for the training, selection, monitoring and evaluation, and advisory support – all such services are provided through its county-level structures. The Payment Agency for Rural Development and Fishing (PARDF) is in charge of financial-related issues within the LEADER program. 123. Several key figures about LEADER in Romania are worth highlighting. There are 163 LAGs established, with over 80% of the eligible area covered. 9.4 million inhabitants live in the targeted areas, with 55,000 – 57,000 inhabitants per area, on average. 2.85 million EUR is the maximum eligible allocated amount per LDS, with a share of 20% for administrative costs for each LAG’s functioning. There are 160 contracted projects so far (with a total value of 3,500,000 EUR) and over 100 LAGs have requested advances for operating expenditures (up to 20%). The geographic coverage of LEADER is presented in the map below. 66 Figure 41. Coverage of LAG during 2011-2012 timeframe LAG LAG Source: NRDP @ http://www.pndr.ro/content.aspx?item=2243&lang=RO 124. Regarding the project proposals submitted by eligible LAGs, a total of 2,633 projects are registered for approval by the PARDF. They cover: i) investments in rural development infrastructure in rural areas (local roads, water supply networks, sewage system, social centers – 311 projects); ii) support for semi-subsistence farms – 360 projects; iii) financial support for young farmers – 85 projects; iv) investments in agriculture holdings – 78 projects; v) investments in non-agricultural activities – 75 projects; vi) vocational training for farmers – 30 projects; vii) investments in agricultural and forestry infrastructure – 18 projects; and viii) investments in rural tourism and agro-tourism – 14 projects. As of October 2013, the used funds ratio for LEADER is 22.43%, representing 16,370,492 EUR. Lessons learned for Romania’s CLLD (2014-2020) 125. Looking back at the very inception phase of the LEADER implementation in Romania, several stages proved to be key development steps. Specifically: i) the pre-selection of the territories in order to select their representatives for the LEADER training, in 2006, using PHARE twinning funds, which led to a total of 121 files selected and training provided through 2 sessions as a means to raise the awareness on the bottom-up approach; ii) the drafting and negotiating of the LEADER approach within the NRDP during 2006-2008, with a final approval from the EC in February 2008; iii) the training for local stakeholders in two phases, for a total of 1,878 participants and the provision of financial support, within a third phase, for drafting the LDSs (in 2010); and iv) the selection of 81 LAGs in 2010 and an additional 82 LAGs in 2012, while the effective implementation of LDSs started in 2011. 126. Despite the constraints in effectively implementing projects through LAGs, lessons learned from the LEADER experience can be grouped into five categories, as noted below:93 93 Most of the information in this section comes from the work carried out by Ms. Cornelia Mihai, World Bank consultant in Romania and Croatia, in February and October 2013. 67  Lessons learned regarding the technical details of the program: The complex approach of LEADER may address only partially the existing situation in a rural area, mainly because LAGs’ institutional capacity is low. There is a need to improve the evaluation of eligibility criteria for LAGs and to focus on the quality of LDSs. The criterion of shared responsibility within LAGs between public and private stakeholders should be reinforced by an additional criterion of some previous experience in a LEADER type approach. The conditions for eligible areas (communes and villages and small towns of up to 20,000 inhabitants) should be maintained, but financing of the selected projects by LAGs coming from the urban area should be done as part of those respective LAGs. There is also a need for better prioritization of interventions, not necessarily opening all axes for the LAGs. Finally, there is also a need to improve knowledge exchanges, in a context where training is useful but has less impact than practical workshops.  Lessons learned regarding the general framework of implementation of the program: The MA and PARDF should have dedicated structures for LEADER, along with sufficient staff, especially at the local level, where the direct contact with the local actors remains vital. The staff at both central and local levels should be continuously trained and updated with relevant information; the same is needed for the managers of LAGs, but also for projects’ beneficiaries. Both training approaches may be reinforced through a broader public awareness effort. The national legislation needs to be drafted and updated to respond better to the requirements of LEADER – the status of LAG, access to funding, use of the 20% pre-financing, etc.  Lessons learned in relation to the implementation system. There is a need to simplify the actual implementation system. A single institution should work with the LAGs in their action plan’s implementation. While rather small projects receive financing, the implementation system remains complicated (administrative bottlenecks, complex financial rules, etc.). The LAGs as a whole have, generally, limited capacity and expertise to be leaders in their area, and there is still strong involvement of mayors. Doing LDS work is valuable step in the process but there is a need to secure funds for the implementation of such strategies in a context where it is very difficult for rural communities to find the appropriate financial budgetary resources. There is no clear articulation between, on the one hand, the actions and priorities established in the LDSs (which appear to be representative for the interests of the local community) and, on the other hand, their membership structure. The whole financial system can be simplified further in terms of reimbursement rules, bank guarantees, bank loans, LAG liquidity, etc.  Lessons learned in relation to the evaluation of local development strategies. The selection procedure for LAGs should allow for improving local strategies before signing the contract. The MA should perform periodical assessments of the LAGs’ performance, with the option to reduce the initially planned budget, if needed. More emphasis should be on LDS indicators, on added value and innovation, on quality of LDS versus geographical coverage and versus areas lagging behind.  Lessons learned on the activities carried out by the LAGs. The legal status of LAGs should be maintained and the private – public share in LAGs should comply with the rule of 51% to 49%. The training of the personnel needs to focus on very practical issues, including on learning how to set up relevant indicators for the measures included in the LDS and collecting appropriate data to measure such indicators. The MA should improve its function to support and advise the LAGs in achieving the action plan included in the LDS. 127. In short, there are several recommended improvements to the LEADER framework, which would also hold for the future CLLD: 68  Simplify the institutional framework in order to avoid overlapping responsibilities among stakeholders, particularly in the area of supervision mechanisms. In this respect, only one implementation institution, either the MA or the PARDF, should be involved in working directly with the LAGs in the implementation of their action plans. Moreover, the PARDF should probably keep only the payment function.  Simplify the implementation procedures through a thorough analysis of redundancies and the relevance of each bureaucratic procedure.  Conduct an in-depth evaluation of LEADER to tackle the following issues: i) how to improve the role of local actors depending on the size of the LAGs; ii) how to better identify the instruments or the methods for collecting ideas; iii) how to decide on the pros and cons of mono-fund or multi-fund approaches; iv) how to improve the national legislation applicable in this area; v) how to assess the impact of the administrative regionalization and decentralization (ongoing); vi) and how to better define the role that local authorities should play in implementing the future LEADER measures. Critical success factors for an effective CLLD approach in urban marginalized communities 128. A CLLD approach in urban areas is currently in its infancy . Its effective implementation in the future 2014-2020 ROP will be a complex challenge for both the EC and Romania. The brief analysis above hints at the fact that the EC still needs to develop a coherent and detailed framework for how the CLLD instrument would work in practice. In the particular case of Romania, there are also systemic challenges that need to be addressed in a strategic and sustainable manner. As ROP stakeholders refine the framework guiding the CLLD’s future implementation for the 2014-2020 programming period, they should keep in mind several critical areas. 129. A first imperative is “simplify, simplify, simplify.” All the recommendations for simplifying processes and procedures (as found in this report and in the assessment of the MA-IB collaboration for the next ROP) should apply to CLLD interventions. The reason is straightforward: CLLD-financed projects will generally be small and the capacity of the future LAGs will remain limited, at least for the 2014-2020 programming period. In Ken Miller’s terms from “Extreme Government Makeover” (see the Component I report on the MA-IB collaboration), CLLD projects need to go through a dedicated, shorter pipe, without the convoluted procedures that tend to slow down the absorption of ROP-financed projects. Otherwise, there is a significant risk that the new LAGs will be unable to shoulder the burden of implementing projects under the next ROP. 130. Second, even under a simplified institutional and operational framework, there is a need to dedicate special efforts to boosting the capacity of the future LAGs. The LEADER experience teaches that it is challenging to set up these groups and make them functional. In addition, marginalized communities face special challenges that threaten to undermine the formation and development of LAGs. Specifically, members of these groups generally have a very limited access to knowledge and, perhaps more importantly, there are certain social dynamics and rigid hierarchies within these communities that may prevent a fully participatory approach. Extensive preparations on the ground will be needed to explain the benefits of LAGs, how they function, and how each member of the community will be able to play a role in future projects. The report on Competitive Cities, also part of the World Bank’s Regional Development Program in Romania, includes a more in-depth discussion of the potential support mechanisms needed to address the issues specific to marginalized communities in Romania. 131. Third, the future CLLD approach will need to clearly define the key technical details of the LAGs. For example, it is unclear which area a particular LAG may cover within an urban setting, how 69 many inhabitants it should target, and what kind of projects it should support. These aspects will need to be established before launching the implementation of the instrument and, once in place, the rules should not change during the game. 132. Fourth, monitoring and evaluation (M&E) are crucial components that need to be in place from the very initial stages of programming, not just as ex-post processes. Their articulation within a coherent strategy, doubled by strong political will and appropriate allocation of funding, appear to be key ingredients of success. Moreover, there is enough evidence to help design future interventions and inspire decision makers, based on the lessons learned for comparable interventions. In addition to the LEADER experience in Romania and beyond, the World Bank has the knowledge and the experience of such types of approaches from its Community Driven Development (CDD) programs, and may constitute a reliable source of support for the future. The ideas that follow are only a preliminary reflection of potential synergies in this area. 133. From the CDD perspective, social mobilization can only be achieved if capacity is developed at all levels, from beneficiaries to decision-makers. Any intervention must rely on a solid needs- assessment process, which will help not only with identifying the needs but also prioritizing them, including investment choices. The community and local development plans represent a crucial success factor: they will need to be articulated within the wider strategies, as a means to ensure effectiveness and efficiency. The fund flow and institutional support arrangements need to be tested and refined through appropriate M&E and control mechanisms. Monitoring and evaluation need to be participatory, as they will reflect not only accomplishments of results but mainly the relevance, the effectiveness, the efficiency, the impact, and the sustainability of interventions. There should be in place sound governance arrangements, along with easy access to information and innovations in accountability relationships and grievance redress mechanisms. 134. As further synergies between the CLLD and CDD approaches are explored, one of key topics is related to making the programs and process easy to evaluate. The difficulty of such task is determined by the very high complexity and diversity of CLLD approaches, but also by the lack of an articulated framework to regulate monitoring and evaluation. For instance, in the CDD approach, demonstrating social and economic impact of the intervention is crucial and its achievement is carried out through rigorous impact evaluations. Acknowledging the diversity of interventions, the World Bank has selected six key areas of interest for the evaluation, with the purpose to further detail relevant indicators for each:  What is the poverty or socioeconomic welfare impact of CDD programs?  Who benefits from program interventions?  Do the interventions effectively reach poor areas and poor households?  Do the programs improve access to and use of basic services?  Do they improve social capital (using the standard proxy measurements for social capital – trust, collective action, association, groups and networks)? and  Do they improve local governance (participation in local meetings, satisfaction and increased confidence with government officials, awareness of program activities, etc.)? 135. Investing in evaluation is particularly important when using an experimental or quasi‐ experimental design based on significantly large samples that allow one to claim a high degree of 70 internal validity, with control groups. Evaluations are also the only means to ensure the funds are spent in a relevant, effective, and efficient manner, and the interventions they are used for have impact on the most vulnerable, in a sustainable manner. From the World Bank’s experience, impact evaluations of CDD programs showed that these interventions: i) have positive economic welfare outcomes; ii) register significant improvements in a range of schooling and health outcomes; iii) are geographically targeting mainly the poor and worst-off population (i.e., they are generally pro-poor); and iv) encourage participation, which improves civil works quality, maintenance, and sustainability. There is mixed evidence on impacts on social capital. 136. The good news is that Romania has had its own CDD programs from which it can now extract the best lessons and apply them to the design of the CLLD approach. Between 1999 and 2012, the Romanian Social Development Fund (RSDF) has implemented over 1,400 projects (small infrastructure investments, investments in educational facilities, investments in social facilities, investments in health facilities, small business grants, etc.), in both rural and urban communities, for a total value of around 100 million USD. At global level, the World Bank’s CDD portfolio surpasses 40 billion USD (see figure below), making this a tried and tested method, with concrete, positive results. Figure 42. WB CDD projects from 2000 to 2013 137. Finally, on the financing side, the Partnership Agreement provides little information on envisaged schemes for funding the CLLD approach. Despite this challenge, in order to better prepare for the future programming, it could be useful to look at the specific eligibility criteria for costs associated to CLLD and the broader funding framework within the whole ESF and its operational programs. The costs of preparatory support are eligible for CLLD interventions, including the implementation of operations under the Local Development Strategies. Moreover, the preparation and the implementation of the LAGs’ cooperation activities are also eligible. The PA also needs to set out 71 implementation and coordination structures for CLLD and to anticipate the funding mechanisms. The member state needs to present the main challenges it intends to tackle, the key objectives and priorities for CLLD (general objectives, thematic focus, and target groups) and the types of territories reflecting strategic choices. There is also a need to approximate the planned budget allocation for each Fund and the role of each ESI Fund in different types of territories. If the LDS will not be supported from multiple ESI Funds, it should be explained how the integrated approach will be achieved at local level. 138. Last but not least, the following critical factors need further consideration when choosing a multi-fund approach: i) the agreement between the funds over the types of areas and projects; ii) common criteria for the lead fund; iii) common calls for preparatory support and final selection of areas and strategies; iv) common criteria and procedures for selection of areas, strategies, and partnerships; v) agreed criteria for selection of projects; vi) agreement over the role and the function of the LAGs and the management authorities; vii) separate but well-coordinated plans for monitoring and evaluation; and viii) acknowledgment of the high degree of difficulty in implementing a multi-fund approach in the absence of a common intermediate body. 139. In conclusion, the CLLD can be a powerful tool for ensuring improved support for marginalized communities through the ROP 2014-2020. Many details are still in flux and will need to be carefully thought through before the start of the upcoming programming cycle, while keeping in mind the time factor: LEADER teaches that forming the LAGs and getting them up to speed (i.e., reaching a phase where they implement projects) is a complex, lengthy process. To minimize the risk of funds decommitment, swift actions will be needed, along with a keen focus on the five priority areas listed below:  Simplification of procedures;  Capacity building of future beneficiaries;  Definition of the technical details (e.g., size of intervention areas for each LAG, structure, etc.);  Monitoring and evaluation mechanisms;  Financing arrangements. 72 V. Pre-Application 140. In the pre-application phase, assistance to potential applicants should target two key areas: programming and pipeline development (i.e., awareness-raising and project development support). In terms of timing, these activities should start as early as possible to prepare for the 2014-2020 programming period. Unlike other phases in the project cycle, which have clear starting points, certain pre-application support activities can commence before the program is officially launched. This is true of programming, which is well under way through the regional working groups and the Regional Development Plans currently prepared by each RDA. As soon as the program design is finalized awareness-raising activities should actively engage potential ROP applicants, particularly those less likely to find out about available financing opportunities (e.g., marginalized groups). At the same time, information and training sessions at the MA and IB levels can begin to prepare future applicants on how to prepare and submit zero-error, high-quality applications further down the road, including through targeted support for pre-feasibility studies. 141. Certainly, the stakes are high: effective programming and pipeline development are more important than ever for a successful ROP 2014-2020. If Romania’s implicit priority was to absorb EU funds as quickly as possible in 2007-2013, the focus is now shifting toward a more balanced mix between two priorities: absorption and impact. After going up the learning curve during the pre- accession period and the first post-accession cycle, there is reasonable certainty that the ROP faces a more manageable risk of funds decommitment. This is true of 2007-2013, based on interview feedback and recent data on absorption, and there are positive expectations that it will remain true going forward as the MA and the IBs build on the experience accumulated to date. That said, effective, swift absorption should remain a priority for Romania’s ROP, and many of the recommendations in this report and in the previous one on the MA-IB collaboration emerge precisely from this desideratum. Additionally, greater impact can be achieved through more strategic selection models (the focus of another ongoing World Bank assessment) and from improved programming and pipeline development, which this chapter addresses. Figure 43. Balancing absorption and impact Absorption Impact • Appropriate focus for 2007-2013 to reduce • First priority for 2014-2020, after the risk of funds decommitment 2007-2013 learning curve • In hindsight, optimal decision to prioritize • Impact and value for money as the quantity and speed new key objectives 73 Beneficiary Engagement in Programming 142. Programming refers to the process of establishing the strategic directions and implementation framework for structural funds in Romania generally and, in particular, for the Regional Operational Programme (ROP). The fundamental question revolves around the level of beneficiary engagement in this process, including through inputs provided into key documents like the Regional Development Plans, the ROP 2014-2020 programming document, and the Framework Document for Implementation (FDI). Involving potential future beneficiaries in these activities does not only lead to better outputs that incorporate the direct feedback of those who actually implement ROP projects, but also ensures that these critical stakeholders are fully informed of the program’s objectives, requirements, and procedures, as early in the process as possible. Activities for engaging potential beneficiaries include formal and informal exchanges – primarily local and regional-level meetings for specific thematic areas. 143. Romania’s programming framework for structural funds shows a number of strong points that decision-makers can build on going forward. First, it is praiseworthy that, from the beginning, Romania chose a bottom-up model for defining programmatic priorities for structural funds, essentially adopting the lessons learned by other Member States that have shifted to more participatory approaches. These have proven successful in building a common development vision, involving a range of public/private/nonprofit actors, and strengthening collaboration among various levels of government. Second, the Romanian ROP follows a logical sequence of activities, from the identification of regional development needs and opportunities through the establishment of thematic working groups, and the generation of project ideas. The current programming methodology is also appreciated as clear and thorough. 144. The planning process for 2014-2020 has been reasonably functional to date, engaging beneficiaries through an established platform for dialogue and consultation. Particularly major public beneficiaries who are expected to continue to absorb a large proportion of ROP funding going forward (e.g., growth poles city councils, county councils, etc.), have been closely involved in programming activities. Importantly, the June 2012 Government Memorandum specifically notes that “until the establishment of new programs and the designation of new managing authorities, the preparation of future programming documents will be ensured by current managing authorities.” This is particularly useful for ensuring some degree of predictability and ownership, particularly in the context of the evolving conversation on Romania’s regionalization, which has introduced additional variance into the system. Coupled with the lack of decisive agreement on the EU budget, such potentially sweeping changes could have given beneficiaries (as well as MA/IB staff) reason to postpone proper preparation of the next period. In practice, at least in what concerns the ROP, things have moved forward, albeit at a slower pace than in 2007-2013. 145. In parallel, at the European level, a couple of new, critical developments have shaped the current thinking on structural funds and cohesion. First and foremost, the Europe 2020 strategy established three priorities around smart, sustainable, and inclusive growth, each supported by several flagship initiatives (e.g., Youth on the move, European platform against poverty, etc.). Five fundamental objectives cover employment, research & development, climate change and energy efficiency, education, and poverty, backed by specific targets like 3% of GDP invested in R&D and at least 20 million people fewer in or at risk of poverty and social exclusion. While Europe 2020 significantly shapes EU- level thinking, its objectives remain vague, and it is not entirely clear how they were selected as critical to the Community’s development at this point in time. Particularly for interviewed beneficiaries in the field, there is a sense that the EC has certain goals in mind, but there is little visibility into the process of 74 defining and prioritizing them, which sometimes results in a lack of ownership. In the words of one interviewee: “We are aware of what Brussels wants, we know the Europe 2020 strategy and what it seeks to achieve. We just feel like we did not have much say into it all and now we need to align our own plans at the local level with priorities set elsewhere.”94 146. Mirroring the main priorities of the Europe 2020 strategy, a legislative package was adopted by the EC on October 6, 2011, establishing 11 thematic objectives. These were the foundation for the Common Strategic Framework (CSF), presented by the EC on March 14, 2012, and designed to enable more impactful spending of structural funds. As noted by the EC, “national and regional authorities will use this framework as the basis for drafting their 'Partnership Contracts' with the Commission, committing themselves to meeting Europe's growth and jobs targets for 2020.” The next step in the process is the drafting, evaluation, negotiation, and signing of the Partnership Agreement (PA) with each Member State, the document that is meant to replace the NSRF 2007-2013 for the next programming period. As of June 2013, a draft PA by the MEF is open for public debate. Key stakeholders, including current and future beneficiaries of structural funds, are asked to provide feedback. 147. Previously, the EC had prepared position papers customized for each Member State, summarizing the Commission’s views on funding priorities, challenges, and recommendations for 2014-2020. This document has been the best indication of what the EC expects from Romania in terms of how it will use available structural funds over the next seven years. As it specifically notes in its introduction, “the purpose of [the] position paper is to set out the framework for dialogue between the Commission services and Romania on the preparation of the Partnership Agreement and Programmes.” The paper goes on to say that “the Commission proposes to group and limit EU funding to key challenges outlined in [the] position paper.” Indeed, the position paper has been rightly cited by both MA and RDA representatives as the basis of current programming efforts. 148. Insights from conversations with beneficiaries show a mixed picture regarding their awareness of the broader, evolving conversation around EU structural funds, cohesion objectives, and priorities for 2014-2020. At one end of the spectrum, public authorities with dedicated technical teams for EU-financed projects are generally well informed and knowledgeable of the current agenda and key documents (e.g., Europe 2020, the EC position paper for Romania, etc.). At least 1-2 key people in these departments demonstrate a keen understanding of the fact that their success in how they prepare projects for the future programming period depends on keeping up-to-date with these broader developments. The same assessment holds for the top consulting firms that are active in this field. At the other end of the spectrum, local authorities of smaller towns, representatives of non-profits, and most private-sector beneficiaries appear much less conversant on such issues. This does not imply that all potential beneficiaries should be equally aware of the broader agenda, but it does underscore the importance of contracting competent consultants for those ROP applicants who do not dedicate internal resources toward staying fully informed. 149. Partly as a result of internal political changes, partly due to delays in the adoption of the EU budget, Romania has started the programming for the 2014-2020 cycle relatively late, particularly compared to the 2007-2013 period. In June 2012, the Government approved a “Memorandum on the Approval of Actions and Documents for the Preparation of Accessing and Implementing European Funds in 2014-2020.” Based on this framework, the Ministry of European Funds (MEF) created a set of methodologies that describe the main elements of programming documents, as well as the stages of the 94 Interview with public-sector ROP beneficiary, North-West Region 75 planning process, including methods for analyzing development needs, formulating strategies, and consulting stakeholders. Like the Ministry of European Integration in 2007-2013, the MEF is responsible for the overall coordination of the programming process, including related technical assistance projects and, additionally, is in charge of developing the draft Partnership Agreement (PA). The Government has responded to EC requests in this area promptly: for instance, four days after the EC presented its position paper, the Government adopted a note on fulfilling ex-ante conditionalities and, on January 31, 2013, it participated in a first discussion with the EC on the position paper and its implications. This dialogue has continued over the past months and the PA is expected to be approved by the EC by January 2014, according to the official schedule. This timeframe is still relatively delayed compared to other Member States (for example, the Czech Republic, Belgium, Poland, and Slovenia aimed for finalizing their PAs at the end of 2012).95 150. The overall consultative system for developing Romania’s priorities for 2014 -2020 is similar to the previous programming period and involves major public beneficiaries at the level of thematic working groups. At the national level, an Interinstitutional Committee for Elaborating the Partnership Agreement (ICPA) draws from the work of 12 sub-groups: 10 thematic consultative committees (TCCs), spanning topics like transport, environment, education, health, tourism, administration and good governance, etc.; and 2 committees on regional development (CCRD) and territorial cohesion (CCTC). Each committee is coordinated by central authorities (i.e., line Ministries). The CCRD is most directly tied to the future ROP, although some of the other sub-groups (e.g., education, health, tourism, territorial cohesion, etc.) may well influence the program’s priority axes and strategic directions. The CCRD is the main link between the national and the regional levels, and MA-ROP ensures its Secretariat. At the regional level, as for the preparation of the 2007-2013 programming period, there are eight Regional Committees for Elaborating the RDPs (RCRs). They maintain consultative roles and are coordinated by the region’s respective RDA, bringing together a wide range of stakeholders, including key future beneficiaries (e.g., city halls, county councils, etc.). As for 2007-2013, RCRs and RDPs draw from the work of several working groups (WGs) at the level of each region, including RDA staff, major beneficiaries, and socio-economic partners. Figure 44. The current structure for 2014-2020 programming of EU structural funds in Romania 95 “A Balancing Act: Managing the Programmes, Closure, and Preparations for 2014 -2020,” IQ-Net Review Paper No. 31 (1), November 2012. 76 151. Interviewed ROP beneficiaries and RDA staff who have participated in WG sessions for 2007- 2013 and more recently for 2014-2020 appreciate them as useful. Indeed, they are the main institutionalized channel for beneficiaries to influence how the ROP funds are channeled toward priority areas. Importantly, the group of beneficiaries involved in WG discussions is usually limited to public authorities at the county level and from relatively major cities. While the process is open to feedback from a wider range of stakeholders, not all potential beneficiaries have had the resources to participate to these interactions in person. The de facto solution has been to involve other actors who could speak on behalf of different interest groups, including prominent non-profits, representatives of the business environment, and academic experts. 152. At the same time, for the 2007-2013 programming period, the engagement of beneficiaries varied throughout different stages of the process: specifically, they played a much more active role in shaping the ROP programming document compared to the framework document for implementation. Going forward, it is recommended that working groups of beneficiaries stay engaged throughout the entire programming process, which should aim to become increasingly transparent and participatory. This is meant to ensure that all potential beneficiaries are fully involved and continue to have a say into what kinds of projects can be financed through the ROP 2014-2020, as well as the relative allocation of funds for different axes. It is also recommended to ensure full alignment between the RDPs (essentially a synthesis of beneficiary priorities) and the ROP programmatic documents, which would likely be easier under regional-specific ROPs, in the broader regionalization context. Last but not least, beneficiary preferences should also be reflected in the program’s implementation framework (e.g., procedures, definition of eligible vs. ineligible expenditures, etc.). 153. In particular, decision-makers responsible for drafting the PA, the ROP, and related documents should begin experimenting with online consultation tools, following the example of other Member States. For instance, between March and April 2012, the UK Department for Business, Innovation and Skills asked people to offer their feedback on the delivery of future structural instruments, including: objectives; geographical focus; administration and management mechanisms; integration across funds; and “other issues.”96 Online consultations have been used in other countries as well, including Germany and Scotland, and they promise to increase the transparency of the process and make it much more accessible to various stakeholders. In fact, some interviewed beneficiaries noted that it is difficult to participate in the WG sessions and other in-person consultations that may not be close by, particularly given the high time pressures in the last stretch of the current programming period (i.e., with many projects under implementation, others in preparation, etc.). In this sense, online feedback instruments would be very welcomed and, if the participation rate to the survey conducted for this report is any indication of expected efficiency of such tools, there is a good likelihood that beneficiaries will respond positively and provide valuable input. Pipeline Development 154. After the program-level objectives and strategy are set, MA/IB assistance efforts should concentrate on: (a) promoting the financing opportunities; and (b) working with potential applicants to develop viable ideas and create the conditions for successful applications. On the former, reaching 96 See Stefan Kah, “Planning for the Future while Maintaining Focus on Spending,” IQ -Net Review Paper No. 30 (1), June 2012, p.39 77 potential beneficiaries is part of the program’s communication plan at the MA and IB levels, and includes various information channels, along with objectives, indicators, and budgets. As for actual project development support mechanisms, these can vary from limited – e.g., horizon scanning (i.e., asking likely future beneficiaries to provide a list of co-funded projects they are planning to develop and submit for consideration) – to more extensive (consultations, advisory sessions, field visits, etc.). The more complex a project is expected to be, the more extensive the pre-application support required. Awareness-Raising Activities 155. At a basic level, in the absence of proper awareness regarding ROP financing opportunities, potential applicants would not be able to submit projects for consideration and the program’s absorption and impact targets would suffer. On the one hand, a wide pool of people who know about the ROP generates a more competitive evaluation process and a more diverse project portfolio. On the other hand, a large number of irrelevant/ineligible applications creates significant work for the ROP system (primarily at the IB level), without adding much value to the process. For the ROP in particular, the inherent characteristics of the program suggest that a significant proportion of the funds go toward public-sector beneficiaries that are already well informed of the available opportunities. It follows that targeted outreach and awareness-raising activities should concentrate on those axes and intervention areas that are available for more a diverse beneficiary pool, particularly those that have not implemented ROP-financed projects in the past (e.g., NGOs, private companies, smaller public authorities, etc.). 156. In general, the MA and the IBs are required to provide basic information to all potential beneficiaries of ROP funds, including: eligibility conditions, required procedural steps and documentation, selection criteria, and contact points at MA and IB levels. A desk review of the ROP’s communication plans (for the entire program and at the IB-level) suggests that current awareness- raising mechanisms include a wide range of tools: conferences and information sessions, websites, helpdesks in each IB, advertising campaigns, and promotional materials. Survey results imply that key stakeholders are mostly satisfied with these activities in the pre-application phase, although one should keep in mind that the survey only targeted actual beneficiaries and, hence, the sample is biased toward those who found out about the ROP financing opportunities and took advantage of them. Nevertheless, respondents qualified the MA’s efforts in this respect as “satisfactory” (29%), “very good” (33%), and “excellent” (14%). As for RDAs, their support at this stage was characterized as equally positive: “satisfactory” (23%), “very good” (38%), and “excellent” (24%). 157. At the same time, there are ways to further improve the targeting of potential ROP beneficiaries. In particular, it would be preferable to prioritize informational activities for specific target groups. As noted earlier in this report, MA/IB staff should undertake special efforts to reach marginalized communities and generate interest in applying for financing. More broadly, the communications plan should be able to adapt in line with the program’s evolution: for instance, if certain axes become over-subscribed, informational activities related to these areas should dial down, rechanneling funds to other areas; also, if a certain major change occurs in applicable legislation and/or procedures (e.g., in the area of public procurement), proper informational activities should be carried out to make potential applicants fully aware of what an ROP-financed project involves. 158. Moreover, awareness-raising activities should be calibrated to specific target segments. As noted above, such efforts benefit new applicants the most, but can also play a role in expanding the pool of people who are likely to submit an application. Based on this project’s field research, a barrier 78 that tends to be more important than the lack of knowledge about the ROP is the skepticism generated by a perception of generalized corruption – not related directly to the program itself, but more broadly to the Romanian society. Countless beneficiaries from the private and non-profit sectors, i.e. the main target of awareness-raising efforts, have confessed in interviews that they could not believe that their applications could be approved without some sort of “intervention.” They were often trying their luck or “going into it for the experience.” As one interviewed medical professional noted: “To tell you the truth, I never expected my project to win. I never bribed anyone, I never called anyone, and I never asked anyone to intervene on my behalf. Everyone I interacted with in the RDA and the MA was so professional, so different than what I was expecting. It was a positive shock. And guess what? To this day, none of my colleagues believe me when I tell them that my application was successful purely on merit. And perhaps I wouldn’t have believed it either if I hadn’t been through the whole process myself.”97 159. These are precisely the stereotypes that the MA and the IBs should actively deconstruct. There are potentially thousands of viable, impactful project ideas that never materialize because people have a false impression of the system’s fairness. One solution is to constantly improve the ROP’s transparency to leave no room for doubt around selection criteria, the order of project evaluation and contracting, etc. Another option is to promote very diverse success stories to show that any viable idea can receive financing, regardless of who the beneficiary may be. In addition, the ROP should catalyze word-of- mouth networks: the vast majority of interviewed private firms and nonprofits noted that they heard about the financing available from a friend, a relative, or a consulting company of an acquaintance. In that sense, TV spots are useful to show the value of the program to the general public, but word-of- mouth campaigns are much more powerful tools for actually getting people to submit projects for consideration. 160. One final recommendation is to enable eligible projects to enter the pipeline, while channeling ineligible ones toward other opportunities. There is little reason why someone should go through the trouble and expense of putting together an application that will be deemed as clearly ineligible by the RDA staff. This is not only inefficient and discouraging for the applicant, it is also over- burdening IB resources with more work than is necessary and can in fact distract from high-quality, eligible projects. In addition to explaining ROP requirements clearly during information sessions and offering the option of an informal opinion on an idea’s eligibility before the application is put together, an easy, automated electronic validation should be made available to all interested applicants (for instance, on the official inforegio website). Other Member States have experimented with pre- application filters, but Romania could lead the way in creating an electronic system that would check the information submitted and issue a recommendation for eligible financing sources. The purpose would not be to turn someone away from the ROP, but to direct the investment idea to the right instrument. 161. In sum, ROP awareness-raising efforts are generally strong, but would continue to improve if the MA/IBs facilitate the following:  More dedicated resources for targeted informational activities relative to pure promotional efforts aimed at the general public;  Improved customization to address, first and foremost, the needs of new ROP applicants;  Focus on word-of-mouth and combating of stereotypes through increased transparency and more strategic use of successful case studies; 97 In-person interview with ROP private sector beneficiary, South-East Region 79  Adoption of pre-evaluation filters to ensure that interested applicants only invest time and resources in the process if they are eligible for ROP financing. Project Development 162. Project development efforts go beyond informing the public and require ROP experts working closely with potential applicants to identify, prioritize, and refine investment ideas. IBs are best positioned to provide this kind of close, local, customized support – acting as what the EC has called in the past “one-stop shops.” Such assistance is particularly useful for beneficiaries who seek to implement large, complex projects that require early actions to ensure a quick, efficient absorption during the implementation phase. These also tend to be the types of projects that require substantial work at the MA/IB levels, so close cooperation starting with the pre-application phase is beneficial to all parties. This should not come to the detriment of a fair evaluation; competitive calls are of particular concern in that respect. The solution is not to refrain from such support activities for fear of biasing the results, but to make it available to all interested applicants, so that the principle of equal treatment is fulfilled at all times. Picture 3. Proper early planning ensures a smoother implementation later on 163. In the ROP’s case, project development efforts have been tied to the process of drafting Regional Development Plans (RDPs). Public beneficiaries were asked to send to RDAs their list of top strategic projects that they seek to implement during the 2014-2020 period. These ideas have also been discussed, to various extents, during thematic WG sessions in each region. This “horizon scanning” effort has been deployed by other Member States as a way to assess the potential project pipeline and tentative budget requirements for various priority areas. Beyond any doubt, such processes should continue, for the next programming period and the ones after that, although some room for improvement exists, as revealed by beneficiary interview and survey feedback. 164. Mechanisms for project development vary somewhat across regions and there is little sharing of best practices across RDAs. Some of them maintain mostly an administrative focus and centralize information submitted by potential beneficiaries through standard forms that were provided by the MA in the “Methodological Framework for Identifying and Developing Strategic Projects at the Regional Level.” Other RDAs go several steps beyond this formal exchange of information and engage in close conversations with local authorities, together with growth pole coordinators, sometimes conducting 80 field visits and providing in-depth guidance on the types of projects that would be most feasible and impactful. In some cases, RDA representatives have taken on the active role of facilitator by trying to bring multiple local authorities together to define projects spanning multiple counties and generating truly region-wide effects. In regions where the RDA’s approach was limited to requesting project ideas, beneficiaries did not find the process particularly useful. There is also some variance across regions with respect to beneficiaries’ understanding of the kinds of projects that they should be generating at this stage; indeed, many have noted that there is confusion around the definition of a “strategic project with regional impact.” Some RDAs have clarified such aspects based on the guidance provided by the MA, while others appear to have left the issue up for interpretation. 165. Several other areas would require improved IB support for potential applicants. For one, the IBs can act as a sounding board for planned investments, making sure that these foreseen projects are based on actual needs on the ground and also closely align to existing territorial and sectoral strategies. Such efforts would help limit “supply-driven,” inefficient investments that may not reflect local or regional priorities. Also, the issue of sustainability is of special concern. While evaluation grids evaluate long run expenditures, in principle, project proposals are often lacking high-quality assessments of costs and benefits, particularly when it comes to estimating operations and maintenance (O&M) costs. Stronger support in the pre-application phase could help ensure that potential beneficiaries consider all relevant aspects of planned investments and prioritize them based on the full picture of expected returns, ensuring that they can continue to maintain a particular investment even in the absence of EU- level support. 166. For its part, the MA has the opportunity to play a more active role in collecting and promoting a set of best practices for boosting project generation at the regional level. As the key stakeholder with a system-wide perspective, the MA can enable IBs to share information and ideas on how to best engage beneficiaries during the pre-application stage. Particularly for large public beneficiaries, a close collaboration spanning the whole range of possible support mechanisms (from horizon scanning to field visits) is highly necessary. The MA should also ensure that all consultations and WG sessions are based on a clear, shared understanding of the types of projects that beneficiaries should generate for the next programming period (e.g., thematic focus, definitions of regional/strategic impact, etc.). At the regional level, IBs should be encouraged to act as true facilitators of collaboration among various future beneficiaries, promoting coordinated action for integrated projects that cross administrative borders of cities and counties. No matter what Romania’s future administrative model may be at the regional level, RDAs have the ability to act as independent brokers of various interests and many of them have been effective in bringing political decision-makers around the same table. In the words of one RDA General Director: “We sat down with county council presidents and mayors and we tried to bring all their views together into a coherent regional strategy. We didn’t want one local administration to submit a project taking over the entire budget available, or all of them to invest in the same kinds of projects. Beneficiaries trust us because we are seen as impartial. This model of coordination has already proven its viability. We don’t need to keep reinventing the wheel; we just need to replicate elsewhere what we have accomplished [in our region].”98 167. In addition to more proactive engagement overall, it is useful to appoint a liaison person at the RDA level, for every project in the pipeline. Currently, this only happens once a project is contracted out (i.e., through the monitoring officer), but not as early as the pre-application phase. The 98 In-person interview with RDA General Director 81 LP would be able to act as an advisor to the potential applicant, providing informal guidance on project feasibility, eligibility, and alignment with program and regional-level goals. To be sure, this person should not be involved in any way in the project’s evaluation, upholding the principle of the so -called Chinese walls (or paper walls) to preserve the process’ confidentiality and fairness. Such contact points exist in multiple other Member States, including Scotland, Wales, and Flanders (Belgium).99 There is also an opportunity to collaborate closely and rely more on business groups, nonprofit coalitions, and associations of public authorities (e.g., the Association of Romanian Municipalities and the Union of Romanian County Councils). Examples from France and Austria demonstrate that networks of knowledgeable peers can be highly successful in “mediating between different actors and offering support packages comprising domestic and EU funds.”100 While similar actors are currently involved in some programming activities in Romania, they do not typically act as partners and advisors to potential beneficiaries during the pre-application phase. The IBs have the potential to engage them more closely and encourage them to act in that capacity. 168. Last but not least, there is room for improvement in the training offered to potential beneficiaries, particularly around how to properly fill out the financing proposal and how to prevent pitfalls during the subsequent implementation. It is crucial to recognize that the content of a financing application determines the result of the evaluation, which means that it is hard to adjust ex-post. This is partly why changing key aspects of the project is so cumbersome if there is any chance that modifications might alter the initial scoring. As such, IB staff should work closely with interested applicants to make sure that the information submitted through the financing proposal is accurate, complete, and high-quality. A common example is around impact indicators; applicants are under the impression that the more they include in that section, the higher the likelihood of their application getting approved. They can then be held responsible for reaching these goals, even though sometimes they are completely unmeasurable or unrealistic. It is also important to assist potential applicants, on demand, during other pre-application procedures (e.g., drafting the requirements for prefeasibility studies). Certain best practices could be replicated across the country if only authorities would know what to ask of technical designers and contractors: for example, there are few projects for road and sidewalk rehabilitation, if any, which included underground passages allowing easy access for subsequent repairs and maintenance. This technical solution has been adopted in many other developed countries, yet Romanian authorities continue to intervene on newly built roads from the surface, which reduces the quality and reliability of the work and affects the comfort of citizens in the community. The IB liaison person could work with each applicant who requests assistance to ensure that the project achieves the best possible value for money. 99 “Turning Strategies into Projects: The Implementation of 2007-2013 Structural Funds Programmes,” IQ-Net Thematic Paper 20 (2), European Policies Research Centre, p. 75 100 Ibid., p. 76 82 VI. Application, Evaluation, and Contracting 169. This phase begins once a potential beneficiary is ready to prepare the actual financing proposal, including all required supporting documentation. Five areas are most relevant in terms of providing enhanced assistance to ROP applicants at this point in the process: applicant guides; eligible expenditures; helpdesk activities; post-contracting training; and ongoing communication. The main purpose is to help all potential beneficiaries submit proper proposals for consideration. The application process should be straightforward, accessible, and transparent, so that no potential beneficiary is dissuaded from applying for ROP funding because of perceived bureaucratic obstacles or unclear procedures. At the same time, the better a potential beneficiary is at filling out properly the financing application with its all required elements, the less work is generated for IB staff who currently verify the documentation and, often, trigger repeated rounds of clarifications to bring it to a point where it can be assessed by the independent evaluators. 170. It is also useful to note from the beginning that the stakes during the application phase are high: this is the first significant interaction of a potential beneficiary with the ROP system and also sets the tone for all subsequent phases of the project. No matter how engaged and interested people are during pre-application, it is during the application stage that they need to put in substantial effort to ensure proper submission of a viable, impactful proposal. Instead of focusing on critical content, applicants tend to divert their attention to navigating through complex procedures, while rushing to submit and make the cutoff under the first-in-first-out system, as is the case of most ROP 2007-2013 axes. So it is hardly surprising that some of the proposals received have been of lesser quality, including either glaring errors that required further exchanges or clarifications, or subtler, suboptimal decisions, as in the case of selecting unrealistic, unmeasurable project indicators. Moreover, any mistakes or inopportune choices made during the application phase can result in either the rejection of the proposal or a range of problems during the implementation and post-implementation stages. 171. Overall, surveyed beneficiaries are generally satisfied with the support received from Regional Development Agencies (RDAs) in the process of preparing and submitting ROP applications. The survey conducted as part of this report shows encouraging results: of a total 470 respondents, 24% of consider the support received as “excellent,” and another 38% label it as “very good.” Only 8% think the RDA provided “bad” or “unsatisfactory” assistance. Interestingly, responses vary somewhat by beneficiary type: public institutions are most content with the assistance received, on average, followed by private companies and non-profits. Specifically, 71% of public-sector respondents characterized application support provided by RDAs as “very good” and “excellent”, compared to 58% of private companies, and 47% of non-profits. But, all in all, across various categories of surveyed beneficiaries, the RDAs’ beneficiary assistance during this phase appears strong. 172. This is in line with feedback received during over 150 field interviews: with very few exceptions, beneficiaries praised the professionalism and responsiveness of RDA staff. They also noted that RDA counterparts generally keep them informed of the status of projects, notifying them at various stages in the process, and are also quick to react to any questions, to the extent that they can provide answers. Some delays have been reported when an official response was needed from the MA – which, for its part, has faced staff shortages and has been trying to balance operational tasks and strategic 83 responsibilities.101 Importantly, beneficiaries share equally positive opinions of the staff they have interacted with in the MA, but there is a general perception that the central authority is over-burdened and understaffed. As argued in an earlier section, plans to devolve increased responsibilities at the regional level are welcomed, both from the perspective of splitting much of the work into eight region- specific workstreams and because shortening the distance between the ultimate decision-makers and beneficiaries will inevitably increase the responsiveness of assistance mechanisms. Figure 45. Satisfaction with RDA support during the application phase is generally high Applicant Guides 173. A first resource for potential ROP beneficiaries is the applicant guide, customized for each program axis and intervention area, and elaborated on the basis of the ROP programming document and the Framework Document for Implementation (FDI). Beneficiaries generally appreciate the guides as comprehensive, useful, and sufficient for understanding the process of filling out and submitting a financing application. It covers several sections: general overview of the ROP/priority axis/intervention area; applicable rules for the financing application (eligibility and selection criteria, filling out and submitting the documentation, evaluation and selection processes); pre-contracting conditions (e.g., technical documentation, field visit); beneficiary rights and obligations; and annexes (forms, verification checklists, etc.). ROP beneficiaries appreciate the fact that the guide goes beyond what is required of them strictly as part of the application process, but also explains how their project will be evaluated, including eligibility and selection criteria, the technical and financial evaluation checklist, a general template for the financing contract, etc. Additionally, the guide also includes templates of the typical documents that would be required during the implementation phase for applicants who move forward in the process: prefinancing request, reimbursement request, and progress report. 174. The current structure of the applicant guide should be kept for the future programming period. At one level, it is sufficiently thorough and considered to be very helpful by most interviewed subjects. There is also a level of acquired comfort with the current template for more experienced beneficiaries of ROP funding and it provides a good starting point for the next ROP for less experienced applicants as well. Of course, content changes between the current version and its future iteration will be needed to reflect the program’s updated priorities, evolving selection models, and other applicable requirements, including a full section on how to use ROP-related electronic platforms if decision-makers push adopt the recommended switch to an e-based system. To the extent possible, all the different 101 See “ROP 2.0: The MA-IB Collaboration and Communication” (December 2013) 84 elements currently included in the applicant guides should be kept in order to provide applicants with as much visibility into what the process is like, what they are expected to deliver and to report on, how the project’s budget will evolve, etc. The overall objectives of the guide should be, as fo r 2007-2013, to explain how one can apply for ROP financing and to minimize “surprises” in subsequent stages of the process. This can also act as a filtering device, making sure that only beneficiaries who understand the complexities of the program and are committed enough to follow through the entire project cycle apply for funding. 175. Indeed, beyond their actual structure, applicant guides are valuable if they provide a predictable, clear framework for the application and implementation process. In a February 2010 survey by the fonduristructurale.ro portal, beneficiaries of structural funds – not just the ROP – have pointed out the “rigid and unclear applicant guides, with last-minute corrections” as the 4th most significant obstacle against the effective absorption of EU funds. This is not entirely surprising: qualitative feedback from field interviews mentioned repeatedly that it was hard to plan for a specific project given changes in eligibility standards, eligible expenditures, prefinancing and co-financing levels (e.g., for private-sector beneficiaries under axis 4.3), etc. 176. There are two separate issues that beneficiaries have signaled out: first, differences between the consultation draft and the final version of the applicant guides; and second, formal corrigenda to the guides adjusting some of the rules “during the game.” The first type of potential adjustments is hard to avoid and the good practice of publishing draft versions of the guides for consultations with future applicants should be maintained going forward, for all axes and priority areas. At the same time, the MA and the IBs should clearly specify that consultative versions are subject to change, advising beneficiaries to hold off on going too deep into developing their proposals. For instance, crafting a full budget for a project should wait until the guides are finalized, but other aspects that will not change – e.g., the need to secure approval from the State Inspectorate for Constructions (SIC) – can and should be completed as early as possible. 177. As for the second time of adjustments, through formal corrigenda of finalized guides, this should be avoided as much as possible to uphold the predictability of the enabling environment, as argued throughout this report. Particularly in the first-in-first-out system, proactive applicants have started the process in advance, which in some cases backfired as the “early-risers” were forced to make adjustments and spend additional resources to align their proposals to the requirements stated in the updated guides. Of course, there are situations where adjustments are the result of broader legislative changes, so reflecting the latest legislation is a must. There have also been situations of active learning as the process of implementation moved forward, which helps explain why most corrigenda were adopted early on in the program cycle. As the next ROP builds on the accumulated lessons learned of the current programming period, it is expected that the number of corrigenda will decrease and, as a result, the predictability of the overall application process should increase. 178. Other recommended improvements to the applicant guide are at a more technical level, but remain related to the overall goal of increasing applicants’ understanding of the full process and their corresponding obligations. For example, according to the audit track in the Procedures Manual, the duration of the contracting phase is estimated to take between 37 and 92 days, including clarifications and signing the actual contract by both the beneficiary and an MA representative (in the past, the Minister has personally endorsed these contracts). This stage is not clearly described in the applicant guide, despite the fact that it has the potential to significantly slow down the process. Moreover, based on the updated audit trails for all stages in the process, the applicant guide should include a section with 85 the potential timeline (minimum and maximum duration) of each step. One possibility is to publish audit trails for all applicants to access and understand, to the extent that by doing so the fairness and accuracy of the evaluation and control processes are not affected. 179. Equally important, the section on “beneficiary rights and obligations” is not fully aligned with the equivalent articles in the financing contract. Even though the latter is an annex to the applicant guide, to avoid any confusion among applicants, it is recommended to maintain the same content for both documents referenced here. In particular, article 9, sections (2), (9), (11), (12), and (23) should be clearly spelled out in the applicant guide because they refer to critical beneficiary obligations with respect to taking full ownership over the project’s implementation, following the set calendar of planned activities, fulfilling all advertising requirements, and ensuring full “conformity with European Union and national policies on public procurement, […] equality of opportunity, sustainable development, information and publicity.” Article 9, section (24), in the Financing Contract, stipulates the following: “In attributing [contracts] for implementing the project referenced in the current Contract, the Beneficiary, as a Contracting Authority, is required to abide by national public procurement legislation. Failing to fulfill this obligation leads to ineligible expenditures [related to contracts awarded through public procurement], or to financial corrections, according to applicable current legislation.” Such requirements are particularly critical to include front and center in the applicant guide, in light of public procurement issues and associated corrections over the past few years. Eligible vs. Ineligible Project Expenses 180. A critical component of the application phase is the “order of eligible expenses,” which is an annex to the applicant guide and defines what kind of project components are to be financed with ROP support. Commission Regulation 1685/2000 specifically notes that “rules on eligible expenditure should be the relevant national rules unless the Commission deems it necessary to adopt rules at the Community level.”102 The same act defines some broad categories of eligible and non-eligible expenditures (e.g., value-added tax, salaries, etc. As such, within reasonable limits, the decision on which project-related expenses to cover from structural funds belongs to national-level authorities, as the EC only provides general guidance in this respect. For the ROP, the list of eligible expenses is established through a common order (“the Eligible Expenses Order” – EEO) by the Minister of Regional Development and Public Administration and the Minister of Finance. This procedure duplicates national legislation on the topic of expenditure items included in a particular infrastructure project, namely Government Decision 28/2008, and has also generated suboptimal outcomes in the implementation of projects. For example, chairs in classrooms are not eligible for ROP support, but tables, computers, maps, and other equipment are covered. Alleys and benches in front of health centers for elderly people are also ineligible expenses, and a range of permits and authorizations also have to be paid out of pocket because they are not specifically included in the EEO. The same goes for different measures related to setting up the construction site, connecting to the power network, etc. The list of examples is long and it goes to show that over-regulation typically generates unintended consequences, which in this case manifest in increased bills for project beneficiaries, higher budgetary pressures, and diminished capacity to implement current projects and co-finance new ones. In truth, despite the commendable intention of ensuring that ROP funds go toward critical project-related activities, it is a daunting task to define, for all types of eligible ROP projects, all possible eligible expenditures. 102 See http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2000:193:0039:0048:EN:PDF 86 Picture 4. Even in the most modern classrooms, some of the old chairs were kept 181. Interviewed ROP beneficiaries urge decision-makers to either (A) improve the Eligible Expenses Order (EEO) based on their feedback; (B) eliminate the EEO and rely on applicable national- level legislation instead; or (C) define a list of ineligible expenditures. Given the accumulated experience in the current period, the ROP 2014-2020 would need to take into account extensive inputs from beneficiaries regarding the revision and expansion of EEOs (option A). At the same time, it is doubtful that one can regulate every single project-specific instance, so option B appears more feasible. This would imply eliminating EEOs for ROP projects and using the framework provided by Government Decisions 759/2009 and 28/2008, which include the structure of the general expenditure form (“devizul general”), with all the key headings and sub-headings. This legislation is already in place, so the only additional measures by the MA should target risk management procedures in project evaluation and implementation. Details on a potential solution are provided in the box below. As for option C, compared to the status quo, it would still be more viable to define what cannot be financed (as warranted by EU and national law) than the entire universe of potential expenditures. Box 2. . An alternative system for determining eligible and ineligible expenditures The current system that relies on EEOs is too rigid and leads to suboptimal outcomes in project implementation, where expenses clearly needed and related to the investment cannot be covered by ROP funds. Relying mainly on Government Decision (GD) 28/2008 instead provides for a more flexible framework, but also requires independent evaluators to actively assess the list of expenditures requested by applicants during the evaluation stage:  Establish through desk review whether requested expense item fits under the scope of GD 28/2008. If it is beyond what the legislation mandates, it should be considered ineligible.  If it is within the scope of GD 28/2008 but still appears irrelevant for the project at hand, evaluators can seek clarifications and approve the expense.  If a clear-cut assessment is not possible, a field visit may be necessary to determine and present to the applicant the final list of eligible expenses. Technical evaluators would also be required to sign statements holding them accountable for faulty assessments, to minimize the risk of vetting expenses as eligible when they should really be out-of-pocket. This solution promises to address project-specific needs and ensure that beneficiaries are granted all the ROP funding they should receive as part of completing a particular investment. 182. What is more, the current EEO sets a maximum value of 5% for expenses related to project design and technical assistance (Article 3), which has contributed to the low quality of technical 87 documentation and construction work.103 International benchmarking suggests that this artificial cap on this type of expenses is inadequate and should be raised to at least approximately 10% of the project’s value. Currently, the 5% is supposed to cover everything under: field studies to determine feasibility (e.g., geotechnical, topographic, etc.); expenditures related to authorizations and approvals (e.g., construction permit, connections to major utility networks, etc.); project design and engineering (e.g., feasibility study, technical project, technical project verification, technical expertise, etc.); consulting services (e.g., project management, public procurement advisory, etc.); and technical assistance, including construction site supervisors (“diriginte de șantier”). All these items are critical for the success of the project, yet under the current cap they are underfunded and underpaid and, as a result, service providers end up cutting corners. 183. One telling example concerns construction site supervisors, who play a critical role in project implementation but remain significantly underpaid under the status quo. To be exact, they are supposed to ensure the quality of work and materials and report all potential issues to beneficiaries, all vital attributions that require full-time presence on-site. But because contracts usually go to the lowest- price bidder and beneficiaries do not hold supervisors accountable for staying on site, they often serve in this role for multiple contracts at once, even in different localities. In some cases, they endorse progress and inspection reports without actually verifying the situation, which later turns against the beneficiary in the form of unanticipated, ineligible additional expenses. The box below presents further details on the factors responsible for the current situation, as well as potential solutions to be explored. Box 3. The 5% total cap for key services is inadequate Situation: In cases where future beneficiaries opt for contracting an external consultant for developing the financing contract (FC) and the feasibility study (FS), this expert is also responsible for putting together the project’s budget and the list of cost estimates (“devizul general”). In the next stage, the same consultant often bids to win the contract for developing the technical project and/or the project’s management, which means that, when drafting the list of cost estimates, it is in their best interest to allocate as much of the budget as possible to line items 3.3 (Project Design and Engineering), 3.6 (Technical Assistance from the Project Designer throughout the Execution Phase), and 3.5b (Project Management Services). Implication: This means that only a small proportion of the total budget (capped at 5% of the total project value, as described earlier) is available for line item 3.6b, which covers expenses for construction site supervisors responsible for verifying the quality of work performed. Along wit h the “lowest bid” selection criterion, this situation forces supervisors to look for work on multiple sites. Key Observations: International benchmarking suggests that it is common to allocate 3-5% of the project value for feasibility studies and supporting assessments, 3-7% for detailed technical projects, and 3-5% for construction supervisors or “engineers.” The exact proportions vary depending o n project scale and complexity. This compares unfavorably with the 1% typically available for supervisors of construction contracts in Romania and also suggests that the overall 5% cap on these critical components is artificial and inadequate, contributing to poor-quality work by various service providers. Recommendation: The 5% cap should be increased to reflect international best practices. This should be complemented by a public procurement system that moves beyond the “lowest bidder” standard to more adequate selection criteria. 184. As a best practice, the Romanian decision-makers should consider the Spanish equivalent of the EEO, which allows for great flexibility in defining eligible expenditures. In simple terms, this legal document lists mostly those items that are not eligible to be financed out of structural funds, specifically 103 See “Monitorul Oficial al României,” No. 327, from May 11, 2011 88 noting that all expenditures need to abide by EU and national-level regulations. For example, there are only two conditions for a “general expenditure” to be considered eligible: (1) to have been included in the application selected for financing, according to selection criteria approved by the Monitoring Committee; and (2) to be based on real costs attributable to the project. This flexible approach has facilitated smoother implementation of EU-financed projects in Spain and may well serve as an example for how EEOs for the next ROP could look like, if they are maintained as such. The following box provides additional details on the Spanish EEO. Box 4. The Spanish equivalent of the Eligible Expenditures Order is much more flexible 104 Excerpt from Order 524/2008 issued by the Ministry of Economy and Finance: Eligible expenditures: General Norm. Eligible expenses must match the financed operation with certainty, without limitations other than those derived from applicable Community law and national legislation, as well as those contained in this order. Actions cofinanced by the ERDF, which fall within the scope of Article 3 of Regulation (EC) no. 1081/2006, will have to comply with rules on eligible expenditures provided for in Article 11 of the Regulation and, where necessary, those permitted under Article 56.4 of Regulation (EC) no. 1083/2006 for the ESF. Ineligible costs. The following expenses will not be entitled to funding from the ERDF or the Cohesion Fund: interest on debt; other financial expenses, excluding interest rate bonuses included in subsidy programs; headquarters; dismantling of nuclear power plants; VAT recoverable and general indirect tax recoverable in the Canary Islands, and other similar recoverable taxes; personal income tax; contributions in kind; interest, surcharges, and administrative and criminal sanctions; expenses for legal proceedings; expenses for bank guarantees or otherwise, except the cost of bonuses guarantee contained in subsidy schemes; and, for the purchase of goods and services through public contracts, reductions and/or payments made by the contractor to the state (management fee work, quality control, etc.), deriving from the project’s execution, and/or payments resulting from changes to the procurement contract, provided they are not approved by DG Community Funds. […] General expenditures will be eligible provided they fulfill the following requirements: a) appear explicitly in the application selected in accordance with criteria approved by the Monitoring Committee; b) are based on real costs attributable to the execution of that operation, in accordance with the principles and generally accepted accounting standards or actual average costs for operations of the same type. Lump sums based on average costs may not exceed 25 % of the direct costs of an operation that can affect the overall spending. Calculation of lump sums will be properly documented based on a technical report, before the first statement of expenses and will be reviewed at least once a year. Expenditures incurred by public bodies in the preparation and execution of operations : Besides the available technical grant for the operational program, in accordance with Article 46 of Regulation (EC) no. 1083/2006, the following will be eligible: a) the cost of professional services provided by a public authority other than the beneficiary during the preparation or execution of an operation; b) the cost of services related to the preparation and execution of an operation provided by a public authority when it itself is the beneficiary of the services and performs an operation on its own, without recourse to external service providers. Source: Ministry of Finance and Public Administration (Spain) Helpdesk 185. Each RDA has established a functioning helpdesk that offers assistance to interested ROP applicants, with positive results to date. RDA support covers making beneficiaries aware of 104 Translation from Spanish. Full text (Romanian translation) is available upon request 89 opportunities offered by all available structural instruments in a particular region, and they can refer inquiries to various MAs and IBs, on a case-by-case basis, depending on the intended project focus. Still, most helpdesk activities remain concentrated around the ROP. The main channels for assisting beneficiaries include email, phone, and in-person meetings at RDA regional headquarters or at one of the county offices. Discussions with RDAs reveal that helpdesks have helped alleviate some of the pressure from staff in the evaluation department, which is typically under heavy time pressures and workload requirements, particularly when new axes and intervention areas are launched and applications start coming in. It has been much more effective to separate “customer-interface” functions under helpdesks, making sure that each question received is accounted for and addressed promptly. To be sure, helpdesk personnel cannot respond to all types of inquiries, as some may be very technical and project-specific. But they can address all questions that are relatively basic and can be answered by referring to the ROP’s programming and implementation documents and to the applicant guide. Figure 46. Beneficiary perceptions of RDA/MA helpdesk support (# of ROP projects implemented) 100% 90% 80% 70% 60% excellent very good 50% satisfactory 40% unsatisfactory poor 30% N/A 20% 10% 0% RDA MA RDA MA RDA MA <2 2-5 >5 186. In some cases, IBs can also direct inquiries to the MA’s helpdesk, which is focused on providing general information about the ROP program, its axes, intervention areas, etc. For their part, IB-level helpdesks are able to act as a first point-of-contact for applicants who need guidance around how to fill out an application and how to obtain the required supporting documentation, functions that the MA has delegated to RDAs. As noted earlier, the closer this assistance gets to beneficiaries, the better it can address their needs in a timely manner, which suggests that most of the helpdesk activities should be concentrated at the IB level. Sure enough, feedback from interviews confirms this hypothesis: most beneficiaries contact their local RDA representative first, in the county or regional office, and only turn to the MA staff under special circumstances, where they feel the need to reach someone in the Ministry. This may happen if the RDA cannot provide a definitive answer of if the applicant wants to 90 inquire about a particular point in the process that depends on the MA (e.g., appointing an independent evaluator after the initial administrative and eligibility check by the IB). 187. Beyond traditional helpdesk functions, i.e., having a dedicated staff available to answer on- demand inquiries, RDA helpdesks also organize training and informational activities for potential applicants. The purpose of these sessions is to present available financing opportunities through the ROP and answer any questions that may arise around applicant guides, required documentation, procedural steps for obtaining different permits, etc. Training 188. Both RDAs and the MA organize post-contracting training sessions with beneficiaries who have signed contracts or have already started project implementation. In an ideal-case scenario, every contracted beneficiary should receive in-depth training on applicable procedures as well as common challenges and solutions during implementation and post-implementation phases, including public procurement and project management. The advantage of scheduling sessions at this point in the process (i.e., after contracting) is that the target group is narrower and better defined compared to the whole population of potential beneficiaries and applicants. The importance of these sessions for ensuring smooth project completion is reflected in both survey results and interview feedback from field visits. Going forward, the MA and the IBs should continue to organize this training and continue to improve its delivery, keeping in mind the need for: effective timing and coordination; increased frequency; and enhanced customization for different types of beneficiaries. IB-led 189. One best practice adopted by RDAs is the organization of post-contracting workshops with future beneficiaries of ROP funding. These sessions are scheduled periodically: as approved applicants come through the evaluation pipeline and head toward implementation, the RDAs form groups of 12-15 future beneficiaries and ask them to come to the regional headquarters for, typically, a day-long training workshop. Some RDA managers have noted a conscious effort to match workshop participants according to a clear set of criteria (e.g., experience, focus area, private vs. public, etc.). In some instances, however, time constraints have led to less strategic selection of cohorts. Also, there is no standard methodology for these sessions, but in principle they all share the goal of preparing project managers for common implementation challenges. As this report later shows, the two most significant obstacles are related to public procurement and required adjustments to the technical documentation. In addition to presenting on these topics, trainers (from the RDA or external consultants) also go over typical reporting requirements, including progress reports, monitoring visits, post-implementation durability assessments, etc. 190. Based on survey results and interview feedback, ROP beneficiaries consider these sessions as useful and productive. Beyond the content itself, these are opportunities to interact with other beneficiaries present, discuss potential challenges and solutions, and learn from one another. The numbers tell a positive story: 52% of respondents evaluate RDA-led workshops as “excellent” and “very good,” and another 23% qualify them as “satisfactory.” Interviewed IB representatives are also strong believers in the value of these sessions and have noted that many errors were prevented by taking the time to go over various procedures with future beneficiaries. A special case is IB-Tourism, which was forced to give up on organizing these sessions in recent months due to its overburdened, understaffed capacity that has been predominantly focused on operational issues. The result was a decrease in 91 quality for key documents submitted by beneficiaries. For instance, the IB received reimbursement requests from the same beneficiaries making identical errors time and again, which in turn ended up generating even more work for IB staff. This goes to show that training beneficiaries in the post- contracting phase does not only help them complete their project and absorb the funds available more efficiently, but it also reduces the burden of operational tasks that MA and IB teams have to address. 191. At the RDA level, post-contracting training should maintain the current small-group format and continue to cover the entire pool of future beneficiaries through increasingly customized and continuously relevant workshops. A common methodology or at least a set of guiding principles for selecting workshop participants is recommended, across all RDAs, to make sure that discussions remain at an adequate level. Two important criteria are the beneficiary’s level of experience with ROP/EU funding and the organization type (private/public/non-profit). The strategic selection of participants will increase the quality of the workshop itself and can also facilitate longer-term interactions and knowledge sharing among cohort members, as they are expected to face similar issues later on during the implementation phase. At the same time, course materials should always be updated and speaker profiles should be aligned with the particularities of the Romanian environment, reflecting the latest legislative framework. This is not an easy task because organizing the workshops – and possibly contracting them to an outside resource – takes time and involves a range of resources that have to be mobilized in advance (e.g., experts, location, equipment, etc.). But the relevance of the information presented should always take priority. 192. Improved customization and relevance also come from addressing project-specific concerns during pre-implementation sessions, which is precisely where the competitive advantage of RDAs lies. As the main driving force behind Regional Development Plans (RDPs), RDAs have acquired superior knowledge of the local context, including regional issues that go beyond the ROP. They also have critical resources available close to beneficiaries – i.e., “boots on the ground” in county and regional offices that are able to build trust-based, collaborative relationships with beneficiaries. In many cases, RDA teams know very well the capacity of a particular beneficiary and can anticipate challenges before they materialize. For all these reasons, whenever possible, training sessions organized by RDAs should include one-on-one coaching components to discuss the specific questions of each project manager. MA-led 193. The MA has also organized training sessions with ROP beneficiaries at several points during the 2007-2013 programming period, including with the help of external experts. For example, at the end of 2009, the MA contracted a consortium of firms to lead trainings with people from beneficiary organizations. These happened in two rounds – July 2010-July 2011 and June-December 2011 – and covered over 1,000 participants from beneficiary organizations. The sessions were dedicated to two broad themes, “ROP Project Implementation” and “Public Procurement,” and were three days each, allowing for in-depth training and support. These types of sessions typically involve larger groups of participants and cover broader themes. Compared to the RDAs, which can focus on local and project- specific issues, the MA’s comparative advantage lies in seeing the evolving country-wide picture on complex issues like public procurement and technical documentation and updating beneficiaries on how to address common challenges. Training sessions were organized during the same period with IB and MA personnel as well, to ensure a standardized approach to technical and public procurement issues. 194. Beneficiary feedback regarding MA-led training is generally positive. 43% of surveyed respondents qualified the sessions as “excellent” (13%) and “very good” (30%), while 16% were not 92 satisfied and 14% could not provide an answer, which may imply they did not participate in the courses offered. The numbers, as well as the qualitative feedback in interviews, suggest that such training activities are helpful and should continue in the future, especially because beneficiaries seek closer interactions with MA staff. These are opportunities to get closer to the ground-level implementation and exchange views with project teams on challenges and acceptable solutions. Of course, the MA cannot be expected to go into as much detail and project-specific feedback as the RDA during the post- contracting workshops, but the value the MA can add is in providing the latest guidance on pressing challenges (e.g., public procurement legislation, adjustments of technical projects, etc.). Coming from those who hold the final authority over implementation matters, such feedback is extremely important for beneficiaries. 195. For trainings at this level, room for improvement still exists in terms of two key aspects: frequency and differentiation by beneficiary competencies. The former refers to how often beneficiaries from around the country are called in to attend the seminars. The 2010 and 2011 sessions were welcomed, but they also came late in the process, over three years after the ROP’s start date. For beneficiaries who rushed early on to submit projects competitively – and benefit from the FIFO selection model – the timing was not entirely helpful. If such sessions had been contracted early on and organized in 2007-2008, it is likely that at least some of public procurement issues and associated corrections could have been avoided. The delays were also related to the lengthy tendering procedures that Romania is struggling with in general, but better planning can also help better align the sessions’ timing with the project cycle. Depending on the available budget, such seminars should be organized more frequently to include every beneficiary who is beginning to implement an ROP project, as well as on a case by case basis, on specific themes, if a major change requires it. For example, adjustments like the 2011 clarifications to the public procurement legislation should prompt the organization of seminars to explain the impact to beneficiaries and prevent further mistakes. 196. As for differentiation of sessions by beneficiary competencies, it is recommended to organize seminars for at least three levels: beginner, intermediate, and advanced. Under the current one-size- fits-all system, the material presented has been too challenging for some participants and too basic for others. Looking at the numbers in the figure below, one can speculate that, under the current format, the value of the training decreases with a beneficiary’s level of experience: only 28% of respondents with over five ROP projects in their portfolio found the sessions to be “excellent” or “very good,” compared to over 40% for respondents with less experience. As such, a straightforward way to increase the quality of MA-led training is to offer it for different competency levels. Budgetary constraints should not apply, as these activities can and have been financed from the ROP’s technical assistance axis, which still leaves room for requesting additional sessions from contracted trainers. Last but not least, with proper planning, seeking beneficiary feedback into course curricula and the format of sessions is highly recommended and would ensure that trainings respond to beneficiaries’ most pressing needs, as perceived by the staff in the field. This should also help to differentiate between the content of sessions for the three levels (beginner, intermediate, and advanced). 197. All in all, the training resources available at the MA and the RDA level have met an important proportion of beneficiaries’ needs and expectations. It is true that some of them are now more knowledgeable and better prepared than in 2007. But it is equally true that the overall need to train the system’s key human resources continues to be pertinent: on the one hand, the environment is dynamic and there are always new elements that should be learned and adopted; on the other hand, there are new people coming into the system, and the risk of turnover in beneficiary organizations remains relatively high, as argued in an earlier chapter. 93 198. As the regionalization and decentralization processes move forward, it is worth evaluating which activities should stay at the national level and which should move closer to beneficiaries. Based on the observations above, national and regional training sessions offer quite different value propositions: the former are less frequent and serve larger groups of beneficiaries, taking on broader themes that affect projects across Romania, including as a result of emerging legislative changes; the latter can delve deeper into project-specific challenges and coach participants within smaller groups, facilitating more interactive knowledge sharing. It is also true that national-level workshops are harder to organize and typically require more complex, lengthy procurement procedures, so whenever possible further decentralization to RDAs is advisable. Some of them have already taken the lead by formalizing course offerings through their own training centers, covering key themes like project management, procurement, auditing, etc. Figure 47. Level of satisfaction with MA-led sessions by beneficiary level of experience 100% 90% 80% 70% 60% N/A poor 50% unsatisfactory 40% satisfactory 30% very good excellent 20% 10% 0% <2 2-5 >5 Number of ROP projects in beneficiary portfolio Ongoing Communication 199. A major recurring theme in survey responses and interviews is the lack of transparency and predictability of the evaluation and contracting process. This is reflected in the 40% of surveyed beneficiaries who assessed “lack of transparency” as an “extremely” or “very important” obstacle in accessing ROP funds. This perception persists, despite good practices currently available, including easily accessible lists of criteria for how an application is verified and evaluated (administrative and eligibility checklists, technical and financial evaluation criteria, and technical project assessment). Beneficiaries understand the procedural steps and selection models, but they have much less visibility into the 94 timeline and progress of their application throughout the process and particularly once their documentation is sent to MA-ROP. The RDAs send notices to applicants at all key stages (administrative and eligibility check, technical and financial evaluation, technical project assessment, and contracting), but it is not uncommon for staff to be unable to provide thorough updates upon request. This creates frustration among beneficiaries, with some of them also trying to get information directly from the MA. As argued repeatedly throughout this report, such inefficient mechanisms end up generating more work at MA and IB levels. Instead of focusing on pushing applications through the process, some resources have to be dedicated to answering applicant inquiries. 200. A solution with benefits for all sides would be to increase the transparency of the flow of documents, developing a simple online tracking system. Applicants would be able to access the platform to determine where their application is at any one point, how long the process is expected to take, and who the main point of contact is at each step in the process.105 This could increase the timeline’s predictability and transparency, mitigate concerns among beneficiaries, and free up capacity in the MA and RDAs helpdesks, which could refer applicants to the online system. Several processes could be automated, including letting people know via email and SMS when their application has completed a particular phase, expected timeline for next steps, required clarifications, etc. It would also help improve accountability up and down the process chain within the MA and the IBs, ensuring visibility around bottlenecks and capacity constraints. If management notices that an unusual number of applications get stuck with a particular unit, it will be able to take immediate measures to correct the problem (e.g., by reallocating resources more dynamically). These kinds of data would also enable an improved pay-for-performance system, an idea that has been considered recently by the Ministry of European Funds for all institutions involved in the absorption of EU funds (MAs and IBs across different structural programs). 201. Special care should be devoted to increasing the transparency of the contracting process. Some interviewed beneficiaries have noted that it is not clear how applications get prioritized for contracting once they pass the evaluation phase. As shown in the other report, some projects require additional clarifications before actual contracting, primarily provided by the IBs and initiated by the MA, which at this point in the process has access to the entire application file. Beyond reducing this administrative duplication between MA and IB staff, there should be a clear, transparent process for future beneficiaries to know what to expect in terms of the probable timeline. This is important for a range of reasons, including mobilization of capital, human resources, efficient preparation of tendering documentation, etc. The same portal used to track the application progress should provide information on how long the contracting stage is supposed to take, when its completion can be expected, potential causes for delays, and what, if any, further clarifications are needed. It would also be useful to publish real-time information to the general public on the list of projects that are waiting to be contracted and on those that have been contracted. A similar list does exist on the MA’s website, but it is not updated in real time. 105 A simple, comparable example is the USCIS immigration service, which includes a useful platform for showing the evolution of an application, expected timelines, and other requirements. 95 Figure 48. Example of online tracking system (USCIS) 106 Source: U.S. Citizenship and Immigration Services Online Case Status 106 A simple, comparable example is the USCIS immigration service, which includes a useful platform for showing the evolution of an application, expected timelines, and other requirements. 96 VII. Implementation 202. The implementation phase is considered by most beneficiaries, as well as by RDA staff, as the most complex juncture in the project cycle. It starts after the contract for a particular project is signed, and ends when the investment is complete and the last reimbursement request is accounted for. At a basic level, during the implementation phase, the beneficiary of the ROP funds is required to implement the financing contract and abide by all its requirements. Based on extensive interview feedback with beneficiaries, as well as RDA staff, and taking into account survey results, the greatest challenges faced by beneficiaries during project implementation include: inadequate public procurement framework and corresponding financial corrections; low quality of technical documentation and construction works; lack of standardized contract forms; and monitoring and reimbursements. Several of these issues are, to some extent, interrelated, and the sections below present both an assessment of the current status quo and multiple recommendations for the future. 203. But before focusing on each topic area, an overall recommendation endorsed by the vast majority of beneficiaries is the adoption of an official Implementation Manual. This would include sections on all major issues that may be encountered during a project’s implementation: from public procurement through managing the contractual relationship with contractors and service providers. Any attempt for publishing such a material would need to involve close MA-IB interactions and multiple rounds of consultations with experienced beneficiaries. It is also recommended to engage to EC on any aspects that would require further clarification or advice. While it would not be an easy task, it is certain that the demand for a comprehensive Implementation Manual is very high: 54% of interviewed ROP beneficiaries consider it “extremely useful” and an additional 29% view it as “very useful.” Only 2% do not see any use in such a support mechanism. Public Procurement and Corrections 204. By far the greatest concern expressed by ROP beneficiaries in field interviews centers on issues related to public procurement and associated corrections. The inadequate public procurement framework also features as the 6th most important obstacle in accessing ROP funds in the survey implemented as part of this work. When segmenting results by beneficiary type, it becomes obvious that this affects public authorities more than private firms – indeed, the former implement larger projects, face more complicated rules, and run into bigger problems related to the public procurement legislation. Over 70% of public beneficiaries rate the inadequate public procurement framework as an extremely or very important obstacle, compared to fewer than 40% of the private beneficiaries. 205. Financial corrections related to public procurement procedures are also widespread among local authorities, who commonly refer to them as “a witch hunt,” “a national obsession,” and “the biggest deterrent against ever seeking EU financing again.” To clarify in simple terms, a correction is a mechanism for withdrawing awarded funds from beneficiaries as a result of irregularities found in the process of making project-related expenditures. If the funds under question have already been reimbursed to the beneficiary, they would be requested back or subtracted from future reimbursement tranches; if they are still to be reimbursed, the beneficiary would just receive the total requested minus the established correction. Several institutional bodies are mandated to identify irregularities and propose corrections: the MA, the IBs, Unit for the Coordination and Verification of Public Procurement (UCVPP), the National Authority for Regulating and Monitoring Public Procurement (NARMPP), the Certifying and Payment Authority (CPA), the Department against Fraud (DLAF), and the European Commission (EC). At the national level, the MA makes the final decision on the actual correction level, 97 but the cases where it differs from NAA proposals remain rare. The backstory goes to the EC’s audit findings from 2011, which noted that the system of verifying public procurement procedures was flawed and had permitted a significant amount of breaches of applicable EU-level regulations governing the implementation of structural instruments. Figure 49. Beneficiary perceptions of the public procurement framework as an obstacle 100% 90% 80% 70% N/A 60% Not important 50% Hardly important Relatively important 40% Very important Extremely important 30% 20% 10% 0% Public institution Private company 206. The key issue is that Romanian legislation on public procurement procedures related to EU- financed projects failed to incorporate until 2011, 4.5 years into the ROP’s implementation, several key mandatory requirements (e.g., clear definitions of restrictive criteria, applicable corrections, etc.). Particularly beneficiaries who launched public tenders early on during the ROP’s implementation were largely unaware of such requirements and simply followed Romanian legislation applicable at that time (ordinance 34/2006 with its subsequent adjustments). Government Emergency Ordinance (GEO) 66/2011 later sought to fill the legislative void and translated nearly ad literam the COCOF guide on applicable financial corrections for breaches of public procurement regulations, which was unusual, to say the least, when compared to other Member States. The COCOF guide is meant to provide overall guiding principles, but it is up to each state how it defines criteria and procedures for imposing corrections to abide by the principles of sound financial management and proportional sanctions. Other 98 acts, such as NARMPP order 509 on acceptable criteria for awarding public procurement contracts, further completed the national-level legislation in 2011, clarifying several cases of “what not to do.” 207. After the publication of GEO 66/2011, the Romanian Government mandated the re- verification of public procurement procedures on all ROP contracts, with the RDAs checking 100% of them, and the MA and the Audit Authority (AA) checking a subset based on sampling. According to feedback from interviews with the AA, RDAs, and beneficiaries, most of the issues discovered were related to “restrictive criteria” in the tender documents, accelerated bundled procurement without sound justification, breaches of advertising requirements (e.g., failure to advertise a tender above a certain threshold in the Official Journal of the European Union), and “unpredictable” additional work. Restrictive criteria were reported in interviews as the most common irregularity, often originating in the beneficiary’s honest intention to select an appropriate goods/services provider or contractor (e.g., construction company with experience in EU-funded projects so that it would be familiar with the specific forms and reporting requirements). In a few instances, one mistake (e.g., specifying the brand name of a certain construction equipment required without the added phrase “or similar”) was unintentionally propagated from beneficiary to beneficiary in cases where local authorities collaborated in the preparation of tender documentation. 208. At the end of the day, however, a very small number of instances led auditors to suspect fraudulent intentions, but corrections were applied to a staggering number of projects, for various reasons. The 2012 annual report published by the Directorate General for Finding and Establishing Irregularities within the MRDPA notes that, between April and December 2012, a total number of 491 potential irregularities were recorded. Of these, the European Antifraud Office was notified in only three cases. Even so, financial corrections are extremely prevalent, as the following section shows. 209. Recently, the Institute of Public Policies (IPP) and the Țucă, Zbârcea & Asociații (TZA) law firm performed a comprehensive assessment of proposed corrections, painting the first in-depth picture of this phenomenon. The study looked at projects under implementation in 2011-2011. The total sum imposed in penalties by December 2012 had reached over EUR 300 million, “double compared to Spain and 15 times the level of Poland or Germany.” The same paper provides some key data on corrections, as follows below, with the caveat that they apply to all operational programs, but most of the corrections (84%) are actually related to ROP projects.  The average value of a correction on a EU-financed project was EUR 400,000, or about 13% of the total average budget for public procurement contracts;  Corrections varied from 5% to 100%. In 43% of the cases, the penalty was 5%, followed by 10% (in about one quarter of cases) and 25% (in about one third of the cases);  Equally interesting is the distribution of corrections based on the type of contract: in 62% of the cases, penalties were imposed on construction works, followed by auditing (14%), project design (7%), technical assistance and consulting (each with 6%), and advertising (5%);  As the study notes, the most interesting data comes from segmenting corrections based on the type of irregularity detected, as summarized in the figure below. By far the most common problem is related to “illegal qualifying criteria” – these are also referred to as restrictive/disproportionate criteria and often result from requesting a particular level or type of experience from the bidders. As the next paragraph explains, beneficiaries characterized the vast majority of these errors as purely non-intentional or wrongfully imposed. 99 Figure 50. Corrections by type of irregularity detected 3% Illegal qualifying criteria 6% 6% Breaches of equal treatment principle 9% Winning offer failing to meet criteria Single-offer awarded contract 9% 56% Unlawful increase in contract value Qualifying criteria used as selection criteria 11% Breaches of transparency principle Source: “Collection of Cases with Irregularities and Financial Corrections among EU-Financed Projects,” IPP and Țucă, Zbârcea & Asociații (2013) 210. A vast number of beneficiaries who were interviewed as part of the current field research were frustrated with the control institutions’ understanding of restrictive criteria. In many cases, they defended their choices by pointing to the need for contracting high-quality work, particularly in the broader context of contracts awarded only to the “lowest-bidder,” as recommended by the NARMPP. Examples are numerous: a company was required to have experience renovating historical buildings in order to know how to protect their value and perform complicated operations; one contractor had to have worked on EU-financed projects before to understand the reporting system; another firm had to show that it relied on resources within a particular range (e.g., 250 kilometers) to be able to perform the work swiftly; another one was required to be ISO certified – and the list can go on and on. In some cases, beneficiaries showed that a certain criterion did not constrain participation, as there were multiple bidders present (e.g., up to 5). While control institutions argue that there could have been other bidders that did not participate because they were dissuaded from the start, beneficiaries reply that they did not receive any clarification requests from any such bidders during the tender proceedings. It is beyond the scope of this report to discuss case-specific issues and debate on their resolution. Instead, the next paragraphs present the consequences of the current system, summarize the main challenges, and recommend solutions. 211. The immediate outcome of financial corrections is that local authorities with small budgets face serious financial risks and their long-term appetite for ROP funding diminishes drastically. Whether they have to return part of the disbursed money or take a cut from future disbursements, sanctions are effective immediately and regardless of whether beneficiaries challenge the decision in court. Particularly in the case of sanctions imposed on large public works contracts, a 25% or higher penalty can be a significant burden on local budgets. For cities like Băilești or Brăila (see figure below), or for counties like Tulcea with a EUR 7 million sanction, the total value of EU-financed projects under implementation far exceeds the annual budget. As such, there is a serious risk that corrections will push some local administrations into bankruptcy, affecting their ability to complete current projects or apply for new ones due to the simple lack of funds for co-financing and revolving expenses. According to the 100 IPP-TZA study, through December 2012, 80% of sanctioned beneficiaries challenged the decision through internal ROP mechanisms and 41% have sought remedy through legal court action, often arguing that retroactive application of law is unconstitutional. Many of the initial decisions from Appellate Courts are favorable to beneficiaries, though most final definitive verdicts by the Supreme Court are only expected in 2014 or 2015.107 As a result, a recent initiative of the current Government was to cover corrections from the state budget, in those instances where a favorable “approval” by relevant bodies existed, although this solution is more targeted at the symptoms of the public procurement saga, and less at its root causes. Figure 51. Corrections as % of total budget of projects under implementation (Oct. 2012) Source: http://www.ipp.ro/library/fse/Propunericorectiifinanciare.pdf 212. Funds from corrections get put back into the total ROP budget pool and become available for projects in the reserve list, to be used before December 31, 2015. Still, given Romania’s low level of absorption to date, some of this money may be eventually lost. But the bigger problem is that some beneficiaries are losing trust in the system. They perceive it as unfair to be punished despite the fact that they had abided by Romanian legislation at the time of the public tender and, moreover, several institutions had validated their proposed procedure through ex-ante and ex-post controls. Even if beneficiaries get the money back as a result of court orders, it will take years until final verdicts are issued and the sums are reimbursed, and they will ultimately come from the state budget anyway. There is a substantial cost in terms of legal fees and wasted time, as local authorities sue the MA and the RDAs, but, regardless of what happens eventually, the money comes from the public budget anyway (either local or national). 213. The key issue of public procurement procedures is the suboptimal, inefficient ex-ante and ex- post verification system, with a chronic lack of accountability of all stakeholders involved, with the exception of beneficiaries, who are left to bear the entire burden of potential mistakes . For their part, beneficiaries often lack the technical capacity to avoid all pitfalls of a very complex public procurement framework (Government Ordinance 34/2006 and its many subsequent changes). The MA and the IBs 107 In 1-2 cases, beneficiaries actually had final favorable verdicts as of April 2013. 101 cannot help much because that would create a conflict of interest with their role in evaluation, while the NAA only has a mandate for ex-post audits. There are instead dedicated institutions that are mandated under Romanian law to oversee and regulate public procurement, particularly the National Authority for Regulating and Monitoring Public Procurement (NARMPP), the National Council for Resolving Disputes (NCRD), and the Unit for the Coordination and Verification of Public Procurement (UCVPP). Currently, none of these actors is willing to formally “endorse” a clear set of general rules or resolutions for specific cases, so beneficiaries are left to bear the entire responsibility over the process without the certainty that the other actors involved help eliminate the risk of subsequent corrections. Beneficiaries are also the only ones getting penalized even though past procurement procedures had passed through the NARMPP and UCVPP filters. In quite a few cases, corrections apply long after the sums had been reimbursed (i.e., after another ex-post set of verifications by the IB, the MA, and possibly the NAA), and even after projects have been fully finalized. 214. A related concern is the lack of technical competencies among auditors. Beneficiaries note in their interviews that many auditors come from a finance/economics background and do not fully understand the technical reasons behind certain procedures (for example, unpredictable but required works that are not related to errors in project design). In practice, at the end of the day, many beneficiaries do not have a clear understanding of why they are getting sanctioned and how the level of penalties is decided. The lack of predictability affects their ability to make cash flow projections and in some cases imposed corrections have directly impacted local authorities’ capacity to co -finance additional ROP projects for 2007-2013 and the preparation for 2014-2020. There is also a strong perception that the level of corrections is decided somewhat subjectively by various auditors, without clear guidance for beneficiaries, and with differences across similar cases around the country. 215. Not only are clear standards lacking for how corrections should apply, but there is substantial overlap in the current system for verifying the legality of public procurement procedures. Sparked by the EC’s decision to pre-suspend the ROP as a result of identified deficiencies in this area, a multitude of filters and checks have been imposed on beneficiaries, who are already overburdened with operational tasks and have to accommodate multiple auditors on top of their regular tasks. The following quote from a beneficiary is telling: “We are treated as guilty until proven innocent. Auditors come with their minds set on finding errors and punishing us, as if they have a certain quota they need to fill. It is incredibly hard to focus on implementing our current projects and preparing the 2014-2020 portfolio when we are always under a barrage of auditing missions, which require us to be available at all times and promptly put any document to their disposal. We understand that there are a few cases of fraud and they should be properly punished. But we are simply trying to do our job and should not be treated as potential criminals by default.”108 216. More broadly, the overall public procurement system appears broken. Procedures take too long and are subject to numerous legal proceedings, often times initiated by “professional challengers.” There are known firms who participate in biddings and later challenge them, even without any serious reason. If a challenge is struck down by the National Council for Resolving Disputes (NCRD), some firms will take the case to court and can generate delays of up to 1-2 years. A solution that has been suggested before is fining repeated challengers who dispute public procurement procedures without just cause. Currently, they only stand to lose a small sum of money deposited at the time of the tender, which is obviously a stick that is not big enough to disincentivize malevolent legal action. 108 Interview with public sector beneficiary, South-West Oltenia Region 102 217. First and foremost, any changes to GEO 34/2006 should keep in mind the need to align national standards with EU-level ones and avoid the large number of changes to this legislation recorded in recent years. If a major overhaul of the system is approved, changes should be thoroughly discussed and adjustments should be made from the start and negotiated with the EC, as Romania recently did with respect to a new law on compensating former property owners whose assets had been confiscated under Communism. For its part, the EC has also announced proposals to revamp the Community’s public procurement framework with the aim to simplify procedures and help beneficiaries, so the Government is well-advised to wait until the broader EU-level legislation is finalized. Once that happens and if GEO 34/2006 is also adjusted or replaced entirely, there need to be wide-ranging programs training beneficiaries on the latest acceptable standards for public procurement. That will require a quick and substantial mobilization of resources by the MA and other bodies, including the Ministry of European Funds. 218. In the meantime, a key priority to ensure the system’s proper functionality is to allow a departure from the current focus on the lowest price as the default selection criterion. The NARMPP has often shown reluctance to approve documentation that was based on technical-economic standards for awarding contracts, and many beneficiaries were requested to review such methodologies. A common practice has emerged of using price as the sole selection criterion, which was found by beneficiaries to be palatable to the NARMPP, whose approval is needed before publishing the public procurement procedure on the Electronic System for Public Procurements (SEAP). The consequences have been often regrettable: some companies have submitted low-ball, dumping price offers and beneficiaries were not able to disqualify them, so they ended up with contractors trying to (a) provide low-quality work and (b) increase the price through various tactics like adjusting the technical project or claiming “diverse and unforeseen expenditures.” For their part, many beneficiaries have been stuck with poor-quality service providers and are weary of canceling the contract and organizing new public procurement procedures, for fear of going beyond the agreed calendar for the project’s implementation, which can also result in financial corrections. As such, they are often in a catch-22 situation: break the contract and face the possibility of losing the ROP financing, or keep working with cheap service providers who do poor-quality work. 219. In addition to allowing the inclusion of defined indicators of technical quality as selection criteria, decision-makers should apply regulations allowing for the disqualification of dumping-price offers. In practice, public tender commissions have had a hard time proving that particular offers were unreasonable or unfounded, despite the fact that they used their right to ask for clarifications. There is a need to involve experienced experts in these commissions to determine and explain more clearly whether a particular offer is feasible or not. 220. A complementary, much needed improvement is needed in the system for dealing with challenges of procurement procedures. The current system, which allows any participating firm to challenge a result and go through the NCRD and then the court system for definitive resolutions, takes a very long time and remains somewhat duplicative, failing to punish companies that contest outcomes without just cause. In addition to imposing higher fines to disincentivize such actions, a solution that has been considered by the current government and is worth exploring further is the establishment of dedicated courts with exclusive responsibilities in the area of public procurement. These bodies would be required to address any challenges as quickly as possible, which would not only ensure that Romania absorbs EU funds more quickly, but also relieve beneficiaries of the pressure to avoid canceling contracts when service providers fail to deliver on what they had signed up for. The important benefit 103 would materialize in the form of higher quality services and better-implemented projects, with improved impact on the ground. 221. Regarding the control system and its associated corrections, the MA should work with the NARMPP, the UCVPP, and other stakeholders to clarify and better communicate applicable rules and standards. Interviewed beneficiaries call attention to the urgent need of publishing an official collection of cases and their related corrections. Maximum transparency is required in order to help beneficiaries avoid making the same mistakes in the future. A complementary measure is to put together lists of acceptable selection criteria that would not be penalized further down the road. Equally important, there need to be clear-cut standards for which level of sanction applies in a particular case, with consistent application of these rules regardless of which body makes the proposal for corrections. As the final filter before imposing the sanction, the MA must make sure that resolutions are consistent across the board. Moreover, as far as control institutions are concerned, they should take responsibility for their recommendations and share the costs of potential corrections if they failed to prevent beneficiaries from going forward with a faulty public procurement procedure. 222. Beyond improved accountability along the chain and standardization of proposed corrections, controls must be coherent and duplication across various bodies should be eliminated. There is currently an over-regulation of the public procurement area, coupled with unclear, duplicate responsibilities across multiple institutions, which burdens beneficiaries’ capacity to design and implement projects and also makes it hard to enforce accountability. In particular, for ex-ante controls, the value add of the UCVPP is debatable, as projects have to pass through the NARMPP filter anyway. As for ex-post verifications, it is inefficient to check the public procurement procedure with every reimbursement request submitted by beneficiaries. A single round of verifications should suffice if there are clear checklists and standards. 223. Also, for cases where proposals to apply corrections go through, the conciliation procedure should continue to award beneficiaries an opportunity to present their case. Of 271 surveyed beneficiaries who have been through a conciliation process at the MA, 16% assessed it as “excellent” and 31% as “very good,” while 28% were unsatisfied. The qualitative feedback from interviews goes one step further in showing that some beneficiaries do not believe their point of view is taken into account during the conciliation. Part of the problem is, again, the lack of a standardized methodology for assessing potential irregularities and applying corrections based on clear-cut criteria. Some beneficiaries are eventually able to convince MA representatives that there were no issues with a particular public procurement procedure, while others with similar cases receive unfavorable resolutions and end up taking the matter to court. 224. At the end of the day, the fact that there are very few projects without corrections, combined with the insight that very few corrections are tied to suspicions of fraud, urges decision-makers to consider a different approach. Beneficiaries should not suffer financial penalties if they followed applicable legislation and passed through multiple filters implicitly confirming that they were on the right path. In such cases, a flat correction imposed on all projects, negotiated with the EC and supported by the state budget, is likely a better, quicker, and fairer option. The final outcome is similar when all is said and done: once beneficiaries win in court, and they typically win when there is no clear fault of their own, the state budget still has to cover the cost of corrections. It would there be much more efficient to agree on this course of action from the beginning, which would probably unlock payments from the EC faster and also avoid creating substantial frustrations and mistrust within the ROP system. Relations built over years of collaboration can be significantly damaged by perceptions of unfairness, particularly 104 when the most proactive beneficiaries who started projects before the 2011 clarifications (e.g., GEO 66/2011 and NARMPP Order 509/2011) have also been the most exposed to the risk of corrections. Detailed recommendations for improving the public procurement framework 225. Public procurement law must strike a compromise between excessive regulation, meant to ensure strict compliance with governing principles, and freedom of decision, which would allow contracting authorities to select those tenders that suit best their particular needs. The following list of recommendations offers potential solutions for addressing the issues related to the current public procurement framework and ease the burden on ROP beneficiaries implementing projects. It is important to note that some of these ideas are particularly technical, while others lie beyond the MA/IB mandate and require a national-level resolution. That said, the hope is that these suggestions help advance the conversation on an extremely complex and sensitive topic – public procurement – and enable ROP applicants and beneficiaries to organize efficient and effective procedures for awarding contracts. At a minimum, some of these ideas could form the basis of upcoming changes to Romania’s public procurement law (Government Emergency Ordinance 34/2006), in line with corresponding dynamics at the EU level.  An improved public procurement framework should aim to reduce, to a minimum, the number of ROP contracts awarded on the basis of the “lowest price” criterion and instead rely, in most cases, on the “most economically advantageous tender” (MEAT) criterion. This evolution, which concerns the whole public procurement field, resonates with similar concerns at the EC level, which are presented in the next sub-section. There could be a minimum share (expressed as a percentage of the total value of contract award procedures launched yearly by a contracting authority) for the contracts awarded on the basis of the MEAT criterion. These measures involve some changes in the national legislative framework, but they align with the envisaged developments of the Community rules on public procurement. Because they are based on administrative levers, these measures possess the advantage of having an immediate effect on the change of the proportion of contracts awarded on the basis of the MEAT vs. lowest price criterion and enable, at the same time, the unbiased, rigorous verification of their compliance (as a % of total contracts awarded).  Contracting authorities would benefit from recommended sets of evaluation factors, based on broad types of public contracts. These factors should be synthetic and allow for an objective assessment of the tenders, but they have to be, at the same time, relevant to the object of the contract. In this respect, a counterexample is the practice of some contracting authorities who use, as an evaluation factor, “the risks identified with respect to the performance of the public contract,” and respectively the recommendations to reduce/eliminate the identified risks. The highest score is achieved by the operator who identifies the largest number of risks. In the case of these evaluation factors what matters, in the end, is the degree of detail of the risks/recommendations taken into consideration in the technical offer, so it favors that bidder that can divide a truly relevant, general risk, in a plethora of particular risks, which are then counted multiple times when the score is calculated. In such a situation, if the contracting authority does not assume the active role of discarding those irrelevant risks/recommendations, a distortion of the evaluation factors is taking place, favoring quantity instead of quality. This is just a specific example, but it can be extrapolated to a range of situations. Given that, in the past, in exercising its role in the ex-ante verification of procurement documents, NARMPP 105 ordered, in a significant number of cases, the revision of the evaluation factors proposed for the MEAT award criterion, it would be advisable that the sets of evaluation factors be developed and proposed by this institution.  ROP beneficiaries would like to have access to standard methodologies for verifying the explanations provided by those who submit abnormally low tenders. At present, if an abnormally low tender is identified, most contracting authorities limit themselves to acknowledging the justification transmitted by the economic operators. This is due to the lack of appropriate expertise, but also due to the lack of official guidance on specific methods and tools to be used in such circumstances. The fact is that, in the overwhelming majority of cases, contracting authorities verify only if the justifications transmitted contain discrepancies in relation to other documents submitted (whether there are arithmetical errors or some non- budgeted expenses related to mandatory activities) and proceed, where appropriate, to the rejection of tenders, solely for this reason. An adequate identification of abnormally low tenders and the consequent sanctioning of economic operators that use them can be achieved only with a more extensive support provided to contracting authorities by the authorized stakeholders in the field (e.g., the NARMPP).  Better support is needed to help contracting authorities prevent and avoid potential conflicts of interests. This can be achieved by providing ROP beneficiaries with detailed checklists as a mandatory component of the tender evaluation process. These should be drawn up on the basis of the template used by MA-ROP. Raising awareness on conflicts of interests, by the act of filling out such checklists, will allow contracting authorities (in good faith) to avoid omissions due to negligence or to lack of training/experience of the staff. Consequently, a higher degree of accountability of contracting authorities will be achieved, taking into account that, by filling out checklists with clear points, they will no longer be able to invoke an alleged negligence or the absence of wrong intentions. In addition to a minimum methodology for the verification of conflicts of interests (e.g., guidance notes on how to fill out the proposed detailed checklists), it is necessary that the relevant legal provisions be synthesized and clarified. The NARMPP has already begun this effort, as demonstrated through the issuance of Order No. 170/2012 concerning the interpretation of art. 691 of the GEO 34/2006 governing the conflict of interests between economic operators participating in a procurement procedure and people who hold decision-making positions within the contracting authority.  At the same time, ROP beneficiaries should take on an active role in the process of preventing and avoiding conflicts of interests. Besides the self-declarations on the non-existence of conflicts of interests, filled out by the members of the evaluation commission and by the economic operators, it is imperative that contracting authorities verify the accuracy of these statements by using all information available. From this point of view, it is advisable that contracting authorities require participants to the procedures to submit a certificate issued by the Trade Registry (or an equivalent document, in the case of foreign economic operators), in order to have access to the relevant information concerning the structure of ownership and the governing /supervisory bodies of the economic operators. In the case of subcontractors, the following practical problem arises: although the existence of the conflict of interest is verified, according to Section 8 of Chapter II of the GEO 34/2006, also by reference to subcontractors, 106 pursuant to art. 11 (7) of GD 925/2006109, the contracting authorities may not require the subcontractors, by the procurement documents, to submit a certificate issued by the Trade Registry, as this would represent an imposition of a qualification criterion to subcontractors, which is deemed illegal and sanctioned as such. Possible solutions to solve this impediment would be either that the contracting authorities bear the costs necessary for obtaining, from the Trade Registry, the certificates for sub-contractors, or that a mechanism be put in place through which the National Trade Register Office makes available to contracting authorities, free of charge, the information related to subcontractors.  The usage of professional liability insurances should be expanded. According to art. 185 (1) (a) of GEO 34/2006, the economic and financial situation of a particular agent can also be demonstrated with evidence related to the existence of professional risk insurance. With regard to this qualification criterion, NARMPP has decided, by Order 509/2011, to restrict the possibility of using this requirement only to cases expressly regulated by legal provisions (e.g., technical design services.)110 In the past, some contracting authorities have included the requirement concerning the professional liability insurance in the template for the contract, annexed to part of the procurement documents. Although this practice has not been expressly approved by NARMPP, the generalization of this requirement is necessary to improve the delivery of services and, as argued elsewhere in this report, to “clean up” the market by charging higher premiums of those providers who do not fulfill their obligations. In order to apply this requirement, the request for professional liability insurance must be regulated for additional types of services (following the example of design services, mentioned by NARMPP) and rules must be adopted on how to calculate the ceiling of professional liability insurance, in order to avoid situations in which this requirement would be deemed excessive or restrictive.  The time period for conducting a public procurement procedure can be further reduced, as follows:  Setting a maximum period for finalizing the selection/pre-selection stage provided for in the case of restricted procedure, competitive dialogue, and negotiated procedure with prior publication of a contract notice. At present, this stage is not regulated, from the point of view of timeline, which leads to cases of unjustified extensions of the public procurement procedures. In fact, the indication of maximum time limits for the selection/pre-selection stage of the contract award proceedings mentioned above would be a natural complement to the legislative initiative represented by GEO 35/2013 for amending and supplementing GEO 34/2006, which has set out such time limits, but only in case of procedures for awarding the concession contracts for public services, public works, or public-private partnerships. 109 The contracting authority may not require the fulfillment of qualification criteria by the possible subcontractors, but the human and material resources of the declared subcontractors shall be taken into account according to their involvement in the contract which is to be executed, if relevant documents are submitted in this respect. 110 According to NARMPP's interpretation, as it follows from Order 509/2011, the following qualification requirement is unrestrictive: “the bidder must produce evidence regarding the professional risk insurance related to the contract to be awarded, only if there are legal provisions governing such a situation.” At the same time, according to the same legal act, it is deemed restrictive a requirement concerning “the submission by the economic operator of professional risk insurance.” 107  According to art. 200 (1) of GEO 34/2006, within 25 days from the date of the opening of the tenders, the contracting authority shall determine the winning tender. According to paragraph 2 of the same Article, in duly justified cases, the contracting authority may extend only once the evaluation period, but this regulation did not set out a maximum period for this extension (in fact, prior to the amendments brought by GEO 76/2010, GEO 34/2006 provided a specific cap on the extension of the evaluation period). Therefore, in order to avoid the unjustified extension of the evaluation procedure of the tenders, it is advisable to return to the legislative solution prior to the amendments made to art. 200 by GEO 76/2010, respectively to specify the maximum period allowed for the contracting authority to extend the evaluation period. By specifying a precise period, the contracting authorities will be required to finalize the evaluation of tenders and will not be able to further delay this stage of the award procedure. This would also a lead to a higher degree of predictability from the point of view of the economic operators, which will have the opportunity to better plan their activities.  Beneficiaries should explore the solution of concluding framework agreements. In order to eliminate the risk of excessive extensions of a public procurement procedure, contracting authorities may conclude framework agreements for those types of public products/services/works for which a continuous or periodic need is anticipated. The conclusion of a framework agreement has the advantage that the contracting authority goes through a single tender procedure. Further, on the basis of that framework agreement, it concludes, depending on its needs, subsequent contracts; thus, the contracting authority is no longer forced to launch, in order to meet each need (of the same kind), a new public procurement procedure (each procedure, in part, is subject to the risk of unpredictable extensions or delays. This method has also been recently used by the Ministry of European Funds for contracting project evaluators. Essentially, contracting authorities using framework contracts obtain significant savings in time and resources, as well as the significant improvement of their ability to react, increasing the likelihood that projects will be implemented according to the set timetables.  Public procurement procedures can be launched under a “suspension clause,” particularly useful in the context of projects financed from structural funds . Taking into account the significant periods of time related to the selection, the evaluation, and the contracting stages, the deferred launching of the award procedure, after the signing of the financing contract, can lead to failure to comply with the schedule of activities, because of possible delays in public procurement. From this point of view, a possible solution consists in the launch of award procedures under a “suspension” conditional clause (“clauză suspensivă”), the effect intervening after the designation of the successful tender, but prior to the signing of the public contract. Basically, the conclusion of the public contract is delayed/suspended until a particular condition is met (i.e., the financing request submitted by the beneficiary is approved by the MA). This solution requires specifying in the call for tenders and in other procurement documents that the failure to sign the financing contract will determine the annulment of the procedure, on the basis of art. 209 (1) (c) of GEO 34/2006, due to the fact that the signing of the public contract has become impossible. This will ensure an appropriate degree of transparency and protect the 108 contracting authority against a potential liability claim made, in the case of annulment, by the successful bidder.  Another critical issue refers to the need to deter unwarranted challenges of procurement procedures and award decisions. The solution is to retain the participation guarantee from those economic operators whose claim is rejected as unfounded, which provides a fair compromise between the deterrent effect and the prohibitive effect of such a measure, and is aimed exactly at reducing vexatious litigation. This version is not free of all criticism, taking into account that, according to the Romanian Constitution, any special administrative jurisdiction, like the one exerted by National Council for Resolving Disputes (NCRD) through the jurisdictional-administrative procedure carried out on the basis of the GEO 34/2006, should be free of charge and optional. Therefore, the solution to increase the value of retentions applied in case of rejection of complaints as unfounded, should be dealt with caution, in order not to limit, excessively, the access to the administrative-jurisdictional procedures. It may be more advisable to apply some precautionary measures (to stimulate the “self-censoring” on the part of prospective complainers), like publishing on the website of NCRD the decisions adopted, with the extended reasoning, and to organize the case-law of NCRD based on the typology of cases resolved, both in order to provide an unitary character to the decisions taken by NCRD in similar situations and to allow the possible complainers to be aware of the NCRD’s interpretation, thus increasing the degree of predictability of its decisions.  On a related note, there is a need to optimize the court settlement of disputes that concern appeals against the decisions of NRCD. According to legal provisions in force, the filling of an appeal against the NRCD decision shall not suspend, “de jure”, the effects of that decision or of the public procurement procedure in question; moreover, the court cannot be requested to order these measures as provisional measures, the case-law rejecting consistently such requests as inadmissible (in fact, according to art. 2831 of GEO 34/2006, in the framework of the appeal procedure against a NRCD decision, the courts may order, at the request of the interested party, only the suspension of the performance of the contract). However, the contracting authorities tend to delay the conclusion of the public contract until a definitive decision is issued by the court, and avoid to exercise their right (or even their obligation, according to some interpretations of the provisions of art. 93 (2) of GD 925/2006111) to sign the contract, in order not to take the risk of being compelled to pay compensation for the damage caused to the complainer, which has become an appellant. In view of the specific requirements of such judicial proceedings, which restrict the scope of measures that can be applied, the solution is to set up specialized courts at the level of the Courts of Appeal.  Recommended templates should be designed for contractual variation clauses. In the case of public works contracts, the actual performance of the contracts may reveal differences against the quantities estimated by the technical documents. In order to regulate such changes of the object of the contract, where they do not involve a review of the technical solution adopted or of the technical requirements laid down in the specifications agreed to by the contractor, the contracting authorities can include, in the contract template, certain variation clauses according to which the final price of the contract is to be determined as the product between the prices 111 In accordance with art. 93 (2) of GD 925/2006, the contracting authority may not delay the signing of the public contract/framework agreement by more than 7 days from the date of expiry of the time limits provided for in art. 205 of the emergency ordinance, for reasons of its own making. 109 per unit, initially tendered, and the actual quantities, resulted from works on the ground. Obviously, this option is valid only if the price of the contract has not been determined in its absolute value, but only provisionally, as a product between the prices per unit and the quantities of works estimated by the technical documentation. The legality of variation clauses has been confirmed also by the letter no. MARKT/C3/EP/kr (2012) 677516 of the General Directorate Internal Market and Services of the European Commission, but is subject to the cumulative fulfillment of the following requirements: a) the formula for calculating the final price has to be defined, objectively, and it should be communicated to all prospective bidders, by including it in the procurement documents or in the contract template; and b) variations of the quantities of work and, implicitly, of the price actually paid have to result from the mechanical application of the variation clauses. This solution has the advantage of being suitable to both the current tendering method for public works contracts, based on detailed items, as well as to the method based on consolidated components, which is recommended by this report. The adoption of the solution would be facilitated by the development of standard contractual clauses (preferably by NARMPP), as well as by defining the specific implementation mechanism, in order to avoid applying it in the case of modifications deemed as “substantial.” It is important to emphasize that, as the European Commission has also specified, by using such contractual clauses, the variations of the quantities of works and of the final price do not constitute modifications of the contract per se, but only the application of the provisions of the initial contract.  All actors involved in public procurement should demonstrate standardized interpretations of the legal provisions on public procurement. This can be done by issuing instructions (like instruction no. 1/2013 of NARMPP) or by publishing good practices guides with detailed examples of the most problematic issues of public procurement, with an emphasis on those elements that tend to generate the largest number of financial corrections. A particular emphasis should be given to clearly specifying the level of financial correction warranted by different circumstances; this exercise is all the more necessary since the nature, the purpose, and the organization of institutions with control prerogatives differ significantly. Funding beneficiaries and the consultants should be adequately informed about the official interpretation applied in these various situations; through its essentially preventive effect, this type of measure would contribute to the improvement of the quality of the public procurement process and to the reduction of those deviations determined by the equivocal character of some legal provisions, corroborated with the lack of experience of contracting authorities.  Agents involved in the verification of public procurement procedures should demonstrate an adequate standard of professionalism and control procedures should be standardized. The basic purpose is to warrant that conclusions drawn from the review conducted by one of the institutions be accepted and respected by the other institutions in order to avoid multiple administrative controls.112 Of course, this mechanism should be refined according to the 112 As an example in this respect, it can be mentioned the procedure MA-ROP - PO/VIII/3 Procedure for the verification of Public procurement procedures from the Manual for the verification and the authorization of expenditure (M VIII/I - 2nd Edition (I) (1), which provided that, in case of those tender procedures for which it has been issued a Point of View by NARMPP/ Advisory Opinion by UCVPP , the IB experts perform the administrative check of the tender dossier and draw up a checklist of the contract, while only a copy of the report of activity UCVPP/NARMPP and the checklists drawn up by the IB experts are submitted at MA-ROP. 110 regulations specific to each actor involved in the verification of public procurement procedures. At a minimum, each institution should take ownership over the conclusions formulated by one of its agents, in the form of a mechanism similar to the “res judicata”. The repeated controls carried out by the same actor involved in the verification of public procurement procedures, through different agents, have an effect as harmful as the heterogeneous practice of control institutions, and maintain the status of uncertainty among beneficiaries.  The NARMPP’s role in the ex-ante control of procurement documents requires optimization. By amendments brought to GD 525/2007 by GD 801/2011and to the GEO 34/2006 by Law no. 279/2011, the NARMPP has been entrusted to evaluate the compliance of procurement documents with applicable laws in the field of public procurement. Although the positive effects of this measure are indisputable (despite the initial blockage in carrying out the public procurement procedures), the implementation of this mechanism has also revealed the existence of some deficiencies, which are also mentioned in the Position Paper on proposals for the improvement of legislation in the field of public procurement in Romania, drawn up by the Romanian Association of Management Consultants. Specifically:  The rejection of procurement documentation is often insufficiently explained, which means that economic operators do not always have sufficient clues on the elements that should be reviewed. In order to remedy this situation, the NARMPP should provide a detailed explanation of the reasons for rejecting procurement documents, by reference to the specific facts presented and not just by invoking the applicable legal basis;  Following the resubmission of the procurement documents, adjusted on the basis of the comments made by the NARMPP, new grounds for rejection sometimes are communicated, which had not made the object of the initial rejection. To avoid a long and unpredictable process, it is advisable that the NARMPP be required to rule, from the first verification, on all non-conforming aspects of the procurement documents, and the subsequent verifications be limited exclusively to checking those aspects that have been notified initially.  Another recommendation regarding the activity of NARMPP concerns the reintroduction of the prerogative (referred to in art. 2 (1) (k) of GD 525/2007) regarding the issuance of opinions in response to requests for clarification from contracting authorities/prospective bidders/managing authorities. This is essential for devising a uniform practice and for avoiding subsequent delays and corrections. The effectiveness of this measure would be considerably enhanced if, according to previous recommendations, the interpretative statements communicated by the NARMPP would be officially accepted and would be enforceable against both the issuing agency, as well as against other actors involved in the verification of public procurement.  The institutional framework of ex-ante control of public procurement procedures should be unified. According to GEO 30/2006, the public procurement ex-ante control body is the Unit for Coordination and Verification of Public Procurement (UCVPP). The conclusions issued by the UCVPP observers have advisory status; only NARMPP is allowed to determine irregularities and impose sanctions provided for by GEO 34/2006. As mentioned above, through the amendments brought by GD 801/2011 and by Law 279/2011, NARMPP was entrusted, for its part, with ex- 111 ante control competences. At present, there are two institutions involved in this type of control. In principle, the delimitation of competences between UCVPP and NARMPP is relatively clear: the former controls the process prior to the submission for publication of the invitation to tender/contract notice, and the latter takes over the process after this point onward). The opportunity of transferring in full the ex-ante control activity to NARMPP has to be analyzed, as this would represent a further step towards the simplification of the institutional framework of public procurement. Obviously, a decision in this respect will have to take into account the major implications that the implementation of such measures will have, from a strictly organizational point of view, since the ex-ante control mandate exercised by UCVPP requires an adequate presence in the territory, which NARMPP is currently lacking. 226. A few other potential recommendations have surfaced during interviews with beneficiaries and other stakeholders, but – for various reasons – these would be particularly challenging to implement.  One possibility for addressing the negative effects of dumping-price tenders is to eliminate the tender with the lowest price and/or the one with the highest price. The purpose of such a measure would be to encourage those tenders with prices that are close to the average of the financial proposals, and to penalize those economic operators that offer prices considerably lower than the estimated value of the public procurement. The measure is severe and burdened by numerous problems, therefore its use is not recommended, for the following reasons:  The potential incompatibility with the EU public procurement provisions, taking into account that art. 55 (1) of Directive 2004/18/EC provides, specifically, in the case of tenders that seem abnormally low in relation to the tendered products, works or services, the obligation of the contracting authorities to request, in writing, details regarding the components of the tender before it may reject it.113 Therefore, if the bidder is not granted the possibility to justify its price, the elimination of the tender with the lowest price (which may not necessarily fall within the scope of the legal definition of the abnormally low price) and the elimination of the tender with the highest price (as long as it falls within the estimated value of the contract), risks to be contrary to Community rules;  The need to amend the public procurement legislation both at Community level (taking into account the issues mentioned at the previous point), as well as at the national level (if such a measure is not expressly provided for, it cannot be applied only on the basis of a recommendation of best practices);  The artificial limitation of competition (there is a risk to restrict competition, in the case of procedures with few bidders), which is all the more inequitable as it could lead to the elimination of a tender with a marginal difference from the next admissible tender (in the increasing order of prices).  Another idea is to set minimum standards of cost or discount limits against maximum standards of cost. This solution has the advantage of defining a clear reference to which the contracting authorities can refer in the process of evaluating tenders, thus providing an 113 Besides, the same obligation for the contracting authorities is regulated also by the Practical guide to contractual procedures within the framework of external actions of the European Union (PRAG) (the version applicable starting 14.03.2013, available at http://ec.europa.eu/europeaid/work/procedures/implementation/practical_guide/). 112 adequate protection against economic operators offering prices that are not the result of a real competitive environment. However, the stipulation of a national rule for automatically rejecting tenders below the limits laid down could also be considered incompatible with Community law, for the reasons outlined under the previous point. In addition, the use of standards of cost may be unfit in the case of certain types of public procurement (in particular, in the field of services), and, in those areas that are suited for such an approach, the definition of standards of cost would imply a considerable administrative effort. New dynamics in the EU public procurement framework 227. The proposals for the three directives on public procurement will bring a significant reform to the Community law framework.114 Some of the new provisions, which will become mandatory if the Proposal for a Directive on public procurement is approved, are already included in the Romanian legislation on public procurement (like the right of the bidders to provide, initially, only a self- declaration as preliminary evidence that the economic operator fulfills the relevant qualification criteria), so transposition should be easy. Nonetheless, more attention should be given, at the national level, to how existing legal mechanisms and instruments can be applied in practice. The following list is not a comprehensive review of EU-level changes in public procurement, but merely highlights some issues that will interest ROP beneficiaries and IB/MA staff. 228. The Proposal for a Directive on public procurement provides for the reduction of the minimum time limits for receipt of requests to participate and for receipt of tenders . The Proposal for a Directive on public procurement, as set in the compromise text, compared to Directive 2004/18/EC, establishes the following minimum time limits: Figure 52. Time limits for key procurement steps Procedure Legal basis Minimum time limit Minimum time limit Minimum time for receipt of for receipt of limit for receipt requests to tenders (without a of tenders (with participate prior information prior notice) information notice) Open Directive 2004/18/EC N/A 52 22 procedure Proposal for a Directive N/A 35 15 Reduction N/A 17 7 Restricted Directive 2004/18/EC 37 40 22 115 procedure Proposal for a Directive 30 30 10 114 The Proposal for a Directive of the European Parliament and of the Council on procurement by entities operating in the water, energy, transport and postal services sectors – COM(2011) 895 final, the Proposal for a Directive of the European Parliament and of the Council on public procurement – COM(2011) 896 final and the Proposal for a Directive of the European Parliament and of the Council on the award of concession contracts – COM(2011) 897 final. 115 The Proposal for a Directive on public procurement states that Member States may provide that all or specific categories of sub-central contracting authorities may set the time limit for the receipt of tenders by mutual agreement between the contracting authority and the selected candidates, provided that all selected candidates have the same time to prepare and submit their tenders; In the absence of agreement on the time limit for the receipt of tenders, the time limit shall be at least 10 days from the date on which the invitation to tender was sent. 113 Procedure Legal basis Minimum time limit Minimum time limit Minimum time for receipt of for receipt of limit for receipt requests to tenders (without a of tenders (with participate prior information prior notice) information notice) Reduction 7 10 12 117 Competitive Directive 2004/18/EC 37 - - procedure Proposal for a Directive 30 30 10 with Reduction 7 N/A N/A 116 negotiation Competitive Directive 2004/18/EC 37 - - dialogue Proposal for a Directive 30 - - Reduction 7 - - The significant reduction of time limits for receipt of requests to participate and for receipt of tenders will accelerate the public procurement procedures. Although the time limits set by the Proposal for a Directive on public procurement are minimal, the Member States having the right to establish longer time limits, in Romania’s case, it is recommended to make the adjustment to the new minimum time limits. This adjustment will also speed up the process of implementing the projects financed through the ROP and will minimize the risk of incurring delays versus the timetable of activities, as set in the financing application. Nonetheless, contracting authorities should be made aware of the fact that the actual time limits must be set in accordance with the complexity of the contract and/or its specific requirements. This obligation is already provided for by the Romanian legislation (art. 71 of GEO 34/2006), but it will gain even greater importance if the time limits are reduced. In order to help contracting authorities, it would be useful to define guidelines for determining the proper time limits for a public procurement procedure, so that these are adequate and sufficient, as required by the aforementioned legal basis. 229. The draft for the Proposal for a Directive on public procurement, initially submitted by the European Commission, included an article which defined, in detail, the main aspects of the conflict of interests (art. 12).118 In the compromise text, this article was extensively trimmed and is limited to defining the persons (staff members of the contracting authority or of a procurement service provider acting on behalf of the contracting authority) and the interests (financial, economic, or other personal interests) that may cause a conflict of interests. The fact that, unlike Directive 2004/18/EC, the future Directive regulates the conflict of interests, albeit to a lesser extent that the initial draft, is relevant for 116 The negotiated procedure with prior publication of a contract notice, provided by Directive 2004/18/EC, was renamed, in the Proposal for a Directive on public procurement, the competitive procedure with negotiation. 117 Directive 2004/18/EC does not establish a minimum time limit for receipt of tenders, for the negotiated procedure with prior publication of a contract notice. The proposal for a Directive on public procurement, as set in the compromise text, provides, in the case of the competitive procedure with negotiation, for a minimum time limit for receipt of tenders, from the date on which the relevant invitation was sent. 118 In order to ensure that a public contract is awarded impartially and objectively, the initial draft submitted by the European Commission, required that candidates provide a declaration on honour that they have not undertaken and will not undertake an action qualifying as illicit conduct [like (i) unduly influencing the decision-making process of the contracting authority or obtaining confidential information that may confer upon them undue advantages in the procurement procedure, or (ii) entering into agreements with other candidates and tenderers aimed at distorting competition, or (iii) deliberately providing misleading information that may have a material influence on decisions concerning exclusion, selection or award]. In the compromise text, this article was eliminated. 114 the attention given to preventing, identifying, and remedying such aspects. Even though the transposition of these provisions will not change considerably the current national legislation (apart from the definition of the types of interests, the other aspects pertinent to the conflict of interests are already regulated by GEO 34/2006), special care should be given to the way in which these regulations are actually applied. 230. The Proposal requires contracting authorities to exclude from participation in a procurement procedure the economic operators that are in breach of their obligations related to the payment of taxes or social security contributions. A direct consequence of this fact will be the obligation of contracting authorities to request from economic operators the relevant documents in order to check if the aforementioned exclusion grounds apply. Noncompliance with this may lead to serious outcomes, including awarding a contract to an economic operator that should have been, in fact, excluded. In the case of public procurement procedures organized under Romanian law and, in particular, those financed through the ROP, contracting authorities have usually made use of the right provided for by art. 181 (c) of GEO 34/2006 [which transposes art. 45 (2) (e) and (f) of Directive 2004/18/EC], so the future change of the national legal framework, pursuant to the Proposal for a Directive on public procurement, will only give legal force to an already existing good practice. But since not all contracting authorities require that economic operators participating in a procurement procedure prove that they have fulfilled their obligations relating to the payment of taxes or social security contributions, it will still be necessary to raise awareness among contracting authorities regarding this obligation. As an exception, the compromise text on the Proposal for a Directive on public procurement provides that the exclusion grounds shall no longer apply when the economic operator entered into a binding arrangement with a view to paying the due taxes or social security contributions, including, where applicable, any interest accrued or fine.119 This exception is designed to help economic operators that have not paid the owed taxes or social security contributions on time, but which have entered into agreements to pay those arrears, improve their economic standing. 231. A new reason for excluding economic operators from a procurement procedure includes significant or persistent deficiencies in the performance of a substantive requirement under a prior contract which led to early termination of that prior contract, damages or other comparable sanctions [art. 55 (3) (e)]. This provision allows contracting authorities to sanction those economic operators that have not fulfilled, properly and according to the quality standards agreed upon, their previous contractual obligations, by excluding them from the procurement procedure.120 Although, in theory, it may relate to any contract concluded by any economic operator, in fact, due to lack of databases on the 119 Similar provisions are already available in the national legislation: according to art. 9 (1) of Order 509/2011, „criteria regarding the obligations relating to the payment of taxes or social security contributions to the state budget are to be considered fulfilled if the economic operators benefits from payment facilities, like installments or compensations, approved by competent authorities.” 120 In the initial draft, submitted by the European Commission, this reason for exclusion did not refer to the early termination of the prior contract or to damages or other comparable sanctions, and it regarded only similar prior contracts, concluded with the same contracting authority. It also stated that, in order to apply this exclusion ground, contracting authorities should provide a method for the assessment of contractual performance that is based on objective and measurable criteria and applied in a systematic, consistent and transparent way, and that the contractor in question should be given the opportunity to object to the findings and to obtain judicial protection. This provision came under criticism in the Report of the European Parliament (http://www.europarl.europa.eu/sides/getDoc.do?type=REPORT&mode=XML&reference=A7-2013- 7&language=EN) and in the Opinion of the Committee of the Regions on ‘Public procurement package’ (2012/C 391/09), both documents arguing for its elimination. 115 performance of all contractual agreements, this ground for exclusion will, more likely, prove its efficiency in relation to the contracts that the same contracting authority has previously. Even with a limited applicability, this reason for exclusion will be very useful in eliminating unreliable economic operators and protecting contracting authorities faced with the risk of awarding public contracts to operators that have already proved they are not able to fulfill their obligations. A positive side effect of establishing this exclusion ground will be the increased accountability of both parties involved in a public contract: the economic operator, for the way in which it performs its contractual obligations; and the contracting authority, for the way in which it supervises the contract’s fulfillment. Even though a similar regulation already exists in the Romanian legislation,121 its effects have been limited. Under the ROP framework, the exclusion of unreliable economic operators could be achieved by creating a readily available database (black list) with the operators that have shown significant or persistent deficiencies in the performance of a substantive requirement under a prior public contract. 232. The Proposal for a Directive on public procurement, as set out in the compromise text, provides for the wider use of self-declarations by economic operators, in the form of the European Single Procurement Document (art. 57). 122 This provision states the obligation of contracting authorities to accept that economic operators submit, instead of certificates issued by public authorities or third parties, the European Single Procurement Document consisting of an updated self-declaration as preliminary evidence for fulfilling part of the selection criteria.123 The main aspects of the European Single Procurement Document are already regulated in the national legislation on public procurement: art. 11 (4)-(6) of GD 925/2006 stipulate the right of economic operators to submit, initially, only a self- declaration by which they attest the fulfillment of the qualification criteria set in the tender documents. Still, establishing a standard form and a detailed mechanism for the European Single Procurement Document may lead, in Romania’s case, to a wider use of self-declarations, by creating a unitary framework of interpretation and implementation (the use of the provisions of art. 11 (4)-(6) of GD 925/2006 has been hampered especially by the lack of an official point of view for interpreting the legal requirement for the economic operator to state, within the self-declaration, “in a succinct, but precise manner, the actual way in which it fulfills” the qualification criteria). The next step in reducing the administrative burden put on economic operators participating in a public procurement procedure is to develop national databases, in each Member State, directly available to contracting authorities, free of charge. In this way, economic operators would not be required to always resubmit supporting documents or other evidence, as the contracting authorities could obtain the documents related to the European Single Procurement Document directly by accessing those databases. 121 1 According to art. 181 (c ) of GEO nr. 34/2006 the contracting authority has the right to exclude from a public procurement procedure any bidder/candidate which, in the last 2 years, did not fulfill or has poorly fulfilled its contractual obligation, due to reasons pertaining to the economic operator in question, and thus has caused or could cause damages to the beneficiaries of those contracts. 122 The draft initially submitted by the European Commission provided also for the introduction of an European Procurement Passport, issued by national authorities, which would have included information on economic operators, relevant from the point of view of public procurement [like, for example, certifying that the economic operator has not been the subject of a conviction by final judgment for one of the reasons listed in Article 55(1) or that the economic operator is not the subject of insolvency or winding-up proceedings]. 123 The self-declaration, which the European Single Procurement Document consists of, confirms that the economic operator: (i) is not in one of the situations referred to in Article 55 in which economic operators shall or may be excluded; (ii) meets the relevant selection criteria that have been set out pursuant to Article 56; (iii) where applicable, fulfills the objective rules and criteria used for reduction of the number of otherwise qualified candidates to be invited to participate in restricted procedures, competitive procedures with negotiation, competitive dialogue procedures and innovation partnerships. 116 233. The growing Community trend of favoring „the most economically advantageous tender” (MEAT) award criterion is easily discernible. While the initial draft of article 66 (1), as submitted by the European Commission, provided for two award criteria – the „most economically advantageous tender” and the „lowest cost”124 – the compromise text reads that „contracting authorities shall base the award of public contracts on the most economically advantageous tender.” But the compromise text brings an important nuance: although it mentions only one award criterion – the „most economically advantageous tender,” the actual definition of this criterion reveals the role that the price or cost still plays. Thus, the most economically advantageous tender from the point of view of the contracting authority shall be identified on the basis of the price or cost (using a cost-effectiveness approach, such as life-cycle costing125) and may include the best price-quality ratio, which shall be assessed on the basis of criteria including qualitative, environmental, and/or social aspects linked to the subject matter of the public contract. Consequently, contracting authorities have the possibility of applying the revised MEAT criterion, based solely on the cost or price offered. Notwithstanding this provision, the proposal also states the right of Member States to provide that contracting authorities may forbid using price or cost as the sole award criterion or may restrict their use to certain categories of contracting authorities or types of contracts. It comes thus to the decision of the Member States whether they make use of this provision or not; in view of the recommendations relevant for the ROP, Romania should exercise this right and limit the possibility for contracting authorities to use the lowest cost as the single award criterion to certain well-defined, justified situations (for example, procurement of goods with a high degree of standardization, like fuel). Picture 5. Complex projects require contractors with special competencies (not always the lowest price offer) 124 The European Commission proposed replacing the term price with cost. 125 This refers to the possibility given to contracting authorities of taking into account parts or all of the costs over the life cycle of a product, service or works (like costs relating to acquisition, costs of use, maintenance costs, end of life costs, such as collection and recycling costs, and costs imputed to environmental externalities linked to the product, service or works during its life cycle). 117 234. As for criteria used to determine the best price-quality ratio, the Proposal introduces another major change: the possibility of using criteria like the organization, qualification, and experience of staff assigned to performing the contract. This is particularly useful in situations where the quality of the staff employed can significantly impact the level of performance of the contract, allowing contracting authorities to evaluate and score tenders based on the staff designated for performing the work. Romanian contracting authorities have a poor experience defining and using selection criteria based on staff qualification/experience, as irregularities linked to these issues have often led to corrections. Therefore, it will be necessary to draft and formally endorse national guidelines for defining appropriate evaluation criteria related to the staff assigned to perform the contract, in order to avoid the use of disproportionate or inadequate criteria. A useful example (albeit long overdue) is Instruction 1/2013 issued by NARMPP, which explains how to draft qualification criteria regarding the studies, professional training, and qualification of staff assigned to performing a contract. 235. Additionally, the proposal for a Directive on public procurement regulates, for the first time, the right of Member States to enable the contracting authority to transfer due payments directly to subcontractors, where possible (art. 71).126 This provision may prove useful to combat the effects that arise when the main contractor does not pay subcontractors on time, even if the contracting authority has already sent the payments; it also ensures the protection of the subcontractors’ economic and financial standing by keeping up the estimated cash-flow. In the past, the non-payment of subcontractors (which, in turn, refuse to perform their contractual obligations) has caused delays and, thus, affected the implementation of the projects, making the opportunity cost incurred by the contracting authority considerably higher than the damages owed by the main contractor. Transposition of this provision into national law will entail an in-depth analysis of the Romanian legal system. Special attention will have to be awarded to establishing the competent jurisdiction and the settlement procedure for the objections raised by the main contractor with respect to undue payments. 236. Finally, the proposal defines more cases in which contracts and framework agreements may be modified without a new procurement procedure [art. 72 (1)]. Out of these, of particular significance is the one stipulated by art. 71 (1) (a), according to which modifications may occur, irrespective of their monetary value, where they have been provided for in the initial procurement documents in clear, precise, and unequivocal review clauses (which may include price revision clauses), or options. In the case of public procurement procedures conducted by Romanian contracting authorities, and in particular in the case of public procurement financed through ROP, this Community law provision might prove useful especially when dealing with public works contracts, because it would represent the legal basis for modifying the quantities set in the initial contract. Like with other changes of the Community legal framework, the proper use of this provision depends on the development of guides or recommendations regarding the actual wording of these clauses in order to avoid their misuse, which would lead to financial corrections. In addition, the proposal states that a contract may be modified without any need to verify whether the conditions for a substantial modification are met and without a new procurement procedure, where the value of the modification is below both of the following:  The thresholds set out by art. 4 of the Proposal for a Directive on public procurement (which determine the procurements that fall under the scope of the Directive); and 126 The Proposal for a Directive on public procurement provides also that such measures may include appropriate mechanisms permitting the main contractor to object to undue payments. In order to ensure transparency, the arrangements concerning that mode of payment should be set out in the procurement documents. 118  10% of the initial contract value for service and supply contracts and below 15% of the initial contract value for works contracts. In order to be deemed acceptable, these modifications may not alter the overall nature of the contract or framework agreement, and where several successive modifications are made, the value shall be assessed on the basis of the net cumulative value of the successive modifications. Together with the provision regarding the modification of the contract/framework agreement by review clauses or options, this provision will give more flexibility to contracting authorities, including those implementing projects financed through the ROP. Especially when it comes to public works contracts, this will allow authorities to adapt the contract to the actual situation on the ground. The Design Stage: Technical Documentation 237. Beneficiaries often mention the low quality of technical documentation as a leading cause of implementation delays and ineligible expenditures. A total of 48% of the interviewed subjects consider this factor to be an extremely/very important obstacle against effective absorption of ROP funds, for many reasons, which were presented at length during field interviews. Simply put, beneficiaries are currently exposed in their relationship with both project designers and construction contractors: the latter may seek to point out faults in the technical project, hiking the agreed price set through the public procurement process; the former are reluctant to agree to changes of the project and put the blame on the contractors in the field for failing to apply the technical documentation. The following sections describe the main challenges of the status quo and a range of potential recommendations. Accountability mechanisms 238. To begin with, some issues related to poor quality technical documentation have originated in having experts hired by a central body, without a direct link to ultimate beneficiaries of investment projects. Specifically, Government Decision 811/2006 mandated the then Ministry of European Integration (MEI) to finance the preparation of project documentation to be financed from ROP 2007- 2013. These were actually “back-up projects” inherited from the PHARE CES 2004-2006 program. Several problems emerged from this set-up: technical experts contracted by MEI elaborated the documentation without paying much attention to why the projects had scored less than 50 in the first place; also, in 2006 the list of eligible expenditures had not yet been issued for the ROP, so the beneficiaries ended up incurring significant non-eligible expenses during the implementation phase. Moreover, this documentation was generally of poor quality, but beneficiaries had little opportunity to correct it, both because of lack of capacity and lack of time: when they received the technical projects financed by the MEI, they rushed to apply to the recently launched ROP. It is recommended that such situations are avoided in the future by always establishing the direct link between beneficiaries and technical experts, with possible technical support from RDAs. 239. Even when beneficiaries contract technical experts directly, the overall quality of technical documentation for ROP projects tends to be unsatisfactory as a result of lack of capacity and lack of proper accountability mechanisms. For one, technical experts in charge of feasibility studies and technical projects are often selected based on the lowest price criterion – again, this is because NARMPP tends to reject documentation based on other criteria or because beneficiaries simply fear future potential corrections due to “restrictive” conditions, so they generally award the contract to the lowest bidder. Also, the procedural deadline for elaborating the technical project is six months, and few 119 beneficiaries decide to start on that before then. This is mainly because they are weary to invest in producing technical documentation for projects that may not receive financing. Otherwise, auditors from the Court of Accounts may challenge their decision to spend taxpayers’ money on investments that do not materialize. Some beneficiaries have considered launching public procurement procedures for technical documentation in advance, awarding the contract, and ordering the winning bidder to commence work only after the financing is secured. Further research is needed to verify the legality of this potential solution given currently applicable legislation. But the deeper question is around the technical designers’ responsibility for delivering high-quality outputs. Under current legislation, designers are only liable under extreme circumstances (e.g., a building collapses as a result of a faulty project). This should be amended to cover errors and poor-quality work, even if discovered after the project is received by beneficiaries. 240. While technical experts are sometimes poorly trained and often lack sufficient time to develop solid technical projects, public authorities also lack the capacity to draft stronger contracts and properly verify submitted documentation. For their part, technical project evaluators mainly perform an “administrative” check on the documentation to see if all supporting documents are included, without having the capacity to check chosen technical solutions in depth, the lists of required materials and quantities, etc. There are also technical project verifiers, who should be contracted directly by beneficiaries and look out for their best interest, in accordance with currently applicable legislation (Law 10/1995 for quality of construction works). In practice, numerous beneficiaries receive technical documentation already stamped with the seal of approval of a verifier, paid by the designer of the same project. Such situations can be avoided through more rigorous contracts that hold parties accountable for errors at every step of the chain (beneficiaries for failing to abide by law 10 for quality in construction works; technical experts for any clear mistakes in the project; verifiers for approving faulty documentation; engineers for not supervising work progress properly, etc.). 241. Poor technical documentation eventually leads to significant ineligible expenditures when there are differences between the technical project (TP) and the situation on the ground during implementation phase. EU-level regulations do not accept all such discrepancies as “unpredictable.” More specifically, technical errors in developing the documentation are not considered to be unpredictable and do not qualify for the 10% margin of expenses that can vary from the original budget baseline. Only natural disasters or truly unexpected “discoveries” when conducing renovation works in old buildings have generally been accepted as eligible additional work. To some extent, it is reasonable to say that errors in project design should not be overlooked and expensed from EU funds. Preventing them would require holding project designers accountable for the quality of their output. 242. Equally important, differences between EU and state-funded projects still persist, well after Romania’s EU accession, which confuses beneficiaries and crowds out EU investments. Typically, state- funded projects face more flexible regulations, including with respect to what qualifies as unpredictable required works. It would be advisable that all public investments follow the same rules, regardless of the funding source, with the important note that such rules and their enforcement should be reasonable. On unpredictable expenditures specifically, an optimal approach would allow beneficiaries to claim them as eligible whenever they cannot be anticipated, despite best efforts by beneficiaries and their contracted service providers (including technical project designers and verifiers), as mandated by current legislation (particularly law 10/1995 on the quality of construction works). 243. In particular, MA and IB teams of experts should help beneficiaries avoid discrepancies between the technical documentation and the situation on the ground by endorsing standard contract 120 forms. These would be part of the public procurement documentation and help ensure that qualified, professional bidders enter the competition. Best practices would include the following provisions:  A clear stipulation of the designer’s responsibility to deliver a high-quality output: This would include applicable fines and sanctions in the eventuality of clear mistakes that can be attributed to the design team. One requirement could be to have professional insurance for designers, which some beneficiaries are already beginning to request through the contract form. There are existing companies that will provide this type of professional insurance. Over the long-term, insurance schemes are particularly helpful for rewarding competent service providers and penalizing those who deliver inadequate technical documentation.  A complete timeline for every deliverable, including payment terms: Some beneficiaries are already applying a system of payments in tranches, making sure that the designer only receives the last payment when the project is successfully completed. This ensures that technical designers stay close to beneficiaries throughout the implementation phase and collaborate with engineers and construction crews in order to find optimal solutions.  Clear requirements that all property rights over the technical documentation belong to the beneficiary. In some cases where this was not explicit in the contract, designers have refused to allow any changes to the projects, invoking their proprietary rights over the delivered content, despite the fact that they had been paid for a particular output. Picture 6. A good technical design is essential for a high-quality execution phase 244. Another solution for reducing frictions among designers, beneficiaries, and construction firms during the implementation phase is to allow design-and-build contracts. This has not been feasible under the current ROP system, which required beneficiaries to deliver the technical project as part of their application. In principle, this idea rested on good intentions – i.e., to compensate for beneficiaries’ lack of technical capacity to receive and verify complex documentation submitted by designers. However, as noted earlier, since evaluators did not perform field visits as part of the process and they 121 faced their own timing and cost constraints, evaluations were mostly administrative checks to make sure that all required documentation that should be part of a technical project was indeed included. Beyond that, evaluators could not intervene on the actual technical solutions chosen, the list of materials, or other substantial content. Moreover, the technical project does not become an annex to the financing contract. In principle, design-and-build contracts should be possible for ROP projects, where beneficiaries decide to opt for this alternative, which promises to clear much of the back-and-forth between designers and contractors. This would also require, however, the elaboration of improved and more detailed feasibility studies, which form the basis of public tenders for awarding design-and-build projects. The winning bidder takes on the responsibility of developing the technical project and implementing it, so any discrepancies during implementation must be solved without incurring additional costs for the beneficiary, unless particular construction works are truly unpredictable, in which case they would qualify as eligible. Bill of Quantities, Schedule of Rates, and Detailed/Consolidated Items 245. Depending on the type of contract chosen by the beneficiary, different methods may work best for defining the work components to be executed by the contractor. If the beneficiary prefers to design a project and then contract out its execution (FIDIC Red Book), the technical designer will be mandated to develop a bill of quantities (BOQ). The BOQ is used in the tendering procedures and helps contractors price the work required, and also ensures consistency and comparability across different offers. Designers in Romania typically develop the BOQ with the help of a computerized system that has data on standard, defined items (based on a collection of construction items of work – “indicatoare de norme de deviz”). Interviewed beneficiaries have pointed out multiple problems with the current system, including the fact that the database of items is not always updated and the software programs used by designers and contractors are not always compatible with each other. 246. More importantly, the current method preferred by designers is to use BoQs based on detailed items instead of consolidated components (“articole comasate”), which generates difficulties during the implementation phase. To give a basic example, ROP beneficiaries and monitoring officers should not be interested in how much cement goes into a particular wall for a foundation. Rather, they should track the quality of the cement used and whether that particular wall fits the required indicators (depth, width, length, etc.), within the total budget allocated. This allows contractors greater flexibility in how they deploy resources and greatly reduces the burden of tracking every single expense and making sure that it matches the corresponding budget line. The approach based on detailed articles dates from the Communist era, when the government implemented all projects and required extensive explanations for every material used, including its quantity and its price. This is not needed under a system where a beneficiary and a service provider agree on a defined list of major components to be delivered as part of a construction project, along with set indicators and aggregate costs. As one expert put it: “When I buy a car, I am not interested in how much plastic went into it, how many screws are under the chairs, or, for that matter, how many people worked at it and for how long. I agree to pay for a list of defined technical characteristics and I need to feel confident enough that the product delivers on what is promised.” Beyond these simple examples, the essential point is that the best practice of using consolidated items should be adopted by both beneficiaries and designers. 247. Compared to model described above, the design-and-build contract (FIDIC Yellow Book), should provide even more flexibility for contractors with respect to the quantities used, within the limits defined by the beneficiary. In this case, there is no need to define specific quantities ex-ante if the contractor is also in charge of developing the technical project. According to a recent JASPERS 122 report, if a bill of quantities BOQ is included by the designer, “it can [then] be difficult to agree on the cost of variations if the work defined in the BOQ is higher or lower than that shown on the tender drawings or defined in the general specifications.”127 Ultimately, the principle enunciated above remains valid: the simpler it is for beneficiaries to define construction work components and monitor their completion, the more efficient the whole system becomes. This also translates in beneficiaries and the construction site supervisor having more capacity to focus on other critical items, instead of tracking every single item and material that goes into producing a certain deliverable. 248. At the end of the day, the type of contract and BOQ used should be above all the beneficiary’s choice based on which model delivers greater benefits. A few supporters of the current preference for detailed items note that it is a method for preventing corruption and ill-intentioned cost changes. In practice, however, this does not work because contractors who would resort to such practices have easy ways of tricking the system. What is more, it should not be important whether the list of detailed items is implemented exactly as requested, but rather that the final product is according to all requested specifications. Keeping in mind the various contract forms in use, MA and IB experts should advise beneficiaries of ROP funds to rely on the system that is easiest and most efficient to implement. As with the other improvements in contract forms suggested in this chapter, these best practices can become part of the Implementation Manual and can also be taught in specialized seminars and workshops. The Execution Stage: Contracts, Monitoring, Reimbursements The FIDIC standard contract 249. In particular, the FIDIC standard contract forms would help beneficiaries manage complex projects better, while protecting the interests of all stakeholders involved. FIDIC, which stands for the International Federation of Consulting Engineers, is an international private organization that has developed since the 1950s various contract forms to facilitate the implementation of infrastructure and construction projects. Gradually, the superior quality of these proposed forms has translated into more and more people adopting them around the world. The main benefits of the FIDIC contracts revolve around a balanced sharing of risks between the beneficiary and the contractor and a greater degree of familiarity of international investors with the specific conditions and procedures mandated by these contracts. Indeed, the FIDIC contracts have already demonstrated their usefulness by covering a broad range of situations that may arise during the project’s implementation and having clear mechanisms for resolving them in a fair and transparent manner. Since both beneficiaries and service providers benefit from adopting the FIDIC contract (compared to other formats that may expose one of the parties to greater risks), this practice has taken deep roots in the vast majority of developed countries, as well as for projects implemented by multilateral development banks. 250. Recognizing the benefits of the FIDIC forms, the Romanian Government has made a previous attempt at institutionalizing this practice. A 2007-2008 PHARE project was meant to translate the FIDIC framework into Romanian and adapt it to the local context, which further led to a common order by the Ministers of Finance, Transportation, and Regional Development in June 2008. This act made the FIDIC form required for beneficiaries of EU-funded projects, in one of three versions: red, for executing construction works based on a technical project provided by the beneficiary; yellow, for design-and- build contracts; and green, appropriate for projects under EUR 500,000 and subcontracting work. By 127 “Support for the Preparation of Romanian Standardized Tender Documents for the Transport Sector 2010 132 RO ROD Ref. PO42217/CC. 4195,” JASPERS 123 May 2009, it became clear that the application of FIDIC was not optimal and another order by the three aforementioned Ministries eliminated the requirement of using FIDIC. The following key reasons were responsible for the ultimate failure of this first attempt at expanding the use of the standard contract forms:  The initial order making FIDIC required was issued in June 2008 and began applying in July of the same year, leaving virtually no room for proper training of beneficiaries on how to use this complex instrument.  The order made the application of FIDIC extremely rigid, stipulating both “general” and “specific” conditions. Normally, the latter type should be customized by beneficiaries for each project under implementation, but this was not possible given the order’s format at the time, which included an actual contract template. As JASPERS noted in a recent study, “it is recommended not to impose special conditions by law but to provide guidance and leave contracting authorities adapt special conditions to each project particulars.”128  The order was insufficiently aligned to applicable legislation and, moreover, it was superseded by legislative acts with superior power – including Government Decisions. Some of the terminology was poorly translated from the common law system, without a proper equivalent in under Romanian law. This created significant confusion among beneficiaries, who did not always know which rules to follow. 251. This brief analysis suggests that the reintroduction of FIDIC for ROP-financed contracts should be much more carefully planned and delivered. First, there should be ample time for public consultations and trainings regarding how to apply and enforce different types of FIDIC contracts. Second, the process should be officially launched only after fully aligning the FIDIC contract form to the current Romanian legislation and, moreover, there should be a Government Decision mandating the use of FIDIC (as opposed to a Minister order, which is superseded by any government decisions that are at odds with the standard contract form). This was the solution adopted by the Ministry of Transportation in order to facilitate the use of FIDIC for the Transport Operational Programme. Last but not least, only the general FIDIC conditions should be specified by law, leaving the specific conditions flexible enough to be customized to the needs of each project. 252. Undoubtedly, aligning FIDIC fully with Romanian legislation is a precondition of any efforts to promote beneficiary use of this contract form, and indeed this will not be an easy task. As noted by interviewed experts in this field, the FIDIC system was developed based on the UK’s legal system (common law), which is markedly different than the Romanian one (civil law). In the past, there have been challenges in translating certain terms (e.g., “reasonable profit”) and in making sure that Romanian courts can understand and uphold FIDIC. As noted earlier, extensive preparation will be needed in some areas (legal, capacity development, etc.). 253. Beyond such challenges, the good news is that based on survey results the majority of ROP beneficiaries are in favor of using the FIDIC forms. Of 470 respondents, 22% see the requirement of using FIDIC as an “extremely useful” solution, and another 41% qualify it as “very useful” (23%) and “relatively useful” (18%). This is further testimony to the benefits that managers of ROP projects would accrue by balancing the risk sharing through a FIDIC-based agreement with contractors, compared to the status quo in which they are accountable for most issues that can arise during implementation. As the chart below shows, more experienced project managers are, on average, more comfortable with 128 “Support for the Preparation of Romanian Standardized Tender Documents for the Transport Sector 2010 132 RO ROD Ref. PO42217/CC. 4195,” JASPERS 124 using FIDIC. This further underscores the need for training and capacity development, particularly for those beneficiaries who are newer to the field. Figure 53. Beneficiary perceptions of requiring FIDIC contract forms for ROP-financed projects 100% 90% 80% 70% 60% N/A not useful 50% hardly useful relatively useful 40% very useful 30% extremely useful 20% 10% 0% <2 2-5 >5 Number of ROP projects in beneficiary portfolio 125 Figure 54. Selecting the right type of contract (i.e., FIDIC color) is relatively simple Source: http://fidic.org/node/149 254. In a welcomed recognition of the potential benefits involved in the adoption of red and yellow FIDIC, the Ministry of European Funds (MEF) has launched a significant effort to reintroduce these contract forms into the Romanian legislation. The first step was the May 2013 the World Bank report on Improving the National Framework for Preparing and Implementing Public Investment Projects. This work built on the request expressed by consulting firms, technical designers, as well as public and private sector beneficiaries, who manifested a strong preference for FIDIC. The MEF initiated several rounds of consultations and, in August 2013, published a draft document on three types of FIDIC contracts (red, yellow, and green), noting its intention to adopt these forms through a legislative act with a higher authority than a simple Minister Directive (i.e., a Government Decision). This would prevent conflicts with the current Romanian legislation and improve the chances of beneficiaries adopting the FIDIC contracts, leveraging all the corresponding benefits described above. 255. The new FIDIC proposal promises to solve a number of critical issues that have slowed the implementation of public investment projects (including investments with ROP co-financing). Specifically:  Unpredictable circumstances: The new FIDIC General Conditions clearly define what conditions need to be fulfilled for a particular situation to be qualified as “unpredictable,” a matter of great debate and confusion during the 2007-2013 period. 126  Role of the engineer: The Engineer has to obtain the beneficiary’s approval in a number of instances, such as: particular technical solutions leading to increases in the contract’s price (clause 3.5); the appointment of subcontractors (4.4); the extension of the execution timeline (8.4); the suspension of works, partially or in full (8.8); and increases in the contract’s value beyond the approved budget for eligible expenditures, with the potential to minimize the number of instances when a particular needed expense is considered “ineligible” and has to be incurred by the beneficiary (clauses 13.1, 13.3., 13.5, and 13.6). Importantly, however, an issue left unresolved even under the new proposal is the amendment of Law 10/1995 that should reflect the role of the engineer, as warranted by FIDIC.  Professional insurance for designers: It is recommended to add a clause (to article 18) requiring technical designers to have professional insurance. A potential formulation of this clause follows: “The contractor has to own a professional insurance policy that would cover all risks associated to professional negligence in the design of the project. The minimum level of the coverage should be equal to the total sum presented in the Annex to the offer. The contractor is required to maintain this policy up to the deadline mentioned in the same Annex.” As noted earlier in the report, this requirement would significantly boost the designer’s ownership over the quality of the product and also gradually “clean” the market of those firms that are not abiding by rigorous standards.  Dispute Adjudication Board: The proposal introduces for the first time the Dispute Adjudication Board (DAB), as the main body responsible for mediating between the key parties (beneficiary, contractor, engineer, etc.). The DAB is fundamental for accelerating the pace of projects’ implementation and can prevent lengthy court procedures.  Existing utility networks: The new FIDIC adds a special clause to the following effect: “The contractor is required to know in advance, based on documentation submitted by the beneficiary and relevant entities, the physical location of all utilities, before the start of any works that may affect the existing networks.” While this clause is , this would also require that the beneficiary allow sufficient time for contractors to prepare their offers and become fully aware of the reality on the ground (i.e., under the minimum timeframe of 45 days).  Reception of partial works: The new FIDIC contracts would permit beneficiaries, upon the Engineer’s recommendation, to receive partial works, as long as they are physically and functionally independent components of the larger project. This would create a win-win situation: the beneficiary could begin using certain parts of the investment sooner; the contractor would be able to cash in the warranty for proper execution; and the overall absorption rate would accelerate. 256. The benefits of reintroducing FIDIC in the implementation of ROP-financed projects (and beyond) are straightforward: simpler procedures, faster absorption, and reduced risk of financial corrections and ineligible expenditures due to poor quality technical documentation. More broadly, the FIDIC contracts ensure that there are balanced responsibilities between all the key parties – beneficiaries, consultants, designers, and contractors. In addition, when finalized, the legislative proposal should be correlated with the latest changes in this area: for example, the NARMPP recently issued Order No. 138/2012, which regulates certain provisions for FIDIC contracts applicable to road works. 257. Given this welcomed progress, it is important to launch in the near future detailed preparations for boosting beneficiaries’ capacity to implement FIDIC contracts. This would require a 127 series of technical workshops on contract management, with the participation of key staff from the MA, the IBs, and public beneficiaries (particularly the personnel of Investment, European Affairs, and Project Management Units, etc.). It is recommended to select participants based on need, but also based on their competence – specifically, they should have a background in civil engineering, economics, and/or law. Other key service providers and corresponding accountability mechanisms 258. To further protect beneficiary interests, it is recommended to strengthen the mechanisms holding accountable other actors that can play critical roles in a project’s implementation: construction supervisors, auditors, and project managers. The main instrument for achieving this is through improved contract forms, which beneficiaries should put forth in public procurement procedures. This will ensure that the services they contract are of high quality and that they share the risks of potential errors that originate from external consultants and collaborators. With respect to construction site supervisors, recommendations made in an earlier section urged beneficiaries to hold them accountable for staying on site full-time. Also, if something goes wrong because of improper supervision, they should bear the costs required to remedy the situation, in part or in full. 259. As for auditors, they currently play a limited role, with low accountability: they certify expenditures by checking accounting documents, but do not typically go beyond desk assessments. First, auditors should evaluate every reimbursement request, including those procedures that have led to particular expenses. For instance, increased accountability for auditors may have reduced the number of corrections applied to public procurement procedures, as beneficiaries would have learned what to do better, from one reimbursement to the next. Equally important, in other countries, a best practice is to require these actors to also perform technical audits. This would act as an additional filter verifying the quality of the technical documentation and construction works performed. The same principle of stronger accountability should also apply to project managers, who must be held accountable for any mistakes resulting from their own work, as in the reimbursement requests. 260. For all these actors, increased accountability should be complemented by a requirement to have professional insurance. This would facilitate the process of recuperating at least some of the financial corrections that pass through these additional filters. Further research is needed to determine how exactly this model could work in practice in Romania across these different categories of service providers. Monitoring 261. At key points in the project’s implementation, MA and IB staff performs monitoring visits in the field. These are complemented through progress reports submitted by beneficiaries. The overall purpose of monitoring interventions is multiple-fold: to verify the investment’s progress according to the financing contract, anticipate and preempt issues; to identify successful project components; and to encourage communication with beneficiaries.129 In this view, the ROP procedures manual defines monitoring as a support mechanisms in of itself. Based on interview feedback and survey results, these objectives have been fulfilled by MA/IB staff with professionalism and diligence. Beneficiaries appreciate the close relationship they have with monitoring officers (MOs) from the RDAs and there is always an open, close interaction between these parties. With few exceptions, most MOs are reported to be very 129 ROP Procedures Manual – “PO XVI: Project Monitoring,” December 2008 128 active in supervising projects in their region and remain close to beneficiaries throughout an investment’s cycle. Visits by MA personnel are also appreciated by beneficiaries, although to a lesser degree, possibly because they are less frequent and the central agency continues to be perceived as somewhat more “removed” from the field. Figure 55. Beneficiary satisfaction with visits by officers from RDAs (left) and the MA (right) 6% 3% 21% 19% 5% 27% excellent excellent very good very good 19% satisfactory 4% unsatisfactory unsatisfactory N/A poor poor N/A 17% satisfactory 30% 40% 9% 262. These high satisfaction rates and additional qualitative feedback strongly suggest that monitoring visits should continue as a valuable assistance mechanism for beneficiaries. The key area for improvement is enabling MOs to offer definitive answers to most issues faced by the teams implementing projects. Currently, in too many instances, beneficiaries need to turn to the MA for a clarification or approval, which generates increased pressure on the central staff and also runs the risk of incurring significant delays. The solution is to continue devolving responsibilities to the regional level, along with ensuring proper training for MOs to enable effective assistance for the full range of issues that might arise during a project’s implementation. Reimbursements 263. Most beneficiary complaints related to reimbursements have to do with procedural delays. In some situations, this has caused significant financial strain on certain public authorities, private firms, and nonprofits, which were not able to pay for the services provided on time. They faced penalties and/or were forced to contract expensive loans to provide bridge financing until reimbursements came through. The recent mechanism adopted by the Romanian Government, which ensures that beneficiaries receive funds based on submitted bills, has been highly welcomed by interviewees, as noted earlier in this report. This should avoid some of the problems encountered and, along with the measures recommended in the other report corresponding to this project (on the MA-IB collaboration and communication), will reduce the administrative burden and simplify the processing of reimbursement requests (e.g., avoiding duplication in verifying the public procurement file). In most cases, delays are caused by an overburdened capacity at the MA or IB level – some suggestions on addressing bottlenecks are provided in the report on the MA-IB collaboration (Component I of this project). 264. On the beneficiaries’ end, it is recommended to adopt a transparent online tracking system that provides real-time updates on a particular request’s status, required clarifications, and expected timeline for completion. If any corrections are applied, there should be clear information on the percentage of the funds withheld and from which source of funding. Some beneficiaries note that under the current system they are not always properly notified that a correction has been applied, but merely receive less money for a particular request. An online accounting system correlated with the tracking platform described earlier should address all such concerns. 129 265. Last but not least, it is important to acknowledge the government’s recent efforts aimed at short-circuiting the process for reimbursing public-sector beneficiaries, many of who have been struggling to ensure proper cash flow for projects under implementation. The GEO 27/2013 allows beneficiaries to submit “payment requests” based on work completed and charged by service providers. The MA verifies the documentation and sends the funds covering approved receipts to beneficiaries, who further send it to the service providers within five days. After that, beneficiaries submit a reimbursement request to the IB for those services for which the payment was approved and wired. This mechanism reduces public beneficiaries’ need to have a substantial financial buffer and also ensures that service providers are paid on time and can continue to execute the project. Interviewed beneficiaries were extremely enthusiastic about this measure and praised the new initiative, while private-sector companies and NGOs expressed their hope that they would be eligible to take advantage of the alternative payment mechanism at some point in the future. 130 VIII. Post-Implementation 266. There are two priority areas for improved beneficiary assistance in the post-implementation phase: ex-post monitoring and evaluation (M&E); and enhanced knowledge sharing. In this context, post-implementation is defined as the period following the completion of a project, including the submission of the last reimbursement request. Formally, CE 1083/2006 mandates ex-post monitoring for five years for most financed projects, with the exception of investments led by SMEs, which are monitored for three years upon finalization. The main purpose is to ensure the “durability of operations,” including to confirm that the property ownership remains the same and there are no substantial changes to the project. Currently, most of the interventions at this stage are entirely focused on the required M&E activities: the beneficiary provides annual written reports and receives the visit of IB or MA officials. Going forward, it would be valuable to: demonstrate more flexibility in applying CE 1083/2006; clarify the rules around project-level indicators; and complement M&E activities with knowledge-sharing opportunities among beneficiaries and from beneficiaries to key MA/IB staff. Ex-Post Monitoring and Evaluation (M&E) 267. The ex-post monitoring for project durability entails two annual components: a beneficiary report and a field visit. The ROP Procedures Manual on Monitoring provides complete details around both processes, which are meant to verify that projects have not suffered major changes and indicators defined during the application stage are still fulfilled. In general, beneficiaries understand the need to continue to report on project results even after completion. In the words of one interviewee, “it’s normal for the EU to check and see what happened to the money they put toward my project and how I’m doing, and we’re glad to report back.”130 RDA/IB staff is responsible for annual monitoring visits, along with representatives of the Directorate for Coordinating Field Activities (DCFA) within the MA, who follow the same procedure as the IB. Beneficiaries describe their collaboration with these officers as positive and productive, although some of the same problems encountered during implementation may still arise: the lack of technical knowledge of some monitoring experts and their overly rigid interpretation of rules. The templates for beneficiary reports and for the visit reports by monitoring officers are brief but clear, allowing beneficiaries to explain any special circumstances, if needed, and should be maintained going forward. The final decision on whether to impose any penalties belongs to the MA, based on the reports received. 268. Article 57 of CE 1083/2006 requires “the Members State or Managing Authority [to] ensure that an operation retains the contribution from the Funds only if that operation does not [for a period of 3 or 5 years] undergo a substantial modification.”131 The stakes are very high: if a project is found to have changed the nature of ownership, to have ceased the productive activity, or to have modified its implementation conditions, the EC’s entire contribution to an ROP-financed investment, or a significant part of it, could be claimed back from the beneficiaries. Interviews with people who have accessed ROP funding suggest that they are fully aware of the need to abide by these conditions in order to safeguard their investments. Indeed, for most of them, given the large sums involved, paying a potential ex-post penalty would be extremely problematic. At the same time, interview feedback suggests that MA and IB personnel are relatively rigid in applying the requirement to avoid changes to the project in the ex-post 130 In-person interview with private-sector ROP beneficiary from the Center Region 131 CE 1083/2006, http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2006:210:0025:0025:EN:PDF 132 period. Fundamentally, this has to do with the interpretation of what qualifies as “substantial modification” in article 57 of CE 1083/2006 and, in practice, there are multiple situations where beneficiaries avoid any adjustments, even though good reasons may exist to perform them. 269. A more clear definition of the ROP’s interpretation of Article 57 in EC 1083/2006 would be very helpful to beneficiaries, along with a procedure for allowing changes to the projects, when justified by special circumstances. It is important to note that EU-level regulations do not expressly forbid any modifications, but only those that would qualify as “substantial.” Most interviewed beneficiaries are under the impression that, for 3 or 5 years, depending on the type of project, they are not allowed to move “a single nail.” As described by the case-study in the box below, it can lead to absurd situations where a beneficiary faces the risk of bankruptcy because of restrictions on changing elements of the project. On one level, this comes from MA/IB monitoring officers’ interpretations of EU- level requirements; on another level, auditors from the Courts of Account in Romania also verify whether projects continue to follow indicators set in the technical documentation. In any case, at least to increase the flexibility of the ROP system, there should be a clear procedure that beneficiaries could follow to explain the circumstances of potential changes needed. Decisions by the MA should be based on a clear, comprehensive definition of “substantial changes,” a definition that should also be agreed with EC counterparts in Brussels. If the required changes also affect project-level indicators, the bar for accepting the adjustment should be raised even higher, but even in those situations there needs to be a clear path that beneficiaries should follow to resolve a potential impasse in the post-implementation phase. Otherwise, there is a serious risk of hurting the beneficiary’s future appetite for accessing ROP funding. Box 5. The unintended consequences of rigid rules The primary school in Livada village (Iclod commune, Cluj County) was modernized with ROP funding. It currently absorbs a large proportion of the locality’s budget because an oversized heating boiler was installed, in accordance with the technical project. The boiler was financed through the ROP, but since it began operating it has added substantially to the current heating bill paid by local authorities. Interview feedback suggests that the beneficiary cannot replace the current boiler, even as an out-of- pocket expense. The initial fault lies with the technical designer, but local authorities in this small administration did not have the capacity or knowledge to intervene on the technical project. Now the commune is incurring very high expenditures and cannot do anything about it before five years elapse since the completion of the school’s renovation. There are multiple similar examples across the country – a telling consequence of rigid interpretation and application of broader principles. 133 Picture 7. An oversized boiler aggravates budgetary pressures for the school and local authorities in Livada 270. Clarifications are also needed when it comes to the types of acceptable project-level indicators and the methodology and standards used in assessing them after the completion of an ROP investment. Feedback received from interviewed beneficiaries suggests that there is a lack of clarity regarding how monitoring officers are evaluating whether projects have delivered on their indicators and how potential financial corrections may be decided for beneficiaries who fall short of the targets. The first problem is that there were no adequate controls on the set of indicators defined by beneficiaries. The applicant guides and the financing application template instruct applicants to carefully select indicators correlated with program or axis-level ones. In practice, however, overzealous or inexperienced beneficiaries (and/or their consultants) often included a wide set of targets, believing that this increased their chances to receive financing. Some of the indicators chosen were either unrealistic or impossible to measure and attribute, particularly with respect to the indirect impact of particular investments. For their part, evaluators did not call attention to this problem and a number of projects ended up receiving financing and committing their beneficiaries to achieving the indicators they had chosen. Further down the line, particularly in the post-implementation phase, these beneficiaries have a hard time (a) reaching particular targets; and/or (b) demonstrating that those targets were reached. This is not always the direct fault of those who put together the application. In a few instances, there are objective reasons for failing to get to a particular number, as in the case of public-sector beneficiaries who promised to create new jobs but cannot fill them because of hiring restrictions imposed through GEO 34/2009. Such cases should have a clear, objective, and fair resolution. 271. On the choice of indicators, applicants should be encouraged to select from a pre-defined list aligned with program- and axis-level targets, which would be easy to implement particularly through an e-application form. At a minimum, chosen indicators should abide by the “SMART” set of criteria: specific, measurable, attainable, relevant, and time-oriented. An electronic application system would not allow users to submit a financing proposal with indicators beyond the pre-defined set. As an alternative, applicants could be free to choose their own targets, but evaluators should be allowed and instructed to correct these based on the principles enunciated above. These solutions would avoid instances of beneficiaries over-promising on the impact of their projects and would also save time when preparing the monitoring reports. Currently, beneficiaries have trouble proving that they are reaching set targets and either need to invest additional funds into this, or hope that they do not get penalized. 134 272. Interviewed beneficiaries have also requested a detailed methodology applicable to how project-level indicators are monitored and evaluated throughout the implementation and post- implementation phases. For instance, they do not know what point in time counts for which indicator; if they promised to hire, say, five additional people, it is unclear whether they are required to keep these positions filled for the entire monitoring period or under what conditions there could be exceptions to this rule. While generally simpler is better when it comes to regulation, in this case there need to be more details provided around what is evaluated and how, along with potential applicable sanctions in those cases that fail to reach set objectives. Some beneficiaries fear that they will be subject to “retroactive corrections” based on an evolving M&E framework for ROP projects, a situation one of them compared “to the public procurement saga that started in 2011.”132 Whatever is decided, it should be fully aligned with EU-level and national legislation to make sure that if any corrections are imposed they are upheld by courts in potential legal actions. In this context, it is useful to note that applying corrections for failing to meet indicators remains a call for Romanian authorities to make, as EU-level regulations applicable to the 2007-2013 programming period do not require such sanctions. 273. For projects that fail to reach their targets, a cautious approach is recommended in imposing sanctions. The MA could demonstrate flexibility in assessing such cases, particularly given beneficiaries’ relative lack of experience with performance indicators. Although it is not recommended to enforce financial corrections, the MA may still decide to activate this mechanism. If that is the case, sanctions should only apply to program indicators and remain proportional to the calculated gaps between target and actual values. In other words, the entire sum contributed by the ROP should only be charged as a penalty for highly exceptional circumstances where a project completely missed all targets. In general, if any penalty is deemed necessary, it should be, at most, equivalent to the difference between the target (100%) and the achieved level. 274. Such clarifications are fundamental to enhancing the ROP’s assistance provided to beneficiaries because, simply put, they nurture predictability in the system and make it clear which targets projects should accomplish, in what time horizon, and what the possible consequences are for failing to get there. Additionally, beneficiaries should always have easy and clear access to conciliation proceedings within the ROP system for any and all proposed sanctions. The current procedures manuals may need updating to detail the steps for conciliations and it would also be useful to include this information in applicant guides to both make beneficiaries aware of this instrument and assure them that there is an institutionalized process within the ROP system that allows them to argue their case. While current procedures include details on conciliations before contract cancelations, they should also clearly stipulate what beneficiaries can do when a sanction is proposed as a result of failing to reach project indicators. Knowledge Exchange 275. The ROP’s performance to date can and should be replicated and improved upon during the next programming period, which depends fundamentally on leveraging the resources that already exist in the system. This is certainly true of the program’s “graduates” – i.e., beneficiaries who have successfully completed ROP-financed projects. Some of them have many finalized investments in their portfolio, others are less experienced, but they all could contribute tremendously to the ROP’s improvement by effectively closing the feedback loop and feeding back their accumulated insights into 132 In-person interview with private-sector ROP beneficiary from the SW-Oltenia Region 135 how new projects in the pipeline are implemented. As the saying goes, “smart people learn from their mistakes, but really smart people learn from the mistakes of others.” Beyond beneficiaries’ best intentions, this is hard to accomplish under the status quo. 276. Currently, once a project is complete, graduates interact with the MA and IB staff mostly through monitoring visits and reports, but there is no institutionalized, effective way of collecting their suggestions for improving the program or for ensuring smoother project implementation. This is a loss for the entire ROP system, as there is ineffective knowledge transfer across beneficiaries and between beneficiaries and MA/IB staff. Moreover, given available data from SMIS and other databases, the MA and the IBs would have the possibility to select only the top-performing projects and beneficiaries, seeking to draw as many lessons learned as possible for how to replicate these experiences. By the same token, selecting the projects with the greatest implementation challenges also promises to lead to interesting and useful results on how to avoid common pitfalls and unintentional errors. To be sure, knowledge sharing among beneficiaries is still happening in the absence of formal mechanisms, but on an ad-hoc basis only and with suboptimal results, as not all knowledge exchanged currently is useful. As demonstrated by what happened with the drafting of public procurement documentation, beneficiaries copied mistakes from one another and, as a consequence, incurred painful corrections. This goes to the point that knowledge among peers can be a simple, but very powerful, valuable tool if implemented correctly – i.e., properly collected, organized, verified, and distributed. 277. There are multiple possibilities for leveraging program graduates and facilitating knowledge exchange, and further assessment will be needed to provide a list of detailed solutions. For instance, one simple possibility is to add to the final progress report and to ex-post monitoring reports submitted by beneficiaries to the IBs a number of questions asking for suggestions for future beneficiaries. If this report’s recommendation on switching to an e-based ROP is followed through, it should be easy to receive and collect this feedback at the level of each region, and then send it to the MA for further processing and analysis. Of course, this best practice can be expanded to all progress reports, throughout a project’s cycle, but there is value in doing it particularly during the post -implementation stage when ROP graduates typically face more manageable time pressures and also have the full picture of how their project performed. 278. Another option is to host in-person workshops with successful ROP graduates acting as trainers and mentors for new beneficiaries. In some regions, success cases are featured in promotional materials and project managers are sometimes asked to give talks to various audiences, but this practice is not institutionalized. In addition to the proposed ROP Corps of Practitioners, which would include the most experienced staff in public-sector beneficiaries – the IBs and the MA could engage program graduates to deliver effective trainings customized to fit the needs of different beneficiary types (e.g., public, private, and non-profit), as well as thematic workshops on topics like road infrastructure, public parking development, renovating fragile historical buildings, etc. ROP graduates could also play a role in the training sessions that are already planned through the existing communication plans for 2007-213, and the plans for 2014-2020, which are expected to be similar. Specifically, graduates could lead parts of the sessions organized before the official launch of a particular axis and upon contracting of new projects. The same recommendations formulated under the implementation chapter above apply here as well: customized trainings for different types of beneficiaries depending on their experience, applicable legislation (e.g., public vs. private organizations), and priority areas of their projects. 279. In addition, graduates could play a key role in developing a range of information and communication technology (ICT) solutions for knowledge management and sharing. The possibilities 136 under this heading are nearly endless, but the guiding principles should be ease of input and ease of access. Whether it is a simple online forum with postings for problems encountered and approved solutions, or a more elaborate search catalogue based on key topics (e.g., “public procurement,” “feasibility studies,” etc.), it will only be successful if users do not spend a lot of time figuring o ut how to use the platform. To borrow a well-known example from this field, Google devised the world’s most popular search engine by simplifying everything, down to a single button. This is why it is important to keep testing, piloting, and refining ICT solutions based on users’ feedback to reduce the risk of developing a really complex tool used by only a few ROP graduates and beneficiaries. Box 6. The positive power of knowledge sharing is beyond doubt “I’ve learned from practical experience that in the public tender for this particular project I should have put out to bid both the cost for the total distance and the cost per meter. That way, when additional works were needed, I would have already been able to contract them out, without going through another tender. I know this for my future projects, but I’d like to caution others to adopt this best practice as well. There’s just no formal channel to share these kinds of lessons…” - Public-sector ROP beneficiary, West Region 280. For all its benefits and positive potential, it is important to note that knowledge exchange is not a silver bullet in the implementation of structural funds and it cannot solve all challenges faced by the ROP. For it to be effective, it also requires careful quality control to avoid situations where the shared information is either wrong or would expose project beneficiaries to additional risks. It is useful to caveat all knowledge exchange initiatives with the note that advice from ROP graduates should always be adapted to the specific circumstances of a project and, naturally, what worked for one beneficiary may not be appropriate for another. Despite such risks, the benefits of leveraging the experience of beneficiaries who complete ROP projects far outweigh the potential costs. If the ROP system is to sustain and enhance its future performance, it needs to facilitate knowledge exchange at the beneficiary level. This would not only have a direct positive impact on future projects, but would also reduce some of the burden on MA and IB staff, who would have to deal with fewer minor questions and will be able to focus more on their critical assigned tasks. 281. Finally, it is encouraging to note that the enabling conditions to implement ROP knowledge exchange initiatives exist, at a basic level. In fact, one of the most positive signals resulting from the field research for this project is that beneficiaries are keen to speak up and contribute to the success of the program beyond their own projects. There is also an implicit sense of partnership among beneficiaries, founded on a common understanding that the complexities of an ROP project require collaboration and proactive engagement with others in search of optimal solutions. 137 IX. Roadmap of Priority Recommendations FACILITATION OF PROACTIVE AND DIRECT SUPPORT FOR APPLICANTS AND BENEFICIARIES OF THE REGIONAL OPERATIONAL PROGRAMME 2014-2020 Legend Short term (0-3 Months) Low Low $$ Timeline Medium term (3-12 Months) Difficulty Medium Resources Medium $$$ Long term (>12 Months) High Significant $$$$ Note! All recommendations marked with (*) can be implemented starting with the current programming period. Put differently, recommendations lacking the (*) symbol apply exclusively to the ROP 2014-2020. MA-ROP Difficulty of Challenges/ Needed Main area Recommendations Output indicator Timeline role implementation Risks resources Priority area #1: HORIZONTAL MEASURES Human Resources, Financial Resources, Electronization, and Targeted Support for Marginalized Communities 1. Address the 1.1. Amend GEO 32/2010 Enabler  Workload vs.  Lack of $$$$ significant risk to allow flexible capacity of political of turnover of hiring of staff at beneficiary support existing beneficiaries’ staff  Insufficient qualified staff level133*  Number of financial within public new hires resources beneficiaries  Uncontrolled due to low spending on salaries new hires (especially for  Hiring of 133 Applicable to public beneficiaries (local /central authorities, public institutions etc.) 138 young inadequate employees), staff based on unsustainable opaque time political pressures and decisions workloads 1.2. Amend Law No. Enabler  Law amended  Lack of political $$$$ (taking into 284/2010 to expand  Bonus system support account the bonus system to updated  Insufficient challenges of cover all phases of financial managing two project cycle to resources programming ensure a high-quality  Uncontrolled periods for project preparation* spending / 2014- poor targeting 2015/2016), of bonuses and a 1.3. Organize trainings Enabler  Number of  Lack of staff $$$ demotivating with new hires to trained availability to working ensure transfer of specialists participate environment knowledge, within  Number of the same beneficiary workshops organization and across beneficiaries to share best practices, boost motivation, etc. (see also 2.1-2.3 below)* 2. Strengthen the 2.1. Facilitate the Enabler134  ROP CoP  Lack of 0 partnership establishment of an established willingness to between MA- ROP Corps of (including a be part of the IBs- Practitioners (CoP) full members’ CoP beneficiaries by from among the most list)  Lack of time capitalizing on successful ROP availability the existing beneficiaries* 134 It is recommended to involve IBs in the selection of the most successful beneficiaries 139 knowledge of 2.2. Support the  Number of  Lack of $$ experienced organization of Enabler meetings in financial practitioners national /regional each region resources level CoP meetings to and nationally  Unavailability share experiences, of practitioners provide feedback to to participate MA/IBs, identify in meetings solutions to emerging issues, etc.* 2.3. Organize training Initiator  Number of  Lack of $$ sessions with new or meetings in financial beneficiaries, with Enabler each region resources CoP members as and nationally  Unavailability trainers*  Number of of practitioners trained to participate beneficiary in meetings staff  Unavailability/ lack of interest among beneficiaries 3. Improve 3.1. Publish the name of Initiator  List of projects 0 beneficiaries’ consulting companies selected for access to that elaborate financing quality service selected/financed includes the providers by project(s)135* name of boosting the consulting market’s companies (on transparency MA and IB and promoting websites) a set of best 135 If the beneficiary is a public body, the consultant (in the elaboration and in the implementation phases of a project) is selected following public procurement rules (and the consultant name is published in SEAP); however, private beneficiaries are not bound by the law to launch a public tender for contracting a consultant if they do not fulfill the condition of “Contracting Authority,” as defined by the law 140 practices in 3.2. Encourage Enabler  Platform  Lack of willing 0 contract professional (discussion project lead drafting associations of forum) from within consulting companies developed and private sector to create a user maintained by (since MA/IB feedback mechanism player(s) in cannot take on to enable the field136 this role) beneficiaries to comment on and rate the performance of service providers (consultants, designers, etc.)* 3.3. Promote best Enabler  Guide on / list  Time constrains $$ practices in drafting of best  Lack of contracts with service practices specialized providers to protect  Number of expertise on the interests of seminars best practices beneficiaries and organized in designing hold consultants contracts accountable for the quality of their work137* 4. Reduce 4.1. Implement an Enabler  Electronic  Overlap with $$$$ excessive electronic exchange system in mySMIS bureaucracy data system for the place (currently and entire ROP project under documentation cycle (submission of development involved in applications, by MEF) developing and submission of  Low capacity of implementing reimbursement beneficiaries to 136 Associations of consulting companies, chambers of commerce, etc. 137 See also the ANRMAP project “TA to draft technical specifications for standard tender documents for managing Authorities” 141 EU-financed requests, etc.), in use the system projects under addition to and mentality the ROP by procedural of reliance on promoting simplifications hard copies electronization  Long period of testing/ implementing the e-system  Significant costs 5. Improve access 5.1. Inform ROP Enabler  Number of  Lack of interest $$ to finance in all applicants and awareness – of banking stages of a beneficiaries about raising system to project by financial products sessions for participate in acting as a tailored for EU ROP applicants events “bridge” projects (e.g., invite and/or between banks banks beneficiaries  Lack of interest and ROP representatives to  Number of of banking applicants/ regional/national events with system to beneficiaries awareness events, banks’ develop periodically update counterparts specific the newsletter/ products for EU brochures and/or use beneficiaries other informational tools to explain banking products, etc.)138* 6. Boost support 6.1. Define targeted Initiator  Investment  Lack of 0 for marginalized selection criteria for operations commitment to communities, ROP 2014-20 to address clearly support ensuring that the need for supporting identified in marginalized ROP projects marginalized ROP to groups through 138 All banks should be invited 142 target their communities and/or address issues the ROP specific needs establish dedicated specific to  Lack of investment operations/ marginalized knowledge on priority axes communities effective  Applicant interventions guides and selection criteria reflect these priorities 6.2. Develop the capacity of Initiator  Number of  Lack of $$ marginalized groups to and ROP projects knowhow on design/implement their Enabler submitted for appropriate own ROP projects (e.g., (with IB financing by capacity through the CLLD help) marginalized building instrument), including communities techniques through a potential electronic platform Priority area #2: PROJECT CYCLE-SPECIFIC MEASURES 7. Application, 7.1. Analyze/review the Initiator  Proposals  Lack of political $$ evaluation, and legislative framework adopted willingness for contracting applicable to EU-  Limited legislative financed projects and number of framework propose necessary changes to amendments amendments to ROP Applicant  Time pressures eliminate gaps/ Guides contradictions/ overlaps before starting the 2014-20 period 7.2. Revise or eliminate the Initiator  New eligible  Lack of political $$ eligible expenditure expenditure commitment order (EEO) based on order  Inability to 143 the 2007-13 experience approved create a to cover all necessary  Evaluators’ framework for expenses related to an grids updated managing risks investment and also to require a of “over- increase the 5% thorough spending” maximum cap on verification of project design and proposed technical assistance budgets 7.3. Implement an electronic Initiator  Online  The system is $$$ system to inform ROP tracking not integrated applicants and system in with mySMIS beneficiaries on the place  Low capacity of status of their critical beneficiaries to requests (submission of use the system applications, submission of reimbursement requests, etc.) 7.4. Elaborate a Initiator  Number of  Lack of $$$ comprehensive and beneficiaries financial training139 plan for Enabler attending the resources beneficiaries and (with IB courses/  Delays in organize sessions help) workshops contracting tailored to their  Number of specialized needs/experience with events consultants EU projects and organized  Unavailability scheduled to match the  Level of of Corps of cycle of the participants’ Practitioners to programming period satisfaction participate in (i.e., explain the rules of events the game before the game starts) 139 Recommended 3-4 years framework contract; however, if financed under current ROP priority axis 6, the framework contract should last until July 2015 144 8. Implementation 8.1. Introduce FIDIC Enabler  Standard  Time constrains $$ standard contract to contract in  Lack of address the poor quality place knowledge on of technical  Low number FIDIC rules or documentation and the of corrections inability to low capacity of on technical adapt FIDIC to beneficiaries to documentatio Romanian verify/develop such n legislation documentation  Low number of contract amendments 8.2 Deliver effective Initiator  Number of  Unavailability $$$ trainings customized to beneficiaries of beneficiaries fit the needs of different attending the to participate beneficiary types (e.g., courses/ public, private, and non- workshops profit) and /or thematic  Number of workshops on sensitive events topics recommended for organized beneficiaries who have  Level of signed contracts or have participants’ already started project satisfaction implementation 9. Post- 9.1. Clarify the methodology Initiator  Ex post  Different 0 implementation for ex-post monitoring monitoring opinions and assessment of procedure among indicators*140 updated AA/MA/CE  Instruction regarding the issued to interpretation beneficiaries of EC Regulation 140 Analyze the effect of financial crisis on ROP indicators 145 9.2. Establish a long term Initiator  Number of  Insufficient $$ plan for quarterly 1-day beneficiaries financial meetings among attending the resources for MA/IBs/successful courses/ organizing beneficiaries/new workshops meetings beneficiaries*  Number of  Unavailability events of beneficiaries organized with successful  Level of completed participants’ projects to satisfaction participate 146 Annex A: Survey of ROP Beneficiaries (April-May 2013) Introduction At the initiative of the Ministry of Regional Development and Public Administration (MRDPA), the World Bank is currently working on several in-depth studies aimed at improving the implementation of the Regional Operational Programme (ROP) for the current period, and especially in view of the 2014-2020 period. This survey is conducted in collaboration with the information portal in the field of European funds www.fonduri-structurale.ro, facilitating an overview of the ROP beneficiaries' perspective on the implementation programme (benefits, obstacles, recommendations for the next period). The survey objectives: 1. Assessing how ROP beneficiaries have collaborated with the Regional Development Agency and respectively, the Management Authority of the Regional Operational Programme (ROP); 2. Assessing support mechanisms available to beneficiaries throughout the entire project (from concept development to post-implementation); 3. Collecting information about the difficulties encountered in accessing ROP funds and possible recommendations from the beneficiaries for their resolution during 2014-2020. Note: All answers are anonymous. The main purpose of this survey is to collect complete responses from beneficiaries, with direct impact over the recommendations for improving ROP in the future. Your contribution to this effort is appreciated as particularly important. The estimated time for filling-out this survey is ~ 12-15 minutes. *Mandatory General Questions 1. As an ROP funds beneficiary, you represent: * Mark a single option. A public institution A non-governmental organization (NGO) A private company Other: …………………………………………………… 2. Which is the region where you have carried out one or more projects funded through the ROP? * Mark a single option. Bucharest - Ilfov Centre q North-East North-West South-East South-Muntenia South-West Oltenia West 148 3a. How many projects funded through the ROP have you IMPLEMENTED during the current period? * Mark a single option. 0 1 2 3 4 5 6 7 8 9 10 > 10 3b. How many projects funded through the ROP have you FINALIZED in the current period? * Mark a single option. 0 1 2 3 4 5 6 7 8 9 10 > 10 4a. What are the main benefits of the funds obtained through the ROP? * Assign a rating for every possible benefit, according to the following scale: Mark a single option for each row. Do not Hardly Relatively Very Extremely know/cannot Not important important important important important assess Non- reimbursable funds Capital for investments Opportunity to motivate the team Greater institutional capacity Professional development 149 Networking 4b. Other benefit/other benefits (optional) If the list from the previous question does not include one or more relevant benefits, please include them here (description and rating, on the same scale as above) ....................................................................... 5a. What are the main obstacles you have encountered in accessing the funds available through ROP and in implementing the projects? * Assign a rating for every possible obstacle, according to the following scale: Mark a single option for each row. Do not Not Hardly Relatively Very Extremely know/cannot important important important important important assess Poor quality Applicant Guides Frequent legislative changes Excessive bureaucracy/ documentation Assessment/project selection delays Lack of transparency Lack of quality consulting services Technical documentation of poor quality Insufficient level of pre-financing Absence of capital for co-funding and/or cash-flow Reimbursement delays Improper framework for carrying out public procurements Poorly trained MA staff and improper communication with beneficiary Poorly trained RDA staff and improper communication with beneficiary 150 5b. Other obstacle/ other obstacles encountered (optional) If the list from the previous question does not include one or more relevant obstacles, please include them here (description and rating, on the same scale as above) ...................................................................... ...................................................................... ...................................................................... ...................................................................... Collaboration with the RDA 6. How would you describe your level of satisfaction with the support provided by the Regional Development Agency (RDA) in different phases of the project? * Assign a rating for RDA's support in each phase, according to the following scale: Mark a single option for each row. Do not know/not Poor Unsatisfactory Satisfactory Very good Excellent applicable Promoting funding opportunities through ROP Preparing and submitting the financing proposal Evaluation of the financing proposal Potential changes to the funding agreement Filing reimbursement requests Checking procedures Monitoring visits 151 7a. How effective were the main mechanisms that the Regional Development Agency has used to support you in accessing the funds available through ROP? * Assign a rating for RDA's support in each phase, according to the following scale: Mark a single option for each row. Do not Hardly Relatively Very Extremely know/cannot Not important important important important important assess Informational materials Website Information sessions Seminars and training Site visits Answers to questions addressed through Helpdesk Interactions at personal level 7b. Other support mechanism/mechanism from RDA (optional) If the list from the previous question does not include one or more support mechanisms used, please include them here (description and rating, on the same scale as above) ...................................................................... ...................................................................... ...................................................................... ...................................................................... 152 Collaboration with MA-ROP 8. How would you describe your level of satisfaction with the support provided by the Management Authority (MA-ROP) throughout different phases of the project? * Assign a rating for MA-ROP's support in each phase, according to the following scale: Mark a single option for each row. Do not know/not Poor Unsatisfactory Satisfactory Very good Excellent applicable Promoting funding opportunities through ROP Evaluation of financing application Contracting Reimbursement of eligible expenses Monitoring Verification procedures (e.g. of procurements) Conciliation/mediation procedure (if applicable) 9a. What are the main mechanisms that the MA-ROP has used to support you in accessing funds available through ROP? * Assign a rating for MA-ROP's support in each phase, according to the following scale: Mark a single option for each row. Do not Not Hardly Relatively Very Extremely know/cannot important important important important important assess Informational materials Dedicated website (www.inforegio.ro) Information sessions Answers to the questions addressed through Helpdesk Site visits 9b. Other support mechanism/mechanism from MA-ROP (optional) 153 If the list from the previous question does not include one or more support mechanisms used, please include them here (description and rating, on the same scale as above) ...................................................................... Solutions and recommendations for the future 10. What are the main ways the Regional Development Agency could support you more/more effectively in accessing the ROP funds in the future? * ...................................................................... 11. What are the main ways the Management Authority could support you more/more effectively in accessing the ROP funds in the future? * ...................................................................... 12. How would you rate the following ideas for facilitating access to funds available through ROP? * Rate each of the following proposals, according to the following scale: Mark a single option for each row. Do not Not Hardly Relatively Very Extremely know/cannot helpful helpful helpful helpful helpful assess Introduction of electronic documentation for application and/or reimbursement requests Introduction of online status tracking system for applications/requests submitted by the beneficiary to RDA/MA Simplification of procedures (assessing/contracting/verifications/etc.) Standardization of the public procurement documentation (criteria, etc.) Support for checking and improving the technical documentation Online rating system of service providers (designers, auditors, consultancy companies, project managers, etc.) based on reviews of other beneficiaries More frequent meetings between the beneficiaries and the RDA staff More frequent meetings between the beneficiaries and the MA staff Development of an ROP Project Implementation Manual Mandatory FIDIC type contracts (red/yellow) 13. As an ROP beneficiary, do you consider that the implementation costs of a project outweigh the benefits of the structural funds available through the program? * 154 Mark a single option. YES NO 14. Upon assessing the process of accessing ROP funds and implementing your project(s), would you seek new financing from this operational programme during the next financial period?* Mark a single option. YES DON'T KNOW/ HAVEN’T DECIDED NO 15. Other suggestions/remarks: ...................................................................... ...................................................................... 155 Annex B: Beneficiary Interview Guide Introduction  Meeting objectives: o Understanding how beneficiaries collaborate with the RDAs and MA-ROP; o Assessing support mechanisms available to beneficiaries throughout the entire project (from concept development to post-implementation); o Collecting information about the difficulties encountered in accessing ROP funds and possible recommendations for their resolution. Collaboration with the RDA and MA-ROP o How would you describe, briefly, the project/projects for which you have sought (and eventually received) ROP funding? Short history, impact, major problems. o How would you describe the collaboration and communication with the RDA over time?  Frequency  Type of relationship (formal vs. ad-hoc)  Specific examples (positive and/or constructive) o What are the main 3-5 positive aspects of the collaboration with the RDA? What are the elements that should be maintained for the 2014-2020 cycle? o What are the overall 3-5 most important aspects that should be improved with regard to your collaboration with the RDA? o How would you describe your collaboration and communication with MA-ROP? o In view of the 2014-2020 period, how would you assess the decentralization/regionalization perspectives for a new regional administrative level?  Expectations, hopes, fears  Possibilities and perspectives for the MA-ROP role  Possibilities and perspectives for the RDA role Support mechanisms for beneficiaries o What are the beneficiary support mechanisms (formal or informal) you know of and/or you have used directly?  Informational materials, seminars, conferences, helpdesk, formal and informal meetings etc.  At RDA vs. MA-ROP level o How would you assess the effectiveness of these mechanisms?  Responsiveness  Clarity and usefulness of answers 156  Impact of recommendations (positive/negative) o How could the RDA and/or MA-ROP support you more effectively from the project concept phase to post-implementation?  What are the main 3-5 measures you would adopt to strengthen the support given to ROP beneficiaries?  For example: better promotion of opportunities; greater transparency in application process; clearer explanations/ direct support for implementation problems/ others 157 Annex C: List of Interviewed Beneficiaries Note: The list below includes interviewed ROP beneficiary organizations. Public-sector institutions were typically represented at the level of top leadership (i.e., Mayor / County President and or Vice-President) and through technical team staff (i.e., Technical Director / Investment Director / EU Funds Director). For private-sector and nonprofit beneficiaries, interviews were conducted with CEOs and high-level executives. The table is ordered by interview date. The selection of interviewees was based on both RDA recommendations and the Bank team’s own preferences. In addition to the list below, the team held meetings with a range of key stakeholders whose inputs are reflected throughout this report: financial institutions, consulting companies, NGOs, law firms, and other public institutions (e.g., the National Audit Authority, the Authority for the Coordination of Structural Instruments, etc.). Region Date Beneficiary Institution / Organization type 01/21/2013 Public Cluj-Napoca City Hall 01/23/2013 Public Cluj County Council 01/24/2013 Public Cluj-Napoca Recovery Hospital 01/25/2013 Private Media Young 01/28/2013 Public Elementary School in Livada, com. Iclod (Cluj County) North-West 01/29/2013 Public Zalău County Council 01/29/2013 Public Zalău City Hall & Mihai Eminescu Elementary School 01/29/2013 Nonprofit Rainbow Foundation, Cehu Silvaniei (Sălaj County) 01/30/2013 Private ArcParc (Cluj County) 01/31/2013 Public Bistrița City Hall 01/31/2013 Public Sângeorz-Băi Town Hall 02/04/2013 Private SC Devi Plus SRL (Alba-Iulia) 02/05/2013 Public Alba-Iulia City Hall 02/05/2013 Public Aiud Town Hall Center 02/06/2013 Public County Council Sibiu 02/06/2013 Public Sibiu City Hall 02/06/2013 Private Hotel Casa Turiștilor (Păltiniș, Sibiu County) 02/07/2013 Public Alba County Council 158 Region Date Beneficiary Institution / Organization type 02/07/2013 Private Hotel Parc (Alba-Iulia) 02/08/2013 Public Târgu Mureș City Hall 02/08/2013 Private S.C. Romex SRL (Târgu Mureș) 02/11/2013 Public Brăila City Hall 02/12/2013 Public Brăila County Council 02/12/2013 Private SC One Brand SRL (Brăila) 02/13/2013 Public County Council Galați 02/13/2013 Private SC Cabinet Stomatologic Addent Dr.Purdea Adrian SRL (Galați) South-East 02/13/2013 Public Măcin Town Hall 02/14/2013 Public Tecuci Town Hall 02/14/2013 Public Focșani Town Hall 02/15/2013 Public Galați City Hall 02/15/2013 Private SC Viper SRL (Brăila) 02/20/2013 Public Chitila Town Hall Bucharest- Ilfov 02/20/2013 Private Siriului Residence SRL (București) 03/11/2013 Public Piatra Neamț City Hall 03/11/2013 Private Hotel Central SC RO COM CENTRAL (Piatra Neamț) 03/12/2013 Nonprofit Euronest Inter-Community Development Association (NE Region) 03/12/2013 Private SC Consult Alga SRL (Piatra Neamț) 03/13/2013 Public Bacău City Hall North-East 03/13/2013 Public Bacău County Council 03/13/2013 Public Moinești Town Hall 03/14/2013 Public Suceava City Hall 03/14/2013 Public Suceava County Council 03/14/2013 Public Botoșani City Hall 159 Region Date Beneficiary Institution / Organization type 03/15/2013 Public Neamț County Council 03/15/2013 Public Buhuși Town Hall 03/19/2013 Public Călărași City Hall 03/19/2013 Public Călărași County Council 03/20/2013 Public Slobozia Town Hall South- 03/21/2013 Public Pitești City Hall Muntenia 03/21/2013 Public Argeș County Council 03/21/2013 Public Mioveni Town Hall 03/22/2013 Public Teleorman County Council 04/10/2013 Public Dolj County Council 04/10/2013 Public Craiova City Hall 04/12/2013 Private SC Cadworks MFG SRL (Craiova) 04/12/2013 Private SC Policlinica (Craiova) 04/16/2013 Public Slatina City Hall South-West Oltenia 04/16/2013 Public Olt County Council 04/17/2013 Public Târgu-Jiu City Hall 04/17/2013 Public Gorj County Council 04/18/2013 Public Mehedinți County Council 04/19/2013 Public Vâlcea County Council 04/22/2013 Public Timiș County Council 04/22/2013 Public Timișoara City Council 04/22/2013 Public Arad City Council West 04/22/2013 Public Arad County Council 04/22/2013 Public SC Profi Beton SRL 04/22/2013 Public SC Balagri SRL 160 Region Date Beneficiary Institution / Organization type 04/26/2013 Public Deva City Council 04/26/2013 Public Hunedoara City Council 04/27/2013 Private SC Jecza SRL 161 ANNEX D: Spatial Distribution of ROP Projects (contracted as of December 2012) (1) Regional Level 162 (2) County Level A. Total 163 B. Public institutions 164 C. Private organizations 165 (3) Local Level A. Total 166 B. Public bodies 167 C. Private organizations 168 (4) By Key Intervention Area 169