Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00003912 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD – 48110) ON A LOAN IN THE AMOUNT OF EUR 110 MILLION (US$ 130 MILLION EQUIVALENT) TO ROMANIA FOR THE JUDICIAL REFORM ( P090309 ) September 29, 2017 Governance Global Practice Europe And Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective September 20, 2017) Currency Unit = Romanian Leu 3.83 = US$1 US$ 0.26 = 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Cyril E. Muller Country Director: Arup Banerji Senior Global Practice Director: Deborah L. Wetzel Practice Manager: Adrian Fozzard Task Team Leader(s): Klaus Decker ICR Main Contributor: Georgia Harley ABBREVIATIONS AND ACRONYMS CEPEJ Commission for the Efficiency of Justice COTS Customized Off-The-Shelf CPS Country Partnership Strategy CSM Superior Council of Magistracy CVM Cooperation and Verification Mechanism DIEFP Department for the Implementation of Externally Financed Projects EC European Commission ECA Europe and Central Asia ECRIS Electronic Court Record Information System database EMP Environmental Management Plan ENPV Economic Net Present Value ERR Economic Rate of Return EU European Union EUR Euro currency FIRR Financial Internal Rate of Return FM Financial management FNPV Financial Net Present Value FY Fiscal Year HCCJ High Court of Cassation and Justice HR Human resource management ICT Information and communications technology JHA Justice and Home Affairs JRP Judicial Reform Project JSIP Justice Services Improvement Project KPI Key Performance Indicator LA Loan Agreement LSAT Law School Admissions Test M&E Monitoring and evaluation MOJ Ministry of Justice MoPF Ministry of Public Finance MTR Mid-term Review NAP National Administration of Penitentiaries NIM National Institute of Magistracy NSC National School for Clerks O&M Operation & Maintenance ONCR Office of the Trade Register PAD Project Appraisal Document PAL Programmatic Adjustment Loan PDO Project Development Objective PIP Project Implementation Plan PSC Project Steering Committee QER Quality Enhancement Review RMS Resource Management System RON Romanian Lei ROSC Reports on the Observance of Standards and Codes STS Special Telecommunications Service TOR Terms of Reference ToT Training of Trainers USD US dollars TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9 II. OUTCOME .................................................................................................................... 11 A. RELEVANCE OF PDOs ............................................................................................................ 11 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 11 C. EFFICIENCY ........................................................................................................................... 19 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 21 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 21 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 23 A. . KEY FACTORS DURING PREPARATION.................................................................................. 24 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 25 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 26 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 26 C. BANK PERFORMANCE ........................................................................................................... 28 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 29 V. LESSONS AND RECOMMENDATIONS ............................................................................. 30 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 32 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 57 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 61 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 62 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 67 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 68 ANNEX 7. BENEFICIARY SURVEY RESULTS ............................................................................. 69 ANNEX 8. STAKEHOLDER WORKSHOP .................................................................................. 85 ANNEX 9. SUMMARY OF BORROWER’S ICR .......................................................................... 88 ANNEX 10. BEFORE AND AFTER PHOTOS .............................................................................. 96 The World Bank Judicial Reform ( P090309 ) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P090309 JUDICIAL REFORM ( P090309 ) Country Financing Instrument Romania Specific Investment Loan Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Ministry of Public Finance Ministry of Justice Project Development Objective (PDO) Original PDO The project's development objectives are: (i) to increase the efficiency of the Romanian courts; and (ii) to improve the accountability of the judiciary which should result in reduced corruption and a more transparent act of justice. Page 1 of 104 The World Bank Judicial Reform ( P090309 ) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 130,000,000 121,018,005 130,926,282 IBRD-48110 Total 130,000,000 121,018,005 130,926,282 Non-World Bank Financing Borrower 0 0 0 Total 0 0 0 Total Project Cost 130,000,000 121,018,005 130,926,282 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 15-Dec-2005 26-May-2006 15-May-2009 01-Apr-2011 31-Mar-2017 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 15-Oct-2010 12.87 Change in Results Framework Change in Loan Closing Date(s) Change in Implementation Schedule Other Change(s) 31-Aug-2011 30.62 Change in Implementation Schedule 28-Dec-2012 53.86 Change in Results Framework Change in Implementation Schedule Other Change(s) 16-Mar-2015 103.52 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Legal Covenants KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Modest Page 2 of 104 The World Bank Judicial Reform ( P090309 ) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 29-Jun-2006 Satisfactory Satisfactory 0 02 06-Apr-2007 Satisfactory Satisfactory .64 03 22-Jan-2008 Moderately Satisfactory Moderately Unsatisfactory .99 04 05-Jun-2008 Moderately Satisfactory Moderately Unsatisfactory 1.51 Moderately 05 19-Sep-2008 Moderately Unsatisfactory 1.96 Unsatisfactory 06 13-Apr-2009 Unsatisfactory Unsatisfactory 1.86 07 30-Jul-2009 Unsatisfactory Unsatisfactory 3.31 08 26-Oct-2009 Unsatisfactory Unsatisfactory 4.74 09 04-Jun-2010 Unsatisfactory Unsatisfactory 8.27 10 15-Mar-2011 Moderately Satisfactory Moderately Satisfactory 19.63 11 27-Dec-2011 Satisfactory Satisfactory 41.12 12 01-Dec-2012 Moderately Satisfactory Moderately Satisfactory 53.86 13 24-Jun-2013 Moderately Satisfactory Moderately Satisfactory 67.97 14 29-Dec-2013 Moderately Satisfactory Moderately Satisfactory 78.39 Moderately 15 12-Jul-2014 Moderately Unsatisfactory 90.40 Unsatisfactory Moderately 16 16-Jan-2015 Moderately Unsatisfactory 97.28 Unsatisfactory Moderately 17 03-Aug-2015 Moderately Unsatisfactory 103.52 Unsatisfactory 18 04-Jan-2016 Moderately Satisfactory Moderately Satisfactory 111.45 19 15-Jun-2016 Moderately Satisfactory Moderately Satisfactory 121.58 20 22-Dec-2016 Moderately Satisfactory Moderately Satisfactory 125.27 21 27-Mar-2017 Moderately Satisfactory Moderately Satisfactory 127.98 Page 3 of 104 The World Bank Judicial Reform ( P090309 ) SECTORS AND THEMES Sectors Major Sector/Sector (%) Public Administration 100 Law and Justice 100 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Public Sector Management 100 Rule of Law 100 Judicial and other Dispute Resolution 78 Mechanisms Legal Institutions for a Market Economy 11 Personal and Property Rights 11 ADM STAFF Role At Approval At ICR Regional Vice President: Country Director: Senior Global Practice Director: Practice Manager: Task Team Leader(s): Irina L. Kichigina Georgia Harley ICR Contributing Author: Page 4 of 104 The World Bank Judicial Reform ( P090309 ) EXECUTIVE SUMMARY The Judicial Reform Project (JRP) was a USD 130 million stand-alone specific investment loan. It was approved on 15 December, 2005, and closed eleven years later on 31 March, 2017. The project’s objective was to improve efficiency and accountability in the Romanian judicial sector. Project performance is summarized below. Relevance Efficacy Efficiency Overall Bank M&E Quality Efficiency Accountability Outcome performance Substantial Substantial Substantial Modest Moderately Moderately Modest Satisfactory Satisfactory JRP’s development objective was highly relevant for Romania’s context and Romania’s justice reform agenda at the time of approval. Improvements in efficiency and accountability were seen as critical to address key shortcomings in the final stages of the EU accession process. A stand-alone investment loan was the right instrument to achieve those objectives. The project objective remains relevant today. A follow-up Justice Services Improvement Project (JSIP) will support the Romanian judiciary in its next phase of reform. JRP was effective in achieving these objectives. Efficiency in the Romanian courts improved through reductions in backlogs, reductions in case processing times, upgrades to facilities, optimization of workloads, streamlining of processes and a range of training activities that helped increase workplace productivity. Accountability of the judiciary improved through the overhaul of fundamental legislation, introduction of audio recordings in all courts, training, accreditation tests for magistrates, and a range of transparency initiatives. Several project interventions improved efficiency and accountability simultaneously. In particular, the legislative overhaul and the introduction of the audio recording system across all courts were significant and sensitive reforms, which would not have been possible in the absence of the project. They were relatively low-cost yet achieved profoundly positive impacts across the justice system. Three out of the five PDO indicators were achieved, and the other two were partially achieved. Several of JRP’s major achievements were not highlighted as PDO indicators. JRP’s efficiency was modest. This was largely due to significant delays to the infrastructure and ICT activities. The project was restructured four times, with a cumulative extension of six years. The PDO, objectives and components were not changed. Delays did not increase the total project cost. It simply took much longer than expected to achieve the results. In the case of one large activity, the integrated resource management system (RMS), the benefits are still yet to be visible. JRP contributed to significant reforms in the justice sector that would not have been achievable in the absence of the project. Net benefits of the project show an Economic Internal Rate of Return (EIRR) of 71.2 percent. However, quantitative analyses should be treated with caution because JRP’s most significant achievements – especially those related to improvements in the accountability of the judiciary – are those most difficult to quantify. Bank performance is rated as moderately satisfactory. Project preparation was inadequate. JRP was far from ready for implementation at approval: it took approximately three years for the project to start disbursing. Gradually, an effective PIU and an Action Plan contingency mechanism put the project on track. FM, procurement and safeguards arrangements were satisfactory throughout project implementation. Monitoring and evaluation (M&E) arrangements were moderately unsatisfactory, primarily because of delays in setting the baseline for the results framework, and delays in reviewing and updating the indicators Page 5 of 104 The World Bank Judicial Reform ( P090309 ) and data throughout project implementation and the four restructurings. More careful M&E could have better highlighted JRP’s significant achievements. This ICR highlights several lessons. M&E should be a top priority, rather than an afterthought seen as necessary to comply with Bank rules. Projects that combine hard and soft investments should be sure to provide each type of investment sufficient focus, document how each project intervention contributes to the results chain and communicate the progress being made. An effective PIU, with the authority to take and implement decisions, is critical to success. Complex IT and infrastructure projects require effective preparation to ensure technical readiness and adequate implementation arrangements. Wherever possible, the simplest solutions for IT systems should be chosen. Projects should monitor budgetary allocations carefully to prevent delays. Slow-moving projects should consider introducing action plans with contingency mechanisms to focus attention on key milestones. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country Context: Romania was well on its way to accession into the European Union (EU) at the time of project appraisal in 2005 and on January 1, 2007 Romania joined the EU as its 25th Member State. The accession process drove the direction, timing and progress of legal and judicial reforms in Romania. In the final stages of accession negotiations, Romania committed to implement eleven specific reform commitments, seven of which related to the Justice and Home Affairs (JHA) chapter. In 2005, the Government adopted a Strategy for the Reform of the Judiciary 2005-2007 and an Action Plan which focused on these remaining challenges, including: (a) strengthening the institutional capabilities of the main judicial institutions; (b) increasing efficiency of courts: reducing duration of court proceedings and improving quality of judgments; (c) strengthening the accountability and integrity of the judiciary: the need to step up anti- corruption efforts and develop performance monitoring mechanisms, and; (d) upgrading court infrastructure and working conditions. The Justice Reform Project (JRP) was designed to address these challenges. Approval of the project helped the Romanian Government to demonstrate progress and continued commitment in the final stages of EU accession. Since accession in 2007, Romania has been subject to a Cooperation and Verification Mechanism (CVM), under which the European Commission (EC) continues to monitor the country’s progress in addressing shortcomings in judicial reform, corruption and organized crime. The CVM is a transitional measure aimed to ensure that Romania develops the administrative and judicial system needed for the correct application of EU laws, policies and programs. The EC produces reports assessing progress against benchmarks and issuing recommendations on an annual basis. On judicial reform, progress in Romania has been generally positive and visible. 2. Bank Rationale: Given the strategic importance of justice reform in Romania, there was a strong rationale for Bank investment in the justice sector. Before this project, the Bank had initiated a series of Programmatic Adjustment Loans (PALs) to support the Government’s rule of law agenda. The Bank’s Country Partnership Strategy (CPS) for FY2006-09 focused on assisting Romania to successfully integrate within the EU. The CPS was founded on the three main priorities: (a) achieving sustainable and equitable private sector led high growth; (b) upgrading fiscal management systems and reducing fiscal vulnerabilities, and; (b) enhancing governance and upgrading public institutions, including the judiciary. The JRP Page 6 of 104 The World Bank Judicial Reform ( P090309 ) contributed to all three CPS priority goals, especially the third. Subsequent CPSs for Romania have similarly highlighted the importance of judicial reform to the country’s long term growth. Theory of Change (Results Chain) 3. The Project Appraisal Document (PAD) did not describe in detail the results chain or the logic behind the operation. This was not required by Bank procedures at the time of project preparation. Nonetheless, the linkages can be discerned in several areas. Backlog reduction and targeted efforts to optimize workloads, streamline processes and improve timeliness in case processing were clearly aimed at improving efficiency across the court network. Improvements in court infrastructure were intended to increase efficiency through improved work flows and a larger number of court rooms, while also strengthening accountability by meeting EU standards and offering greater workplace standardization and increased transparency in the user experience at targeted court locations. The introduction of the court recording system was clearly intended to strengthen accountability and build public trust and confidence in the judiciary. 4. In other areas, the theory of change and the links between the PDO and the project’s approach, components and activities had to be inferred by the ICR team. Though not stated, the ICR team assumes that the training activities were intended to increase efficiency by building the capacity of training participants to work more productively. The introduction of new Codes was presumably intended to achieve large-scale legal reform that would reduce ambiguities in the law, bottlenecks in processing and generally increase legal certainty for citizens and businesses, which would in turn improve both efficiency and accountability of the judiciary. Presumably, better resource management, through a sector-wide integrated resource management system (RMS), would increase efficiency while also strengthening accountability by limiting vulnerabilities to corruption within justice institutions. 5. To achieve progress across these areas, the project worked with stakeholders across the sector, particularly with the Superior Council of Magistracy (CSM), High Court of Cassation and Justice (HCCJ), Ministry of Justice (MOJ), National Institute for Magistracy (NIM) and the National School for Court Clerks (NSC). Project Development Objectives (PDOs) 6. The PDO, as stated in the PAD and Loan Agreement (LA), is: “(i) to increase efficiency of the Romanian courts, and (ii) to improve accountability of the judiciary which should result in reduced corruption and more transparent act of justice”. The PAD and subsequent project documentation refers to these two limbs in shorthand as efficiency and accountability. The ICR team has assessed the project based on the extent to which it has achieved these two objectives. 7. The prepositional phrase ‘which should result in reduced corruption…’ suggests a second-order result rather than a direct project objective. Following consultations and review of the PAD, project design and related documentation, the ICR team concludes that this phase refers to higher-level outcomes which are outside of the project’s control. This ICR therefore reviews the project based on the two direct objectives: efficiency and accountability. Page 7 of 104 The World Bank Judicial Reform ( P090309 ) Key Expected Outcomes and Outcome Indicators 8. The project’s key indicators were: a) improved capacity of the court system to adjudicate disputes (in terms of fairness, speed, affordability and ability to enforce decisions); b) improved court facilities, in line with international standards; c) improved public image of the judiciary, and; d) enhanced competence, professionalism and integrity of judges and court staff. Components 9. The JRP project had four components: 10. Component 1: Court Infrastructure Rehabilitation (Cost USD 85.0 million). Activities included: a) rehabilitation and limited new construction of prioritized court buildings, and; b) development of uniform space planning and design standards for court facilities. 11. Component 2: Strengthening of the Administrative Capacity of Courts (Cost USD 11.05 M). Activities included: a) development and implementation of a program to reduce case delays and backlogs; b) development of a framework for economic management of the courts, including regulatory and organizational arrangements for economic managers and training; and c) optimization of courts’ operational processes, including transfer of non-adjudicative tasks from judges to court personnel, and upgrading the functioning of case registries, archives, recording of court proceedings, court statistics and case monitoring; and (d) public education/ information, strengthening the CSM’s and courts’ capacity in the area of public information and external relations. Midway through the project, activities were added including work on the new Civil Code, Civil Procedural Code, Criminal Code, and Criminal Procedure Code, as well as impact assessments of the four Codes. In 2010, work on a new Insolvency Code was also added. 12. Component 3: Court Information System (Cost USD 21.5 M). Activities included the development of a comprehensive RMS for the judicial system. (Case management systems were implemented through EU financing.) The RMS would cover financial, physical, and human resource management functions for the justice sector. It would also cover management support functions, both reporting and analytic. The RMS would support management functions at the level of the individual courts, as well as at the MOJ, CSM, and the HCCJ. 13. Component 4: Institutional Development of Judicial Institutions (Cost USD 6.61 M). This component would provide assistance to the following judicial institutions: a) CSM – in the area of development of long- term judicial policies, monitoring judicial performance, and public communications; b) MOJ – in the area of capital investment planning, judicial statistics, human resources management, budget planning and internal and external communications; c) NIM – in the area of development of new qualification tests for judges’ selection and promotion; development of training courses and curricula; (d) National School of Clerks (NSC) – in the area of strategic planning; development of training courses and delivery of training through e- learning programs; and (e) HCCJ – in the area of budget planning and IT upgrading. This component would also provide funding, to develop and implement specific tools for monitoring project results, including public surveys and court user surveys. Page 8 of 104 The World Bank Judicial Reform ( P090309 ) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 14. The PDO remained unchanged for the duration of the project. Revised PDO Indicators 15. The results framework was revised twice, first in 2010 and again in 2015. These changes are outlined in Annex 1. 16. The 2010 restructuring revised and reduced the level of ambition of the outcome indicators to take into account what can be achieved by the extended project closing date. The results framework was also modified to include more quantifiable indicators. For example, while the original framework included an indicator for “Improved capacity of the court system to adjudicate disputes (in terms of fairness, speed, affordability, and ability to enforce decisions),” the revised indicator called for a 10 percent reduction in case backlog. 17. The 2015 restructuring added a PDO indicator, "Audio recordings available to any trial participant upon request" to strengthen accountability and better link the implementation of the court recording system within the results chain. The 2015 restructuring also dropped three intermediate results indicators (IRIs), namely: a. 10% decrease of average time required to dispose of cases in 5 courts rehabilitated under this project: the restructuring paper argued that this kind of data is neither available in Romania nor could it easily be generated because there was no systematic system for data collection in place. The paper also noted that the indicator was no longer relevant because the JRP had supported the development of a country-wide performance monitoring system based around five key performance indicators (KPIs), and results are made publicly on annually on the MOJ website. b. CSM effectively assumes functions of budget planning, policy formulation and performance monitoring: this indicator was premised on the anticipated transfer of these responsibilities from the MOJ to the CSM and was intended to strengthen judicial independence. However, the implementation of the law transferring this responsibility has been suspended each year and responsibility remains with the MOJ still in mid-2017. The indicator was therefore also dropped. c. NIM and NSC design and apply web-based user feedback survey for all courses and publish quarterly results: this indicator could not be implemented technically because the open source platform used for the development of the surveys was outdated and its functions were very limited and could not be configured to request user feedback related to the online courses Revised Components 18. The project components remained the same for the duration of the project, although some of the sub-components were revised. Other Changes 19. The JRP was restructured four times during implementation. Extensions were granted to allow the completion of all project activities under implementation and the achievement of project objectives. Restructurings cumulatively extended the project duration by six years. Several changes were made to reduce Page 9 of 104 The World Bank Judicial Reform ( P090309 ) the scope of activities and the level of ambition of the indicators. The project components, focus, core activities and costs remained the same. 20. Restructuring 1 (October 2010) – This restructuring extended the project duration by two years and introduced an Action Plan conditionality mechanism. The Action Plan outlined an agreed and clear timeline for completing a selected number of key project activities, and was included in the Loan Agreement, which stipulated that failure to complete any of these activities would result in the cancellation of the associated financing for those activities. It also dropped 2 out of 20 court sites (Campina and Valeni de Munte) from the infrastructure component because they were at the early stages of approval and thus unlikely to be completed within the foreseen timeline by April 2011 and component budget available at that time. 21. Restructuring 2 (August 2011) – This restructuring extended the deadline for one of the activities in the Action Plan, the Optimization of Court Procedures, from April 1, 2011 to October 31, 2013 in order to avoid the cancelation of funds for that activity. 22. Restructuring 3 (December 2012) – This restructuring extended the project duration by another two years to allow the project finalization of civil works and other engaged activities. Similar to the first restructuring, an Action Plan was defined with the borrower. The plan included a timeline for each measure as well as a defined loan that amount the World Bank could cancel in case a target was not met. 23. Restructuring 4 (March 2015) – This restructuring extended the project duration by a further two years, narrowed the scope of the RMS under component 3, Court Information System, and revised related result indicators and disbursement projections accordingly. In addition, the Project Implementation Plan (PIP) was revised to include only those activities that could be completed within the new closing date. The plan included a risk-management or "contingency" mechanism intended to allow for the cancellation of funds under the RMS by the MOJ if the associated activities were not completed by deadlines agreed with the contractor. In case of failure to comply with the PIP, funds could be reallocated from the RMS to other activities under the project through a restructuring. 24. Financing. The financial management (FM) and disbursement arrangements for the project continued in largely the same manner throughout project implementation. Two improvements were made. First, from 2009, the project pre-financed expenditures from the State Budget through the Treasury system. Second, from 2010, the share of project finance for civil works was raised from 85 to 100 percent and the remaining undisbursed amounts were grouped into one new category to ensure effective use of the loan funds. The 2010 and 2015 restructurings revised the withdrawal schedule. The restructurings also reallocated funds across categories but not between project components. This did not entail significant changes to the project. Page 10 of 104 The World Bank Judicial Reform ( P090309 ) Rationale for Changes and Their Implication on the Original Theory of Change 25. The changes made during project implementation were focused at the activity level. They arose mainly due to poor project preparation and extensions that were due to delays associated with infrastructure and ICT activities (discussed in detail below under Factors Affecting Implementation) as well as underfinancing of the project for several years. The changes did not affect the rationale, objectives or key outcomes expected of the project. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating Rating: Substantial 26. From project preparation through to completion, project objectives aligned well with Bank and borrower’s development priorities. The PDO aligned with Romania’s national program for EU accession. The Bank agreed to focus on improvements in accountability and efficiency, given their high priority and linkage with EU accession, and they built on the Bank’s earlier projects and reflected in the Bank’s comparative advantage in justice reform. The PDO was aligned closely with the main theme of the FY06-09 CPS to enhance governance and upgrade public institutions, including the judiciary. These objectives remain relevant, align with the current CPS, and are an integral part of the follow-up project, the JSIP. 27. While the objectives were clear, the alignment and linkage between project activities and components and their contribution to increased efficiency and accountability of the Romanian justice system could have been clearer. It would have been helpful if the PAD and subsequent project documentation had articulated in detail what was meant by “efficiency” and “accountability” in this context and how the different components contribute to these objectives. Without this, the Bank, borrower, implementing agency and beneficiary institutions lacked a shared understanding of the results chain, why it was important and how each project activity contributed a link in the chain. 28. The choice of instrument, investment project financing, was appropriate, given the project’s focus on infrastructure and ICT procurement. An important factor contributing to successful implementation of the infrastructure component, and others to a lesser extent, was the application of the Bank’s procurement procedure. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 29. Several of the project interventions contributed to both efficiency and accountability of the Romanian judiciary simultaneously. However, to align with the ICR template, this section analyzes each development objective in turn and the key project interventions that contributed to its achievement. Achievement of Objective 1: Increased Efficiency of the Romanian Judiciary Rating: Substantial 30. The project increased efficiency through five groups of interventions: civil works to increase court capacity to deal with cases; targeted efforts to reduce backlogs and improve timeliness; optimization of workloads and processes; training and capacity building; and the integration of resource management across the sector. 31. Civil works. The JRP financed construction and rehabilitation of 16 courthouses, some of which are integrated court complexes and so represent 21 courts. Before the project, these courts were in provided sub-standard working Page 11 of 104 The World Bank Judicial Reform ( P090309 ) conditions, often without functioning heating, windows, and toilets, and with suboptimal work flows. Office space was cramped, and several judges worked in shifts. Archiving arrangements were ad hoc, insecure and cumbersome, with files often stored in offices and public hallways which caused a range of case processing problems. There was a shortage of courtrooms, which caused delays in hearings. Security and safety were a concern. Operations and maintenance costs were high (some before and after photos are at Annex 10). Of the 16 courts, 12 were renovated and extended and 4 were new buildings. The number of courtrooms across these 21 courts has increased from 27 to 42, and the number of counsel rooms – where lawyers can meet privately with clients before hearings – increased from 6 to 43 rooms. As a result, more cases can be heard and hearings can be held in parallel, maximizing the use of judge and staff time. Archiving is now appropriate for a courthouse. As the Iasi Court President noted, “We see a direct link from the infrastructure to the quality of justice and shorter time to disposition. Without proper infrastructure, it would not have been possible to deal with this many cases.” 32. Civil works have also contributed to greater efficiency by offering more sensible use of space, streamlined work flows, user-friendly workspaces improved staff morale and motivation. As the Court President in Pitesti expressed it, “on a scale from 1 to 10, it would be 1 for the old building and 10 for this one.” The fact that the judges no longer need to work in crowded offices, as the design standards set out the minimum space requirements for a judge to enable them to work quietly and productively, is particularly appreciated. 33. Civil works faced major delays due to obstacles common to construction projects in Romania (discussed below under Factors Affecting Project Implementation), but in the end, the project exceeded its target of building 15 courthouses. If the same works had been procured using local procurement rules, it is unlikely that the construction would be undertaken, let alone completed. With this in mind, the ICR team concludes that the efficiency gains of these infrastructure upgrades would not have been possible without the support and leverage provided by the project. 34. At project closure, a further three courthouses were under construction and will be completed under the new JSIP: Sibiu Tribunal (finalization expected end of June 2018), Prahova Tribunal (finalization expected end of October 2017) and Prahova Palace of Justice (finalization expected by the end of 2017). 35. Efforts to reduce backlogs and improve timeliness. The project supported the development of backlog reduction plans for courts and related efforts to improve timeliness of case processing. The results framework tracked the number of cases disposed of in five pilot courts, although it is not clear why these five were targeted in the results framework. Initial efforts exceeded the target of 10 percent: starting in 2008 until 2015, an increase of at least 16 percent in the number of cases disposed in the five pilot courts was registered. However, in 2016, the work volume of Orsova First Instance Court dropped significantly (from 3,584 in 2015 to 1,810 in 2016), which led to an overall drop in efficiency from 18.52 percent in 2015 to 6 percent in 2016. 36. Across the court network, there also appears to be some improvement in backlog reduction and timeliness in case processing over the period. According to the 2016 EU Justice Scoreboard (2010 to 2014 data), the overall clearance rates1 in Romania have increased in first instance courts. The Scoreboard also shows that the clearance rate for resolving litigious civil and commercial cases in first instance courts has increased from approximately 90 percent in 2010 to approximately 110 percent in 2014. The Scoreboard also shows that the time needed to resolve litigious civil and commercial cases (using the Saturn method) has been steadily reducing in Romania, from more than 200 days in 2010 to approximately 150 days in 2014. The MOJ shared with the ICR team more detailed court statistics by court type, case type and location over different time periods covered by the project. The detailed data revealed a number of discrepancies 1 The clearance rate is the number of resolved cases as a percentage of the number of incoming cases in a reporting period. Values higher than 100 percent indicate that more cases are resolved than come in, while values below 100 percent indicate that fewer cases are resolved than come in. The clearance rate indicates whether backlogs are increasing or decreasing. Page 12 of 104 The World Bank Judicial Reform ( P090309 ) but was broadly consistent with the conclusion in the Scoreboard that backlogs reduced and timeliness improved somewhat over the project period. 37. Performance management and optimization of workloads and processes. At project start, a range of inefficiencies caused frustrations in courts and other justice institutions, but the sector lacked analysis or plans to optimize and streamline workflows to solve the problem. Under the JRP, the judiciary assessed the optimal volume of work for judges and court clerks. Based on the analysis, cases are now distributed more equitably among judges and tasks have been delegated from judges to staff to boost productivity across courts. Standards were also set for ensuring and monitoring the efficiency of courts’ activity. Three KPIs were developed (clearance rate; case stock pending for more than one year, and; cases closed within one year) and a tracking system was designed for monitoring the indicators. Various causes for delays in case resolving process were also analyzed in detail. The tracking system was piloted at the level of 11 Romanian courts covering all types of jurisdictions (3 courts of first instances, 3 tribunals, 4 courts of appeal and High Court of Cassation and Justice) while the Pilot Court Implementation Plan organized work areas into all three separate but complementary main categories: Court Management, Case flow Management and Court Records. In 2015, implementation was extended to all Romanian courts. Two more KPIs (average time for disposition; and drafting of decision in excess of legal deadlines) were added by the CSM and all five are tracked annually by Court Presidents/management and by CSM. The KPIs are published on courts’ portals and are also available to the public. The system was put in place in 2015 so it is too early to assess quantitative efficiency gains. However, the ICR team heard feedback that was very positive. Lessons from other jurisdictions also suggest that a performance management system like this one is likely to improve both efficiency and accountability. The 2017 CVM report notes that on the national level, progress on equalizing workload within and between courts has been slow, including continuing discrepancies of workload between large and small courts, and this continues to have an impact on the consistency of court decisions. However, the European Commission notes that the quality of information in this area has improved and is now at the disposal of the MOJ and SCM, allowing them to continue the work in this area. 38. A Case Flow Management Guide was also developed and disseminated to all judges. The guide includes good practices that courts use in managing the flow of cases and good practices used by individual judges to manage each assigned case. 24 actionable recommendations were made in the fields of: (a) resource allocation and court structure; (b) key performance indicators for workload, time standards, data quality and ECRIS; (c) pilot courts; (d) training needs; and (e) CSM organizational and capacity enhancement. These recommendations set the groundwork for the 2015-2020 Judicial System Development Strategy (Government Decision No. 1144), which lays out the direction for judicial reform and modernization over the next five years. An Action Plan for the Implementation of the Strategy (2015-2020) which incorporated the recommendations was adopted (Government Decision No. 282) in April 2016. This means that the recommendations will be used for the development of Romanian judiciary over the next five years. The uniform standards for operational processes have been institutionalized since 2015. 39. Updating the Codes. JRP supported the adoption of a new Civil Code that entered into force in 2011, a Civil Procedure Code in 2013, and Criminal Code and Criminal Procedure Code in 2014. This achievement is discussed in detail below as an accountability measure, but it also improved efficiency. Among other things, the drafting and adoption of new Codes helped to shorten the time of resolution by simplifying and improving consistency in decision making, and clarifying roles and responsibilities in the justice system. Since the introduction of the new Civil Code, the time to disposition of cases has shortened by 43 percent. In the 2017 CVM Report, the EC also noted that the new Codes have contributed to faster court proceedings, greater respect for fair trial rights and more consistent judicial decisions. This activity cost approximately 2 million USD, a small part of the total project cost, yet this work was transformative. Almost all beneficiaries interviewed for this ICR report highlighted the positive impact that the new Codes have had on their daily work, and emphasized how the Codes have improved the transparency, predictability, availability of consistent case law Page 13 of 104 The World Bank Judicial Reform ( P090309 ) and the simplification of proceedings. According to the Court President in Iasi Court of Appeal, there has been a visible improvement in efficiency since the introduction of the new Codes. 40. A related project intervention was to conduct impact studies on these new Codes to ensure that Romania was meeting its responsibilities and to prevent implementation gaps in the Codes. The ICR team concludes that, in the absence of the JRP, these impact assessments would have been unlikely, or would not have been conducted to the level of quality to achieve their purpose. The four impact assessments (one for each Code) assessed what resources were needed to ensure the effectiveness of the new Codes. Four reports were drafted: a technical report on the new procedures and their impact; a report on human resources, assessing how many new staff would be needed and new positions created; a report on budget and infrastructure; and a final report with conclusions. The impact assessment reports facilitated a smoother transition to the new Codes, thus strengthening the administrative capacity of courts and improving the efficiency of cases processed under the new legal framework. 41. Training and capacity building. The project supported a range of training activities that helped to improve the capacity and productivity of judges and staff working across the sector, while also increasing the efficiency of training delivery. The project supported the NIM and NSC to introduce e-learning, including an e-platform and training of trainers (ToT) to use the platform. At project start, there was no e-learning platform, and by project closure more than 1,215 magistrates and court clerks had undertaken e-learning using the platform at NIM and NSC. The ToT helped the NIM to create a pool of experts for e-training of magistrates. 42. The NIM then created 24 courses on the newly adopted Codes: four larger courses (one for each Code), which take up to one year, and; 20 smaller courses on discrete topics, which normally last 8 to 12 weeks. The NSC has also drafted guides and handbooks on the four new Codes for court staff. The material is also available in electronic format and posted on their website also. The training activities, particularly training on the new Codes, have had a positive impact on the work in courts. Interviews with judges and magistrates during the ICR mission revealed that most of them had participated in courses on the new Codes and appreciated their content. 43. Since 2016, the number of courses has doubled and the NIM is currently providing 54 e-learning modules on a range of topics, with 20 to 40 people trained per each course. The NIM is also offering more than 300 face-to-face trainings. The NSC trained 40 trainers on the e-courses for court clerks. In 2015, the NSC then trained 420 clerks, organized 7 e-learning sessions and 3 blended learning sessions on topics such as civil procedural law, criminal procedural law, English for legal purposes, and Romanian legal language. The Romanian legal language e-learning course also had 100 participants from the Republic of Moldova in 2015. The e-learning courses have been a great success, and both NIM and NSC now want to team up with the Bank to upgrade and customize the e-learning platform, which would cost around 10,000 EUR. The e-platform was developed in 2008 and is outdated and currently only supports three courses operating simultaneously, whereas the ambition of the NSC and NIM, and the demand of participants for courses, has grown. 44. With JRP support, the NSC has also organized several ToT seminars, and then training programs, on soft skills for court clerks and prosecution clerks on topics such as communication, time management, and file management. These training programs has been very popular among court clerks, as training on this kind had not been available before the JRP. 45. Unfortunately, the training platform does not support course evaluations, and this was not insisted upon during the activity design stage. As a result, the NIM and NSC do not have any aggregated user feedback that they can use to reliably assess training impact or inform future trainings. (For the same reason, training records are not used in the project’s M&E system, nor relied upon in this ICR.) The NIM and NSC, however, report that anecdotal feedback on the training is positive. Page 14 of 104 The World Bank Judicial Reform ( P090309 ) 46. Introduction of an integrated RMS. The RMS system was intended to transform the way that human and financial resources are managed in the justice sector. In retrospect, the RMS was overdesigned and too costly for the functionality it offers. It is also not clear that the full system works and the full ambit of change that it will create, as the RMS is still not complete. The idea was to replace approximately 160 legacy applications with an integrated system that would allow the different institutions to use the data for planning and management purposes. For example, the HR module allows the user to see how many judges will retire over the coming years and help the CSM to plan the recruitment of new magistrates. The RMS introduced new solutions which did not exist before, such as a solution for payroll and HR. It also has integrity features, such as the inability to erase entered data and the ability to track who is working in the system on what and when. 47. The RMS is a complex system comprising several modules/sub-systems. These are: the Human Resources and Payroll module for the judiciary personnel (approximately 35,000 people), meant to manage about 90 percent of the judiciary budget, with a payroll component; the Financial Module, to manage all other resources of all judiciary institutions, except personnel; the Budget module, intended to be used by all institutions for budget development, allocation, tracking, and execution, in line with the provisions of the State Budget Law; and a Special sub-system for National Administration of Penitentiaries (NAP), through which all claims of detainees and their families could be more easily managed. 48. By project closing, some of these components had been implemented, some were still being rolled out. The implementation of the RMS system has been marred by a series of delays. The HR/Payroll solution is operational at the CSM, HCCJ and the PM and its subordinated units. However, the old, existing systems continue to run in parallel at the CSM and HCCJ. The DIEFP will continue working with the RMS beneficiaries on transition plans. The RMS solution for NAP is currently at the user acceptance stage; and data migration for the RMS solution at the HCCJ and MOJ is underway. NAP deployed the document management solution (ER/DR) in March 2017. The RMS activities which remained incomplete after project closure on March 31, 2017 will be supported by State budget. Negotiations are currently ongoing between the MOJ and the RMS supplier to establish a final deadline for the remaining activities. 49. As of May 2017, the Human Resources and Payroll module is live and functioning at the level of CSM, HCCJ and PM, while its roll out at MOJ, courts, NAP and all prosecutors’ offices is ongoing. This particular module adds the most immediate value to the judiciary institutions, since it manages approximately 35,000 members of judiciary personnel and 90 percent of the annual judicial budget. The Financial module has passed all the functional tests and hence was accepted from a functional perspective by the following beneficiary institutions: MOJ, PM and HCCJ. Testing is on-going for NAP and CSM. The budget module passed all the functional tests and hence was accepted from a functional perspective by the following beneficiary institutions: MOJ, NAP, CSM, MPA and HCCJ. Testing is on-going for the Public Ministry. The ERDR has passed all the functional tests and was accepted from a functional perspective by the only beneficiary of this module, NAP. The RMS will continue to be implemented at all beneficiary institutions, funded from the State budget, until successful completion. 50. On a positive note, feedback on the RMS is promising as stakeholders start to use the system. The CSM reported that the system is better than the old one and has already decreased workload, contributed to transparency and enables the users to respond to certain queries to which they were not able to respond before. The HR module allows for projections on staffing needs. CSM representatives noted to the ICR team that the RMS “ has a huge advantage over the old system. It is something that should have been done a long time ago.” In a few years, it is possible that the RMS could become a key success of the JRP, but it is difficult for the ICR team to gauge the time impact trajectory of this significant reform while it is not quite complete and only six months after project closure. Achievement of Objective 2: Increased Accountability of the Judiciary Page 15 of 104 The World Bank Judicial Reform ( P090309 ) Rating: Substantial 51. The project increased accountability through five groups of interventions: overhauling the legislative framework; introducing audio recording of all court hearings; upgrading facilities to help ensure the right to a fair trial; magistrate certification; and a range of performance management and transparency initiatives. 52. Overhauling the legislative framework. Legislative reform had a pervasive impact on the justice system of Romania. The project assisted the MOJ update the Civil and Criminal Codes and Civil and Criminal Procedure Codes, altering how citizens and businesses are regulated and how cases are processed in Romania. At project start, legislation and procedure were in urgent need of update, as the four fundamental laws had barely changed in 150 years. Several unsuccessful attempts at reform had been made, the last in 2005, but each had failed to produce substantive legal reform. In 2007 and 2008, the JRP implemented six related technical assistance projects, where professionals from Romania and abroad provided their expertise to the Government appointed commission drafting the new Codes. This entailed drafting and correlating 4,846 articles incorporated in the new Codes and analyzing their interaction as a framework. The reform, including its scale and scope, would have been unlikely without JRP intervention. The 2017 CVM report notes that the adoption of the Codes has led to the acceleration of court proceedings, greater consistency in judicial decisions, and enhanced respect for the right to a fair trial. The CSM reports that there is now greater accountability in the justice system thanks to the new Codes in two ways. First, at an individual level, judges working on cases know that are more answerable to the new law than before. Second, at an institutional level, more cases are reviewed by the Supreme Court, and published and analyzed by the profession and build a more consistent approach to the law (often referred to as case law harmonization). 53. In addition, JRP supported reforms to insolvency legislation, which increased transparency, improved clarity in case processing and reduced the number of insolvency cases. This activity was not included in the original PAD but was added in 2010 following consultations with stakeholders. The MOJ had on several previous occasions attempted insolvency reform, each time to no avail, so it was perceived that reform had a greater chance of success with Bank and JRP support. From 2011 to 2014, the Romanian Insolvency framework and related mechanisms were strengthened thorough the elaboration of the first national Insolvency Code, in line with European ROSC standards. The courts and parties are now obliged to send all case documents to the National Office of the Trade Register (ONCR), which are then published in an online bulletin available to the public and the courts. All procedural documents, minutes of meetings of creditors, documents drawn up by liquidators and any other official documents relating to the file are published in the ONCR bulletin, which provides useful information to courts, regulators, consumers, and businesses, including banks, creditors and clients. The number of insolvency cases fell because the Code introduced a minimum threshold of 40,000 RON. The new Code has also clarified issues which were ambiguous, eliminated bottlenecks and simplified the liquidation of assets. A further enhancement was the introduction of the enforcement procedure from the new Civil Code for insolvency cases. The World Bank’s Doing Business data indicates some improvement in Romania’s performance in terms of resolving insolvency. Romania’s distance from the frontier for resolving insolvency has improved from 27.65 in 2011 to 59.16 in 2017. Recovery rates have increased from 25.7 cents in the dollar in 2011 to 34.4 cents in 2017. However, the Doing Business indicators for time, cost and outcome have remained stagnant. Romania remains behind the Europe and Central Asia (ECA) average with regard to the strength of insolvency framework index 2, where on a scale from 0 to 16 Romania ranks 13. 54. Improvements in the quality of Romanian law also increased accountability indirectly. Since the adoption of the new Codes, the HCCJ has undertaken effort to promote the consistency of court decisions. HCCJ already published some 2The index is the sum of four component indices: commencement of proceedings index, management of debtor’s assets index, reorganization proceedings index and creditor participation index. The index started to be collected only in 2015, so it is not possible to compare the current score with the situation that existed prior to the reforms supported by JRP. Page 16 of 104 The World Bank Judicial Reform ( P090309 ) of its decisions before the adoption of the new Codes but decisions were often scattered, un-searchable and sometimes contradictory. Today more decisions are published and there is a greater consistency of case law. With more of the case law being available online, judges can access court decisions and apply the findings on similar cases. Under the Codes, judges are also able to ask the Supreme Court directly when there is need for clarification. While the JRP did not finance the online availability of court decisions, the procedural Codes supported by the JRP have provided the HCCJ and other courts with the necessary tools for a more transparent and predictable act of justice, consistency in decision making and greater accountability of judges. 55. Audio recording of all hearings. One of JRP’s most important achievements was the introduction of audio recording for all court hearings throughout Romania. The audio recording system is one of the success stories of JRP, not only for the greater transparency and accountability it provides, but also for being highly valued by its users as a practical tool. At project start, none of the Romanian courts had audio recording systems in place. Handwritten notes were kept by clerks but were unreliable for accountability purposes. This is a challenging and sensitive reform – both in Romania and in judiciaries around the world – and such initiatives often fail. In Romania, attempts had been made to introduce these systems for more than five years, each to no avail, due to stakeholder resistance and procurement problems. The ICR team concludes that this intervention would not have been possible without JRP’s support and leverage. By JRP closure, integrated audio recording systems were operational in all 243 national courts and all 699 courtrooms and available within the day upon request by any interested party. Hundreds of audio recordings are requested and used in every day. The National Judicial Inspection uses these recordings to verify judges’ behavior, and audio recordings have been used in a range of corruption and misconduct cases. Stakeholders report that the courtesy and professionalism of judges has improved since hearings have been recorded. In addition to strengthening accountability, the recordings greatly facilitate the work of judges, prosecutors and lawyers and thereby increase efficiency in case processing. Court Presidents and judges reported to the ICR team their satisfaction with the court recording system. 56. Civil works. The efficiency gains from the civil works are discussed above, but improved courthouses also strengthen accountability by enabling Romania to meet its responsibilities for quality justice based on various EU and international standards. The counsel rooms enable lawyers to meet with their clients confidentially in a professional space before and after their hearing. This is particularly important for defendants on pre-trial detention in criminal cases, where this may be their only opportunity to meet with an attorney. The number of holding cells, usually in the basement of the courthouse, more than doubled. Standards are now met for the treatment of prisoners, including the separation of men, women and juveniles, and adequate space, hygiene, and safety of all parties, especially defendants, guards and lawyers. Common areas are more conducive for the public: they are clean, safe and offer useful information for parties awaiting hearing. Appropriate archiving spaces and practices prevent the disappearance or misuse of court records. Each new courtroom has been designed to offer separate entrances for judges, defendants, prosecutors, witnesses, lawyers and the public. This improves workflow, offers a more secure and safe environment, and increases role clarity among justice stakeholders. Importantly, it also limits the informal interactions between judges, prosecutors and lawyers at the courthouse. Stakeholders also report a general improvement in the tone and atmosphere of court proceedings. The 2016 CVM Report highlights that future structural reforms to the Romanian justice system need to be accompanied by the modernization of buildings and IT systems, and notes that the World Bank is supporting such work in Romania. 57. Interestingly, while the quality of court decision-making has improved during JRP implementation, the difference is outsized and most marked in courts that received the infrastructure upgrades. Poor quality of judgements has been a major problem for several reasons, including uneven workloads of judges, backlogs, limited access to case law, poor quality and inconsistent legislation and insufficient training and specialization. While courts across Romania received support to address these challenges, only 21 courts received the infrastructure upgrades. According to the borrower’s report, the quality of the judicial services delivered in these 21 courts was analyzed from two different angles, namely the legal quality and the perception of the external court users who interacted with the courts. The legal quality is measured Page 17 of 104 The World Bank Judicial Reform ( P090309 ) by the annual cassation index which is calculated in percentage, as follows: total number of cases for which appeals/second appeals or any other type of claim were admitted by a superior court (regardless of the appealed matter, such as civil, administrative, criminal, fiscal), divided by the total number of cases closed with a court decision by that specific court for which the index is calculated, multiplied by 100. To capture the perception of the users, the borrower carried out three surveys, presented at Annex 7. As per 2015, data indicates that all courts part of JRP registered a low cassation index, below 8 percent of the total number of decision taken illustration a high quality of the decision. Some of the courts issued decisions of a very good quality, such as CA Iasi, as only 0.56 percent of them were modified in appeal or second appeal or by any other extraordinary legal means of attach. The interval varies between 0.56 percent (Pitesti and Iasi CA) to 7.56 percent (Tribunal Iasi), the rest of the courts being in this interval. Table 1: Cassation Index 2015 for 21 courts financed under JRP project Name of the court Total no. of cases Number of cases for which Cassation Index (Total no. of cases/Number of resolved/closed appeals were successful cases for which appeals were successful as %) CFI Blaj 1,918 46 2.39% CFI Vatra Dornei 2,407 119 4.94% CFI Viseu de Sus 2,163 118 5.45% CFI Tulcea 9,022 320 3.55% CFI Slobozia 6,047 158 2.61% CFI Orsova 3,289 80 2.43% CFI Bolintin Vale 4,982 135 2.70% CFI Saliste 1,289 74 5.74% CFI Oradea 22,074 685 3.10% Tribunal Arges 7,766 569 7.32% Tribunal Maramures 10.326 602 5.82% Tribunal Ialomita 7,766 569 7.32% Tribunal Cluj 13,931 NA 1.27% Tribunal Tulcea 4,417 233 5.27% Tribunal Iasi 18,641 1,411 7.56% Tribunal Dolj 23,159 612 2.64% Tribunal Suceava 6,800 386 5.67% Tribunal Bihor 12,723 541 4.25% CA Iasi 8,018 45 0.56% CA Pitesti Info not available Info not available 0.56% CA Oradea 642 24 3.73% 58. Magistrate certification. Certification increased accountability by ensuring a more objective and transparent admission process for magistrates and prosecutors. At project start, there was no law students’ admission test (LSAT) in place. Under JRP, the LSAT was developed and has become embedded into practice. During the first pre-test in 2009, 100 law students participated, and by the latest round, the number had increased to over 2,000. The CSM and NIM have adopted regulations requiring the continuous use of the LSAT test for magistrates. The test has a considerable impact on admission as it represents one-quarter of the final grade of the candidate. Greater accountability is also reflected in the number of appeals from unsuccessful candidates, which has fallen since the introduction of the LSAT. Candidates now limit complaints to the subjective parts of the admission test, such as the interviews. While the number of candidates who are admitted as magistrates is approximately the same, stakeholders report that the test has gradually increased the caliber of candidates who are admitted. Stakeholders report that the existence of the LSAT also promotes greater public trust in the admission process and in the perception of quality and reliability of magistrates and prosecutors across Page 18 of 104 The World Bank Judicial Reform ( P090309 ) Romania. While the EC’s 2017 CVM Report does not explicitly mention the LSAT test, it notes that the rigor of entry procedures into the magistracy, together with obligatory training by NIM, has played an important part in consolidating the professionalism and independence of the magistracy in Romania. 59. Performance management, surveys and transparency initiatives. JRP supported a range of activities to boost transparency in the judiciary that contributed to greater accountability across the justice sector. At project start, the judiciary did not publish any court performance data and did not conduct any surveys of court users or institutional stakeholders. As outlined in the PAD, it was impossible to tell what was a high-performing court, judge or staff member, let alone what to do to address under-performance. Under JRP, the websites of courts and CSM began to publish court performance data periodically. They also published data on their KPIs, discussed under the efficiency section above, which the EU considered to be a key accountability mechanism. JRP also conducted and published data from the surveys of individuals and firms on access to and satisfaction with judicial services (including fairness and timeliness) for courts financed by Part A of the project. Three periodic user surveys were conducted and published: a baseline survey; an intermediate survey, and the final survey just after project closure (See Annex 7). The surveys were shared with the European Commission for the Efficiency of Justice (CEPEJ) and were used to certify that Romania conducts surveys for purposes of CEPEJ’s annual EU Justice Scoreboard and biennial Evaluation of Judicial Efficiency Reports. JRP also financed a survey relating to the reform of the four new Codes. MOJ published the impact studies on these Codes in 2012. Although worthwhile, these transparency activities could have been more ambitious to further pursue the accountability objective. Given the exponential increase in online transparency among governments globally, JRP could have pressed for more than annual publication of basic data. Also, once the targets in the results framework were met by 2012, the level of ambition could have been raised in any of the three subsequent restructurings that extended the project duration. The surveys did little to inform policymaking on ongoing reform. They were undertaken primarily to ensure compliance with Bank results monitoring. Furthermore, attribution is only partial given that Romania, as an EU member state, was always likely to increase transparency to meet rising EU and citizen expectations. Nonetheless, JRP interventions were worthwhile and contributed to greater transparency, and therefore lay groundwork for greater accountability, across the justice sector. Justification of Overall Efficacy Rating Rating: Substantial 60. JRP substantially achieved the two objectives set out. Several of JRP’s interventions are notable successes, in particular the JRP’s support to: the overhaul of Romania’s legislative framework; the introduction of audio recordings in every Romanian courtroom, and; the upgrading of court facilities. Several of these activities contributed to improvements in efficiency and accountability simultaneously and in a symbiotic manner, which helped to overcome sensitivities, so that reforms are now embedded in practice. 61. The RMS has yet to contribute significantly to the achievement of these two objectives because the work is not yet complete. Feedback from the existing modules suggests that it may, in the end, achieve its purpose, but this has not been reflected in the efficacy ratings. C. EFFICIENCY Assessment of Efficiency and Rating Rating: Modest 62. The efficiency of the project is rated as modest. This is largely due to implementation delays which caused the project duration to more than double. Despite the extended length of the JRP, costs remained in line with those set out Page 19 of 104 The World Bank Judicial Reform ( P090309 ) in the PAD. This is largely due to a fall in construction prices and very low inflation associated with the global financial crisis. Implementation was hampered by the lack of State budget allocation from 2012 to 2015. 63. No ex ante economic and financial analysis was conducted of the likely benefits and costs of the JRP. The PAD argued that it is hard to quantify the benefits resulting from improved justice. The expected efficiency gains resulting from project outcomes included reductions in administrative expenses and a higher number of cases being resolved in less time, leading to better use of public resources. 64. The ICR team conducted an ex post economic analysis at Annex 4. This ex-post economic analysis of the project considers the direct benefits and costs associated with changes in economic welfare that were a result of project implementation. The analysis focuses only on the period of project implementation, and future benefits beyond project closure are not included. The lengthy project period, and the early period of idle funds and slow implementation progress, are factored in to the analysis. Two benefit streams are identified 65. First time savings from more efficient use of time of judges, court staff and lawyers enabled by various project interventions. This part of the analysis estimates the joint effect of civil works done by the project (16 courthouses built and/or reconstructed), related increases in counsel rooms used by lawyers, codes overhauling (Civil and Criminal Code and the Civil and Criminal Procedure Code as well as Insolvency Law), introduction of a court recording system which is operational in all 243 courts across Romania, training activities performed by the project and finally work load optimization. These interventions were split into those affecting only the judges, staff and lawyers in the jurisdiction of the 16 courthouses where significant civil works were done, and those affecting the court network across Romania. Variation in years of implementation of certain actions is accounted for in the analysis. Finally, the effect of these interventions is calculated through assumed time savings of judges, court staff and lawyers and the official data on gross hourly earnings of these categories. Estimates for time savings are at the lower bound of the expected range of time savings for each category. Even so, the aggregate undiscounted amount of economic benefits estimated was EUR 54.6 million. 66. Second, Improved human and financial resources management as a result of the introduction of the Resource Management System (RMS). The RMS replaces approximately 160 existing applications by introducing a single platform which would include HR and payroll module, financial module, budget module and the module for NAP. Since the RMS is only partially implemented, the benefits are assumed to represent only a small fraction of its potential, currently 0.1% of the total court budget for 2016 and 2017. As a result, this part of the project is assumed to generate aggregate undiscounted EUR 6.5 million of economic inflows. 67. Figure 1 below shows the distribution of net economic benefits that resulted from JRP over the investment horizon. Page 20 of 104 The World Bank Judicial Reform ( P090309 ) Net Economic Benefits (Euro) 16000000.0 14000000.0 12000000.0 10000000.0 8000000.0 6000000.0 4000000.0 2000000.0 .0 -2000000.0 68. As a result of these two quantifiable elements of the project, the Economic Net Present Value (ENPV) of the total project amounts to EUR 47.4 million at a 5 percent discount rate with an EIRR of 71.2 percent. The ENPV of the project also remains positive at much higher discount rates. For example, a discount rate of 10 percent yields an ENPV of EUR 30.8 million, while the switching value of the discount rate (i.e. the value that makes the ENPV equal to zero) is 72 percent. 69. Nonetheless, the PAD raised a valid point: several of JRP’s benefits are those that are least capable of being quantified. In addition to the project components for which a formal EA was done, JRP produced several benefits that are difficult to quantify. These include for example: benefits from an enhanced legal environment achieved through higher legal clarity and certainty as a result of redesigning the Civil Code, Civil Procedure Code, Criminal Code, Criminal Procedure Code and Insolvency Law; construction works at the 16 courts and the substantial upgrade of working conditions contributed to improved staff morale and motivation, more sensible use of space, and streamlining of work flows; the introduction of court recording system improved the overall transparency, accountability and legal certainty of the court system in Romania. D. JUSTIFICATION OF OVERALL OUTCOME RATING Rating: Moderately Satisfactory 70. The ICR team rates the overall outcome of the JRP as moderately satisfactory. JRP supported significant and fundamental reforms that contributed to improvements in the efficiency and accountability of the Romanian judiciary. However, a satisfactory rating is precluded by the various delays in project implementation, and the setbacks that dogged the RMS, which necessitated multiple project extensions. The RMS amounted to 15 percent of the project budget, but has so far produced very few tangible outputs or benefits. This has also increased the risk of the project. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 71. The justice sector in Romania is overwhelmingly comprised of women. In 2016, women comprised 74 percent of all judges in Romania, higher than the averages in the EU11, EU27 and in ECA. The gender disparity is even more pronounced for court staff, clerks and other administrators. Unfortunately, gender-disaggregated data was not collected Page 21 of 104 The World Bank Judicial Reform ( P090309 ) under the JRP to demonstrate the positive gender impact of project interventions. However, viewing the JRP in the aggregate, the project appears to have had a positive gender impact by improving the working conditions, professionalism and productivity of justice institutions, and the skills and capacities of the women who work in them. Institutional Strengthening 72. JRP helped to strengthen justice institutions, especially the courts, MOJ, CSM, HCCJ, NSC and NIM by improving efficiency and accountability. JRP did so in several ways, including by improving working conditions, streamlining process and work flows, updating foundational laws, supporting performance management and providing training. Mobilizing Private Sector Financing 73. JRP did not mobilize private sector financing. Justice is a core state function which is typically financed from the state budget. Poverty Reduction and Shared Prosperity 74. Improvements in the efficiency, transparency and predictability of the justice system help to foster a more conducive business and investment climate. Unfortunately, data has not been collected to measure private sector impacts beyond the economic analysis at Annex 4. However, faster and more predictable processing of cases, especially relating to insolvency, helps lower the costs of doing business. For example, according to the Business Environment and Enterprise Performance Survey (BEEPS), the percentage of Romanian firms reporting that the courts are a problem for business operations fell from 72 percent in 2008 to 27 percent in 2013. The percentage of firms reporting that bribery is frequent in dealing with the courts also fell from 14 percent in 2008 to 1 percent in 2013 and is now better than ECA and EU11 averages. Newly established and rehabilitated court buildings in accordance with international design standards also contribute to better public access to the facilities as well as to court materials, forms, records and archives, and thus contribute to improved access to justice for the general population, in particular for vulnerable groups. Other Unintended Outcomes and Impacts 75. The new or rehabilitated court buildings have become an external motivator for staff, which has also had an impact on efficiency. Unfortunately, quantitative data on the efficiency impacts of staff engagement was not collected under the JRP. However, qualitative feedback is overwhelmingly positive. During the site visits to Pitesti and Ploiesti, the court Presidents highlighted the general improvement in motivation among staff and that the atmosphere and tone of work, in hearings and in back offices, have also improved. In Oradea, the rehabilitated court building (which had not been renovated since it was built in 1890) has attracted new judges to the court, including judges who had never been to Oradea. Page 22 of 104 The World Bank Judicial Reform ( P090309 ) III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME Page 23 of 104 The World Bank Judicial Reform ( P090309 ) A. . KEY FACTORS DURING PREPARATION 76. Project preparation was accelerated. There were critical shortcomings in the preparation of the infrastructure component, the RMS and M&E baselines. Poor preparation took years to rectify and caused lengthy implementation delays. 77. Infrastructure delays were foreseen during preparation and not adequately addressed. Concerns were raised both at the Quality Enhancement Review (QER) in September 2005 and the Decision Package Review Meeting in October 2005 regarding the readiness of the infrastructure and whether the timelines were feasible. The QER recommended that a rigorous assessment should be conducted of the feasibility of the proposed rehabilitation program within the implied four-year time frame. The preparation team then conducted an assessment, but only focused on the private sector capacity and MOJ capacity (which were both found to be adequate), and did not review issues such as readiness of documentation, designs and permits. Although the PAD had indicated that preparation of infrastructure procurement packages was advanced and would be ready by loan effectiveness, no such packages had been prepared. According to the DIEFP, it took almost two years to update the required technical documentation before they could start the procurement process. Moreover, at project appraisal, the DIEFP did not have the internal authority and structure to approve procurements on behalf of MOJ of the various design phases (such as Feasibility Study and Technical Detailed Design). So they had to rely on another department of the MOJ, which did not fully ensure and facilitate the bureaucratic process of internal approvals. Only in late 2008 did the DIEFP become in charge of MOJ internal technical and financial approvals, as a separate structure was created inside the DIEFP – The Technical and Economic Committee of the DIEFP (TEC – DIEFP). The first civil works were finalized only in 2010. 78. The Bank’s Cultural Property safeguard policy had not been triggered during preparation, even though eight of the twenty courthouses initially covered by the Project were designated as historical sites under Romanian law. It appears that the Bank and client simply had not undertaken the necessary due diligence to find out their status. In the case of the Sibiu Tribunal, it appears that the authorities notified the DIEFP that the courthouse had been declared a historic building only after the designs were finalized and works were ready to begin. This had significant implications for the approval process for technical design and construction authorization. The time taken for such works was far longer than initially estimated during appraisal due to special requirements for restoration sites, including spatial and operational restrictions. Thus, the team had seriously underestimated the level of effort needed to finalize the technical packages for these types of civil works. 79. Preparatory work on the RMS was also inadequate. The Terms of Reference (TOR) for the RMS was only prepared in 2007, two years after project start. The initial idea behind the RMS, which was apparently advocated by a Bank consultant, was too ambitious: it aimed to build a single integrated system for five different institutions, each of which had with their own internal complexities with different approaches to resource management. The RMS was modeled on a similar system in France, without taking sufficient account of the Romanian context. There was no compelling business case for these institutions and these functions to be merged in one system. The Customized Off- The-Shelf (COTS) system that was originally chosen did not have a number of functionalities specific to the Romanian context. Consequently, iterative changes were needed to adapt the original design, which caused further and repeated delays. In 2011 the whole system was redesigned from a single system to several that were adapted to each institution’s needs. Stakeholders report that the RMS is rigid and difficult to understand compared to the simple r, albeit obsolete, legacy systems, and that the complexity and delays limited stakeholder buy-in from the outset. 80. Lastly, the project team did not focus on results and their monitoring. There was no baseline for the project, nor any documented efforts to create one during the preparation phase. This had a knock-on effect across the results framework and all components. Page 24 of 104 The World Bank Judicial Reform ( P090309 ) B. KEY FACTORS DURING IMPLEMENTATION Adverse Factors 81. Inadequate Preparation. JRP was at an almost complete standstill during its first three years of implementation. Issues contributing to the delays were described in the Project Launch Aide Memoire from March 2006. These included: outdated and inadequate feasibility studies; lack of prepared procurement packages; outdated cost tables and procurement plan; and delayed design standards (due to problems with contract negotiations). These problems were compounded by changes in the court requirements during implementation. Design standards were only approved after project approval causing delays because court designs drafted during project preparation had to be revised significantly to comply with the new standards. CSM’s Rationalization Plan listed two of the courts selected for rehabilitation by JRP (in Horezu and Orsova) among those to be closed. In the end, these two courts were not approved for closure by the authorities and works could commence. Changes to projected staffing in several of the courts called for revision of space requirements and adjustments in design. 82. High Staff Turnover. Significant changes to personnel in the early years – with 3 TTLs during the project’s first four years, and 16 Ministers and 8 project coordinators during the project implementation – slowed project implementation as incoming decision-makers developed their understanding and ownership of the project objectives. By 2009, the borrower had still not developed an Operational Manual, which also led to delays, in particular for the civil works component, due to insufficient day-to-day coordination with the committees outside of MOJ. By the later years though, the DIEFP comprised the right personnel and skills, and implementation became much smoother. 83. Inadequate Budgets. Under-funding of projects is a systemic issue in Romania. Projects in Romania receive budget allocations at the beginning of the fiscal year which then are adjusted at the state budget rectifications in the second half of the year. Initial budget allocations were often inadequate to cover payments of contractors and consultants. Sometimes shortfalls were corrected in the budget rectifications, sometimes allocations were cut. Budget constraints caused significant delays throughout and caused the Implementation Progress rating to be downgraded to Moderately Unsatisfactory in the period 2013 to 2014. Underfunding creates uncertainty for contractors who are reluctant to deploy staff and equipment, or only do so late in the year when funding becomes available. This was a particular problem for the RMS and for infrastructure contractors. In three cases, Cluj Tribunal, Iasi Palace of Justice and Tulcea Tribunal, contractors stopped or postponed scheduled work due to lack of budgetary allocations. Budget issues were addressed by the CMU as part of their country dialogue and some improvement was noted from 2015 but the problem was never fully resolved. 84. Lengthy Approval Processes. JRP suffered delays due to a lengthy approval processes even for small investments, especially from 2006 to 2009. According to Art. 42 (1) of the Public Finance Law, projects with a value over RON 30 mil. (approx. USD 7.4 mil.) must ensure that all investments are approved by the Government. This affected several projects in Romania, but particularly affected the JRP, which included 19 court investments, many of which, especially in the last years of the project, with values over RON 30 million. Each activity had to go through the three-stage approval process, which routinely lasted 2.5 years to 4 years. 85. Audit Risk. Stringent compliance audits by the Court of Accounts and civil servants’ personal liability for decisions creates an environment where civil servants are reluctant to approve procurements, accept deliverables and authorize payments. Decisions are deferred up to the highest levels of Government, where there is less ownership, frequent changes and less knowledge of the project. Among Bank project implementation units, the DIEFP was also one of the most effective in obtaining internal approvals within the MOJ, but it was not immune to this chilling effect. Positive Factors Page 25 of 104 The World Bank Judicial Reform ( P090309 ) 86. PIU Capacity. Once the DIEFP was fully functional, it became a significant positive factor in project implementation. As one stakeholder noted, over time ‘It became a well-oiled machine, addressing countless challenges in its path as best it could’. By the later years of the project, the DIEFP developed a reputation for the highest-capacity PIU in the Bank’s portfolio with a team of dedicated and knowledgeable local professionals, including a judicial specialist, civil engineer/architect, IT specialist, procurement specialist, and financial specialist,. Several of the staff had taken Bank-organized procurement training as well as a FM seminar for Bank-management projects. Due to the Bank’s strengthening of its application of its safeguard policy, the DIEFP was recently found to have insufficient expertise in the area of Environmental Safeguards (more on that in section 2.4). 87. Bank Procurement Procedures. DIEFP argues that the Bank procurement procedure as one of the most important factors for successful implementation. Bank procurement templates were helpful and the Bank’s feedback and clearances secured support of top management and allayed fiduciary concerns. Bank procurement procedures reduced delays from contested processes. Under Romanian procurement law, any bidder has the right to file a claim against the process. The ensuing court procedures can last years, while the winning contract is blocked and the works cannot start. Under the Bank procurement procedures, filing a claim does not block the award and the works can commence. The Bank’s procurement procedure is also more transparent because each bidder is informed why they failed. Finally, the Bank allows a certain degree of flexibility regarding amendments without having to re-tender the parts of the contract subject to the amendment. The Bank allows for amendments of the contract. While 11 years is a long time, stakeholders report that many of these project interventions – especially the infrastructure and ICT components – would have been further delayed, or would not have possible, in the absence of Bank procurement procedures. 88. Contingency Mechanism. The Project Implementation Plan revised during restructuring in 2010, and 2015 introduced a risk-management or “contingency” mechanism which allowed the Bank to cancel portions of the loan in case of failure to achieve dated milestones. Initially this focused on infrastructure and led to cancellation of two out of twenty civil works sites. The contingency mechanism subsequently focused on the RMS with the renegotiation of the contract in August 2015. State funds have been allocated to ensure completion of the RMS after the closure of the Bank loan. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design and Implementation 89. According to the PAD, the CSM and MOJ project teams were responsible for monitoring of the project results. DIEFP was responsible for consolidating information provided by various project counterparts and courts to track progress against the monitoring plan. 90. It should be noted at the outset that this was a challenging task during the early phase of the project, because the data environment in the Romanian justice system was very rudimentary and far below EU comparators. Statistical data was either not available or courts gathered data manually. Furthermore, information was not reliable, as there were also no conventions for ensuring data consistency or data integrity across the system in paper or electronic formats. The situation improved somewhat in 2010, from which time much of the M&E data was collected from the case management system used by Romanian courts and then checked for consistency by the DIEFP. Administrative data was also collected directly from the different stakeholders and complemented by periodic surveys. Reliability concerns remained, though data since 2014 is now reliable. Page 26 of 104 The World Bank Judicial Reform ( P090309 ) 91. Three years into project there was no baseline data in the results framework, and no documented attempts to obtain it. A baseline survey was conducted in 2008, two years after project effectiveness. Its existence later became an indicator, but the data contained therein was not used to set targets. Due to the delays in generating the baseline, by the time data was collected, several of the indicators had become irrelevant. Others were vague and proved a challenge to measure. In 2012, the results framework was revised and new, quantifiable, indicators were introduced where possible. The project outcome indicators were also modified to be quantifiable and have direct relevance to the project, while still retaining the essence of the original outcome indicators. Further changes to the results framework were made during the fourth restructuring in 2015, where three results indicators were dropped. 92. In retrospect, several of the targets set out in the Results Framework lacked ambition. For example, the results framework estimated a 10 percent increase in case disposal, which over more than 11 years seems modest. Similarly, the results framework estimated a 10 percent increase in court hearings in 5 courts rehabilitated under JRP. Why five courts were chosen is not clear. The actual increase across court sites was approximately 89 percent, which would have been known from the technical designs conducted early in the project cycle. These early mistakes became compounded, as the level of ambition of these targets was not increased with the successive project extensions even after the target had been achieved. M&E Utilization 93. Once the M&E system was finally established, data was provided regularly and in a timely manner to the Project Steering Committee PSC and the Bank. For its part, the Bank reported data routinely in the project ISRs. 94. However, the ICR team found no evidence to suggest that data was used, either strategically or operationally, by the Bank, the DIEFP or the MOJ to inform project implementation, policymaking or citizen outreach. For example, the three surveys offered insights (See Annex 7) that seem not to have been applied. With more careful M&E, the project might have demonstrated satisfactory achievement of both development objectives. Justification of Overall Rating of Quality of M&E Rating: Moderately Unsatisfactory 95. Throughout JRP’s preparation and early years of implementation, M&E arrangements were clearly an afterthought. Years passed without a monitoring system, definitions or baselines for several indicators. Mid-way through the project, the M&E system was finally established and operational. Thereafter, the DIEFP was diligent in ensuring timely reporting through to project closure. The project conducted three surveys over the life of the project, and these were of good quality. The DIEFP also provided valuable support during the ICR phase when requested. 96. The M&E arrangements for the JRP should not be judged in hindsight, as M&E frameworks generally (and the Bank’s expectations of them) have changed vastly over the last 11 years. Nonetheless, the project missed several opportunities to improve their M&E arrangements over time. While it is true that the Bank was more focused on outputs and outcomes back in 2005, this does not fully explain why baseline data was not collected to feed the original indicators for three years. It is also true that the Bank’s understanding of justice system performance measurement has improved significantly over the last decade, yet the project seems not to have reflected these innovations in the restructurings that were processed since then. The data environment in Romanian justice system was indeed very challenging in the early years of the project. However, more reliable data became available in 2010 and 2012, and was further enhanced in 2014, yet the results framework did not take advantage of these improvements. If these opportunities had been seized, a moderately satisfactory M&E rating may have been warranted. Page 27 of 104 The World Bank Judicial Reform ( P090309 ) 97. Overall, the ICR team was left with the impression that M&E was considered exclusively a compliance exercise throughout project implementation. Data was collected and sent to the Bank for inclusion in ISRs, but the information it produced did not penetrate the strategic or operational level in the Bank, the DIEFP or the MOJ. This is a pity, given JRP’s significant achievements. C. BANK PERFORMANCE Quality at Entry 98. The project was not ready at the time of effectiveness. Shortcomings in preparation are outlined above, but the most significant among them were the inadequate preparation of technical documentation for the civil works, the failure to adequately trigger the Bank’s cultural property safeguards, and the poor initial conception of the RMS. In addition, the PIU was inadequately staffed, and no provision was made for JRP in the State budget for its first year of implementation. Quality of Supervision 99. Bank performance varied over time and between different functions, as can be expected for a long and complex project. Overall, Bank performance was poor during preparation and the early years of implementation, when support was most needed. By mid-way though, Bank performance improved with more proactive monitoring. The Bank team conducted routine missions and reporting was generally timely. Throughout the project, the FM and procurement support was satisfactory. In particular, the team supported the DIEFP through the procurement processes, and the DIEFP appreciated the procurement advice provided by the Bank over many years which helped them to mitigate and manage risks. Bank support to M&E remained lackluster throughout the project. During preparation and the early years of implementation, Bank support to the ICT and infrastructure elements of the project were inadequate, but this improved mid-way through the project, and in the last years the Bank’s support to ICT was highly satisfactory and helped to get the RMS activity on track. Overall, the varied performance resulted in uneven implementation support over the life of the project. 100. The MTR was conducted in 2010, four years after project start. The task team did not heed the advice of the MTR to restructure the project by removing or amending the second project development objective relating to accountability. The reason for inactivity on this recommendation is unclear: doing so would have made JRP’s purpose and results chain clearer. 101. The introduction of the Action Plan and contingency mechanism at the 2010 restructuring was particularly useful. The JRP was among the first projects in ECA to include clear performance targets linked to disbursements. Without this, the project would likely have been cancelled in 2010 without having achieved any results. Although the contingency mechanism had little legal effect, it provided the leverage that the client needed to get activities moving. Stakeholders reported to the ICR team that the conditionality was not resisted by local stakeholders, rather welcomed as a way to overcome inactivity and help navigate Romanian bureaucracy. Combined with highly proactive supervision during this period, the Action Plan was effective in getting the project on track. 102. Considering the length of the project, a second MTR would have been advisable in 2013 or 2014. A second MTR could have provided an opportunity to reflect on the project’s successes and challenges, analyze the M&E situation, consult with stakeholders and set the strategic direction for the last phase of implementation. Instead, the project drifted to a closure, with the final years focusing on the completion of ICT and infrastructure activities. Page 28 of 104 The World Bank Judicial Reform ( P090309 ) Justification of Overall Rating of Bank Performance Rating: Moderately Satisfactory 103. JRP preparation had many shortcomings, most of which could have been foreseen. Preparation was rushed, and although the project team conducted a QER and incorporated the feedback provided, it was not nearly enough to ensure timeliness of implementation. Preparatory work for the civil works component had failed to take into consideration that some of the sites were historical and had to have special permits and designs. There were no international design standards developed at project start, which meant that new design standards had to be developed and the preparation for the sites had to be redone. There was little buy-in from users for the RMS, which was still not completed at project closure. No change management was foreseen for the beneficiaries for the RMS system. Indicators were prepared without taking into consideration that there was no available baseline data for some of them. Indeed, some of the indicators were later dropped as they proved impractical to measure and other obvious indicators added very late in the project. There was also a lack of ambition behind some of the targets. Only after introducing a contingency mechanism four years into the project did JRP start to disburse at a reasonable rate. Supervision of the FM, procurement and safeguards aspects of the project were adequate and appreciated by the DIEFP. D. RISK TO DEVELOPMENT OUTCOME 104. Risk to the development outcome is moderate. 105. The infrastructure upgrades provide permanent improvements to working conditions in 16 locations across 21 courts. The facilities are easier to maintain and more energy efficient, making them more sustainable in the long term. The MOJ has made adequate provision for operations and maintenance costs, which are significantly lower (approximately half) of what they were before the project. In Iasi, for instance, the maintenance costs for the old building were 400.000 RON, while for the new building, which is twice the size, the costs are 490,000 RON. 106. Structural reforms – such as streamlining of processes, equalizing workloads among judges, and delegating tasks to clerks – have made significant improvements. These are not one-shot reforms, and more work is needed, but indications from the CSM suggest that next-generation reforms will continue. 107. The RMS will be completed using State budget resources. The MOJ is now able to provide quality data to inform decision- making and guide reforms. Tools are now in place to monitor the functioning of the courts and human resource management. The HR and Payroll modules of the RMS for MOJ and CSM are embedded in the daily operations of the judicial system. However, no budget has been allocated to the migration of data from the old system to the new one requiring the insertion of tens of thousands of files into the new system. This has led to further delays. 108. The NSC and NIM curricula and the training provided under the project strengthened the skills to judges, prosecutors and their staff. Development of e-learning facilities for both the NSC and NIM had an important positive impact on the training process in both institutions. The online platform has been developed for both institutions and a methodology for course development has been taught to the core trainers of both the NIM and NSC and embedded into the work of these institutions. Enhanced design standards (the SCORM standards) for online courses have been provided on a sample base. Based on the initial assessments and interviews with representatives from both institutions, online courses are popular and effective tools that are used by a significant number of trainees across the justice sector. Over time however, the online platform is becoming slow and outdated, and does not allow the training centers to host more than three courses simultaneously. According to NIM, the necessary upgrade would cost around Page 29 of 104 The World Bank Judicial Reform ( P090309 ) 10,000 EUR but there is no state budget for it. The Government of Romania should carefully consider how to make the most of rapid global advancements in online learning technology which can build on their already strong base. 109. Based on the achievements of the JRP, the Bank and Borrower have started a new follow-on project, JSIP, which became effective in August 2017. The JSIP will focus on improving efficiency and transparency in the delivery of justice services in Romania. The new project continues several of the successful elements of the JRP, such as financing basic infrastructure in poor and under-served communities. It also applies some lessons from the JRP, such as focusing on simple ICT infrastructure, rather than complex software systems. The new project also scales up some of the JRP’s successes, including by focusing more on transparency initiatives and court user needs. The JSIP will be implemented by the DIEFP at the MOJ under similar implementation arrangements as the JRP. V. LESSONS AND RECOMMENDATIONS 110. There are several key lessons that can be learned from a long and complex project such as the JRP. 111. M&E should be a top priority for any project, not an afterthought to comply with Bank rules. Teams should continually review and explain to internal and external audiences how project interventions are expected to lead to results and impacts of importance. As part of the constant dialogue around the results chain, the project results framework should be reviewed and updated periodically, and at least during each restructuring to ensure that indicators are appropriately defined and measured and that the level of ambition of targets matches the project investment. This is particularly important for projects that are extended multiple times. Under the JRP, stakeholders never understood precisely how the project expected that civil works and IT activities would increase efficiency and/or accountability. ICR consultations revealed that the Bank did not discuss the results chain with stakeholders and stakeholders showed little interest. 112. Projects should give adequate attention to institutional reforms and should communicate progress and results. The Bank, the Borrower and the implementing agency focused most of their attention to the ICT and infrastructure activities, likely because they were the largest disbursement categories and the most problematic activities in terms of bureaucratic hurdles and delays. As a result, JRP was perceived by many as simply a financing vehicle for new courthouses and computers, as if that were an end in itself. If the project had solely built courthouses and bought computers, it would not have achieved its development objectives. Much of the achievement of the objectives derived from the structural reforms that the project supported, complemented by investments in ICT and infrastructure. A few key personnel spread across the DIEFP, the MOJ, MOPF, the beneficiary institutions and the Bank seemed to understand this, but it was not well communicated or documented throughout the sector or throughout the Bank. As a result, the JRP had many good stories that had gone untold. 113. Projects with complex IT and infrastructure investments should have a fully operational project implementation unit in place prior to, or soon after, effectiveness. In the case of the JRP, a full DIEFP was not in place at effectiveness, and took years to develop a full complement of experienced staff. As a result, at critical times in the early phases of preparation and implementation, there were no internal qualified technical staff able to prepare relevant documentation or support decision makers in Government. Furthermore, a capable PIU, with the authority to take and implement decisions, is critical to project success. 114. The simplest ICT solutions should be prioritized, wherever possible. In the case of JRP, the RMS was overdesigned and overambitious. Its complexity contributed to costs, delays and limited stakeholder Page 30 of 104 The World Bank Judicial Reform ( P090309 ) support. Teams should be especially wary to incorporate a complex ICT project inside an already-complex institutional reform project. 115. It is essential for a project to ensure sufficient State budgetary allocation. This involves realistic planning, proactive monitoring and follow-up on budget allocations, as well as close and timely coordination between the borrower, the Bank and the implementing agency in case of insufficient funds allocation. In Romania, the MoPF leads the annual budgetary allocation process, based on the forecasts provided by the line Ministries and ceilings and fiscal targets established at macroeconomic level. Experience shows that the allocated project budget is rather decided on historical basis and is likely to be established at the level of the budgetary execution in the previous year. For the years 2012 to 2015, budgetary allocations were inadequate, which caused implementation delays, frustration, complexity and, ultimately necessitated successive project extensions. In order to ensure that sufficient funds are available for project activities on a timely basis, the implementing Ministry should be proactive in discussing financing needs with the MoPF and ensuring that their budgetary requests are realistic and closely aligned to the project procurement and implementation plan. Furthermore, the Bank could agree with the Borrower certain milestones in project implementation and specific financing allocations for their achievement. 116. Slow-moving projects should consider introducing Action Plans with contingency mechanisms. Prior to the introduction of the Action Plan, JRP was at a standstill and at risk of being cancelled. After the introduction of the Action Plan, implementation progress accelerated and the project started to disburse at a reasonable rate. The contingency mechanism helped focus attention on key milestones for implementation and specified consequences if these milestones were not met. The relevant stakeholders – the Bank, the client, the DIEFP, and the beneficiary institutions – each benefited from the Action Plan using the contingency mechanism to focus the attention of stakeholders. This alignment of interests provided the leverage needed to pull the project out of problem status. . Page 31 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS 1. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: The project's development objectives are: (i) to increase the efficiency of the Romanian courts; and (ii) to improve the accountability of the judiciary which should result in reduced corruption and a Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvements to judicial Text 11,224 cases in 5 18.52% (cumulative Improved efficiency 6% (cumulative data – efficiency in pilot courts randomly chosen 5 data - 2015 compared measured by 10% 2016 compared to pilot courts to 2008 baseline). increase in the 2008 baseline). number of cases disposed of or archived in pilot courts. 31-Dec-2008 02-Jun-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Page 32 of 104 The World Bank Judicial Reform ( P090309 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Audio recordings of court Percentage 0.00 100.00 100.00 100.00 hearings available to any trial participant on request 05-Oct-2010 02-Jun-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Integrated audio recording systems for court hearings are operational in all national courts (243) and their courtrooms (699). Audio recordings of court hearings are provided on request to any interested trial participant. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Rehabilitation/construction Text 0 court buildings 16 courthouses are in Met international 16 courthouses are in of at least 15 courthouses in line with international standards in the line with international line with international standards. rehabilitation/ standards. standards (standards for construction of at better accommodations for least 15 court trials, public access and buildings completed. security features). 31-Dec-2008 02-Jun-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Information required to Text RMS not in place. HCCJ, Public Ministry RMS system installed HCCJ, Public Ministry increase the capacity of the and SCM (HR & Payroll and functioning; and and SCM (HR & Payroll judicial system available module) technically data published module) technically Page 33 of 104 The World Bank Judicial Reform ( P090309 ) through establishment of a ready to go-live. quarterly for the past ready to go-live. functioning RMS and such year (since 2012). Implementation information/data published ongoing for MOJ.RMS quarterly by the MOJ, SCM implementation will and HCCJ on thei continue with support from State budget funds beyond the closing date of the project. None of the institutions publish on their websites resource management data originating from the RMS. 31-Dec-2007 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvements to judicial Text No publications of (a) The baseline survey Regular The baseline survey transparency through actual court was finalized. Another annual/quarterly was finalized. Another [annual/quarterly] performance data, or survey related to the publication of survey related to the publication on the websites (b) data from periodic activity of all courts available data in the activity of all courts of the respective courts and user surveys was completed for the past two years (2011 was completed for the SCM of: (a) court conducted, are impact Studies of the & 2012). Impact Studies of the performance data; (b) data available to be four new codes. The four new codes. The from periodic user surveys of published on the MoJ published the MoJ published the websites of the impact studies on its impact studies on its Page 34 of 104 The World Bank Judicial Reform ( P090309 ) ind respective courts and website in January website in January the SCM. 2012. The 2012. The intermediate survey intermediate survey was finalized. The final was finalized. The final survey is under survey is under implementation. implementation. The contract will be extended for two months beyond the Project closing date, because approval by the SCM Plenum is late. 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Court Infrastructure Rehabilitation: (Cost $85.00 M) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Rehabilitation/construction Text 0 renovated court 18.52% (cumulative At least 15 buildings 16 courthouses of at least 15 court buildings buildings. data - 2015 compared completed. finalized. Civil works to 2008 baseline). are ongoing in 3 courthouses to be finalized after the closing date of the Page 35 of 104 The World Bank Judicial Reform ( P090309 ) current Project and with support of the new Justice Services Improvement Project and State budget. 15-Mar-2006 02-Jun-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): The decreased percentage is caused by the fact that in 2016 the work volume (number of files registered during one year) of Orsava First Instance Court (one of the pilot courts dropped significantly - in 2015 the total number of cases was 3584 and in 2016 was 1810. Out of the 1810 cases registered in 2016, 1500 cases were disposed in the respective year (83%). For the other remaining four pilot courts, the volume of activity stayed within the previous levels. It should be noted that for eight consecutive years, starting with 2008 until 2015, an increase of at least 16% in the number of cases disposed in the five pilot courts was registered. Component: Strengthening of the Administrative Capacity of Courts: (Cost $11.05 M) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Uniform standards for Text Realistic The implementation Uniformity in The implementation operational processes time/workload of the indicators operational (case time of the indicators developed and country-wide standards (based on started with 11 pilot handling and optimal started with 11 pilot implementation started EU/International courts. It is an ongoing workload) standards courts. It is an ongoing standards) for process, measures for (for handling court process, measures for handling the cases in its achievement were cases) achieved. its achievement were courts, were not included also in the included also into the developed as yet. Judicial Reform Judicial Reform Strategy for 2015- Strategy for 2015- 2020 (Government 2020 (Government Decision no. Decision no. 1155/2014) and the 1155/2014) and the Page 36 of 104 The World Bank Judicial Reform ( P090309 ) Action Plan for the Action Plan for the implementation of the implementation of the strategy (Government strategy (Government Decision no. Decision no. 282/2016). 282/2016). 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Component: Court Information System: (Cost $21.50 M) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion At least 10 % increase in the Text 277 trainees using the 335 % increase (1205 10 % increase (304 339 % increase (1215 number of trainees using the NIM and NSC e- trainees). trainees) trainees) NIM and NSC e-learning learning platform in platform 2012. 31-Dec-2012 14-Dec-2016 31-Mar-2015 16-Mar-2017 Comments (achievements against targets): Component: Institutional Development of Judicial Institutions: (Cost $6.61 M) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Uniform standards for Text Realistic The implementation Uniformity in The implementation operational processes time/workload of the indicators operational (case time of the indicators Page 37 of 104 The World Bank Judicial Reform ( P090309 ) developed and country-wide standards (based on started with 11 pilot handling and optimal started with 11 pilot implementation started EU/International courts. It is an ongoing workload) standards courts. It is an ongoing standards) for process, measures for (for handling court process, measures for handling the cases in its achievement were cases) achieved. its achievement were courts, were not included also in the included also into the developed as yet. Judicial Reform Judicial Reform Strategy for 2015- Strategy for 2015- 2020 (Government 2020 (Government Decision no. Decision no. 1155/2014) and the 1155/2014) and the Action Plan for the Action Plan for the implementation of the implementation of the strategy (Government strategy (Government Decision no. Decision no. 282/2016). 282/2016). 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Unlinked Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion New design standards used Text No standards were All courts rehabilitated New design standards All courts rehabilitated in rehabilitation and developed. and constructed under used for the court and constructed under construction of court the Project are based buildings constructed the Project are based buildings on designs in line with in the last two years on designs in line with international (2011 & 2012). international standards, 16 standards. 16 Page 38 of 104 The World Bank Judicial Reform ( P090309 ) courthouses are courthouses are finalized. finalized. 3 remain under implementation beyond the Project closing date. 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 10% increase in the total Text Baseline established 99.72 % increase At least a 10 % 77.25 % increase number of court hearings in with 2008 data. (cumulative data - increase (cumulative data - 5 courts rehabilitated under 2015 compared to 2016 compared to the project 2008 baseline). 2008 baseline) 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Recommendations to Text No recommendations Recommendations Recommendations Recommendations optimize courts' activity are in place to optimize were adopted by the adopted by SCM. were adopted by the adopted by the SCM court activity. SCM on March 28, SCM on March 28, 2013 (SCM Decision 2013 (SCM Decision no. 392 for the no.392 for the Page 39 of 104 The World Bank Judicial Reform ( P090309 ) approval of the Final approval of the Final Report). Report). 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion An RMS for core justice Text RMS not in place. HCCJ, Public Ministry Sustained use of the HCCJ, Public Ministry sector entities enables and SCM (HR & Payroll RMS. and SCM (HR & Payroll management of HR and module technically module) technically payroll functions ready to go-live. MOJ ready to go-live. MOJ (central apparatus, (central apparatus, Bucharest Court of Bucharest Court of Appeal and subordinated courts) is in the process of preparing data for migration for HR and Payroll. 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 40 of 104 The World Bank Judicial Reform ( P090309 ) An RMS for core justice Text No RMS HCCJ, Public Ministry Sustained use of the HCCJ, Public Ministry sector entities enables pilot and SCM (HR & Payroll RMS and SCM (HR & Payroll justice sector entities to module technically module) technically collect management ready to go-live. MOJ ready to go-live. MOJ information data (central apparatus, (central apparatus, Bucharest Court of Bucharest Court of Appeal and Appeal and subordinated courts) subordinated courts) is in the process of is in the process of preparing data for preparing data for migration for HR and migration for HR and Payroll. Payroll. 01-Jan-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion The Law School Admission Text No LSAT in place Magistrates' A formal regulation Magistrates’ Test (LSAT) is formally recruitment regulation regarding the recruitment regulation embedded into magistrates# was modified by SCM continuous use of was modified by SCM recruitment procedures decision no. LSAT test for decision no 869/2009, 869/2009, formally magistrates formally embedding embedding LSAT. requirement adopted LSAT. by the NIM. 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Page 41 of 104 The World Bank Judicial Reform ( P090309 ) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Conduct of periodic internal Text No surveys conducted. The baseline survey 3 surveys conducted The baseline survey and external surveys of was finalized. Another and results was finalized. Another individuals and firms and survey related to the disseminated. survey related to the publication of results activity of all courts activity of all courts was completed for the was completed for the impact Studies of the Impact Studies of the four new codes. The four new codes. The intermediate survey intermediate survey was finalized. The final was finalized. The final survey is under survey is under implementation. implementation. The contract will be extended for two months beyond the Project closing date to allow for SCM Plenum approval of final survey report. 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 42 of 104 The World Bank Judicial Reform ( P090309 ) New/amended Criminal Text Codes not in place. All Codes adopted in All 4 Codes adodpted All Codes adopted in Procedure Code and Civil the Parliament and in the Parliament and the Parliament and Procedure Code adopted by enforced. enforced. enforced. Parliament and came into force 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Impact assessment of four Text No impact assessment Impact assessment TA that would provide Impact assessment Codes (Civil Code, Civil conducted. studies finalized on impact assessment of studies finalized on Procedure Code, Criminal November 30, 2011. 4 Codes under November 30, 2011. Code and Criminal Procedure implementation. Code) completed and accepted by SCM and MOJ. 15-Mar-2006 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion At least 50 % increase of the Text 7 128% increase (16 50 % increase (11 128 % increase (16 number of council rooms in council rooms). council rooms) council rooms) 10 courts rehabilitated under this project 31-Dec-2010 14-Dec-2016 31-Mar-2017 16-Mar-2017 Comments (achievements against targets): Page 43 of 104 The World Bank Judicial Reform ( P090309 ) Page 44 of 104 The World Bank Judicial Reform ( P090309 ) A. KEY OUTPUTS BY COMPONENT Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Outputs 2010 – 2011 2011 – 2012 2012 – 2015 2015 – 2017 2006 – 2010 Components Components Components Components Components prior to post post post post restructure 1 restructure 1 restructure 2 restructure 3 restructure 4 D=Dropped C=Continued N=New R=Revised Component 1: Court Infrastructure Rehabilitation (a) Rehabilitation and limited R C C C • Rehabilitation and (re)construction new construction of works in 16 court buildings: FIC prioritized court buildings (25 (a) Bolintin Vale, FIC Blaj, T Maramures, buildings of which 5 were new Rehabilitation T Ialomita, CA Craiova, T Dolj, FIC constructions)3 construction of Orsova, T Arges, CA Pitesti, T Tulcea, at least 15 FIC Saliste, FIC Viseul de Sus, T court buildings Suceava, FIC Vatra Dornei, PJ Iasi, PJ rehabilitated Oradea, T Cluj. Work is still ongoing (b) Development of uniform C C C C on: T Prahova, PJ Prahova, and T space planning and design Sibiu standards for court facilities • Manual with new design standards for all court buildings • TA for Independent Technical Monitoring • Assistance for Site Supervision for Construction Works • Furniture for the court buildings • Technical designs • Site supervision for civil works 3 The Loan Agreement of the Project initially refers to 25 civil works projects. However, five of these were already canceled during the four-year implementation of the project. Page 45 of 104 The World Bank Judicial Reform ( P090309 ) • Independent Technical Monitoring/Review Component 2: Strengthening of the Administrative Capacity of Courts (a) Development and C C C C • TA to assess the impact of the implementation of a program implementation of the four new to reduce case delays and codes backlogs • TA to revise Civil and Criminal Code (b) Development of a C C C C Procedure Code framework for economic • Determining and implementing the management of the courts, optimal volume of work of judges including regulatory and and court clerks and ensuring the organizational arrangements quality of the courts’ activity for economic managers and training (c) Optimization of courts’ C C C C operational processes, including transfer of non- adjudicative tasks from judges to court personnel, and upgrading the functioning of case registries, archives, recording of court proceedings, court statistics and case monitoring (d) Public Education / D ---- ---- ---- ---- Information Component 3: Court Information System (a) Development of a C C C C • High Level System Analysis for RMS comprehensive RMS for the judicial system Page 46 of 104 The World Bank Judicial Reform ( P090309 ) (b) Upgrading the archive D ---- ---- ---- • Integrated RMS – supply & system for Courts installation of IS • Audio Court Recording System • Encrypting equipment for RMS • RMS Data Migration Services Component 4: Institutional Development of Judicial Institutions (a) Assistance to CSM – in the C C C • Provision of distance learning area of development of long- facilities for the NIM and the NSC term judicial policies, • TA for development of LSAT testing monitoring judicial for NIM performance, and public • Revision of curricula for continuous communications education at NSC • Simultaneous translation equipment for NSC • Baseline public opinion survey on judicial performance • Development of Teaching Techniques for NIM • Management software for DIEFP • Software for electronical processing of the NIM • Translation and publication in Romanian language of all 27 EU Member States’ Constitutions • Project audit for the period for 2015 - March 2017 • Support to the CSM - PR Communications strategy development Page 47 of 104 The World Bank Judicial Reform ( P090309 ) • Support to the MOJ - Internal and external communication • Support to the MOJ – Development of quality management system • Project audit for the period May 2006 - 2009 • Project audit for the period 2010 - March 2013 • Support for strengthening the Insolvency mechanism (insolvency code) • LSAT - Phase II • LSAT - Phase III • Support to the NIM – Development of teaching techniques • Revision of NSC’s continuous training activity • Technical assistance – syllabus drafting for NSC curricula and training in 4 new areas • Technical Assistance for the Development of the Network of Trainers Teaching Techniques in Long Distance Training (E-Learning) within NSC • Development of continuous on line training of the court personnel • Technical Assistance for the Development of the Network of Trainers Teaching Techniques in Page 48 of 104 The World Bank Judicial Reform ( P090309 ) Long Distance Training (E-Learning) within NIM • Baseline Survey • Support to the CSM – Development of quality management system • Intermediate Survey (mid project) • Project audit for the period for 2013 – 2014 • Support to the MOJ - Training for DIEFP staff • Final Survey REVISIONS OF THE RESULTS MATRIX Results Indicator Baseline (2008) Actual End of Project Target Achievement Project development objective indicators Improvements to judicial efficiency in 11,224 cases in 5 18.52% Improved Fully achieved pilot courts * randomly chosen (cumulative data – Efficiency measured by 10% pilot courts 2015 compared to increase in the number of cases 2008 baseline) disposed of or archived in pilot courts Rehabilitation/construction of at least Met Fully achieved 15 courthouses in line with 16 (additionally, international international standards4 (standards for 0 works are ongoing standards in the better accommodations for trials, on 3 other courts) public access and security features) * 4International standards are based on a comprehensive Manual ( Manual of Physical Design Standards for Romanian Court Buildings, May 10, 2006) created by the Bank team at the early stages of the Project. Page 49 of 104 The World Bank Judicial Reform ( P090309 ) rehabilitation/construction of at least 15 court buildings completed Information required to increase the RMS not in place HCCJ, Public RMS system installed and Not achieved but capacity of the judicial system available Ministry and CSM functioning and data published substantially met through the establishment of a (HR & Payroll quarterly for the past year functioning RMS and such module) (since information/data published quarterly technically ready 2012) by the MOJ, CSM and HCCJ on their to go-live websites * Audio recordings of court hearings 0 100 100 Fully achieved available to any trial participant on request (Percentage) *** Aggregate case backlogs for Courts Methodology and The baseline Regular annual/quarterly Fully achieved improvements to judicial transparency baselines under survey was publication of available data in through the [annual/quarterly] development. finalized. Another the past two years (2011 & publication on the websites of the One user survey survey related to 2012) respective courts and CSM of: (a) conducted the activity of all actual court performance data; (b) courts was data from periodic user surveys of completed for the individuals and firms on access to and Impact Studies of satisfaction with judicial services the four new (including fairness and timeliness) for codes. The MOJ courts financed by Part A of the published the project* impact studies on its website in January 2012. The intermediate survey was finalized. The final survey is under implementation. Page 50 of 104 The World Bank Judicial Reform ( P090309 ) Intermediate results indicator – Component 1 Court Infrastructure Rehabilitation (a) renovated court buildings with better accommodations for trials, public access and security features; (b) new design standards for court buildings reflecting best international practices and cost efficiency principles Rehabilitation/construction of at least 0 16 At least 15 buildings completed Fully achieved 15 court buildings rehabilitated 10% decrease of average time required N/A At least a 10% decrease. Dropped to dispose of cases in 5 courts rehabilitated under this project*** New design standards used in No standards All courts New design standards used Fully achieved rehabilitation and construction of were developed rehabilitated and for the court buildings court buildings constructed under constructed in the last two the Project are years (2011 & 2012) based on designs in line with international standards. 16 courthouses are finalized. 10% increase in a total number of Baseline 99.72% increase At least a 10% increase. Fully achieved court hearings in 5 courts rehabilitated established with under the project 2008 data. Intermediate results indicator – Component 2 Strengthening of the administrative capacity of courts: (a) improved case management processes; (b) effective integration of economic managers within courts’ structure; (c) new internal working arrangements within courts whereby non judicial tasks are transferred from judges to court clerks; (d) public communication strategy is adopted by CSM Recommendations to optimize courts’ No Recommendations Recommendations adopted by Fully achieved activity are adopted by the CSM recommendations were adopted by CSM. in place to the CSM on March optimize court 28, 2013 (CSM activity Decision no.392 for the approval of the Final Report). Page 51 of 104 The World Bank Judicial Reform ( P090309 ) Uniform standards for operational Realistic The Uniformity in operational (case Not achieved but processes developed and country wide time/workload implementation of time handling and optimal substantially met implementation started standards (based the indicators workload) standards (for on started with 11 handling court cases) achieved EU/International pilot courts. It is standards) for an ongoing handling the cases process, measures in courts, were for its not developed as achievement were yet. included also into the Judicial Reform Strategy for 2015-2020 (Government Decision no. 1155/2014) and the Action Plan for the implementation of the strategy (Government Decision no. 282/2016). New/amended Criminal Procedure All codes are in All Codes adopted All 4 Codes adopted in the Fully achieved Code and Civil Procedure Code place in the Parliament Parliament and enforced adopted by Parliament and come into and enforced force Impact assessment of four Codes (Civil No impact Impact TA that would provide impact Fully achieved Code, Civil Procedure Code, Criminal assessment assessment assessment of 4 Codes under Code and Criminal Procedure Code) conducted. studies finalized implementation completed and accepted by CSM and on November 30, MOJ 2011 Page 52 of 104 The World Bank Judicial Reform ( P090309 ) Economic managers employed dropped dropped dropped Dropped through competitive process in courts of all levels) ** About 50% of economic managers dropped dropped dropped Dropped trained) ** About 50% of economic managers with dropped dropped dropped Dropped satisfactory performance ratings) ** Public communication strategy is dropped dropped dropped Dropped adopted and implemented) ** Page 53 of 104 The World Bank Judicial Reform ( P090309 ) Intermediate results indicator – Component 3 Integrated RMS for the Judiciary: (a) development of RMS for courts; (b) upgraded archive system for Courts An RMS for core justice sector entities RMS not in HCCJ, Public Sustained use of the Not achieved but substantially (MOJ, CSM, courts, Public Ministry, High place Ministry and RMS met Court of Cassation and Justice) enables CSM (HR & pilot justice sector entities to collect data Payroll module) on budget execution, financial assets and technically liabilities, human resources, and ready to go-live. performance (MIS) according to legal MOJ (central provisions and internal regulations**** apparatus, Bucharest Court of Appeal and subordinated courts) is in the process of preparing data for migration for HR and Payroll. No. of key offices at the national level (MOJ, CSM, HCCJ) with an integrated view of up-to-date and accurate information regarding resource use, dropped dropped dropped Dropped resource availability, and court operations performance (in the aggregate as well as court-specific)) ** About 50% of court records stored in fire and waterproof dropped dropped dropped Dropped devices) ** Intermediate results indicator – Component 4 Institutional development of judicial institutions: (a) organizational structure of CSM is completed and fully functional; (b) MOJ’s capacity in capital investment planning is improved; (c) new tests and qualification exams procedures are used by NIM; Page 54 of 104 The World Bank Judicial Reform ( P090309 ) (d) national School of Clerks’ capacity is enhanced. CSM effectively assumes functions of CSM assumed CSM effectively Dropped budget planning, policy formulation and the function of assumes all the performance monitoring Policy indicated functions; formulation And performance monitoring but not budget planning The LSAT is formally embedded into No LSAT in Magistrates’ A formal regulation Fully achieved magistrates’ recruitment procedures place recruitment regarding the regulation was continuous use of LSAT modified by test for magistrates CSM decision no requirement adopted 869/2009, by the NIM. formally embedding LSAT. Conduct of periodic internal (within the No surveys The baseline 3 surveys conducted Fully achieved judicial system) and external surveys of conducted survey was and results individuals and firms and publication of finalized. disseminated results Another survey related to the activity of all courts was completed for the Impact Studies of the four new codes. The Page 55 of 104 The World Bank Judicial Reform ( P090309 ) intermediate survey was finalized. The final survey is under implementation. NIM and NSC design and apply web- No quarterly 3 quarterly Dropped based user feedback survey for all reports reports courses and publish quarterly results published published based on user feedback New strategy for NSCC is adopted and dropped dropped dropped Dropped implemented) ** 30% increase in training delivery by dropped dropped dropped Dropped NSCC) ** 80% of positive evaluation of NSCC dropped dropped dropped Dropped training by court presidents) ** Indicators revised following the 2010 restructuring ** Indicator dropped following the 2010 restructuring *** Indicator created following the 2015 restructuring **** Indicators revised following the 2015 restructuring Page 56 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Irina Kichigina Task Team Leader Amitabha Mukherjee Task Team Leader Nurul Alam Senior Procurement Specialist Frances Allen Communications Officer Laurentiu Anton David Bernstein Lead Public Sector Specialist Genoveva Bolea Bogdan Constantin Constantinescu Senior Financial Management Specialist Ruxandra Costache Senior Counsel Vladislav Krasikov Mirela Mart Craig Neal ICT Consultant Eric Peterson Architect Dorothea Abeline Reiling Senior Counsel Siew Chai Ting Lead Finance Officer Supervision/ICR Klaus Decker Task Team Leader(s) Elena Corman Procurement Specialist(s) Anneliese Viorela Voinea Financial Management Specialist Luisita I. Guanlao Lead ICT Specialist Valerie Morrica Safeguards Specialist Cesar Niculescu Safeguards Specialist Page 57 of 104 The World Bank Judicial Reform ( P090309 ) Victor Neagu Communications Officer Andreea Silvia Florescu Program Assistant Gabriela Grinsteins Counsel Irina Kichigina Task Team Leader Amitabha Mukherjee Task Team Leader Migara De Silva Task Team Leader Nurul Alam Senior Procurement Specialist Frances Allen Communications Officer Kosuke Anan Senior Social Development Specialist Janis Bernstein Senior Social Development Specialist Natasha Bhandari Junior Professional Associate Genoveva Bolea Bogdan Constantin Constantinescu Senior Financial Management Specialist Ruxandra Costache Senior Counsel Claudiu Danciu Jonas Arp Fallov Senior Public Sector Specialist Gabriela Beatrice Gabor Maurizio Gallerini IT Elena Georgieva Andonovska Public Sector Specialist Claire Louise Greer Operations Officer Hans Jurgen Gruss Chief Counsel Asli Gurkan Senior Social Development Specialist Haik Hakobyan Yoko Kagawa Senior Public Sector Specialist Vladislav Krasikov Waleed Haider Malik Senior Public Sector Specialist Gabriela Doina Manea Resource Management Specialist Danijel Marasovic Infrastructure Consultant Craig Neal ICT Consultant Page 58 of 104 The World Bank Judicial Reform ( P090309 ) Valeria Nikolaeva Procurement Specialist Rajesh Pandey Eric Peterson Architect Denisa Popescu IT Mihai Preda Driver Ismail Radwan Lead Public Sector Specialist Steven Reichenbach Consultant Milena Sanchez de Boado Justice Consultant Ella Shagabutdinova Charles Simpson III Olga Sipka Justice Reform Consultant Ramesh Sivapathasundram Lead Information Specialist Ireneusz Smolewski Procurement Costel Dorin Todor Zhanybek Ybraiym Uulu Public Sector Specialist Andrew Vorkink Virginia Yates Program Assistant Barbara Ziolkowska Procurement Specialist ICR Georgia Harley Public Sector Specialist Agnes Said Justice Reform Consultant Elena Gabriela Ardelean Operations Analyst NB: the first table below is pulling inaccurate data based on the new ICR template. As agreed with OPCS, please skip and refer to the next table with the accurate data. 1. STAFF TIME AND COST Page 59 of 104 The World Bank Judicial Reform ( P090309 ) Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation Total 0.00 0.00 Supervision/ICR FY10 0 362,930.50 FY11 0 151,541.21 FY12 0 85,225.41 FY13 0 178,725.65 FY14 18.964 111,487.67 FY15 23.405 149,201.76 FY16 17.088 98,614.17 FY17 19.625 91,593.28 FY18 7.403 41,599.79 Total 86.49 1,270,919.44 STAFF TIME AND COST Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including travel No. of staff weeks and consultant costs) Lending FY 2005 6.5 55.088 FY 2006 24.094 139.553 Total: 30.594 194.641 Supervision FY 2007 8.505 75.749 FY 2008 12.346 91.071 FY 2009 18.383 117.856 FY 2010 46.604 363.670 FY 2011 27.809 155.908 FY 2012 14.56 88.837 FY 2013 23.343 181.017 FY 2014 18.964 110.867 Page 60 of 104 The World Bank Judicial Reform ( P090309 ) FY 2015 24.08 164.131 FY 2016 15.888 98.630 FY 2017 14.542 81.159 Total: 222.024 1,528.900 ICR FY 2017 4.083 22,032.85 Total: 4.083 22,032,85 ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Court Infrastructure 85.0 85.00 100.0 Rehabilitation Strengthening of the Administrative Capacity of 11.05 11.05 100.0 Courts Integrated Resource Management System for the 21.5 21.50 100.0 Judiciary Institutional Development of 6.61 6.61 100.0 Judicial Institutions Total Baseline Cost 124.16 124.16 0.00 Physical Contingencies 5.00 5.00 100.0 Price Contingencies 0.00 0.00 0.00 Total Project Cost 129.16 129.16 100.0 Page 61 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 4. EFFICIENCY ANALYSIS This ex-post economic analysis of JRP considers the direct benefits and costs associated with changes in economic welfare resulting from project implementation. JRP generated several economic benefits; however, quantifying these depends on the availability and reliability of data. This economic analysis provides estimates of benefits and costs using data obtained from the MOJ, DIEFP, the Romania National Institute of Statistics and the IMF World Economic Outlook. The following areas were subjected through economic and financial analysis: a. Time savings from more efficient use of time of judges, court staff and lawyers enabled by various project interventions. b. Improved human and financial resources management as a result of the introduction of the Resource Management System (RMS). As a result of these two quantifiable elements, the ENPV of the total project amounts to EUR 47.4 million at a 5 percent discount rate with an EIRR of 71.2 percent. Benefits and costs are estimated for the period from year one of project implementation until current year (i.e. from 2006 until 2017), spanning a 12-year investment horizon. The horizon of 12 years is usually seen in similar projects in the sector. In our case, increasing the number of years to include in the analysis would add to the efficiency of the project given that economic benefits are expected from the RMS implementation in the future. Estimated economic costs include: the cost of investment (the amount of loan used as a proxy plus associated fees)5 staff time expenses for project implementation and O&M costs. Detailed rationale for the estimation of all categories of economic benefits is outlined below: 1- Time savings from more efficient use of time of judges, court staff and lawyers enabled by various project interventions Project implementation enabled judges, court staff, and lawyers to spend time more efficiently, which was possible through improved working conditions, simplified and more efficient legislative framework, improved performance management, and streamlined business processes. The analysis assumes that time that was left at the disposal of judges, staff and lawyers was otherwise used efficiently (for example, clearing more cases, devoting adequate time to more complex cases, reducing backlogs and professional development education.). This benefit was facilitated through key project interventions that included improvement of courthouse infrastructure, overhauling the legal codes, introduction of a system for court recording, workload optimization, training activities and introduction of a case flow management guide. Figure 2 illustrates distribution of project funds across interventions. 5 Loan repayments are not factored in because the grace period extends past the last year of investment horizon Page 62 of 104 The World Bank Judicial Reform ( P090309 ) The analysis splits the estimates regarding time savings of judges, court staff and lawyers according to: a) the year where benefits from these actions became effective and; b) whether specific interventions affected all judges, court staff and lawyers across the court network in Romania, or only the ones working on the territory of 16 courthouses where the civil works were done. The final estimate is the effect of civil works in the 16 courthouses is 0.33 hours (i.e. 20 minutes) daily saved of the time of 494 judges working in the 16 courts, 0.33 hours daily for each of the 1,200 court staff and finally 0.17 hours (i.e. 10 minutes) daily for each of the estimated 5,415 lawyers working on the territory under the authority of 16 courts. The difference in years when civil works in different courthouses were finished was accounted for. Introduction of court recording affected the entire court system and the estimated time saved for judges as a result of this intervention is 0.33 hours (i.e. 20 minutes) for judges and 0.50 hours (i.e. 30 minutes) for court staff. This system was effective from 2013 onwards. Finally, the other interventions including code overhauling, training, workload optimization and introduction of the CFMG affected the whole system as well and their cumulative effect is estimated to be 0.25 hours (i.e. 15 minutes) saved both in case of judges and for court staff. The value of the working hour of judges, court staff and lawyers was calculated using the average monthly gross salary of judges of EUR 1,800 per month and the average gross salary of court staff of EUR 1,000 per month assuming 8 hours working day and 22 days of work per month. Lawyers’ monthly compensation is assumed to be on average equal to judges. It is important to note that our estimates are at the lower bound of the expected range of time savings for each category. This conservative approach was taken given that it was not possible, beyond qualitative ICR interviews, to determine precisely the extent to which judges, staff members and lawyer was affected by the project. 2- Improved human and financial resources management as a result of the introduction of the RMS. This is a significant project activity and its final costs were estimated at EUR 21 million which is 20.5 percent of total loan value. The system is intended to replace approximately 160 existing legacy applications. It includes modules for human resources and payroll managing around 90 percent of the judiciary budget, financial module to manage non-wage expenses of the system, budget module used for budget formulation, allocations, tracking and execution and finally a special sub-system for the NAP. However, the RMS is not complete, so its full benefits cannot be assumed. Only the HR and payroll module of the RMS has been implemented, starting from 2016 and the roll out of it is still ongoing. Other modules are still under development. The analysis accounts for this by assuming benefits only for this small portion Page 63 of 104 The World Bank Judicial Reform ( P090309 ) of RMS and only on a limited scale. The economic benefits arising from RMS implementation are estimated at only 0.1 percent of the total judiciary budget in 2016 and 2017 while the real benefits from full implementation of the system are yet to come. Table 1 below shows the year by year structure of economic costs and benefits for the life of the project. Page 64 of 104 The World Bank Judicial Reform ( P090309 ) 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Economic outflows Project investment 500,000 251,140 578,871 3,403,097 7,310,080 18,303,143 12,473,804 15,538,251 13,996,540 12,490,631 12,514,401 5,164,339 Staff time expense for project implementation 16,267 16,674 17,091 17,518 17,956 18,405 18,865 19,336 19,820 20,315 20,823 21,344 O&M cost 0 0 0 0 0 74,982 76,857 118,167 161,496 165,533 212,089 217,391 Front end fee 256,311 Commitment fee 255,061 254,433 252,986 244,478 226,203 180,445 149,260 110,415 75,423 44,197 12,911 0 Total economic costs 1,027,639 522,246 848,947 3,665,093 7,554,238 18,576,975 12,718,786 15,786,169 14,253,278 12,720,676 12,760,224 5,403,075 Economic inflows Time efficiency gains (judges) 0 0 0 0 94,109 192,923 2,208,515 2,304,266 5,199,400 5,350,680 5,528,102 5,666,304 Time efficiency gains (staff) 0 0 0 0 114,636 235,004 2,690,243 2,806,879 8,049,083 8,276,250 8,536,333 8,749,742 Time efficiency gains (lawyers) 0 0 0 0 243,636 499,453 511,939 629,685 699,213 771,824 904,136 926,740 Improved human and financial resources management 0 0 0 0 0 0 0 0 0 0 3,402,604 3,131,273 Loan disbursements 500,000 251,140 578,871 3,403,097 7,310,080 18,303,143 12,473,804 15,538,251 13,996,540 12,490,631 12,514,401 5,164,339 Total economic benefits 500,000 251,140 578,871 3,403,097 7,762,461 19,230,523 17,884,500 21,279,081 27,944,235 26,889,384 30,885,576 23,638,398 Net benefits -527,639 -271,107 -270,076 -261,996 208,222 653,548 5,165,715 5,492,911 13,690,956 14,168,708 18,125,352 18,235,323 Total economic costs 1,027,639 522,246 848,947 3,665,093 7,554,238 18,576,975 12,718,786 15,786,169 14,253,278 12,720,676 12,760,224 5,403,075 Total economic benefits 500,000 251,140 578,871 3,403,097 7,762,461 19,230,523 17,884,500 21,279,081 27,944,235 26,889,384 30,885,576 23,638,398 Net benefits -527,639 -271,107 -270,076 -261,996 208,222 653,548 5,165,715 5,492,911 13,690,956 14,168,708 18,125,352 18,235,323 discounted economic costs 75,781,185 discounted economic benefits 123,154,946 NPV at 5% 47,373,761 EIRR 71.2% Page 65 of 104 The World Bank Judicial Reform ( P090309 ) 3- Other Qualitative Benefits Several benefits have not been quantified in this analysis, for example, the benefits brought through an enhanced legal environment achieved through higher legal clarity and certainty as a result of redesigning the Civil Code, Civil Procedure Code, Criminal Code, Criminal Procedure Code and Insolvency Law. Also, the construction works at the 16 courts and the substantial upgrade of working conditions contributed to improved staff morale and motivation, more sensible use of space, and streamlining of work flows. Finally the introduction of court recording system added substantially, transparency and legal certainty in Romania. The financial analysis performed was based on a cash flow approach and considered real cash inflows and outflows that arose from project implementation. The analysis yielded Financial Net Present Value (FNPV) of EUR 18,237 using 5 percent discount rate and Financial Internal Rate of Return (FIRR) of 5.2 percent. Key assumptions of financial analysis include: • The investment horizon starts in the first year when the loan was effective, which is 2006. This is to capture the opportunity cost of non-utilization of the funds for several years, reflected through the commitment fee paid on the undisbursed amount.In order to determine the incremental financial inflow coming from reduction in O&M costs in the renovated courthouses, we took the average O&M bill of a Romanian courthouse which is currently at the level of 380,000 Lei or 68,000 EUR. The average reduction achieved as a result of civil works done under JRP is estimated to be around 60 percent. This estimate is corrected downwards to the average level of 45 percent for the 4 out of 11 courthouses that were both renovated and extended under the project. • Financial outflows include operation and maintenance (O&M) cost of newly built courthouses since they represent an incremental cost which is a result of project implementation. These are assumed to be at the level of 50 percent of the total O&M cost of the existing courthouses due to improved overall condition of the buildings – better insulation, new electrical and sanitary installations, etc. • The cost of government staff time used for implementation is determined based on an assumption that there were 20 employees working on project implementation on average 16 hours per month over the considered horizon. This cost is estimated using the wage per hour calculated based on the assumption that average government staff wage is currently EUR 1,000 per month. Similar to the O&M costs above, this amount was multiplied with relevant CPI indices to arrive at annual wage levels for the years before 2017. • The exchange rate used is Lei 4.6 to Euro as of August 24, 2017. • The discount rate of 5% is used. The choice of discount rate is motivated by the fact that 5 percent rate is suggested to be used in EU Cohesion countries as noted in the 2016 WB GN on Discounting Costs and Benefits in Economic Analysis. • The annual inflation rate in Romania is assumed to be 2.5 percent while the annual inflation rate in EU is assumed to be 1.5 percent. Page 66 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Although we agree in principle with the content of the ICR, we still have some comments, mainly to the final conclusions. As a general comment, we strongly recommend that the figures from the ICR be updated according to the final ones (July 31, 2017). This ICR reflects the current rules and approach of the bank, in place at the moment of doing the ICR, not at the moment of the preparation /negotiation of the loan. I recall that at that time the Bank was more oriented to outputs not to outcomes, the safeguard policy was different and the baseline indicators (mostly inputs) were difficult to be identified either by the bank or by the implementing agency, because of the lack of centralized information. The introduction of RMS, was at the Bank’s initiative, further to a functional review done by it. MoJ agreed, hoping to get more centralized information on the judicial system. Being the first initiative it was normal to begin more slowly, but at the end some important achievements (even partially) could be noted. Regarding the idea of having a PMU in place prior to the loan effectiveness, it will be good to have it, but unfortunately it is not allowed by the Romanian legislation. However, a team may be created in the implementing agency which can be transformed latter on in the core of the PMU, to which more staff can be added. Regarding change management and user support, we recommend more consultation with the final beneficiaries and a public information campaign during the preparation and the implementation of the project, not to have a specific component included. We do not recommend Action Plan accompanied by contingency mechanism, because this will proof not reliance of the Bank on the capability of the implementing agency or to the Borrower, to do not jeopardize any future transaction with the bank, taking in to account that the financing is not for free, but a loan. However, Action Plans should always exist, as well as M&E. Page 67 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) • Project Appraisal Document, Report No. 33987-RO dated November 22, 2005 • Loan Agreement, Loan Number 4811-RO dated January 27, 2006 • Restructuring Paper, Report no. RES18256, dated March 12, 2015 • Restructuring Paper, Report No: 56592-RO, dated October 5, 2010 • Restructuring Paper, Report No. 63895-RO, dated August 16, 2011 • Restructuring paper, Report No. 74121-RO, dated December 11, 2012 • Implementation Status and Results Reports No. 12 to 20 dated between March 3, 2011 and December 22, 2016 • Audit Reports dated between June 30, 2008 and June 8, 2017 • Aide Memoires of Bank missions from 2008 to 2016 • Revised Performance and Monitoring Indicators, Attachment to Supplemental Letter no. 2, dated March 19, 2015 • Conformed copy of Revised Performance and Monitoring Indicators, dated October 6, 2010 • Survey of actual experiences with, and attitudes and perceptions about, implementation of Judicial Reforms in Romania", Reference No. IBRD4811RO-MOJ/ CS-4-12-2 • Operations Manual, dated July 2010, with Annexes from 1 to 3 • Borrower ICR, dated December 2016, annexed in Annex 9 Page 68 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 7. BENEFICIARY SURVEY RESULTS Beneficiary Survey 1. Three waves of surveys were done within the scope of the project: one in 2008 (baseline survey), one in 2014 (intermediate survey), and the final one in 2016 (final survey). They covered the providers’ side (judges, prosecutors, court clerks), intermediaries’ side (lawyers), and users’ side (physical and legal persons using the judiciary system). The surveys covered the “intervention” group, i.e. courts where some project activities were planned, and a “control” group, where activities were not planned. Survey activities included large quantitative surveys to gather more anecdotal data. This approach allowed for two types of references when identifying impact: a longitudinal one (across time) and a transversal one (between control and intervention groups). 2. Infrastructure improvements: Data suggests that there are some significant improvements in the courts’ infrastructure as a result of the project activities. Courts from the intervention group show some significant improvements compare to ones in the control group. The improvements include: (i) usable floor area is larger by 35%; (ii) public waiting areas are three times larger; (iii) 50% larger office space; (iv) almost double archive space; (v) percentage of courts in need of major repairs is 20%, compared to 60% in the control group; and (vi) more computers and better access to the internet. As an addition to these “hard” numbers, judges and prosecutors from the intervention group express higher level of sat isfaction with the existing infrastructure (in 2016) and also more frequently express that the situation has been improving in recent years. 3. Improvements in court administrative capacity: There are no differences in the number of employees between the intervention group and the control group courts, nor there is one over time. An analysis of working hours shows no difference between the two groups. However, one difference that the judges noted was an increase in court clerks’ honesty, integrity and professionalism over time. 4. Improvements in the act of justice: Surveyed judges from the intervention group express significantly higher levels oif satisfaction with the quality of prosecutors’ work and improved perception of lawyers involved in cases, more specifically, their integrity, honesty and attitude. The judges also express higher satisfaction with the Civil Code and Ccivil Pprocedure Code. However, these opinions are not shared by the surveyed prosecutors. Moreover, prosecutors from the control group express higher levels of satisfaction with certain aspects of the act of justice. Nevertheless, there is some improvement over time: the last two years have brought some improvements concerning transparency of the act of justice, acknowledged by prosecutors from both the intervention group and the control group, as well as improved perception of time needed for case disposition. 5. The analysis of data from users’ perspective shows that there is an improvement over time in the perception of the judges’ treatment of the beneficiaries of the court. Data also indicates overall improved perception of all aspects of lawyers’ work. Although differences between intervention and control groups are not significant, there is a tendency towards slightly more positive evaluations from the respondents from the intervention group. 6. Court computerization and training: There were no major differences reported regarding access to new technologies or the need for more professional training. Perception over the efficiency of the Page 69 of 104 The World Bank Judicial Reform ( P090309 ) registration and archiving system is higher in the intervention group (93 percent) compared to the control group (75 percent). Access to jurisprudence, legislation and doctrine databases is reportedly easier for the intervention group (59 percent) compared to the control group (51 percent). Overall, improvements are can be discerned as compared to the situation in 2008 (when the intermediate survey was conducted), which suggests a structural improvement of the entire system and a certainthe pervasiveness of the new technologies throughout the system. 7. The three surveys are summarized below, starting with the Baseline Survey. Baseline Survey 8. A baseline survey was carried out by the Gallup Organization in 2007. The purpose of the survey was to provide a snapshot of the current reality based on the monitoring indicators, specifically: (1) the performance of the judicial system in Romania, specifically, in terms of its efficiency (speed, fairness, affordability, ability of to enforce decisions, etc.), transparency and integrity, and professionalism; and (2) the current status and achievements of the judicial reform process. The survey comprised of a qualitative and a quantitative approach. The qualitative component focused on the actual conditions being surveyed while the quantitative aimed at providing data on the relevant indicators in relation to the objectives of the JRP. 9. For the qualitative approach, a documentary analysis and focus groups and pilot interviews were conducted. Three focus groups with experts from the judicial sphere (judges, prosecutors and lawyers) were foreseen in three different locations (Bucharest and two other cities agreed with the MOJ). For the quantitative part, the participation of 2400 respondents was planned in two of the three quantitative surveys, of which 1200 from the general public, and 1200 experts from the judiciary system (400 judges, 400 prosecutors, 200 lawyers, and 200 court staff). In the third quantitative survey on courts and prosecution offices, a questionnaire was sent to all the courts and corresponding prosecution offices (about 520). 70 qualitative, semi-structured interviews with judges, prosecutors, lawyers, court staff, governmental bodies, mass media and NGOs were also planned in 8 different locations. A preliminary report on the findings of the baseline survey was published in January 2008. However, it is not clear from the report or the background documents how many respondents participated in the surveys. Part I: Results of the Qualitative Survey 10. The qualitative survey analyzed three dimensions: (i) the image of the reform of the judiciary; (ii) the attitudes manifested by the respondents; and (iii) respondents’ expectations from the reform of the judicial system. Respondents included law officers, attorneys, journalists and NGOs. 11. With regard to the first dimension, the image of the reform of the judiciary, the prevailing mental image of all respondents was a negative one. The judicial reform was perceived by respondents as “a form without grounds” which lingers only in the speeches of the politicians. The management of the legal system was perceived as being ineffective, not based upon expertise, debates with all the parties involved or an independent radiography of the current problems performed by experts in the field. Respondents from all categories stated that the implemented changes were many but did not trigger the targeted effects, and sometimes they did not even respond to the real problems of the system. One of the causes, frequently mentioned by respondents, why the reform could not bring any major changes, was the lack of political will, followed by the lack of strategy. However, most respondents agreed that the legal system Page 70 of 104 The World Bank Judicial Reform ( P090309 ) had evolved since the 1990s and that a series of changes had been enforced due to pressure from the EU. Still, the results were not perceived as sufficient to create an effective justice system. 12. While the respondents provided similar answers with regard to the perceived image of the reform of the judicial system, their views of the definition of what the justice reform differed among the different categories. Practitioners (law officers and attorneys) viewed it as the development of daily activities. Journalists defined justice reform as the fight against corruption, solving the files of some high officials, and the independence of law officers. NGO representatives viewed it as a degree of collaboration with the MOJ and the extent of enforcement of recommendations/advice provided by NGOs. 13. Practitioners expressed that the most appreciated changes related to the material equipment of courts and public prosecutors’ departments (IT equipment), random distribution of files, youth promotion, startup of the judge specialization process, and enforcement of exams for management positions. However, with the exception of IT equipment, the respondents expressed that all the positive changes also had negative effects, due to some non-correlated issues or lack of strategy. The effects of the changes were also diminished, according to the surveyed practitioners, by the lack of other measures which should have been adopted simultaneously. Among the different categories of practitioners surveyed, attorneys were most critical regarding the changes, as they felt left aside within the system and their views not taken into account when implementing various reform measures. 14. Journalists expressed the belief that the judicial system remained practically unchanged and the phrase “justice reform” had no actual meaning. For them, justice was still politically controlled and dominated by incompetent, corruptible people. 15. NGOs’ perspective of the judicial system is shaped by the direct interactions with certain actors of the system, with whom they interact. They expressed that there were many changes, but the results were poor and not the ones expected. 16. With regard to the second dimension, attitudes towards the characteristics of the judicial system, the reform was a priority both for practitioners within the system and for the members of civil society. The changes debated referred to efficiency and transparency. Most respondents perceived the transformations positively. The large amount of work determined by the large number of files and insufficient personnel was considered by respondents to rather get worse than better. In some cases, positive changes have been mentioned regarding the duration of procedures, which are explained by the better operation of archives (due to the IT equipment of law courts) or by the change of the summoning procedure (summoning by phone call instead of mail). Changes also occurred by the regulation of the emergency procedures (e.g.: the payment summoning), offering the possibility of some shorter terms, which reduces the duration of the process. 17. Regarding the random distribution of files, the perceptions of respondents were generally positive. However, at the same time, the respondents noticed the quality of justice might suffer as certain files should be distributed on the basis of competences and relevant experience. The IT equipment of public prosecutor’s departments and law courts was perceived as having a positive impact upon the activity of the legal personnel by increasing celerity and efficiency, but there are also many deficiencies regarding the infrastructure of the judicial system which are still pending (such as the problem of the insufficient space). 18. When referring to efficiency, most respondents considered certain guiding criteria, such as: the quality of the act of justice, celerity, professional quality of the personnel, cooperation and Page 71 of 104 The World Bank Judicial Reform ( P090309 ) communication among the various levels of the system, as well as the existence of a material-logistic infrastructure to facilitate the good operation of the judicial system. With regard to the quality of the act of justice, the respondents considered that an efficient judicial system should not leave any space for preferential interpretation of the law by the parties involved, and the rectitude of the act of justice should be directly correlated to the observance of the laws in force. Likewise, the ideal perspective is to issue more decisions which manage to stand if subject to challenges or appeals. Practitioners considered that, with a view to improving the act of justice, the implemented changes should focus on legal stability and human resources. 19. With regard to celerity (short term, rapid case solving), most interviewed subjects considered that the duration of the legal procedure was very long, especially in civil matters. Another factor affecting celerity was the fact that judges were obliged to enter the court within certain periods of time, which were closely regulated, due to the lack of space and lack of offices. The respondents also mentioned the importance of predictability of procedural terms of evidence of management, so that trials may not require too long extensions. 20. As for the cooperation and communication at the level of courts and public prosecutors’ departments, the respondents pointed to a lack of efficient cooperation between these two types of institutions. This, they argued, has a direct impact on the duration of proceedings and the image of the judicial system. The respondents also pointed to a lack of cooperation within the institutions. 21. The material basis needed for the development of the act of justice was considered as satisfactory by most respondents, in particular with regard to the IT equipment. Most interviewees pointed to the lack of space as a problem with major implications for the duration of the proceedings as well as for the image of the act of justice. The interviewed law officers and attorneys expressed that new buildings should be built to house courts, in order to avoid over-crowded rooms. In parallel, measures to increase the number of judges and reduce the number of files assigned to each of them should be taken. 22. With regard to transparency of the judicial system, most law officers underlined that courts are sufficiently transparent. The main criteria defining transparency, according to the respondents, refer to: knowing how public prosecutors’ departments and courts work, as well as the procedures employed by the various departments of the judicial system; providing information regarding the conditions according to which the parties or the public may find about the works initiated to solve cases; and drafting activity reports which are made available to the public. 23. As for the expectations regarding the reform of the judicial system, one of the most prominent expectations of practitioners was of balancing the ratio between the number of cases and the number of law officers and auxiliary personnel handling these cases. Moreover, increasing the number of auxiliary personnel and their training was seen as one of the absolutely necessary conditions to improve the efficiency of courts. Most law officers expressed their wish that court clerks should have their roles expanded so that they may take over some of the attributions of judges. A stricter specialization of judges was also perceived as a major factor which might increase the efficiency of the legal process. 24. The reform of the judicial system was perceived as closely correlated with the reform of the legislative framework. The expectations of all of the groups surveyed can be classified under four large categories: stability of the legislative framework, coherence of the legislative framework, unified practice, and correlation of the national laws with the European laws. Page 72 of 104 The World Bank Judicial Reform ( P090309 ) 25. The interviewees also mentioned the need for decent spaces for court proceedings as well as spaces for offices and archives. Technical equipment, such as computers and printers, should be available in all institutions. They also mentioned the need for increasing the transparency of courts by publishing online information about cases. All these changes (laws, procedures, competences) should be monitored, and, in case of infringements, penalties should be applied. A stricter control of observance of procedures was primarily requested by the representatives of NGOs and media, but also by law officers. Part II: Results of the Quantitative Survey 26. Institutions: The quantitative survey showed that the medium age of courts and tribunals was on average 64 respectively 67 years. 71 percent of the head offices of the courts were of the opinion that their institution required major repairs, reinforcement, upgrading or expansion works. In 64 percent of the case no such works were scheduled in the following year. In 63 percent of the cases, the opinion was that the spaces where the archives were kept were inadequate. 49 percent of the respondents thought the quality of installations (power) were inadequate. The three main problems that the respondents pointed to with regard to the office buildings they were working in were: lack of space, poor quality of installations, and the necessity of building restauration. 41 percent of the respondents thought space constraints influence the duration of the case solving procedures, while 59 percent did not think that. 27. The medium number of files pending at the beginning of 2007 was 1,684 in law courts, 2,629 in tribunals, and 1,315 in courts of appeal. In 97 percent of the cases the cases were assigned randomly in courts, while only in 37 percent of the cases in prosecutors’ departments. 28. The average number of employees was 72 in law courts, 241 in tribunals, and 118 in courts of appeal. 72 percent of the judges were also performing administrative, non-legal activities. This was the case for 51 percent of the prosecutors. The three most common non-legal tasks of judges were: organizing elections and participating in committees, organizing departments, and public relations. 29. The number of computers was one per judge in all instances of the courts. However, the percentage of computers in use by judges was only 32 percent in law courts, 27 percent in tribunals, and 37 percent in courts of appeal. Furthermore, only 65 percent of the computers had internet connection in law courts, 59 percent in tribunals, and 55 percent in courts of appeal. 30. 44 percent of the respondents thought that the archiving system in their court was inadequate, while 53 percent thought it was adequate. 31. 59 percent of judges in law courts, 58 percent in tribunals, and 68 percent in courts of appeal attended training courses. With regard to other staff, 61 percent in law courts, 62 percent in tribunals, and 54 percent in courts of appeal attended training. 79 percent of the respondents thought that the development of distance learning programs was necessary. The three main problems pointed out by the respondents with regard to continuous training of magistrates were: lack of time, small number of courses/ seminars and their poor quality, and subjective criteria of selection. With regard to auxiliary personnel, the three main problems were: small number of courses and their poor quality, lack of specialized personnel/ overwhelming activities, and lack of time. 32. Judges: the average working hours for judges in law courts and tribunals was 48 hours, while it was 47 hours for judges in courts of appeal. The number of files assigned per judge in law courts was 184, 223 in tribunals, and 246 in courts of appeal. The three main reasons for judges to have files older than six months were: lack of court experts and court expert opinions, case complexity, and lack of procedure. Page 73 of 104 The World Bank Judicial Reform ( P090309 ) 44 percent of the surveyed judges thought that compared to a year ago the average duration of a case settlement in their court had reduced. The three main reasons for that were: the active involvement of magistrates, decreased number of files/ more staff, and increase of IT equipment and more specialized magistrates. 33. 61 percent of the judges said they were frequently or very frequently confronted with issues that prevented the proper course of the trial and delayed the settlements in the last year. The three most common issues of this kind that the judges were confronted with related to: lack of experts/ lateness of expertise, parties’ failure to attend meeting/ requests of postponement, and lack of procedure. 34. 98 percent of the surveyed judges said they had access to the internet at their institution. 95 percent said they had access to legislation databases. 35. 59 percent of the judges said they had attended training in past year; 91 percent said they were satisfied with the quality of the training. On average, judges spent 6 days on training in 2006. 84 percent of the judges felt they needed additional training for their activity. The main three courses mentioned by judges that they would like to attend were: community law and case law, various courses (foreign languages, computer skills, etc.), and human rights. 36. Prosecutors: the average working hours of prosecutors in law courts was 46 hours, in tribunals 42 hours, in courts of appeal 44 hours, and in the HCCJ 47 hours. The medium number of files for prosecutors along law courts was 35, 16 in tribunals, 10 in courts of appeal, and 7 in the HCCJ. 29 percent of the surveyed prosecutors thought the files in their office were not assigned depending on the prosecutors’ specialization. 37. 20 percent of the surveyed prosecutors thought that the average time of the criminal prosecutions procedures in their office had reduced a little compared to a year ago. The three main reasons for that were stated as: prosecutors are more specialized/ work is organized efficiently, competency changes and collaboration with other institutions, and supplement of staff scheme. 38. 34 percent of the prosecutors expressed they were frequently or very frequently confronted with issues that prevented the proper course of the trial and delayed the settlements in the last year. The three main reasons for that were stated as: the procedures, lateness of expertise and repeated postponements, and failure of parties or defenders in attending meetings. 39. 96 percent of the prosecutors said they had access to a computer at their work place and 85 percent said they had access to the internet. 77 percent found this access useful. 73 percent had access to email. 93 percent had access to legislation database, and 82 percent found this access useful. 91 percent said they had access to case law databases, and 80 percent said they found this access useful. 40. 69 percent of the prosecutors said they had attended training in the past year. The medium number of training days was 5 for prosecutors along law courts, 6 for tribunals, 10 for courts of appeal, and 9 for the HCCJ. Only five percent of the respondents were not satisfied with their training. 75 percent said they felt that they need additional training for their activity. The three top courses they were interested in attending were: various courses (foreign language, computer skills, management, etc.), criminal law and procedures, and various crimes (economic, financial, corruption, etc.). 41. Staff (court clerks): 41 percent of the clerks said they were working between 41 and 50 hours per week, 16 percent said they were working more than 51 hours. 34 percent were working between 31 and 40 hours. 74 percent had worked at their current institution for more than five years. 82 percent said they Page 74 of 104 The World Bank Judicial Reform ( P090309 ) saw themselves working in the same institution in five years’ time. Only four percent thought their job description did not match their current activities. 42. 99 percent of the court clerks said they had access to a computer in their work place; 75 percent said they had access to the internet. 22 percent said they did not have access to email. 82 percent had access to legislation data bases, while 69 percent had access to case law databases. 56 percent said the access to case law was useful to them. 43. 64 percent of the court clerks said they had attended training in the last year. On average, clerks in courts spent 7 days on training in 2006, while clerks in prosecutors’ department spent 6 days on training. 89 percent said they were satisfied with their training, while only five percent were not satisfied. More than two thirds of the surveyed clerks, 67 percent, felt they needed more training for their activity. The three courses they were most interested in were: various courses (communication, management, foreign languages), legislation/ human rights, and computer science/ using ECRIS. 44. Clerks expressed that three top measures that would positively impact professional performance in their departments were: higher salaries, more employees, and training. 45. Seven percent of the clerks thought there were many corrupt judges, 8 percent thought there were many corrupt prosecutors, 19 percent thought there were many corrupt lawyers, and 17 percent thought there were many corrupt police men. 46. 49 percent of the court clerks thought the government had taken too few or no measures to reform the justice system. 83 percent thought that the measures taken were rather good, while 12 percent thought they were rather bad. 68 percent thought that following these measures, the situation would improve, while 23 percent thought it would remain the same. The three main aspects the clerks considered to have priority in order to improve the Romanian justice situation were: simplifying the procedures by reducing the length of a lawsuit, increasing the transparency of justice, and increasing the judges’ and prosecutors’ independence from politics. 47. Lawyers: With regard to workload, the surveyed lawyers had on average 38,93 pending cases, 19,62 of which had been pending for more than six months. 48. Lawyers’ perception of judges was not always very positive. 44 percent of the surveyed lawyers did not think judges treated people with dignity and respect. 29 percent did not think judges made honest decisions. 86 percent of the lawyers thought Romanian judges were not held liable when settling a case erroneously. 44 thought judges did not treat those who have legal assistance and those who do not equally. 49. Opinions were similar regarding prosecutors. 53 percent of the lawyers did not think prosecutors treated people with dignity and respect. 46 percent did not think prosecutors made honest decisions. 76 percent did not think prosecutors were held liable when providing for an erroneous remedy in a file. And 57 percent thought prosecutors’ offices did not treat the poor and the rich equally. 50. As for corruption within the justice system, 47 percent of the prosecutors thought many judges were involved in corruption (of which five percent thought this was the case for almost all judges). 60 percent thought prosecutors were corrupt, 30 percent thought lawyers were corrupt, 28 percent thought this was the case for notaries, 51 percent with regard to legal officers, 78 percent with regard to police officers, and 35 percent with regard to officers of the court. Page 75 of 104 The World Bank Judicial Reform ( P090309 ) 51. Most lawyers were not very impressed with the progress of justice reform. 64 percent of the lawyers thought the government had taken few or very few measures in order to reform the justice system, while less than a third, 28 percent, thought the government had taken many measures. Most of the lawyers, 83 percent, thought the measures taken had been good, while 17 percent thought they had been rather bad. However, 63 percent thought the situation would remain the same despite the reform measures, 8 percent thought it would worsen, while less than a third, 27 percent, thought it would improve. 52. Less than half of the lawyers, 43 percent, were of the opinion that compared to one year ago, the length of criminal proceedings was the same. However, 36 percent thought that the length had reduced a little. With regard to civil cases, half of the lawyers, 49 percent, thought the average term for the settlement remained the same compared to a year ago, while 39 percent thought it had reduced a little. The three main reasons for the reduction were: involvement of prosecutors/ more attention of solving the cases, changes of laws/ complexity of cases, and more personnel/ decreased number of files. 53. Lawyers were also asked how much of the time they spend in court they consider as wasted. More than a third of the lawyers, 36 percent, thought more than half the time they spent in court was wasted, while 29 percent thought about half their time was wasted. 15 percent respectively thought more than a quarter but less than half, or less than a quarter, of their time was wasted. The three top reasons stated for this were: archives, trial courts, and the registry. The top three activities considered by lawyers as contributing to wasting their time were: call of the file in trial courts/ trial meetings, archives, and file study. 54. Many lawyers were not satisfied with the conditions in the court rooms. 43 percent stated they were unsatisfied, 14 percent were totally unsatisfied, while 40 percent said they were satisfied. The main aspects of the court rooms that lawyers were not satisfied with were: small, insufficient space, lack of equipment, and spaces are not clean enough, insanitary conditions. 55. According to the surveyed lawyers, the three most important problems that justice in Romania was facing were: lack of competent magistrates, corruption, and insufficient personnel/ large amount of cases. The aspects that the lawyers thought had priority in order to improve the justice system were: reducing corruption in the legal system, simplifying the procedures by reducing the length of a lawsuit, and permanent training of magistrates. 56. Finally, the general public was also surveyed regarding the act of justice in Romania. Among the top problems that Romania was facing, corruption was the third after too low salaries and prices. When asked how satisfied they were with the operations of justice, 34 percent of the respondents said they were very unsatisfied, while 37 percent said they were rather unsatisfied. The Romanians had least trust in the police, closely followed by the government and local town halls. 30 percent stated they had little trust in SCM and 18 percent said they had no trust at all in the SCM. Almost two thirds of the respondents, 62 percent, said they had very little or no trust in judges, prosecutors and lawyers respectively. 55 percent had no or very little trust in legal officers. The number was 59 percent with regard to officers of the court. 57. 76 percent of the respondents agreed with the statement that lawsuits last too long in Romania. 66 percent agreed that judges’ decisions are influenced by politicians. 65 percent agreed that courts allow many criminals to be free due to procedural flaws. 62 percent agreed that judges often interpret the same law differently. And 49 percent agreed that judges do not pay the required attention and do not allocate the time necessary to the cases they hear. Page 76 of 104 The World Bank Judicial Reform ( P090309 ) 58. With regard to statements on judges and trial courts, 59 percent of the respondents thought that Romanian judges are not being held liable when settling a case erroneously. 75 percent did not think that courts were treating the poor and the rich equally. 59. When asked about corruption, the public’s opinion was that the most corrupt professional categories were judges (63 percent), prosecutors (60 percent), lawyers (60 percent), and police officers (60 percent). With regard to competence, the public had the least opinion on policemen (38 percent), legal officers (26 percent), prosecutors, officers of the court and judges (25 percent respectively). 60. Most respondents, 56 percent, thought the government had taken few or very few measures with regard to the reform of the justice system. 12 percent thought no measures had been taken at all. Only 15 percent thought many measures had been taken. However, 70 percent thought the measures taken had been rather good, while 23 percent thought they were rather bad. There was also a certain level of optimism, with 63 percent of the respondents believing the measures taken would improve the justice situation in the country in the next 2-3 years. 61. The most important sources of information regarding the system of justice for the general public were TV news (72 percent), press (53 percent) and direct experience with courts (22 percent). 85 percent of the respondents or the members of their household had not been party in a lawsuit in the past two years. Of those who had, 63 percent had employed a private lawyer while 8 percent had an ex officio lawyer. 26 percent did not have a lawyer. The majority, 60 percent, were satisfied with the performance of their lawyer, while 32 percent were not satisfied. 62. 62 percent of the respondents who had had a case in court in the last 2-3 years said they agreed with the opinion that the lawsuit lasted very long. 64 percent thought the decision was made based on the facts and 57 percent thought the judge’s decision was fair. 70 percent thought people were treated with dignity and respect. Two third of the respondents, 67 percent, thought the judge was neutral and objective and 65 percent thought the procedures applied in the court were correct. 57 percent said the court allowed everybody to present their standpoints. 63. The respondents were asked about what they think are the three most important problems that the Romanian justice is facing. The number one problem pointed out by the majority of the respondents was corruption. Second came lack of professionalism, and third vague/ misinterpretation of the legislation. Involvement of politics in justice also scored high, as well as too long duration of proceedings. 64. The three priority aspects to improve the justice situation in Romania were: reducing corruption in the legal system, simplifying the procedure by reducing the length of proceedings, and increasing the judges’ and prosecutors’ independence from politics. Intermediate Survey 65. An intermediate survey was carried out by Metro Media Transylvania in 2014. The purpose of the survey was to collect information on actual court performance and user feedback on the project’s impact in designated courts; the results were to be published on court websites. The sampling was based on the courts included in the JRP: six courts (courts from Viseu de Sus, Blaj, Worcester, Bolintin Vale, Săliştea Sibiu Orsova) and four tribunals (tribunals from Maramures; Suceava, Arges). An additional six courts and four tribunals were selected as a group for control (courts from Târgu-Lăpuș, Aiud, Câmpulung Moldovenesc, Cornetu, Agnita, and Judecătoria Strehaia, and tribunals from Satu-Mare, Călărași, Neamț, and Vâlcea). The sample distribution was as follows (with an equal size for the group of control): 75 judges Page 77 of 104 The World Bank Judicial Reform ( P090309 ) (18 from Courts and 57 from Tribunals), 50 prosecutors (12 from Prosecutor Offices of Courts and 38 from Prosecutor Offices of Tribunals), 25 court clerks (12 from Courts and 13 from Tribunals), 154 lawyers, legal advisers (63 from Courts and 91 from Tribunals), 504 litigants, individuals (195 from Courts and 309 from Tribunals), and 250 litigants, companies (96 from Courts and 154 from Tribunals). A total of 151 judges, 100 prosecutors, 50 court clerks, 306 lawyers/legal advisers, and 1506 litigants were interviewed. Below are the results of the intermediate survey. Part I: Assessment of infrastructure 66. For the judges’ sample, the survey identifies an important gap related to the level of satisfaction determined within the intervention group, which compared to the group of control, varies from 1% to 33%. However, the survey notes a higher degree of satisfaction with respect to the basic infrastructure of the courts, such as: size of the useful area, number of courtrooms, quality of the audio system, furniture, and design. Further, the gap between the two groups is maintained also for the indicator “visible improvements” with 17% in favor of the group subjected to intervention (13 out of 15 aspect analyzed). 67. There are similar results for prosecutors where the survey notes an important variation related to the perception of the two groups participating to the survey. The results obtained at individual level show that there is a slightly higher degree of satisfaction within the intervention group. The same group expressed a higher degree of satisfaction related to the public utilities infrastructure: parking lots, cleanliness of restrooms and public spaces. However, the group of intervention responded to be less satisfied with the visible improvements perceived over the last two years. 68. The results for the court clerks are very similar to those determined for the judges’ profile. The level of satisfaction between the intervention group and the group of control varies from 7% to 32%. Overall, the results show that this sample is satisfied with the basic infrastructure of the courts and there is a more positive perception related to the visible improvements. 69. As for the lawyers and legal advisers, the results indicate that the two groups have a level of satisfaction varying between 8% and 25%. On average, lawyers and legal advisers who are part the group of intervention respond to be satisfied or very satisfied with 8 out of 10 aspects, while lawyers and legal advisers from the group of control only with 7. Also this sample notes a more positive perception related to the visible improvements for all aspects identified in the survey. 70. Further, for the litigants, individuals the results show a level of satisfaction varying from 4% to 27% in favor of the group of intervention. For the two groups, the deviation of perception related to the improvements over the last two years differ from 16% to 31%. 71. Similar to other samples, the survey shows that for litigants, companies the level of satisfaction varies between 3% and 24%, in favor of the group of intervention. The way these groups perceive recent improvements varies from 6% to 24%. 72. Overall, in infrastructure, the survey states that the project has generated a real and significant impact in most cases. In the case of judges and clerks, we note a high level of satisfaction related to the basic infrastructure of the courts, while prosecutors are highly satisfied with the public utilities, and lawyers and litigants are satisfied with the design of the courts. Also, the project increased visibility of improvements in almost all situations, except for prosecutors. Part II: Quality of the staff Page 78 of 104 The World Bank Judicial Reform ( P090309 ) 73. The survey shows that judges have a similar perception towards prosecutors and clerks. Both groups (intervention and control) identify the same needs for the improvement of the quality of the staff, and a higher degree of satisfaction towards court staff compared to tribunals staff. 74. As for the prosecutors, the results show for the intervention group a slightly higher degree of satisfaction towards judges and clerks. However, the groups of control expresses satisfaction with the degree of improvement in the quality of the staff. 75. The results for the clerks’ sample show that both groups express same degree of satisfaction towards judges and prosecutors. There are different perspectives related to the degree of improvement where the variation goes from 0 to 12% in favor of the group of intervention. 76. The collected data for the lawyers show an important satisfaction variation between the two groups, precisely 2% to 13% concerning all types of skills identified in the survey. The differences related to the visible improvements of the quality of the staff is in favor of the intervention group with 16%, compared to 6% for the control group. 77. For the litigants, individuals, the variation in the level of satisfaction towards judges and prosecutors results to be significant between the two groups interviewed. 78. Looking at the litigants, companies, the results show a variation in the level of satisfaction between 0% and 18% in favor of the intervention group. The difference between the two groups is less important when it comes to the level of satisfaction towards judges and prosecutors. Part III: The act of justice 79. The data show in the case of judges, that both groups (intervention and control) are satisfied with the outcome, and good improvement was noted but there no clear tendency between the two groups. 80. In the case of the prosecutors, the differences vary between 6% and 18% in favor of the group of intervention, and there is an overall tendency for increased satisfaction towards lawyers. 81. The results show that the court clerks’ level of satisfaction differs between 4% and 16% between the two groups. Regarding the perception towards improvement both groups expressed differences varying from 12% to 28%. 82. For lawyers, the results show a satisfaction variation between 4% and 11%, in favor of the group of intervention. On average, the group of intervention declares to be satisfied with 3 out of 6 aspects of improvement, while the group of control only with 2. 83. The litigants, individuals sample data indicates differences varying between 7% and 19%, in favor of the group of intervention. On average, the group of intervention declares to be satisfied with 5 out of 7 aspects of improvement, while the group of control only with 4. 84. For the litigants, companies sample data shows that the group of intervention is slightly more satisfied where the level of satisfaction differs from 4% to 18%. On average, the group of intervention declares to be satisfied with 4, 6 out of 7 aspects of improvement, whilst the group of control only with 4, 5. 85. To sum, the survey states that the project has generated a real impact, rising the degree of satisfaction among all categories participating to the survey with the exception of judges. There is a high level of satisfaction identified on all dimensions of justice. Also, the level of satisfaction regarding the Page 79 of 104 The World Bank Judicial Reform ( P090309 ) improvement is significantly higher within the intervention group. Part IV: Workload 86. The survey shows that the way this project influenced the workload of professionals has several factors: (i) it was generally oriented towards reducing the workload. This seems to be less the case for clerks (but here the key factor may be the number of clerks in the court); (ii) reduction of delay (more than six months) in the average time required to dispose a case; (iii) increased efficiency for judges; (iv) and better time management framework for prosecutors and clerks. Part V: Professional training 87. This survey does not identify any major differences between the two groups (intervention and control). However, it illustrates differences related to the 2008 Baseline survey. Thus, for the judges, there is an increase of 20% among those who have followed a training course in the year preceding the survey. The data shows similar values for prosecutors, indicating a systemic growth of this indicator over the last 6 years. For clerks, the survey identifies beyond a structural increase in the number of those who have followed a training course (8% compared to 2008), also an increase of 8% due to the project. Regarding the need to follow professional trainings, the recent data indicate rather a decrease compared to 2008 no matter the group. Part VI: Using new technologies 88. Regarding the access to new technologies, data shows that for all three categories of professionals (judges, prosecutors and clerk) there are significant differences between the two groups (intervention and control). However, one aspect to be noted here is the access to data bases related to the case of law, where there is an increase of 10% for judges. They respond to have access easily to such information, and to the archiving system. Compared to the 2008 Baseline Survey, data shows a significant improvement regarding the access to the internet and to professional information for all categories of professionals participating to the survey. Part VII: History access and costs 89. Although concerning the history access the survey does not reveal any major differences between the two groups, the differences are attributable to the costs. The results show that the group of intervention spent more time in courts than the group of control. The audience time did not change, it remained about three hours. However, total costs (audience start, transport, photocopied documents, etc. but without lawyers’ costs) are significantly lower by 16% in the intervention group (1,250 lei) than in the control group (1,450 lei), for individual litigants, these are 38% lower in the intervention group (490 RON) than control group (680 lei). The difference in costs between the two categories is determined by the proportion of different courts and tribunals to access the first instance in which case the costs are higher. Part VIII: Complaints and claims 90. The survey does not present any data analysis for this part of the assessment. Final Survey Page 80 of 104 The World Bank Judicial Reform ( P090309 ) 91. The first investigated aspect was a presentation of the courts, the analysis of the material and human infrastructure which these courts possess. It revealed that the courts which were included in the project possess a superior infrastructure both qualitatively and quantitatively, as compared to the control group courts. Data indicates useable floor areas larger by about 35%, public waiting area about three times larger, office space larger by approx. 50%, and almost double archive space. The number of public facilities (toilets and parking spaces) is much higher for the courts subject to intervention. Taking this into account, it appears normal that the proportion of courts in need of major repairs is much smaller for the intervention group (20%), compared to the control group (60%) 92. There is no significant difference concerning the number of employees between the two groups, the only difference being the number of interns from the intervention group (5 interns) as compared to the control group (1 intern). 93. Data indicates that, on average, the number of pending cases, at the beginning of 2015 was higher for the courts subject to the intervention (4127 cases, compared to 3105, from the control group). Out of the total which includes new cases (8861 cases) 5359 were disposed (about 60%), while in the control group, out of the 9645 pending cases registered with the courts, 6353 were disposed (66%). These numbers are similar to data revealed in 2014 (68% in both types of courts). 94. The computer network for the intervention group is composed of 133 devices, out of which 98% are connected to the internet. Within the control group, out of the 117 existent devices (on average) 90% are connected to the internet. Comparing these results to the ones of the 2014 study, a more developed computer network can be observed, both in terms of numbers and regarding Internet connectivity. 95. The case archiving system seems to be more problematic in the control group, considering that five of the courts either consider it less efficient, or cannot provide a rating. 96. The questioned judges’ opinions reveal a high satisfaction level and a perception of the improvement in courts’ infrastructure situation, mostly within the intervention group. Furthermore, in what concerns the act of justice, judges observe an improvement in the quality of the prosecutors’ body, but less so for the general efficiency of work. 97. Compared to the judges which were part of the control group, the ones in the intervention group stated to a larger extent that they had noticed improvements in what concerns the progress of the act of justice, especially referring to the lawyers’ honesty, where 24% more judges from the intervention group consider that this aspect improved in the past two years. The smallest difference between the two groups (3%) is regarding the time to case disposition. 98. For generating a better work experience, the judges consider it necessary that a larger number of clerks be employed in courts, but also that a regulatory framework be enacted, to facilitate higher efficiency. 99. On average, the judges from the group subject to intervention had 796 cases, and the ones from the control group, 740 cases. Approximately 7% of the cases assigned to the judges from the intervention group and 4% of the cases of the control group judges are at least 6 months old. Efficiency wise, in 2015, judges from the first group disposed 597 cases (75% of the total assigned to them) and the judges from the control group disposed 523 cases (71%). 100. No major differences were observed regarding the number of work hours, the situations of working from home, professional training and adoption of modern technologies. In what concerns the Page 81 of 104 The World Bank Judicial Reform ( P090309 ) access to legislation, jurisprudence or doctrine, there is a difference between the two groups, 56% of the judges from the group subject to intervention saying that access to this type information is easy, compared to 43% in the case of the control group. 101. Compared to the last two years, judges state that they can observe improvements in all aspects regarding the courts’ qualified human resources, the three attributes of clerks being the most mentioned. Differences could be observed between the two groups within the interval 26% (referring to the honor and honesty of clerks) – 29% (in what concerns the clerks’ professionalism). 102. Prosecutors observed improvements in what concerns the courts’ infrastructure and facilities. Moreover, the opinion of the prosecutors from the intervention group was that most aspects have improved considerably. Major percentage differences could be identified between the two categories of respondents, showing higher satisfaction levels for the prosecutors in the intervention group, concerning the changes the courts have gone through. 103. The perception of the act of justice shows that for the good progress of the courts’ activity, as well as for a proper performance of the prosecutors’ activities, the most important aspects are the number of assigned cases, the time to case disposition, and the provisions of the new Civil Code and of the Civil Procedure Code. 104. The last two years brought improvements concerning the transparency of the act of justice, acknowledged by prosecutors from both groups (74% of the group subject to intervention and 67% for the control group) and concerning the time needed for case disposition (56% for the group subject to intervention and 65% for the control group), less so, concerning the number assigned cases. On average, a prosecutor from a court part of the group subject to intervention, had, in 2015, 623 cases while at the same time, a prosecutor from a court which was part of the control group, had on average, 390 cases. 105. Out of active cases, on average, 52 (8%) were older than 6 months from their assignment and 251 (40%) older than 6 months since they entered the system, as reported by the prosecutors in the intervention group. According to the statistics in the control group, out of all the assigned cases, on average 31 (8%) were older than 6 months from their assignment and 74 (19%) were older than 6 months since they entered the system. 106. There were no major differences reported regarding the access to new technologies or the need for more professional training. Instead, perception over the efficiency of the registration and archiving system was higher in the intervention group (93%), compared to the control group (75%). Access to the jurisprudence, legislation and doctrine databases was easier for the intervention sample (59%) compared to the control group (51%). 107. In what concerns the clerks’ perception, it matches the improvement perception trend and that regarding the satisfaction with the work environment. For these employees, the aspects which take precedence are the ones regarding the infrastructure (useful floor area, number of courtrooms, equipment of the employees’ offices), as compared to the specific aspects which regard the act of justice. 108. The act of justice was perceived as satisfying and improving, especially in the intervention sample. With the exception of the time to case disposition (with a higher degree of satisfaction found in the control group) data indicates differences of 2% (concerning the number of clerks) up to 20% (provisions of the new Civil Code/Civil Procedure Code), in favor of the group subject to intervention. Page 82 of 104 The World Bank Judicial Reform ( P090309 ) 109. The average workload of clerks in the intervention group was higher than in the control group. In 2015, clerks in the intervention group had, on average, 29% more cases to prepare (833 cases vs. 645 cases). Furthermore, the clerks in the intervention group had more cases older than 6 months since their registration in the system. 21% of clerks’ cases in the intervention group were older than six months, versus 6% in the control group. 110. No major differences were identified with regard to access to new technologies, but the registering and archiving system is considered to be efficient in a larger proportion (84%) for the intervention group, as compared to the control group (74%). 111. Regarding the way lawyers evaluated the courts’ infrastructure, a main aspect is that regardless of to which group they belong, they most frequently rated the courtrooms’ audio systems, the possibility of researching the archives, as well as the possibility of looking up information on the internet. The aspects which mostly satisfy lawyers are the building solemnity, the exterior aspect of the building, as well as the cleanliness of public spaces. 112. The aspects most frequently considered as satisfactory by lawyers in regards to the act of justice were: the way cases are assigned; the way judges treat litigants in the courtroom and the provisions of the new Civil Code/Civil Procedure Code. 113. Lawyers from the control group were found to have a higher workload, when looking their number of cases. The main reasons for slowing (and sometimes delaying) lawyers’ work have to do with work procedures, long terms and expertise. 114. The attributes used to rate the lawyers’ perception of the quality of personnel (attitude, honesty and professionalism of judges, prosecutors or clerks), were positively evaluated. Differences between the two groups appeared in regards to the higher proportion of those satisfied in the intervention group, as well as vice versa. 115. Private citizen - litigants identify the following hierarchy in what concerns the visibility of the courts’ infrastructure constitutive elements: public waiting spaces, the audio systems in the courtrooms and the possibility of searching for documents in the archives. Litigants stated that they are mostly satisfied with the building aspect, both internal and external, the solemnity of the building, as well as the cleanliness of public spaces. The same as with lawyers, the number of parking spaces is still seen as a problem. 116. In what concerns the act of justice, the most satisfying aspects, from the perspective of private litigants, are the professionalism, the attitude and the honesty of lawyers. The most frequently improved aspects during the last two years, taken into account by the litigants, are the way judges treat litigants in the courtroom, but also the entire set of attributes applicable to lawyers. 117. 15% of the private citizen litigants from the group subject to intervention and 13% of the litigants from the control group intended to file a complaint regarding certain aspects of their courtroom experience. The main reasons for which they intended to do so were the methods of settlement for cases and the time needed for settlement. The ones that haven’t filed their complaints, although they intended to do it, reasoned their choice by the lack of hope that anything would change, the potential deterioration of their relationships with the courts’ employees, and the fact that they didn’t know where to file the complaint or the time needed to do so. Page 83 of 104 The World Bank Judicial Reform ( P090309 ) 118. The most visible elements form the courts, from the perspective of litigants which are representatives of legal entities, were: the public waiting spaces, the possibility of looking up documents in the archive, and the audio system in the courtrooms. The most satisfying aspects regarding the courts’ infrastructure were the internal and external aspect of the building, as well as the buildings’ solemnity. 119. The perception over the act of justice in this case underlines a higher satisfaction with the attitude of the lawyers; the way judges treat litigants within the courtroom and the lawyers’ professionalism. During the last two years, the most evident improvements were made to the way judges treat litigants in court, as well as all the analyzed attributes applicable to lawyers. 120. 15% of the legal entities representatives from the group subject to intervention and 7% of the representatives from the control group intended to file a complaint regarding certain aspects or situations encountered in court. The main reasons for taking these actions are the methods of settlement or the attitude of the personnel. The reasons for which this population did not file their complaints although they intended to, are the fact that they reasoned nothing would change, or that in doing so, they would endanger their relationship with the court. 121. The litigants, irrespective of category, offered a predominantly positive evaluation regarding almost all tested aspects/attributes (attitude, honesty, professionalism) of judges, prosecutors and lawyers. Page 84 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 8. STAKEHOLDER WORKSHOP 1. The World Bank in collaboration with the MOJ organized a stakeholder workshop on March 07, 2017. The main objective of the workshop was to discuss the achievements of the Justice Reform Project. The World Bank team solicited feedback from stakeholders on how the collaboration of the MOJ and the Bank on the Project could have been improved and in what areas the Ministry could make further progress to improve its efficiency. The lists of participants is included in Table 4 below. 2. The workshop program started with opening remarks by the Bank followed by a short presentation delivered by the Ministry of Public Finance. Further, the DIEFP presented the main components and results of the Project delivered and the workshop concluded with an open discussion. This document reflects the main highlights of the presentations and discussion with the stakeholders. 3. The DIEFP emphasized that the project aimed to improve the efficiency and accountability of Romania’s justice system in many areas, but the key outputs of the project that had influenced or would affect improvements in the justice system the most were: court optimization assignment and impact studies for the four codes, development and implementation of audio systems in courts and RMS, upgrades on ICT in the courts, rehabilitation and construction of 18 sites, and training for court personnel. 4. The DIEFP highlighted the context, initial objectives and final results of the Project sharing with the audience the Ministry’s overall satisfaction with the outcomes of the project, emphasizing the civil works in Oradea and Craiova, as well as the IT component. The DIEFP highlighted the important role of the Bank during implementation in providing continuous support and showing flexibility in changing course. 5. The DIEFP delivered a high-level presentation on challenges encountered during the eleven-year project cycle highlighting the financial crisis that hit Romania around 2009 and affected tremendously the budget allocations. Key highlights of discussion 6. The discussion was structured and introduced five questions, as follows: • What worked well? • What could have been done differently? • Do you think the money was well spent? • Has the project impacted the work of courts? And if so how? • What was left undone and what would you focus on in the future? 7. Overall, the workshop participants were supportive of the project and highlighted the good performance of the DIEFP and an excellent cooperation with the Bank which provided continuous support through the entire project cycle. 8. The workshop participants noted a strong preference for civil works as tangible outcomes of the project. The updates and developments regarding the ICT system were appreciated and participants Page 85 of 104 The World Bank Judicial Reform ( P090309 ) showed satisfaction in using ICT systems. However, they highlighted the urgency to complete what remains for further development in order to implement the integrated RMS system to allow for an accurate and transparent decision-making process based on an efficient resource allocation. Participants also requested training and change management on the RMS system to better facilitate the transition and improve the perceptions of the system. 9. The participants also noted that the project would have benefitted from better outcomes if the preparation phase was better developed. The participants discussed they encountered tremendous difficulties in the implementation phase due to a lack of readiness of technical documents to start procurement and implementation, sometimes outdated such as permits. In the opinion of some participants, some of the issues could have been identified and resolved during the preparation phase of the project. 10. The project encountered important delays due to the lack of sufficient budget allocation, which slowed down the implementation process between 2011 and 2015. The participants also highlighted that the “lowest price” method when awarding contracts does not seem to be the most suitable for some products such as the IT software and civil works. Table 2: List of participants invited to the JRP Workshop Name Title Institution Mr. Flavian Popa Prosecutor CSM Ms. Adina Ghita Judge Bucharest tribunal Ms. Elena Lascu President Corabia First Instance Court Mr. Pătru Dumitru Economic Manager Valcea Tribunal Ms. Liliana Oprea Judge Olt Tribunal Ms. Valerica Virtopeanu President Arges Tribunal Ms. Rodica Mihaela Dobrin President Dolj Tribunal Ms. Carmen Domocos President Oradea Tribunal Ms. Paula Ene Economic Manager Prahova Court of Appeal Ms. Lucia Saulea Economic Manager Ploiesti Tribunal Ms. Carmen Ghita Head of Unit MoPF Ms. Elena Heroiu Senior Advisor MoPF Ms. Roxana Badea Senior Advisor MoPF Ms. Cornelia Munteanu Deputy Manager MOJ Mr. Andrei Pana Judicial System Specialist MOJ Mr. Catalin Spiroiu Jurist MOJ Ms. Laura Rodeanu Financial MOJ Ms. Adriana Gheorghiu Procurement Specialist MOJ Mr. Gabriel Clinceanu Technical Manager MOJ Ms. Monica Oana Architect MOJ Mr. Eugen Dirstaru Technical Advisor MOJ Ms. Maria Arsene European Commission, CVM EC Ms. Mona Popescu Liaison Magistrate French Embassy Mr. Cristinel Buzatu Policy Adviser Justice Embassy of the Kingdom of the Page 86 of 104 The World Bank Judicial Reform ( P090309 ) Cooperation Netherlands Mr. Florian Caimac Former Project Manager Mr. Costel Todor Former Project Coordinator Mr. Adrian Gavrilescu Public Manager Public Ministry Mr. Razvan Capatana IT Specialist ICCJ Mr. Cristian Plesa Head of Co-operation and National Administration of Programs Unit Penitentiaries, MOJ Klaus Decker Task Team Leader, Senior WB Public Sector Specialist Georgia Harley Public Sector Specialist WB Agnes Said Consultant WB Elena Ardelean Public Sector Analyst WB Andreea Florescu Team Assistant WB Rob Boone Adviser WB Carmen Calin Procurement Specialist WB Page 87 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 9. SUMMARY OF BORROWER’S ICR Project PDO The project's development objectives are: (i) to increase the efficiency of the Romanian courts; and (ii) to improve the accountability of the judiciary which should result in reduced corruption and a more transparent act of justice. In order to reach the goals of efficiency and accountability, the following areas require support under the project: (i) upgrading of court infrastructure and automation; (ii) court administration reform, including a program of case delay reduction and reorganization of internal working arrangements in courts; and (iii) institution building for the main judicial governing bodies (e.g. CSM, HCCJ and MOJ). PDO Indicators Project Outcome Indicators – 5 Key indicators (restructured during project implementation) to measure success and the PDO achievement are the following: POI 1) Improvements to judicial efficiency in pilot courts; Target 10% increase in the number of cases disposed of or archived in pilot courts. Actual value 18.52 % (cumulative data – 2015 compared to 2008 baseline) Status Achieved – target exceeded POI 2) Audio recordings of court hearings available to any trial participant on request; Target Integrated audio recording systems for court hearings are operational in all national courts (243) and their courtrooms (699). Audio recordings of court hearings are provided on request to any interested trial participant. Actual value Integrated audio recording systems for court hearings are operational in all national courts (243) and their courtrooms (699). Audio recordings of court hearings are provided on request to any interested trial participant Status Achieved as planned POI 3) Rehabilitation/construction of at least 15 out of 18 courthouses in line with international standards (standards for better accommodations for trials, public access and security features); Target International standards in the rehabilitation/ construction of at least 15 court buildings completed. Actual value 16 courthouses are in line with international standards Status Achieved – target exceeded with 1 court building Page 88 of 104 The World Bank Judicial Reform ( P090309 ) POI 4) Information required to increase the capacity of the judicial system available through the establishment of a functioning RMS and such information/data published quarterly by the MOJ, CSM and HCCJ on their websites; Target RMS system installed and functioning and data published quarterly for the past year (since 2012) Actual value Indicator partially achieved for CSM and HCCJ (HR &Payroll module). Underway for MOJ. None of the institutions publish on their websites resource management data originating from the RMS. Status Partially Achieved POI 5) improvements to judicial transparency through the [annual/quarterly] publication on the websites of the respective courts and CSM of: (a) court performance data; (b) data from periodic user surveys of individuals and firms on access to and satisfaction with judicial services (including fairness and timeliness) for courts financed by Part A of the project. Target Regular annual/quarterly publication of available data in the past two years (2015 & 2016) Actual value (a) court performance data – all courts report publicly on their website, on a yearly basis, their annual activity reports, including the value of the 5 related performance indicators (b) data from Interim user survey and Final survey for courts financed by Part A of the project Status Achieved Project IRIs Project Intermediate Result Indicators - 13 indicators (restructured during project implementation) to measure success are the following: PIRI 1) Rehabilitation/construction of at least 15 court buildings; Target At least 15 buildings completed Actual value 16 courthouses were built Status Achieved – target exceeded with 1 court building PIRI 2) New design standards used in rehabilitation and construction of court buildings; Target All courts rehabilitated and constructed under the Project are based on designs in line with international standards. Actual value 16 courthouses were built in line with international standards Status Achieved PIRI 3) 10% increase in a total number of court hearings in 5 courts rehabilitated under the project; Target At least a 10% increase. Actual value 99.72% increase (cumulative data – 2015 compared to 2008 baseline). Status Achieved - target significantly exceeded with over. 89% Page 89 of 104 The World Bank Judicial Reform ( P090309 ) PIRI 4) Recommendations to optimize courts’ activity are adopted by the CSM; Target Recommendations adopted by the CSM Actual value Recommendations were adopted by the CSM in 2013 Status Achieved PIRI 5) Uniform standards for operational processes developed and country-wide implementation started; Target Uniformity in operational (case time handling and optimal workload) standards (for handling court cases) achieved. Actual value 3 individual Key Performance Indicators, namely 1. Clearance Rate, 2. Case Stock Over One Year Old, and 3. Cases Closed Within One Year were elaborated. The country wide implementation of these indicators started with 11 courts of different degrees of jurisdiction and were extended by end 2015 all across the national courts. Status Achieved PIRI 6) An RMS for core justice sector entities enables management of HR and payroll functions; Target Sustained use of the RMS Actual value CSM, NIM, NSC, JI, PM (central apparatus), NAD, DIOCT and HCCJ (core justice sector entities) are currently using the RMS's HR and Payroll functions Status Achieved PIRI 7) An RMS for core justice sector entities enables pilot justice sector entities to collect management information data; Target Sustained use of the RMS Actual value CSM, NIM, NSC, JI, PM (central apparatus), NAD, DIOCT and HCCJ (core justice sector entities) are currently using the RMS's HR and Payroll functions Status Achieved PIRI 8) The Law School Admission Test (LSAT) is formally embedded into magistrates’ recruitment procedures; Target A formal regulation regarding the continuous use of LSAT test for magistrates’ requirement adopted by the NIM. Actual value The formal regulation regarding the continuous use of LSAT test for magistrates’ requirement was adopted by the NIM, through the Decision no 1131/2014 (Art.4, para.( 11 ), starting with 11.12.2014. Status Achieved PIRI 9) Conduct of periodic internal and external surveys of individuals and firms and publication of results; Target 3 surveys conducted and results disseminated. Actual value 3 surveys conducted (Baseline, Intermediate and Final Surveys) and results disseminated. Page 90 of 104 The World Bank Judicial Reform ( P090309 ) Status Achieved PIRI 10) New/amended Criminal Procedure Code and Civil Procedure Code adopted by Parliament and came into force; Target All 4 Codes adopted in the Parliament and enforced Actual value All 4 Codes were adopted in the Parliament and enforced, the last 2 of them, criminal code and criminal procedure code, started the effective enforcement with February 2014. Status Achieved PIRI 11) Impact assessment of four Codes (Civil Code, Civil Procedure Code, Criminal Code and Criminal Procedure Code) completed and accepted by CSM and MOJ; Target TA that would provide impact assessment of 4 Codes implemented Actual value Impact assessments finalized in November 2011. Status Achieved PIRI 12) At least 10 % increase in the number of trainees using the NIM and NSC e-learning platform; Target At least 10 % increase in the number of trainees using the NIM and NSC e-learning platform. Baseline was 277 in 2012. Actual value In 2015 – 655 trainees (increase of 136%) and in 2016 – 1205 (increase of 335%), compared to 277 baseline. Status Achieved and significantly exceeded PIRI 13) At least 50 % increase of the number of council rooms in 10 courts rehabilitated under this project. Target At least 50 % increase of the number of council rooms in 10 courts rehabilitated under this project. Baseline was – 7 council rooms in 10 courts Actual value In 2010, 16 council rooms were in 10 courts. Increase was of 128%. Status Achieved and significantly exceeded Page 91 of 104 The World Bank Judicial Reform ( P090309 ) Project Components The project has four components: (1) Court Infrastructure Rehabilitation (2) Strengthening of the Administrative Capacity of Courts (3) Court Information (IT) Support and (4) Institutional Development of Judicial Institutions. Total amount of the loan is Euro 110 million. Component 1: Court Infrastructure Rehabilitation – The project will support the rehabilitation and limited new construction of prioritized court buildings, and the development of uniform space planning and design standards for court facilities. Rehabilitation and construction will provide improved public access and rationalization of court facilities, structural and environmental upgrading, enhanced security features and technological capacity and plan revisions required for the judiciary to function with efficiency and independence. Court rehabilitation and construction will also mitigate one of the obstacles to the public’s respect for the court system. The project will provide assistance for physical rehabilitation of at least 15 court buildings, but no more than 18 buildings (in which function 21 courts), depending on available funds after finalized procurement procedures. The project will also support the development of design standards that will serve the judicial system in the long term. Component 2: Strengthening of the Administrative Capacity of Courts – The project will assist the Romanian courts to adopt modern administration techniques to increase their productivity, improve the quality of their services and restore confidence in the judiciary. Support for elaboration and implementation of the 4 new legal codes will also be provided, as well as for the new Insolvency Code. Component 3: Court Information (IT) System – The Government’s Information Technology (IT) Strategy, approved in July 2005, anticipated development of a comprehensive RMS for the judicial system. (Operational level case management systems are being implemented under EU financing.) The RMS will cover financial, physical, and human resource management functions for the entire justice sector. It will also cover management support functions, both reporting and analytic. The RMS would support management functions at the level of the individual courts, as well as at the MOJ, CSM, and the HCCJ. The system will operate over the EU/Government funded wide area network and will obtain reporting information from the court-level operational systems. The use of industry standard technologies and interfaces specifications will minimize compatibility issues. The RMS would serve approximately 5,000 users, comprising 20-25 individuals at each of the roughly 200 judicial facilities, and approximately 200 individuals at the MOJ, CSM, and HCCJ. Technical support will be coordinated at the level of the second instance courts. Additionally, during the first JRP restructuring, the funds anticipated as future savings realized during the procurement of the RMS and cancellation of other assignments, will be used to purchasing the long- awaited (by the MOJ) IT systems established for audio recording of the court hearings. Component 4: Institutional Development of Judicial Institutions – This component would provide assistance to the following judicial institutions: CSM - in the area of quality of internal management, development of long-term judicial policies, monitoring judicial performance, public communications; Page 92 of 104 The World Bank Judicial Reform ( P090309 ) MOJ - in the area of quality of internal management and internal and external communications; NIM - in the area of development of new qualification tests for judges’ selection and promotion; development of training courses and curricula; NSC - in the area of strategic planning; development of certain training courses and delivery of training through distance learning programs and other innovative methods; This component would also provide funding, to develop and implement specific tools for monitoring project results, including public surveys and court user surveys. A comprehensive baseline survey will be carried out immediately after loan effectiveness in order to provide a snapshot for the current reality. Two more surveys would be carried out over the project implementation period. Overall Main Achievements The most important JRP outcomes are the substantial reformation of the fundamental legislation impacting both the Romanian society as a whole (New Civil Code and New Criminal Code) and the functioning of all the national judiciary proceedings and trials (New Civil Proceedings Code and New Criminal Proceedings Code). A similar successful radical transformation of these 4 fundamental laws was done 146 years ago, in 1865, by King Charles the first, while other unsuccessful attempts occurred many times during 1940 to 2005. The huge amount of work done consisted in the drafting and correlation process of 4,846 articles incorporated in the new Codes6. Additionally, through a separate consultancy contract implemented in 2010-2011, all the resources needed for effective enforcement of all 4 Codes were assessed and planned according to a gradual 3 years schedule. As a result of all the above, the 4 new Codes were adopted by Romanian Parliament and gradually entered into force in 2011 (Civil Code), in 2013 (Civil Proceedings Code) and in 2014 (Criminal Code and Criminal Proceedings Code). This achievement was fundamental for the national judiciary and its effects for the society will be seen in the next years. Insolvency related national legislation was also reformed through JRP. During 2011 and 2014, through a separate consultancy contract, the Romanian Insolvency legal framework and related mechanisms were strengthened thorough the elaboration of the first national insolvency code, in line with European ROSC standards. Another highly important outcome was the successful implementation of a consultancy assignment by which the management of courts’ judicial activities were improved. Through this consultancy, during 2011- 2013, the optimal volume of work for judges and court clerks was assess and standards for ensuring and monitoring the efficiency of courts’ activity were set as well. Among others, Performance indicators were elaborated and a tracking system was designed for monitoring the case files clearing rate, the percentage of stock cases over 1 year and the percentage of cases closed within 1 year. Capacity building was also a Project top priority. Through 4 technical advisory contracts7 the e-learning activity within National Institute of Magistracy8 and National School of Clerks was enhanced, as the number of magistrates using the new e-learning facilities increased in 2013 with about 56% in comparison with 2012 baseline (277 trainees in 2012, while 639 trainees in 2013). Moreover, as result of the implementation of other 2 consecutive consultancy contracts9, the recruitment process of the young magistrates (judges and prosecutors) started to be conducted more efficient, also based on LSAT type tests (reasoning tests) and Page 93 of 104 The World Bank Judicial Reform ( P090309 ) not only on pure legal knowledge. Thus, during 2010 – 2014, all 870 new future magistrates were recruited based on LSAT admission tests. Additionally, following 2 consultancies implemented between 2009 and 2012, both MOJ and CSM have in place ISO type quality management systems which uniformed and documented all the business flows and administrative documents at their level. IT component also achieved significant outcomes. As a result of it, an integrated IT court session audio recording system was put in place at the level of all 243 national courts and implicitly in 699 court rooms were court sessions were held across Romania in 201310. The system is operational since July 2013 and it is widely seen as a valuable instrument meant to increase the transparency of act of justice, as all the lawyers and any party involved in a certain a trial/court session, can request the audio recording of the court sessions. Moreover, the national Judicial Inspection can now use the recordings to verify the judges’ behavior during court sessions, whenever necessary. In a nutshell, this system contributed to the change of the act of justice and increased the positive attitude of court staff towards all external court users’ categories. Another IT system started to be implemented for management of all judiciary resources (RMS – ERP type). The project of an extremely high complexity is still on going. In case of RMS, the outcomes are so far the following: 1. Human Resources and Payroll module – is functioning (is Live) at the level of CSM, HCCJ and PM, while its extension at MOJ, courts, NAP and all prosecutors’ offices is on-going. This particular module brings the most significant added value to the judiciary institutions, since it manages approx. 35.000 members of judiciary personnel and approx. 90% of the yearly budget of the entire judiciary system, which is represented by payroll costs. 2. Financial module – passed all the functional testes and hence was accepted from a functional perspective by the following beneficiary institutions: MOJ, PM and HCCJ. Testing is on-going for NAP and CSM. 3. Budget module – passed all the functional testes and hence was accepted from a functional perspective by the following beneficiary institutions: MOJ, NAP, CSM and HCCJ. Testing is on-going for PM. 4. ERDR – passed all the functional testes and hence was accepted from a functional perspective by the only beneficiary of this module, NAP. By using the ERDR, all claims of detainees and their families can be more easily managed in the future. Courts infrastructure remains the most costly project component and yet highly important for improving courts’ efficiency. Thus, additional court rooms at the level of national courts will lead, among others, to more cases solved within a year, due to more adequate spaces for judgment. In total, 16 court buildings were finalized, buildings which represent the premises for 21 national courts, out of which: 9 Courts of first instances (CFI), 9 Tribunals (TR) and 3 Courts of Appel (CA). The efficiency and the quality of judicial services delivered at the level of all these 21 courts was increased and details in this regards are presented in the chapters below. Efficiency of Courts The overall rating of efficiency (ORE) of a selected court is calculated by an algorithm part of the IT statistics system (ECRIS STATIS), which takes into account the monitoring and compilation of the first 5 Page 94 of 104 The World Bank Judicial Reform ( P090309 ) efficiency related sub-indicators, as follows: Clarence rate (efficiency), Stock of cases, Cases closed within 1 year, Average time for disposition, Editing exceeding the legal deadlines, Number of court hearings. The contract financed under JRP, related to Optimization of courts activity, significantly contributed to the identification and putting in place country wide, with the involvement of Superior Council of Magistracy, the mentioned ORE monitoring system. The overall efficiency rate of a court, as well as all the 5 above sub-indicators can render 4 different level of efficiency for any court, as follows: 1) Highly efficient, 2) Efficient, 3) Satisfactory, or 4) Inefficient. For all 21 courts, with only 2 exceptions, which were part of the Project, the overall rating of efficiency (ORE) is either Efficient either Highly efficient, according to the statistical information provided for the year 2015. Only CFI Bolintin Vale and Bihor Tribunal has as ORE the rate - Satisfactory. Lessons Learned The following 7 lessons were learnt, related to civil works component: 1. Work on historical (national heritage) monuments is the most complicated and time consuming situation 2. Difficulty to obtain fast building permits due to 1) MOJ internal reasons and 2) MOJ external reasons - national required legal procedures. 3. Technical readiness of implementation arrangements prior to the Loan signing. 4. Capacity of designers, contractors and site supervision engineer are needed. 5. The “true to life” quality of detailed designs, generated in many cases the need for adjustments/changes during implementation. 6. Designers contracted by the local court Beneficiaries, outside the framework of JRP caused in some cases delays in implementation. 7. Yearly budgetary allocation was not always in line with the request of the DIEFP and MOJ, due to financial crises and other MoPF public funds policy related reasons. The following 5 lessons were learnt, related to IT component: 1. IT projects with a single beneficiary institution, despite their complexity, can be successfully implemented far more easily than IT projects with more beneficiary institutions which do not have by law a common leader or single management factor(s). 2. Capacity of IT contractors cannot be maintained throughout all project implementation, unless a specific/tailor made liquidation mechanism is designed from the beginning/contract signing. 3. Supervision management consultants do not bring significant added value when it comes to complex IT projects. 4. When new datacenters are to be created, already existing infrastructure should be leveraged. 5. For complex IT projects, user support should be included in the contract for the duration of the warranty period (typically 3 years). Financial Status at Completion By the end of Project closing date, the amount of EUR 110 mil was committed and the amount of EUR 102.3 mil was disbursed from IBRD funds. The undisbursed amount of EUR.7.7 mil will be paid from government contributioncanceled. Page 95 of 104 The World Bank Judicial Reform ( P090309 ) ANNEX 10. BEFORE AND AFTER PHOTOS Bolintin Vale Court of First Instance – Rehabilitation Before After Blaj - Rehabilitation Page 96 of 104 The World Bank Judicial Reform ( P090309 ) Before After Vatra Dornei Court of First Instance – Rehabilitation and extension Before After Page 97 of 104 The World Bank Judicial Reform ( P090309 ) Suceava Tribunal – Extension of existing building After Saliste Court of First Instance – Rehabilitation and extension Before After Page 98 of 104 The World Bank Judicial Reform ( P090309 ) Pitesti Court of Appeal - Rehabilitation Before After Page 99 of 104 The World Bank Judicial Reform ( P090309 ) Maramures Tribunal – Rehabilitation Before After Arges Tribunal – New building Page 100 of 104 The World Bank Judicial Reform ( P090309 ) Orsova Court of First Instance – New building Page 101 of 104 The World Bank Judicial Reform ( P090309 ) Tulcea Palace of Justice – New building Iasi Palace of Justice – New building Page 102 of 104 The World Bank Judicial Reform ( P090309 ) Page 103 of 104