Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) Report Number : ICRR0020124 1. Project Data Project ID Project Name P099201 JUDICIAL MOD Country Practice Area(Lead) Additional Financing Azerbaijan Governance P125741,P125741 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-80680,IDA-42090,IDA-49610,TF-56731 31-Dec-2011 55,000,000.00 Bank Approval Date Closing Date (Actual) 29-Jun-2006 31-Dec-2014 IBRD/IDA (USD) Grants (USD) Original Commitment 21,600,000.00 0.00 Revised Commitment 55,000,000.00 0.00 Actual 56,102,149.99 0.00 Sector(s) Law and Justice(100%) Theme(s) Judicial and other dispute resolution mechanisms(50%):Access to law and justice(25%):Legal services(25%) Prepared by Reviewed by ICR Review Coordinator Group Christopher Brian Mahony Clay Wescott Lourdes N. Pagaran IEGEC (Unit 1) 2. Project Objectives and Components a. Objectives The objective of the Project is "to develop capacity of the Borrower's judicial system to achieve incremental improvements in efficiency, citizen information and its ability to handle future demand” (Project Financing Agreement 2006, p. 5). There is a similar statement on the Project Appraisal Document (PAD, p. 10). b. Were the project objectives/key associated outcome targets revised during implementation? Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) No c. Components There were four original components: Component 1: Strengthening the Management Capacity of Judicial Institutions (total at appraisal US$ 4.47 million, of which International Development Association (IDA): US$ 2.47 million and Policy and Human Resource Development (Japanese Trust Fund – PHRD): US$ 1.00 million at appraisal) to finance the following: (i) Development and installation of an integrated management decision making and planned processes system to enhancing operational effectiveness; (ii) Design and implementation of an information network between the Judicial Legal Council (JLC) and other public institutions for communication and coordination; (iii) Ministry of Justice (MOJ) and JLC court system modernization, including evaluation of judges; and (iv) Judicial Modernization Team (JMT) Capacity building for project implementation, including planning and supporting of stakeholder participation. Component 2: Upgrading Court Facilities (total US$ 19.63 million, of which IDA: US$ 11.3 million and PHRD: US$ 1.00 million): to finance the following: (i) Upgrade of courthouses; and (ii) Upgrade of court technologies. Component 3: Human Capital - Strengthening Professionalism of Judges and Staff (total US$ 6.7 million, of which IDA: US$ 4 million and PHRD: US$ 0.5 million): to finance (i) Capacity building for training.; (ii) Development of professional skills and capabilities of new and existing judicial sector personnel; (iii) Preparation and facilitation of annual judicial conferences in order to evaluate impacts, exchange views and share experiences for further learning. Component 4: Improving Citizen Information, including strengthening of registries and notaries (total US$ 4.4 million, of which IDA: US$ 3.4 million and PHRD: US$ 0.5million): (i) Identification of target groups’ information needs; pilot citizen information program development, including information campaigns re: components 1-3; and (ii) Upgrade of the registries’ and notaries’ functions. The original components were revised and a fifth component (Project Management) added. The key revised components and costs are: Component 1: Strengthening the Management Capacity of Judicial Institutions (increased from US$ 2.47 million to US$ 10.5 million). • 1) Provision of ICT and other technical equipment to the JLC, pilot courts and other justice institutions for the creation of automated case and document management system and an information network was added. • 2) Capacity building for judges and staff through the provision of training on ICT systems was added. • (3) Activities were expanded to include court performance management. Component 2: Upgrading of Court Facilities and Technologies (increased by nearly 300 percent from US$ 11.3 million to US$ 37.02 million.) The development of an Infrastructure Plan was dropped. The 2013 restructuring changed the name of the component to Court Infrastructure and ICT. Component 3: Improve Quality of Operations of Judges, Staff and Other Legal Professionals (justice personnel) (increased from US$ 4 million to US$ 4.6 million.) Activities to support the judicial academy were dropped. Added components included: • 1) Carrying out diagnostic work related to improvement of quality of legal services. • 2) Conducting a case file analysis and developing courseware (software) for the Judicial Academy. Component 4: Improving Citizen Information and Access to Justice (increased from US$ 3.4 million to US$ 3.49 million) A cluster of new, mostly analytical, activities were added, including: analysis of the legal aid system; analysis of the market for legal services; the development of the second phase of the e-notary system; analysis of court dispute resolution and alternative dispute resolution mechanisms; analysis of the court enforcement system; and carrying out outreach activities for public awareness. Component 5: Project Management (cost originally estimated US$ 1.9 million, increased to US$ 3.96 million) Strengthening the capacity of the JMT in management, monitoring and coordination including project audit, through the provision of goods, consultants' services, and training, and financing of incremental operating costs. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project approval was 06/29/2006, effectiveness was 12/29/2006, and the mid-term review was 10/18/2011. Two project extensions/restructurings occurred in May 2011 and June 2013. The first, a level 2 restructuring, extended the closing date, refined M&E targets, and provided Additional Financing. The second (level 1) restructuring extended the project and re-allocated and provided some additional financing. The closing date was December 2011 in the original design. It was extended for the original project activities until June, 2013, and for the new activities funded by the additional financing to December 2014. The appraised cost was $21.6 million. Bank financing increased under the 2011 restructuring, with an additional IDA credit of US$9.2 million and an IBRD loan of US$24.2 million, to $55 million. The PHRD provided $3 million in co-financing. The Government of Azerbaijan (GOA) provided initial counterpart financing of $11 million and an additional $39 million after the restructuring, bringing total cost to $96 million. 3. Relevance of Objectives & Design a. Relevance of Objectives The objectives relating to incremental gains in justice sector efficiency, citizen information, and capacity to handle future demand were broadly aligned with the Bank's Country Partnership Framework FY 2016-20 to support the strategic goal of rebalancing the roles of the public and private sectors, with the former becoming a facilitator rather than the main driver of growth (CPF, p. i.). The efficiency and capacity related objectives were consistent with the strategic goal of strengthening governance through better management of public resources, delivery of services, and institutions, including a more transparent and rules-based relationship between the public and the private sectors (CPF p. ii). Azerbaijan’s State Program on Poverty Reduction and Economic Development 2003-2005 had the objectives of increasing legal awareness and access to legal institutions by the poor, improved performance of justice institutions, improvement of execution of court rulings and the independence of judges. The 2005 Presidential Decree No. 352 on Modernization of Court System gave a long-term reform vision to achieve these objectives. Rating Substantial b. Relevance of Design There was a clear statement of objectives; however, the components and intermediate outcomes were tenuously linked to the objectives. The link between upgrading court facilities and efficiency is tenuous if there are not enough efforts to improve quality and professionalism. Increased attention to enhancing efficiency and capacity to handle future demand might have informed a closer design focus on quality and professionalism of judicial sector personnel, rather than providing court houses for increased demand that may not materialize. Construction cost increases might have had less impact had the design better ensured that the project schedule was adhered to, thus allowing construction to be undertaken before costs increased. Rating Modest 4. Achievement of Objectives (Efficacy) Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) PHREVISEDTBL Objective 1 Objective Assist the borrower’s judicial system to achieve incremental improvements in efficiency. Rationale Upgrading of court facilities was achieved, with 11 new courts completed and functioning, meeting the revised 2011 target, though below the original target. 81% of new judges were selected using new procedures in the first instance court, exceeding the refined target of 75%. Information technology (IT) training was provided, but there was no substantive or formalized training provided on the judicial process (ICR, p. 77). As a measure of efficiency, there was no improvement in the clearance rate. The planned resource management system was never introduced, and the reduction in time of court decision enforcement in Baku was not achieved. Rating Modest PHREVISEDTBL Objective 2 Objective Assist the borrower’s judicial system to achieve incremental improvements in citizen information Rationale Key elements of the original plan, particularly information campaigns and strengthening of registries and notaries, advanced the sub-objective of incremental improvements in citizen information. This work targeted the broader population and contributed to citizen information relating to judicial system improvements that may drive increased judicial system engagement. One indicator, to establish at least 3 functioning regional legal information centers, was not achieved, with only 2 centers operated between 2011-13. Furthermore, the activities of establishing information centers and legal aid offices were extended only to those persons already seeking legal aid from the judicial system rather than, as originally proposed, engaging with the public as a whole to encourage greater use of the justice system. Under the project restructuring the relevant components were re-oriented towards those already using the judicial system, diminishing the target audience for information campaigns and the impact on citizens. The only baseline indicator survey conducted targeted judicial system users, not the broader citizenry rendering the observation of change and the evaluation of the objective very difficult. No useful baseline survey existed and the survey taken considered whether users believed in the trial system, which saw a 20% increase. The outcome is non-evaluable (ICR, p.78). However, the team pointed out to IEG that Citizen information on judicial processes has improved, with publication of information on the website and 24/7 live web streaming of civil works to enhance accountability and transparency. Rating Modest PHREVISEDTBL Objective 3 Objective Assist the borrower’s judicial system to achieve incremental improvements in capacity to handle future demand. Rationale A critical indicator for increased capacity is judicial training to increase professional performance. 72% of staff was provided training. Although the EU was funding some of the training needs of the Judiciary, broad and comprehensive training, as required in the PAD, did not occur. (ICR, p. 34, 77). Despite these shortcomings, at least 75% of new judges were selected using new, more objective procedures. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) Rating Modest 5. Efficiency Economic and financial rates of return were not assessed as part of project preparation, or in the ICR. The team provided IEG the economic and financial analysis prepared by the JMT which was not included in the ICR. According to this analysis, there was a negative financial return of -10.4%, and negative net present value of -$65 million. The JMT analysis asserts that when indirect benefits are factored in, there was a positive net present value of $100 million, but the methodology on estimating multiplier effects and attribution to the project is not clear. The project produced fewer benefits and did not deliver on key outputs that the PAD anticipated at approximately three times the initial estimated cost. Design and implementation issues accompanying inadequate M&E and revision (or dropping) of activities deepened the opacity of the project’s financial and economic impact. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives was substantial and design was modest. The causal chain was tenuous between the components and the expected outcomes. The sub-objectives relating to efficiency, capacity to handle future demand, and citizen information are rated modest. Government engagement was initially not deep enough, and was never at the same high level across all areas. Efficiency is rated negligible due to the much higher than estimated cost of the project in comparison to the modest net benefits. a. Outcome Rating Unsatisfactory 7. Rationale for Risk to Development Outcome Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) The level of risk varies across the sub-objectives. There have been signs of commitment, including the 2005 Presidential Decree on Modernization of Court System that　 provided a long-term reform vision. However, the statements of support weren't always followed up by effective actions to implement the vision. The recent 2014 Presidential Decree mandating further modernization of the court system is a promising step forward, with support from the Bank's follow-on Judicial Services and Smart Infrastructure project. The team informed IEG that citizen information on judicial processes has improved under this project, with publication of information on the website and 24/7 live web streaming of civil works to enhance accountability and transparency. a. Risk to Development Outcome Rating Negligible 8. Assessment of Bank Performance a. Quality-at-Entry The project was prepared quickly and led to inexact cost estimations, and insufficient understanding by government of project requirements. There was scarcity of analytic foundation, and limited engagement with key stakeholders. The project design was wide in scope, contributing to a tenuous causal chain between the project’s original components and expected outcomes. The design did not account for the low level of government commitment, causing project design revision that negatively affected the link between activities and project development objectives, particularly the objective of incremental gains in justice sector citizen information. The hurried design also diluted project capacity to adequately assess, manage and mitigate project risks including managing unrealistic expectations, potential inflation and accompanying rise in construction costs, selection of pilot courts etc. Recommendations from the project concept note and decision meeting, including to orient the project more towards quality rather than efficiency, as well as to Quality review at entry were not accounted for. These problems were exaggerated by the absence of a task team leader (TTL) at entry. An engaged TTL might have sensitized the government as to the developmental benefits of the project, foreseen the subsequent government preference for project restructuring and triggered that process earlier. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision The ICR notes significant flaws in supervision, including no supervision during the first 10 months. During the subsequent 18 months supervision focused on the Quality Assurance Group review, as well as project staffing, activities, procurement plans, GOA commitment clarification and engagement, and preparation for restructuring. However, the 2011 restructuring occurred prior to the Mid-term Review and without revisiting the PDO and outcomes, despite the scarcity of analysis during preparation and lack of forceful GOA commitment. A restructuring informed by, rather than in spite of, a Mid-term Review might have led to a revised (and empirically-informed) PDO and deepened GOA engagement and commitment. Supervision refrained from reconsidering the PDO, despite reorientation of the project away from a key sub-objective: citizen information. Supervision over shifting GOA commitment and revised project activities was further complicated by the high turnover of project TTLs, diminishing Bank leverage with the GOA to advance the PDO. Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. Assessment of Borrower Performance Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) a. Government Performance GOA support for this project developed as understanding increased on the nature of project focus and activities. Despite a 2005 Presidential decree ostensibly indicating GOA support for the project, the GOA appeared to misunderstand the project's scope and activities for achievement of the PDO. To some extent this misunderstanding is a consequence of inadequate Bank engagement and communication as part of a hurried design effort. After the Bank-GOA engagement improved, the GOA-preferred approach that prioritized building of judicial sector infrastructure was adopted. The majority of technical assistance reports and recommendations were provided close to the end of the project, diminishing the scope for GOA response as to how it might proceed. GOA implementation throughout demonstrated lower levels of commitment to professionalization of judicial sector personnel and to achieving the sub-objective of citizen information, both of which were pursued sparingly and unsustainably. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The Judicial Modernization Team in collaboration with the Registry Department within the Ministry of Justice and collaborative judges engaged constructively after Bank leadership of the project was established. Government implementing actors became particularly engaged after the project restructuring when project focus shifted to activities more oriented towards building infrastructure. Implementing agency focus advanced increased judicial sector infrastructure gains and efficiency but did not necessarily advance personnel efficiency or other softer reforms including outreach and awareness raising that would advance citizen information. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The rushed nature of project design affected design of M&E. Indicators did not all have clear baselines and targets. The intermediate indicators included staff accessing and usage of IT systems (20% reduction in Baku court decision enforcement time); increased percentage of judicial staff trained in modern methods, court ICT plan and equipment development; and 20% satisfaction improvement in two or more pilot locations. These indicators demonstrate modest efficiency gains. Including the change in number of backlogged cases as a supplementary indicator may have better indicated efficiency gains. The design’s unclear outcome and intermediate indicators conflated indicators with objectives and the means of achieving those objectives (for example improved performance and improved availability of training) rather than material and measurable effect on specified numbers, timing and location against explicitly outlined baseline indicators. b. M&E Implementation Ambiguous indicators undermined ease of M&E implementation. These design flaws were inadequately addressed during implementation. For some indicators, the Bank and the MOJ only recognized the need for baseline data in 2010, but still did not undertake the analysis other than on one indicator for user satisfaction. The absence of baseline data, for example on clearance rates, or the continued use of indicators for activities that were scarcely engaged in or excluded during the two restructurings, indicated that poor design was compounded by disengaged implementation. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) At the 2011 restructuring, PDO indicators were refined, with accompanying ICR-cited justifications. This refinement measured the access to judicial training, rather than the quality of training. The revised final indicator sets a lower bar for citizen information by focusing on availability and access to information rather than public awareness (the original indicator) which indicates awareness on the part of the entire public. Citizen information can reasonably be interpreted to relate to all citizens, rather than only those citizens already in contact with the justice system.. The revised user confidence and percentage of judges selected using objective appointment procedures logically tie together end user-driven increases in demand with capacity to handle increases. Some of the intermediary indicators indicate incremental improvement. c. M&E Utilization Progress Reports were not provided until March 2013, after the 2013 restructuring. The reports that were provided did not cover the whole results framework and did not consider performance throughout the duration of the project. Because of the late provision of the results, they could not be used to make significant changes to the project. M&E Quality Rating Negligible 11. Other Issues a. Safeguards The category B status triggered the requirement of Environmental Management Plans that were to precede selection of sites for courthouse rehabilitation or reconstruction. Throughout the project, the plans were not carried out prior to site selection. The JMT did assign staff members to day-to-day supervision of project environmental compliance monitoring and reporting. One episode of displacement occurred not of significant scale. An appropriate Resettlement Policy Framework was employed. An Environmental Management Plan specific to Physical Cultural Resources was not developed until 2013. Fortunately, no cultural resources were found at construction locations. b. Fiduciary Compliance Fiduciary compliance at entry met minimal IDA requirements and risk was rated moderate. Appropriate procurement procedures and internal controls were put in place. Financial management variance throughout the project was driven by over-commitment and discrepancies in some contract protocols. Throughout and despite supervision, and Financial Management Supervision Reports, gaps were identified in financial management. This included significant over-commitment ($US8.8 million in April 2012, for example, and US$3.5 million finally). Key implementation documentation lagged behind real progress, and there were changes to contracts without Bank agreement and appropriate accompanying amendments. The project significantly overspent in some components/categories. As these risks rose, mitigation measures were introduced. Similar discrepancies were encountered in procurement according to the ICR. c. Unintended impacts (Positive or Negative) - d. Other - Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Unsatisfactory Unsatisfactory --- Risk to Development Outcome Negligible Negligible --- Major shortcomings due to poor Bank Performance Moderately Unsatisfactory Unsatisfactory planning, design, and supervision Significant shortcomings in cooperation on citizen Borrower Performance Moderately Satisfactory Moderately Unsatisfactory information and training components Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Lessons in the ICR are supported, including the need to explicitly acknowledge and address the risks of hurried and elastic project design and its impact on a project's empirical basis, government support, and efficacy. The ICR rightly identifies Bank comparative advantage of smart court infrastructure that drives efficiencies and capacity to handle future demand. A lesson for future Bank engagement is that increasing client public sector capacity must be matched with authentic activities to drive increased demand via Bank comparative advantage in citizen engagement and citizen information. This is particularly applicable for Bank engagement in the justice sector (See World Bank, 2014, Strategic framework for mainstreaming citizen engagement in World Bank Group operations: engaging with citizens for improved results; World Bank, 2011. World Development Report 2011). Another lesson is that pursuit of project components that lack government buy-in, often drives inadequate funding for those components, and may undermine implementation of other project components due to component interdependence. A further lesson for Bank practice in the justice sector is that government commitment may be particularly precarious around areas of justice sector practice that confront executive power. Clarity of government commitment and of Bank intention for project components should be made clear prior to project finalization. 14. Assessment Recommended? Yes Please explain Assessment of a number of recently closed judicial reform projects would be helpful for better understanding pitfalls to avoid, and good practices to take forward. In the case of this particular operation, an assessment would help to analyze the political economy behind expediting the project's design and entry and decisions, and dispensing with serious preceding analytics. An assessment could consider what minimum levels of analysis, client consultation and sensitization, and specificity and selectivity of design　are necessary under situations of significant time constraint. An assessment may also consider specific requirements for implementation where time constraints inform the level of analytics and government consultation preceding project entry. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review JUDICIAL MOD(P099201) 15. Comments on Quality of ICR The ICR provides empirically supported and candid analysis. It is consistent with Bank guidelines, concise and demonstrates consistency. A logical and causal link between the presented evidence and the analysis informs the lessons presented. Where evidence is unavailable, the ICR explains the reasons for data absence. The Bank misdiagnosed the actual strength of initial Government commitment and planning for activities directed towards personnel professionalization and broad citizen information causing poor performance in this area. The ICR does not consider the potential of rigorous evidence-based analysis to procure government commitment to all sub-objectives and project activities. The ICR presentation suffers from few typos (for example, in para. 80, "JMT" is used for "JMP", and in para. 81, the first sentence should read: "At times though, the focus was uneven, particularly in areas where commitment of GOA was weak."). a. Quality of ICR Rating Substantial