ICRR 12785 Report Number : ICRR12785 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/15/2007 PROJ ID : P057813 Appraisal Actual Project Name : Judicial Reform US$M ): Project Costs (US$M): 16.2 16.2 Project Country : Georgia Loan /Credit (US$M): Loan/ US$M ): 13.4 13.4 Sector Board : PS Cofinancing (US$M ): US$M): 2.8 2.8 Sector (s): Law and justice (100%) Theme (s): Legal institutions for a market economy (20% - P) Law reform (20% - P) Legal services (20% - P) Access to law and justice (20% - P) Judicial and other dispute resolution mechanisms (20% - P) L/C Number : C3263 Board Approval Date : 06/29/1999 Partners involved : EU/TACIS, Closing Date : 06/30/2003 06/30/2006 Government of Georgia Evaluator : Panel Reviewer : Group Manager : Group : Konstantin Atanesyan Steven Benjamin James Sackey IEGCR Webb 2. Project Objectives and Components: a. Objectives: Assist in the development of an independent and professional judiciary, committed to high standards of judicial ethics and capable of efficient, effective dispute resolution . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project was comprised of six core components and a secondary component covering project management : A. Court Administration and Case Management: (i) training programs, including workshops and study tours for managing judges and court administrators; TA for implementation of the new court administration system; (ii) TA and training for development and implementation of an effective case management system, workshops to facilitate contacts with representatives of developed judicial systems; (iii) TA and training for establishment of an effective case processing and record keeping systems; (iv) computerization, including establishment of internal networks for selected courts, equipment and training for judges and court personnel; establishment of network for the entire court system, and development of an informative website for the court system and its users; (v) audio equipment for selected courts and provision of associated training and TA . Cost: estimated - $3.06m; actual - $3.09m B. Infrastructure Rehabilitation: Development of planning standards and programmatic instructions for new court designs, adaptation and rehabilitation (completion in some cases) of up to 11 court buildings, facilities for the Judicial Training Center (JTC), offices of the Department of Logistical Services (DLS), and minor works for the Ministry of Justice (MOJ). Cost: estimated - $7.00m; actual - $6.83m C. Enforcement of Court Judgments: TA, training and equipment for the new MOJ Department for enforcement of court decisions (DECD). Cost: estimated - $0.23m; actual - $0.19m D. Assistance to the Ministry of Justice : TA and equipment to strengthen legal drafting and harmonization function of the MOJ. Cost: estimated - $0.28m; actual - $0.04m E. Judicial Training Center: Development of methodology for training of court personnel, organization of training programs, including workshops and study tours; training for trainers . Cost: estimated - $0.80m; actual - $0.53m F. Public Information/Education: TA to launch comprehensive public information and education campaign to raise appreciation among population about the importance of independent judiciary and the judicial reform; development of local capacity to carry out effective communications about the legal and judicial systems; development of baseline study to assess the degree of trust with the judicial system by users; regular monitoring and feedback to the judiciary; seminars, workshops, training, production of TV and radio programs . Cost: estimated - $1.38m; actual - $1.60m G. Project Management. Cost: estimated - $0.65m; actual - $1.08m The original components of the project were not revised formally . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: 1. Closing date of the project was extended three times from the originally planned date of June 30 2003 to June 30 2006. The reasons were: (i) implementation delays and design changes in Components A (infrastructure) and B (case management); (ii) lower financial capacity and reform commitment of previous administration (2002-2003) and (iii) disruption following the Rose Revolution (2003). 2. There were some reallocations of project funds between components and disbursement categories during the life of the project. The contingency of SDR 1.0m and savings of around SDR 0.4m from the allocation for goods purchases were reallocated to cover : (i) a 10 percent increase in the allocation for civil works, allowing to rehabilitate 15 instead of the intended 11 court houses - on top of savings achieved by the implementing agencies; (ii) doubling of operating costs due to the 3-year extension of the project; and (iii) a 25 percent increase in spending on consultant services, training and workshops across most components . 3. Project was co-financed by EU/TACIS - $1.0m 3. Relevance of Objectives & Design: 1. The objective of the project - "assist in the development of an independent and professional judiciary, committed to high standards of judicial ethics and capable of efficient, effective dispute resolution " - was overall relevant in the context of the country's development needs and the Bank's strategy, but also overly broad . The ICR notes that the objective could have been narrowed down to focus more on the aspects of improving capacity, effectiveness and efficiency of the judiciary. The planned and implemented activities were overall relevant to FY 04-06 CAS objective of improving governance; as well as the FY 06-09 CPS goals of (i) enabling greater private investment and growth through addressing corruption and governance issues; and (ii) strengthening the country's public sector capacity for good, clean economic management, policy and investment by modernizing its judiciary . 2. The design of the project was somewhat unrealistic as it overestimated political commitment on behalf of all counterpart government agencies . In retrospect, the existence of multiple counterparts / beneficiaries of the project, including two separate branches of government (the executive and the judiciary ) complicated coordination of various components' implementation. 3. The project lacked a sound results framework, key performance indicators were hard to measure, baseline data were generally absent. No time-bound targets for outcome levels were identified . Objectives of particular components were often confused with outputs . Although a logical causal chain between Bank funding and intended outcomes can be constructed in principle, there is little evidence to establish the link clearly . This, in turn, complicated attribution of project-supported interventions to intended outcomes . 4. Achievement of Objectives (Efficacy): Project had one objective: "assisting in the development of an independent and professional judiciary, committed to high standards of judicial ethics and capable of efficient, effective dispute resolution ". For the purpose of this Review, the overall objective is broken into four sub-objectives: 1) Improving independence of the judiciary . The project did provide tools for improving operational performance of courts, but it is not clear how the project -financed activities contributed or were supposed to contribute to improving independence of the judiciary . Achievement: Negligible. 2) Improving professionalism of the judiciary . Project activities invested heavily in rehabilitating court buildings, providing training, equipment and technical assistance to judges and court staff . However, it remains unclear whether there was any impact on professionalism of the judiciary, as no measurable indicators are provided in this regard . Components A (Court Administration and Case Management ), D (Assistance to the MOJ on legal drafting ) and E (Judicial Training Center, JTC) provided training, some hardware and technical assistance aimed at improving the level of professionalism. Of these three components, only JTC produced tangible outputs, its trainings became systematic and the Center itself had been institutionalized into state system as High School of Justice, supported by state budget. As for Components A and D: (i) despite receiving extensive TA and hardware from the Bank, the Government decided not to implement the new court administration and case management systems; and (ii) activities designed to assist the MOJ in legal drafting were eventually dropped due to inability of the client to decide on specific areas of assistance. Although the 2005 EBRD-World Bank Business Environment and Enterprise Performance Survey (BEEPS) shows increase in confidence of firms in the legal system, there is no clear link to the capacity building efforts supported by the project . Achievement: Modest. 3) Raising standards of judicial ethics . Project supported extensive training for judges through the JTC, but it is not clear whether the project outputs (number of judges trained) had any impact on achieving the objective of better judicial ethics standards. Component F (Public Information/Education) supported the Association for Legal and Public Education (ALPE) - an NGO that coordinated mass media campaign, seminars, workshops and training for journalists aiming to (i) improve public knowledge of the importance of an independent and competent judiciary and (ii) educate the public about the use of the legal system . It can be inferred that a better educated public, aware of its rights and the use of the legal system can potentially put pressure on the judiciary to adhere to higher standards of ethics. The 2005 BEEPS survey shows slight improvement in the firms perception of courts' honesty, fairness and impartiality. Achievement: Modest. 4) Development of efficient and effective dispute resolution capacity . Most of the project resources (Component B) financed construction and rehabilitation of fifteen buildings (courts and other facilities). ICR mentions only one indicator of improved efficiency (violations for cases pending beyond the procedural limits dropped by 23 per cent in the rehabilitated courthouses, as opposed to 5 per cent in others). 2005 BEEPS survey did show improved confidence of firms in the legal system, e .g. ability to enforce decisions, fairness and impartiality, honesty - although fewer firms thought that the courts are affordable and quick . However, increased confidence in courts cannot be directly associated with reconstruction of court infrastructure . Activities under Components A, C, D and E provided inputs to building capacity in the judiciary in form of training, study tours, workshops and seminars, as well as equipment - with little evidence of results relevant to the achievement of project objective . Achievement: Negligible. 5. Efficiency (not applicable to DPLs): No available measures of efficiency (ERR, cost-effectiveness). A substantial part of the project resources (almost 50 percent) was spent on court infrastructure rehabilitation . However, there is not enough evidence that the amount of resources spent was justified in terms of benefits achieved and relevance to the overall project objective . Some savings were achieved in the civil works component, allowing for rehabilitation of additional buildings (fifteen instead of eleven). On several other components (A, D) resources were spent, but objectives not achieved . Components G and F required additional resources due to extensions of the project . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The overall outcome is rated moderately unsatisfactory . The project produced significant outputs - a number of court buildings were rehabilitated, equipment was supplied to the courts, many judges and staff received training, substantial technical assistance was provided -- but data were not collected to show outcomes . Some targets - establishment of new court administration and case management systems were not achieved . Others, such as improved public awareness of the functioning of the judicial system and increased trust in the judicial system were achieved partially. However, it is not clear whether the overarching objective of a more professional and independent judiciary was actually achieved in an efficient manner, commensurate with the level of Bank's financing . Although various data sources (BEEPS, CPIA, Index of Economic Freedom, Transparency International Corruption Perception Index, WBI Governance Indicators ) confirm that Georgia has made visible progress in the area of governance, it is more likely that improvements reflect positive changes in the overall political climate and rule of law, following regime change after the Rose Revolution - a development hardly related to the Bank's project . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Success in legal and judicial reform is highly dependent on the political climate in the country and changes therein . In a politically unstable and relatively volatile environment, without well -rooted traditions of separation of powers and independent and competent judiciary - typical for many transition economies, the risks for sustaining the momentum are rated as significant. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: The project focused on an important aspect of the country's development agenda and was prepared in response to the initial positive steps undertaken by the Government in that area . Design was overly broad, which eventually led to inefficiencies. Monitoring and evaluation arrangements had serious shortcomings : there was almost no baseline information, key outcome indicators were poorly designed . Technical and financial aspects were covered properly. Supervision of the project was intense and well -coordinated within the Bank. However, some of the design deficiencies, identified by QAG at QSA 5 in October 2002 (lack of measurable outcome indicators ) were not addressed. Also, the project objective could have been narrowed down to be more realistic, and some of the components could have been formally revised earlier in the life of the project . a. Ensuring Quality -at-at -Entry :Moderately Unsatisfactory b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government performance is rated moderately satisfactory . Despite overall instability, frequent shifts in priorities and ensuing lack of clarity in reform implementation, and continuous problems with counterpart funding (especially in the early years ), the project was in general supported by the government, especially at its last stages, following the Rose Revolution . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E design, implementation and utilization was visibly the weakest part of the project . No changes were made to the overall M&E framework throughout the life of the project, despite explicit recommendations made in this regard by QAG at QSA5 in 2002. Desired outcomes lacked baseline informatio n, had very few measurable indicators of progress - hence the difficulty to relate outputs to outcomes . The ICR noted that the only exception is "the baseline public opinion study undertaken by an external consultant in 1998 and followed up in 2002, 2004 and 2006" (ICR, p.9), but it there is no mention of the results of that survey later in the document . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): According to ICR, the project received unqualified audits throughout its implementation . According to the ICR, there was an issue of Uses and Sources of Funds and SOE withdrawal figures not matching those of the Bank's Client Connection. The ICR notes that the issue was resolved through discussions with PIU Director . However, the reason for the discrepancy is not described, nor does the ICR explain how a discussion could have solved the problem, and the nature of the agreements reached . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately See Section 6 Satisfactory Unsatisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: 1. Outcomes of projects with overly broad objectives, not equipped with measurable performance indicators, can be difficult to relate to project outputs . 2. In cases when M&E system was not properly designed at the appraisal stage, failure to take corrective steps during implementation undermines eventual relevance of the project . 3. Failure to revise project objectives and components, due to changes in country circumstances during the course of the loan, is likely to result in inefficiencies . 4. Improvements in overall governance environment due to unforeseen positive political externalities (e.g. changes in government, revolutions leading to stronger ownership of reforms and commitment to change ) can be erroneously attributed to unrelated lending activities . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is informative and candid discussing some of the deficiencies of project design (e.g. M&E), as well as overall performance of the Bank and the Government . It could have been more concise . Discussion of project outcomes is hampered by poor quality or absence of meaningful performance indicators in the PAD . a.Quality of ICR Rating : Satisfactory