ICRR 12539 Report Number : ICRR12539 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/15/2006 PROJ ID :P008831 Appraisal Actual Project Name :Legal Reform Project Project Costs 104.47 63.30 US$M ) (US$M) Country :Russian Federation Loan /Credit (US$M) Loan/ US$M ) 57.95 47.0 Sector (s):Board: ): US$M ) PS - Law and justice Cofinancing (US$M) (65%), Tertiary education (35%) L/C Number :L4035 FY ) Board Approval (FY) 96 Partners involved : Closing Date 12/31/2000 12/31/2005 Evaluator : Panel Reviewer : Division Manager : Division : Gita Gopal Chad Leechor Lily L. Chu IEGCR 2. Project Objectives and Components a. Objectives The project objective was to improve the performance of the Russian legal system in areas key to the effective functioning of market institutions . Specific objectives were to: (a) improve the quality of drafting of economic laws, decrees and regulations; (b) assist the Government of Russia (GOR) in designing the appropriate classification of legislation, and disseminate legal information; (c) introduce innovative teaching methods and materials in the legal education system, and improve public understanding of the role of law in a market economy; and (d) improve the quality of judicial training for the Courts of General Jurisdiction and Commercial Courts, analyze impediments to effective court administration and case management, develop a consensus for necessary structural reform, and promote the development of alternative dispute resolution (ADR). b. Components (or Key Conditions in the case of Adjustment Loans ): The project components included : (1) Legal Drafting Component (US$20 million) comprising technical assistance and training to : (a) Assist in the drafting of economic laws, decrees and regulations at the federal level - on the basis of annual priorities established by the GOR and agreed with the Bank; (b) Analyze and improve the mechanisms to coordinate the legislative drafting processes across federal agencies and between federal and regional authorities; (c) Improve legislative drafting in two pilot oblasts (Sverdlovsk and Orel); and (d) Prepare specific legislation at regional levels – on the basis of annual priorities communicated to the Bank by the federal authorities. (2) Legal Information Component (US$24 million) comprising technical assistance and information technologies to : (a) Develop legislative classification and coding guidelines; and (b) Design and pilot a public sector legal information system . (3) Legal Education and Public Education Component (US$24 million) comprising technical assistance, competitively-awarded faculty grants and graduate fellowships, materials, information technologies, and public information services to: (a) Produce innovative courses and teaching materials; introduce innovative teaching methods; and upgrade available teaching and research resources; and (b) Improve public outreach and journalist training, improve public secondary education on legal matters . (4) Judicial Reform and Alternative Dispute Resolution Component (US$10 million) comprising technical assistance, training, workshops, seminars, pedagogical equipment, materials, and information technologies to : (a) Train judges on legal matters and judicial practices; strengthen pedagogical resources of the MOJ ’s Law Academy and the Training Department of the High Commercial Court; (b) Assess court administration and case management practices in the Courts of General Jurisdiction; (c) Build consensus and support for judicial reform among the judiciary; and (d) Develop a cadre of arbiters and mediators for alternative dispute resolution . (5) Other High Priority Activities Component (US$3 million) comprising inputs as required to meet emerging priorities and needs in the reform of the legal system – as would be agreed between the Borrower and the Bank . (6) Project Management Component (US$6 million) comprising technical assistance, small works, equipment, information technologies, and operational costs (on a declining basis) to ensure effective implementation of the project. c. Comments on Project Cost, Financing, Borrower Contribution, and Dates The ICR notes that " . . . the appraised and actual costs of the components diverge substantially . Similarly, the appraised four-year implementation period substantially diverges from the 9.5 years of actual implementation (1996-2005)." The ICR explains that " . . . in part, these divergences reflect the rapid development of a large project, without the benefit of an extended period of country -specific sector engagement and analysis . In part, they reflect the rather poorly understood mechanics of institutional development (especially in legal and judicial systems ). Finally, they also reflect the project ’s opportunistic/introductory interventions in broad areas, each of which could productively absorb many multiples of the loan amount . Overall, the mixed performance of the project led to cancellations of approximately US$8.7 million of the US$58 million original loan amount during implementation, of which an additional US$2.3 million remained undisbursed at closing and was subsequently cancelled ." 3. Relevance of Objectives & Design : Overall, the project goals were relevant, although the combination of objectives made the project objectives highly complex and ambitious. The project was conceived and decisions were made in a span of five months "concept review to decision meeting in 5 months" (ICR, pg. 5). Although there was some consultation, a period of five months would not have been adequate to ensure meaningful consultation with a wide range of stakeholders and build consensus, especially given the unfolding of highly complex political developments in 1996. The project design should also have been more conducive to learning and adaptation by incorporating a M&E system from the beginning. The overall goals and the objectives of the project were relevant for Russia and for the Bank . The ICR indicates that this project was in response to a request from the Legal Department of the Office of the President, which approached the Bank for support in legal and judicial reforms . The project objectives were consistent with the four principal directions discussed at the All -Russian Congress on Legal Reform was held in Moscow in March 1996. This was at a time when several in the international community expressed serious concerns over the poor legal protections in Russia and were looking to the Bank to take a major role in addressing them . Additionally, for the Bank, the 1997 Country Assistance Strategy reiterated the importance of developing the legal infrastructure, as part of Russia’s ongoing transition to a market economy . IEG's 'Evaluation of Bank Assistance to Transition Countries' confirmed that "legal and judicial reform, with an emphasis on implementation, is critical for improvements in the business climate, the financial sector, social protection, and governance in general ." To some extent, the project objectives were moderately selective and included some flexibility . For example, it included only capacity building and provision of TA in the judicial component . A flexible 'other high-priority activities' component was included to allow addressing priority areas as they emerged . However, despite this, the overall objectives remained highly ambitious and spanned both legal and judicial reform, thereby involving a wide (and unmanageable) range of client stakeholders that the ICR notes significantly slowed project implementation. Given the state of the legal and judicial institutions, and the immense challenges of shifting to supporting a market economy in a complex and transitional country such as Russia, a phased approach, such as that of an APL, with implementation being undertaken in phases may have presented a more effective approach (ICR, pg 2 and 19 which expresses this view). A lack of an overall strategic focus was also noted by the Bank (see Initial PID for the follow-up Judicial Reform Project dated June 2003). The Technical Annex indicates that strengthening legal and judicial institutions was key in a context where the Russian Government was issuing new laws and regulations on a regular basis . However, the project did not heed its own analysis, and the focus on institutions was subsidiary to supporting the drafting of key laws, perhaps because of Government pressure . Project assistance was provided in three areas - (i) providing comparative advice in the legal drafting process (building in particular on the experience of bilateral s in financing such assistance ); (ii) carrying out studies which could provide the basis for institutional changes which can improve the process of coordinating legal drafting; and training regional authorities in legislative drafting and (iii) providing technical drafting assistance to regional authorities on a pilot basis . Although there seems to have been ownership about the implementing agency at the President's office, in September 2003, the project literally froze, when the quasi -governmental project implementation agency founded under presidential and governmental decrees was deemed not to have proper budgetary status . Only in March 2005, was the project restored by transferring the implementation responsibilities to an entity under the Ministry of Finance and resolving the project funding problem . The Technical Annex provides details of what other donors have done and an assurance that the Bank will take lessons from these experiences, but there is little effort at collaborating with other key donors in aiming to provide the client with a cohesive and comprehensive package of reform in this area during the preparation stage. This said, donor collaboration has increased during the implementation phase and is currently leading to the joint development of a Multi-Donor Trust Fund for Governance for about US$ 26 million, of which about US$19 million will support complementary judicial modernization initiatives . 4. Achievement of Objectives (Efficacy) : Overall, project efficacy is considered to be 'modest .' The 'Legal Education' component substantially achieved its objectives, but the outcomes were more mixed or limited in the case of the other three components ('Legal Training', 'Legal Information', and 'Judicial Reform and Alternative Dispute Resolution' ). Additionally, if the project is to achieve its intended impact, while legal education is an important objective that can not be understated, the 'Legal Drafting' and 'Judicial Reform' activities would need to progress simultaneously, given that they aimed to rebuild and strengthen key legal and judicial institutional processes and systems . As the project's Technical Annex states : "One cannot begin to encourage the public to turn to law on resolving economic disputes if the courts are not adequately staffed to handle cases efficiently ." (Para 1.5). Therefore despite the substantial achievement of the other components, the overall achievement of project objectives is assessed to be 'modest' because of the modest achievements of the judicial reform /ADR and the legal drafting components . Detailed reasons are provided below : Component 1: Legal Drafting Component (appraised costs (US$20 m); actual costs (US$5.47 m)) is assessed to be unsatisfactory (consistent with the ICR rating) for the following reasons: Of thirty-four prospective legal acts identified at project appraisal, only nine were drafted and passed into law . Key laws such as the civil code; a law on alternative dispute resolution; laws on accounting and auditing; laws on the status of judges; revisions to the privatization law; a law on the insolvency of banks; the tax code; the labor code; the land code; a law on mortgages; a law on execution of judgments; and a law on registration of real property were not passed as planned . Studies to analyze and improve the mechanisms to coordinate the legislative drafting processes across federal agencies and between federal and regional authorities, a key component of the institutional strengthening aspect as indicated in the Technical Annex, did not materialize . Only one study on the gaps in the federal regulations governing economic reforms was submitted to the Ministry of Justice in 2002. It is not clear to what extent the study results were utilized . Training was conducted for sixty local legislative drafters, including lawyers and the staff of the regional legislative body in Sverdlosk and Orel . In the absence of post-training review of legislative quality, the ICR states that it is not possible to assess the efficacy of the activity . The ICR does not provide any information on whether specific legislation were prepared at regional levels – on the basis of annual priorities communicated to the Bank by the federal authorities . In late 2005 the project financed an assignment to draft laws and regulations related to the provision of housing to federal civil servants that did help to address the many gaps in the norms and approaches used to compensate civil servants. A study undertaken also resulted in a draft law on publication of court decisions , which was one of the later key outputs in 2005 that helped lay the basis for the JRSP . These activities were via the Other High Priority Activities Component, Component 2: Legal Information Component (appraised costs (US$24 m); actual costs (US$6.6 m). This component is assessed to be moderately satisfactorily (consistent with ICR rating). It aimed to (a) develop legislative classification and coding guidelines; and (b) design and pilot a public sector legal information system . Although the planned pilot of a database and web -based access to official legal information was aborted, two activities were introduced under the High-Priority component, which were successfully completed . A consultant – working closely with client stakeholders – produced a classification scheme for Russian law to make the laws easily accessed via library research or database searches . The scheme was adopted by presidential decree in 2000. The ICR notes that the classification system is utilized by librarians and database developers, and that as result laws are more rationally organized and more accessible to users, such as judges, lawyers, and the public. The pilot implementation of the electronic format did not materialize, but reportedly private database developers are providing databases in electronic format, filling this need without government involvement or financing (management comments). The Government ICR is silent on the outcomes of the classification system. The sub-national legal and legislative environment was addressed by a feasibility study for a mechanism to collect all the laws, decrees, and judicial decisions that are in force for that region, and to make them available to the public. However, the planned pilot of a competitive database and web -based access to official texts of the laws and regulations and judicial decisions in Moscow, Tula, and Tambov was aborted, due to contractual problems. In the area of enhancing access to legal information for the public, two sub -projects were designed and implemented under the "Other High Priority Activities" component: o The first sub-project established Public Legal Information Centers throughout Russia – in partnership with Ministry of Culture and Federal Agency of Government Communications and Information (FAPSY). The project financed about US$10,000 in computers, other equipment, and educational materials relating to the law to establish each center (mostly in public libraries) around the country. In some locations, the information resources are complemented by assistance from law students, who help citizens find materials and explain the law. As of end of 2005, forty-four centers have been established in eighteen regions . An estimated fifty thousand citizens have utilized their services . About sixty further PLICs have been established in various regions using own -budgetary means. o The second sub-project was: “the Development and Introduction of Methodology for the Automated Collection, Analysis, and Summary of All Judicial Practices for the Entire St . Petersburg Region.� The pilot project followed from a proposal of St . Petersburg State Law School. It built on work by some of its students to monitor and collect important local judicial decisions . This sub-project experienced significant delays, due to conflicts between the courts and the Law School, arising, in part, due to the absence of a legal basis for the publication of court decisions . During 2005, however, the technical and political disputes were largely resolved. Towards the end of 2005, a highly functional web site was made operational . This provides the collection of and public access to information on the activities of over two dozen general jurisdiction courts in the Leningrad region– including, in a number of instances, court decisions . Expansion of the coverage of court decisions would continue during 2006 (based on a November 2005 memorandum of understanding between the Judicial Department of St . Petersburg, St. Petersburg City Court, St. Petersburg Law School, and the St. Petersburg Council of Judges, which also provides for the eventual custodianship of the web portal by the Judicial Department ). A draft law has also been prepared on "Safeguards for the Rights of Individuals and Organizations to Information about the Judiciary in the Russian Federation ." Component 3 - Legal Education and Public Education Component (appraised costs (US$24.82 m); actual costs (US$22.6 m) The outcome of this component is rated as satisfactory (consistent with the ICR rating) based on the ICR findings (Pg. 13-14): "The impact of the subcomponent has been significant . An estimated sixteen hundred schools in sixty -two regions participated in the Project . Approximately sixty conferences and training programs were conducted . Over eighteen hundred teachers have been trained to teach law with materials developed under the Project . Over four hundred and fifty thousand copies of new textbooks and thousands of teacher ’s manuals and other materials have been published . Through interviewing teachers and students, independent evaluators (including from the Eurasia Foundation and Harvard University ) rated the quality of the materials as very high . According to a survey conducted by the Public Opinion Foundation of the level of legal knowledge of schoolchildren, the effectiveness of the teaching methodology developed under the Project was high . A subsequent study conducted by an individual consultant who visited all the pilot schools, corroborated the survey findings . The project’s demonstration effect has also been significant . A number of textbooks were published with the publisher’s own financing and subsequently purchased by schools outside of the project scope . The Moscow Government is using the materials in all city schools, for which some three hundred thousand copies of the materials were printed. More than sixteen hundred high schools in sixty -two regions introduced law courses as a training subject. All the regional training seminars conducted in 2000-2003 were co-financed by the regions." (ICR, Pg. 13) The Public Education Campaign through Russian Mass Media Subcomponent sought to (a) improve the understanding of law and legal matters by journalists; and (b) improve the general public’s understanding of law and legal matters. The ICR lists a host of impressive outputs including through the press (training for journalists), TV ( a successful program 'Household Lawyer' ran for 46 episodes and was reportedly well -received), and the establishment of a web page on legal information (which receives 8-10,000 hits per day.) However the outcomes of this sub-component are not clearly outlined in the ICR . Component 4: Judicial Reform and Alternative Dispute Resolution Compone nt (appraised (US$10 m); actual costs (US$8.34 m). The outcomes of this component is rated moderately unsatisfactory - this differs from the ICR rating of 'moderately satisfactory' for the following reasons : The ICR states that the "deep suspicion of the project by the courts, including accusations of mismanagement/misappropriation of project resources, led to the establishment of a Coordinating Council in early 2000. However imperfect, the advent of the Coordinating Council led to the gradual resumption of subcomponent activities." The Project Status Report (PSR) dated 4/08/2004 rates outcomes of this component as "Unsatisfactory". A US$11 million procurement package for 16 contracts, finalized in the last months of project implementation, resulted in key activities to support this component . While the ICR notes that such activities provided some basis for how "the proposed Judicial Reform Support Project is being constructed " (ICR Pg 21), neither the ICR nor the final Implementation Status Report (ISR) (dated November 05) indicate how this last-minute procurement impacted component outcomes . Component 4 aimed to improve (i) the quality of judicial training for the Courts of General Jurisdiction and Arbitrazh [i.e., commercial] Courts, (ii) analyze impediments to effective court administration and case management, (iii) develop a broad based consensus for necessary structural reform, and (iv) promote the development of alternative dispute resolution (ADR) in Russia. Outcomes were however limited, although the project did contribute to developing a broad -based consensus necessary for structural reform (the third objective). Although a study was conducted to better understand the constraints to effective court administration and case management, the ICR is unclear on the outcomes of this sub -component, and whether the study led to a considered strategy to deal with the issues . The purchase of US$11 million ICT during the last few months of the project, permitted by an extension of the closing date, helped to install ICT in a large number of courts. This is reported to have facilitated court administration and case management in these courts, and the ICR notes that this has increased transparency by providing the public with better access to judicial decisions . While significant training activities and materials (outputs) took place, over the ten year period there has been no evident improvement in the quality of judges that is reported by the ICR . Additionally, the focus of the training seems to have been on the use of ICT (see pg 17 of ICR). As the ICR states: “the cumulative effects of study tours and international and domestic knowledge sharing activities, gradually increased the constituency for judicial reform.� The outcomes are, therefore, limited in this area . The outcome of the ADR sub-component is also limited. The ICR states that “evidence for impact of the ADR Subcomponent is scarce and relatively weak . The knock-on effects of training three hundred forty arbitrators may well be seen in the existence of four hundred eighteen arbitration panels in sixty -two regions. On the other hand, the Russian Center for Arbitration Support continues with a static membership of only about twenty and has attracted only limited donor support .� Management notes, however, that "the project was the first to start training in this area, which has since been picked up by other donors and private companies . Notably, since project completion, the PIU has been receiving numerous requests for training materials developed under the project, and there is a growing cadre of mediators and arbitrators emerging from this effort ." Overall, the outputs may be significant, but the outcomes are limited. The Federal Targeted Program (2007-2011) of the Government, however, is likely to strengthen the reform process. In 2005, the project financed a survey of public perceptions of the judiciary . A survey, using structured interview based on combined sampling model of twenty -five hundred persons in twenty -two regions, indicated that the credibility of courts is low, and the lack of timeliness of completing court proceedings singularly degrades popular attitudes toward the courts as well as that corruption in courts is widely perceived as generalized . See also IEG's study on countries in transition "Legal and Judicial Reform in Europe and Central Asia, " (Pg 12) which states that "Supervision reports and interviews indicated that the Russian legal reform project became effective in September 1996, but between 1996–99, important developments took place in the area of judicial reform. As a result, the project could not provide a significant contribution to the development of judicial reform in the Russian Federation." Neither the Project Information Document or the draft Project Appraisal Document for the proposed Russian Judicial Reform Support Project (Dec 13, 2005) indicate that this project had any significant impact . Component 5: Other High Priority Activities Component (appraised costs (US$3 m); actual cost (US$10.64 m). The outcomes of this component are discussed and assessed under Component 1, 3, and 4. Component 6: Project Management (Appraised costs (US$6 m); Actual costs (US$10.02 m)) This component is rated unsatisfactory (the ICR rates it as marginally unsatisfactory ). As the ICR notes: “poor and uneven support of the Project Implementing Agency also impacted the project quite negatively .� There was significant staff turnover for the first several years. Thirty-one contracts, where suppliers had been selected, went unfunded and were allowed to lapse as late as in 2004, after the new implementing agency (Federal Center for Project Finance ) had taken over. Except for a burst of activity at the very end, when sixteen procurement actions with a total value of $ 11 million� were executed, project management was generally weak and unsatisfactory . 5. Efficiency : Neither the Technical Annex or the ICR discuss the efficiency of the project . The overall efficiency of the project is considered to be modest for the following reasons : Significant project delays (the project took 9.5 years instead of 4.5 as planned); Increased implementation costs (the costs increased by 166 percent and amounted to more than 20 percent of the Bank financing); and Problems with the Implementing agency . 6. M&E Design, Implementation, & Utilization: The Technical Annex in the Memorandum of the President included a Table 5.4 that provided detailed output and outcome indicators for each component . The project never used these indicators . Nor was the project retrofitted. 7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative): None Identified 8. Ratings : ICR ICR Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The high relevance of the project Unsatisfactory objectives was reduced by the complexity and ambitiousness of project design . For reasons stated above, the outcomes and efficiency are rated as modest . The overall project outcome is, therefore, moderately unsatisfactory. Institutional Dev .: Modest Modest Sustainability : Likely Likely Bank Performance : Satisfactory Unsatisfactory Bank Performance is rated unsatisfactory because: 1. Performance at entry is less than satisfactory: (1.1) The ICR notes that "in retrospect, Bank Management should have been more firm in narrowing the broad scope of the project and insisting on more realism in the implementation time table (possibly through a sequence of operations). Also, the technical design of the project should have included more efforts to strengthen the implementation arrangements and to develop of risk mitigation strategies for changes in the source and strength of reform impetus. In particular, the project design would have benefited from plans (and earmarked resources) to engage the more skeptical beneficiaries." (1.2) The Borrower's final project evaluation report also provides some reservations on quality at entry . The Borrower states that the "only but significant drawback at the initial stage of LRP implementation was that goals and objective were worded – in very generic terms, without appropriate level of detail, while the pre-project studies were undertaken in the very course of LRP implementation. This fact impeded the effective disbursement of loan funds on some of the LRP components at the most initial stages, since it took a long time to get agreement from the IBRD for some activities proposed by the Borrower representatives, which was especially obvious in the case with procurement and installation of equipment." 2. Performance during implementation is also unsatisfactory: (2.1) Initial supervision efforts are weak with high staff turnover and distance hindering staff ability to deal with issues . The Borrower's ICR comments that activities and programs undertaken by the project in the initial stages were not in line with the proposals made by the Judicial Department and the RF Council of Judges. (2.2) According to QAG quality of supervision review (2004), Bank Management did not adequately address the financing problems that led to initial implementation delay. (2.3) During implementation, the set of monitoring indicators provided in the Technical Annex was not integrated into project monitoring. Bank performance was weakened by this lack of this results orientation in its supervision Borrower Perf .: Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness . 9. Lessons: Systematic measurement of progress through integration of selected project monitoring indicators is a critical element of ensuring effectiveness in legal and judicial reform . Donor collaboration is essential to enhance the impact of legal and judicial reform activities, particularly in large and complex countries like Russia . 10. Assessment Recommended? Yes No Why? Given the intensive activity in the last few months of project implementation, it would be helpful to assess at a later date whether these very recent outputs translate into sustainable outcomes . 11. Comments on Quality of ICR: The ICR is well-written and candid.