Document of The World Bank Report No.: 55277 CLUSTERED PROJECT PERFORMANCE ASSESSMENT REPORT ECUADOR JUDICIAL REFORM PROJECT (LOAN 4066) GUATEMALA JUDICIAL REFORM PROJECT (LOAN 4401) COLOMBIA JUDICIAL CONFLICT RESOLUTION IMPROVEMENT PROJECT (LOAN 7081) June 30,2010 Country Evaluation and Regional Relations (IEGCR) Independent Evaluation Group (World Bank) Currency Equivalents (national currency per US dollar) Ecuador fsucre) Guatemala fauetzal) Colombia fveso) 1996 O 3,635 . O 5.96 1,005.30 1997 4,428.00 6.18 1,293.58 1998 6,825.00 6.85 1,507.52 1999 20,243.00 7.82 1,873.77 2000 25,000.00 7.73 2,187.02 200 1 I1 8.00 2,301.33 2002 I1 7.81 2,864.79 2003 1 I 8.04 2,780.82 2004 1 I 7.75 2,4 12.10 2005 It 7.61 2,284.22 2006 II 7.62 2,225.44 2007 I1 7.63 1,987.81 2008 1 I 7.77 2,198.09 2009 1 I 8.35 2,044.23 Fiscal Year January 1- December 3 1 Abbreviations and Acronyms ADR Alternative dispute resolution LIL Learning and innovation loan CAS Country Assistance Strategy M&E Monitoring and Evaluation CCM Corporate Court Model NGO Non-governmental Organization CGE Contraloria General del Estado PCU Project Coordination Unit CPC Civil Procedure Code PHRD Policy and Human Resources CSJ Consejo Superior de la Development Judicatura PPAR Project Performance Assessment IBRD International Bank for Report Reconstruction and Development SAR Staff Appraisal Report ICR Implementation Completion UCP Office of Criminal Record Report UMOJ Unit for the Modernization of the IEG Independent Evaluation Group Judiciary IEGWB Independent Evaluation Group (World Bank) Director-General, Evaluation : Mr. Vinod Thomas Director, Independent Evaluation Group (World Bank) : Ms. Cheryl W. Gray Sr. Manager, Country Evaluation and Regional Relations (IEGCR) : Mr. Ali Khadr Task Manager : Mr. Jaime Jaramillo-Vallejo Primary Author : Mr. Mauricio Rubio ... 111 Contents Principal Ratings ...................................................................................................................... v Key Staff Responsible ............................................................................................................ vi Preface..................................................................................................................................... ix Summary................................................................................................................................. xi 1.Background .......................................................................................................................... 1 2. Framework for the Evaluation ........................................................................................... 5 . 3 Objectives and Components ................................................................................................ 7 4.Relevance of Objectives and Design ................................................................................... 9 Relevance of Objectives .................................................................................................... 9 Relevance of Design ........................................................................................................ 10 . 5 Achievement of Objectives (Efficacy) .............................................................................. 13 Ecuador ............................................................................................................................ 13 Efficiency. Effectiveness and Transparency in the Judicial Process ....................... 13 Alternative Dispute Resolution Mechanisms ............................................................ 15 Improving the Access to Justice by Women.............................................................. 17 Improving Court Reform and Research and Legal Education ................................. 17 Guatemala ........................................................................................................................ 18 Effectiveness ............................................................................................................. 18 Access to Justice ....................................................................................................... 20 A Credible Judicial System ....................................................................................... 22 Improve Consistency and Equity in the Application of the Law .............................. 24 Colombia............................................................................... .......................................... 25 Improving the Rendering of Judicial Conflict Resolution Services.......................... 25 Software Development .................................................................................................... 29 . 6 Ratings ................................................................................................................................ 31 Outcome........................................................................................................................... 31 Risk to Development Outcome ........................................................................................ 31 Monitoring and Evaluation: Design ................................................................................ 33 Monitoring and Evaluation: Implementation and Utilization .......................................... 34 Bank Performance ........................................................................................................... 36 Borrower Performance ..................................................................................................... 38 . 7 Lessons................................................................................................................................ 39 References............................................................................................................................... 41 iv Box 1. Willy Perfect ......................................................................................................... 29 Table 1: Nature of Bank Support in the Judicial Reform Projects ....................................... 5 Table 2: Components and Costs of Projects ......................................................................... 7 Table 3. Ecuador: Efficacy by Objective and Outcome Rating......................................... 31 Table 4 . Guatemala: Efficacy by Objective and Outcome Rating..................................... 31 Table 5 . Colombia: Efficacy by Objective and Outcome Rating ...................................... 31 Table 6. Risk to Development Outcome Rating ................................................................. 33 Table 7 . Suggested Key Performance Indicators ................................................................ 33 Table 8. Monitoring and Evaluation Rating ....................................................................... 36 Table 9 . Bank Performance Rating ..................................................................................... 38 Table 10. Borrower Performance Rating .............................................................................. 38 Figures Figure 1. Cuenca. Ecuador: Ratio (outhn) of Civil Court Cases ........................................ 14 Figure 2 . Number of ADR Cases: Quito, Ecuador ............................................................. 16 Figure 3. Guatemala Congestion Rates, 1998 - 2005 .......................................................... 19 Figure 4 . Guatemala: Number of Courts @er 100,000 people) by Department, 1987 and 2006....................................................................................................... 20 Figure 5 . Current State of the Judiciary Compared to 10 Years Before .............................. 23 V Principal Ratings ICR * ICR Review * PPAR ECUADOR: JUDICIAL - REFORMPROJECT PO36056 Outcome Satisfactory _ ....................................................................................................................................................................................................................................................... Satisfactory ................................................................................................................. Satisfactory .................................................. Institutional Development Impact** ~ Substantial Substantial - " ........ " Risk to Development Outcome/ Likely Likely Modest Sustainability*** Bank Performance Satisfactory Satisfactory Moderately Satisfactory Borrower Performance Satisfactory Satisfactory Satisfactory GUATEMALA: JUDICIAL REFORMPROJECT -PO47039 Outcome Moderately Moderately Satisfactory ...... Unsatisfactory.... ......... Satisfactory ... ... Institutional Development Impact** - Risk to Development Outcome/ Moderate Moderate Negligible Sustainability** * Bank Performance Moderately Moderately Moderately Unsatisfactory Unsatisfactory Satisfactory Borrower Performance Moderately Moderately Satisfactory Satisfactory Satisfactory COLOMBIA: CONFLICT RESOLUTION IMPROVEMENT PROJECT -PO57369 JUDICIAL Outcome Satisfactory Moderately Unsatisfactory Satisfactory Institutional Development Impact** Risk to Development Outcome/ Low or Moderate Significant Sustainability*** Negligible Bank Performance Satisfactory Unsatisfactory Unsatisfactory Borrower Performance Satisfactory Moderately Unsatisfactory Satisfactory The Implementation Completion Report (ICR) is a self-evaluation by the responsible Bank department. The ICR Review is an intermediate IEGWB product that seeks to independently verify the findings of the ICR, based on desk work. As of July 1, 2006, Institutional Development Impact is assessed as part of the Outcome rating. "*AS of July 1, 2006, Sustainability has been replaced by Risk to Development Outcome. As the scales are different, the ratings are not directly comparable. vi Key Staff Responsible Division Chief/ Task Managerneader Country Director Sector Director ECUADOR:~ I C I AREFORM J L - PROJECTPO36056 Appraisal.. .......... ................... Maria Dakolias ......."... ....._...._..................................... ................. .................. Krishna Challa ._.......................................... Y. AbeP. Isenman ._ .................................................... ............. ................................................. .............. .................. Completion Maria Dakolias Ronald E. Myers Marcel0 Giugale GUATEMALA: REFORM JUDICIAL - PROJECTPO47039 Appraisal Waleed H. Malik ............ ............ ............ " " Guillermo Perry Donna Dowsett-Coirolo ...... Completion Linn A. Hammergren Nicholas P. Manning Laura Frigenti COLOMBIA: JUDICIAL RESOLUTION CONFLICT IMPRO WMENT PROJECTPO57369 - Appraisal Felipe Saez ........................................................................................................................................... ~ Ernest0 May Olivier...Lafourcade .............. . ........................................................ ~ Completion David F. Varela Nicholas P. Manning Gloria Grandolini vii IEG Mission: Improving development results through excellence in evaluation. About this Report The Independent Evaluation Group assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the Banks self-evaluation process and to verify that the Banks work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEGWB annually assesses some 20-25 percent of the Banks lending operations through field work. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEGWB staff examines project files and other documents, visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, and interview Bank staff and other donor agency staff both at headquarters and in local offices as appropriate. Each PPAR is subject to internal IEGWB peer review, Panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible Bank department. The PPAR is also sent to the Borrower for review. IEGWB incorporates both Bank and Borrower comments as appropriate, and Borrower's comments are attached to the document that is sent to the Banks Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEGWB Rating System IEGWB's use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEGWB evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEGWB website: http://worldbank.org/ieg). Outcome: The extent to which the operation's major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project's objectives are consistent with the country's current development priorities and with current Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, Sector Strategy Papers, Operational Policies). Relevance of design is the extent to which the project's design is consistent with the stated objectives. Efficacy is the extent to which the project's objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. The efficiency dimension generally is not applied to adjustment operations. Possible ratings for Outcome: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for Risk to Development Outcome: High Significant, Moderate, Negligible to Low, Not Evaluable. Bank Performance: The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loanlcredit closing, toward the achievement of development outcomes. The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for Borrower Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. ix Preface This is a Project Performance Assessment Report (PPAR) covering three Bank operations to support judicial reform in Ecuador (Loan 4066), Guatemala (Loan 4401) and Colombia (Loan 708 1). In Ecuador, the Judicial Reform Project, costing $14.3 million, was supported by a loan of $10.7 million, all of which was disbursed. The credit was approved on July 18, 1996 and closed on November 30,2002, five months after the original closing date. In Guatemala, the Judicial Reform Project, costing $49.7 million, was supported by a loan of $33 million, all of which was disbursed. The credit was approved on October 22, 1998 and closed on June 30,2007, three years after the original closing date. In Colombia, the Judicial Conflict Resolution Improvement Project, costing $6.66 million, was supported by a loan of $5 million, US$3.9 million of which was disbursed. The credit was approved on November 8,2001 and closed on June 30,2006, one year after the original closing date. This project was a learning and innovation loan (LIL). This PPAR, however, evaluates this loan following the same principles as for regular investment or technical assistance loans in order to glean as much as possible in terms of lessons and findings from the comparison of the three projects, while taking into account the difference in size.' Work for this evaluation included (i) desk review of Bank documents and publications in the area of legal and judicial reform, as well as the project files for each of the three projects, including the implementation completion reports (ICRs) and the ICR Reviews for the three operations; (ii) a one-week visit by the principal author to World Bank Headquarters (March-April 2009), and (iii) a one-week visit to each of the three countries (April 2009). During the Washington visit the principal author held interviews with Bank staff responsible for implementation of projects in the justice sector. In Guatemala and Ecuador, the main author met with the Bank's Country Manager. No review of documents was made in the country offices. In each country project participants and other key sources of information, including government officials, judges, lawyers, academics and NGO representatives, were interviewed. A list of all interviewees is provided in Annex B. In Guatemala, two visits were made outside the capital: to a Mediation Center in Chichicastenango and to Juzgudos de Puz in PetCn. In Ecuador, outside Quito, there were visits to Guayaquil and Cuenca. In Colombia, a planned trip to Medellin was canceled due to the reluctance of the Medellin judges to meet with anyone from the Bank associated with the project. 1 Following the LIL evaluation methodology would have entailed paying special attention to the monitoring and evaluation (M&E), the learning objectives, and the assessment of stakeholder response. It would also have entailed assuming learning "through an outcome-focused pilot" to test the design for a potential full- scale intervention. In contrast to the focused and limited approach set out in the LIL guidelines, the Colombia project involved 37 civil circuit courts in the five most populated cities of the country. X This report was prepared by Mauricio Rubio (consultant; main author). Jaime Jaramillo-Vallejo (IEG) was the Task Manager. The report benefited from comments made by Jorge Garcia-Garcia (peer reviewer) and Gita Gopal (panel reviewer), as well as support from Nestor Ntungwanayo. It was edited by Helen Chin, and H. Joan Mongal and Agnes Santos provided administrative support. The draft PPAR was sent to the Region and to the Governments of Ecuador, Guatemala and Colombia for comments, but no comments were received from the governments. xi Summary 1. This PPAR is a cross-country assessment of three World Bank projects implemented in three Latin American countries to support judicial reform. These projects and the associated loans, as approved, were: (i) the Judicial Reform Project (1996) amounting to $10.7 million to the Republic of Ecuador; (ii) the Judicial Reform Project (1998) amounting to $33.0 million to the Republic of Guatemala; and (iii) the Judicial Conflict Resolution Improvement Project (200 1) amounting to $5.O million to the Republic of Colombia. The rationale for judicial reform assistance by the World Bank was built on the assumptions that the rule of law promotes economic growth and reduces poverty, and that it does so by enhancing opportunity, empowerment, and security through laws and legal institutions. These three operations were also chosen because they are among the few that can shed some light on how the Bank has performed in post-conflict countries in its support for institutions viewed as critical to underpinning peace and reconciliation, such as the judiciary. 2. Before the 1990s, the three countries had lived through internal conflicts: 0 Ecuador, despite lower levels of violence, suffered greater economic and political instability than the other two countries, partly because of the presence of and the need to integrate an important indigenous population (25 percent). Judicial reform efforts in the country began as part of a state modernization agenda, and the Bank had a leading role both in the assessment of the judiciary and in the design of the national judicial reform. 0 Guatemala ended a forty-year-long armed conflict in 1996 and had to reconstruct its economy and build a state that was better able to take account of the claims of the groups involved in the conflict. Judicial reform was considered to be essential to promote post-conflict reconstruction, social stability, and economic growth. 0 Colombia went through an armed confrontation among outlawed armed groups, and between them and the national army. A new constitution deriving from a peace agreement with one of the guerrilla groups brought important changes to the judicial system, and addressed some of the key judicial issues, but the Bank's project did not deal with any of them. Instead, the project aimed at improving judicial conflict resolution services. Achievement of Development Objectives 3. In Ecuador, the proposed project aimed at increasing efficiency, effectiveness and transparency in the judicial process by improving case administration procedures and the infrastructure, expanding the use of alternative dispute resolution (ADR) mechanisms within the court system, improving access to justice by the public and women in particular, and pursuing court reform and research as well as legal education. Although the main objective to reduce court case load appears to have only been achieved in one province, the objective of increasing access to justice through alternate dispute resolution processes and through the smaller components was fully achieved. Overall, achievement of development objective is assessed to be substantial. xii 0 As regards efficiency and effectiveness, the impact of the project in the reduction of pending cases was not conclusive, was slight and, above all, was transitory. However, the project was considered to be useful in rationalizing management, human resources allocation and even the quality of judicial decisions. Although there was no baseline with which to compare results attained, alternative dispute resolution mechanisms were considered a success under the project. The mediation centers are helping to relieve congestion and delays in the ordinary courts and people prefer them because they settle disputes in two weeks, in contrast with the courts' eight months. 0 The program to help increase women's access to justice not only exceeded initial expectations with regard to the number of users and services offered, but it also improved access to justice for poor women in a sustainable manner. Women who used the legal aid centers are better off legally, economically, and feel reassured, as reflected in qualitative and quantitative measures. The majority of the women polled considered that the services accessed under the project had helped reduce ill-treatment by their partners. 0 The project implementation agency has distinguished itself for its professionalism and for having survived after the project closed. It is also a well-accepted organization, and serves as an effective buffer from the political instability in the formulation of projects and programs for the judicial system. 4. In Guatemala, the project development objective was to create a more effective, accessible and credible judicial system that would foster public trust and confidence in it and improve consistency and equity in the application of law. Overall, achievement of development objective in Guatemala is considered to be substantial, according to results deriving from a survey carried out in 2007. 0 As regard judicial effectiveness, performance was limited. Only 18 percent of the users thought that cases took less time to complete than ten years prior. One small project component that dealt with the Office of Criminal Records was, however, very successful and is currently serving more than 600,000 users a year. Regarding judicial coverage, an increase took place not only in a generalized way, but also with a significant regional reallocation in favor of departments least well served by the justice system. There was also an increase in coverage per capita, and a favorable regional redistribution, targeting those regions that had the largest proportion of indigenous communities. In some regions, the components were extremely well-designed: this warrants their use as models by the World Bank for other countries initiating similar projects. Finally, the increase in the number of mediation centers has been steady, particularly since 2005, especially in provinces with a high proportion of indigenous groups, while the launching of mobile courts was a welcome initiative. 0 Achievements toward a credible judicial system included a higher level of confidence of users in the justice system as well as a favorable perception about the performance of justice throughout the last ten years from internal and external users. In both groups a significant majority believed that the judiciary was ... Xlll working better than it did ten years before, mainly because of improved physical infrastructure, the way the courts are equipped, the quality of the system of criminal records, and the coverage of the courts. The survey concluded that credibility in the judiciary was higher today than a decade ago and that mobile courts have facilitated access to justice as well as decentralization in the administration of justice. 0 Progress in ensuring that indigenous communities have access to state justice as well as recognition of indigenous systems of justice was good. Out of the 82 mediators currently serving in Guatemala, 56 percent were women, and 63 percent came from the indigenous communities. Almost two thirds of the centers were under the responsibility of indigenous mediators. The coordination of official justice and indigenous customary law is taking place, and progress has been made in the appointment of indigenous judges. 5. In Colombia, the Bank-supported judicial project aimed to improve the rendering by the judiciary branch of judicial conflict resolution services through: (i) a reduction in processing time for case disposition; (ii) increased number of cases disposed per judge; and (iii) increased satisfaction of users. Overall, performance in achieving development objectives in Colombia is judged to be negligible. 0 In the 1990s, there was huge congestion in the civil courts in Colombia due to an economic downturn that led to historical peaks in court foreclosures. To address this situation and the backlog in the judicial courts, drastic measures were taken in 2003 and 2004, such as the reform of the Civil Procedure Code, and the creation of decongestion courts-none of these efforts was backed by the Bank project. 0 By 2004, as the crisis came to an end, there was a major decrease in the load of court cases. There were some efficiency gains, but it is not possible to conclude that these were the result of actions undertaken under the Bank project. 0 Three separate surveys provided the following feedback on the clearing of court cases in Colombia: (i) the majority of respondents of the 2005 and 2007 surveys thought that the time required to complete a case was longer than the time they thought to be normal, and the impact of the project was negligible, and when positive, it had been transitory; and (ii) the majority of respondents had no idea that a project for improving the civil courts had been implemented, and almost half of them considered the speed with which services are provided by the courts was deficient or at most fair. Achievement of Development Outcomes 6. Achievement of development objective is rated satisfactory for Guatemala and Ecuador, and unsatisfactory for Colombia, as detailed below. The judicial project implemented in Ecuador helped reduce court case load, but only in one province. The program to help women's access to justice exceeded initial expectations and improved access to justice for poor women in a sustainable manner. The mediation centers helped to relieve the congestion and delays in the ordinary courts and people prefer them because they settle disputes in a matter of weeks instead of months. xiv 7. As regard Guatemala, there has been good progress in the recognition of indigenous justice and a sustained coordination of the official justice system and the indigenous customary law. The increased availability of courts in remote parts of the country was a key step in increasing access to justice. In particular, the mediation centers and the mobile courts have facilitated access to justice, as well as decentralization of the administration of justice. 8. In Colombia, however, performance of the Bank-supportedjudicial project was weak. As noted, user surveys show that the time to complete cases is not perceived as having fallen, that the impact of the project was negligible and transitory. Moreover, the majority was not aware that the project had been implemented. Risk to Development Outcome 9. Because of the magnitude of the reforms, the Judiciary branch in Guatemala assimilated the project well in institutional terms, and there is a perception of continuity between the project coordination unit and the current Government's Unit for the Modernization of the Judiciary. Another indication of sustainability of the project is that some of the actions undertaken continue autonomously, and in some cases, without being identified as having been initiated by the Bank. Risk to development outcome in Guatemala is rated negligible. 10. In Ecuador, several years after the project closed, Projusticia (which served as a Project Implementation Unit [PIU]) is still recognized in the media as "the body responsible for the modernization of the justice system, and the entity responsible for coordinating the agenda between the Judiciary, the Executive, and Congress." The consolidation of this coordination work to support the judicial system took the form of the creation of the Ministry of Justice at the end of 2007. Both the first Minister of Justice and the present one are former directors of Projusticia. Risk to development outcome in Ecuador is rated modest. 11. In Colombia, institutional memory was much more elusive. People asked about the project responded that they were unsure if the project had formally closed, and no one knew what had been achieved, what was missing, what had happened, and what, if anything, was to continue. Risk to development outcome is rated significant. Bank Performance 12. Bank performance is rated moderately satisfactory in both Guatemala and Ecuador. In both countries, project design and objectives matched the Bank's strategy for the country and the government's overall reform program. There were also broad ideas about how performance indicators could be measured, but no baselines or target values were specified. However, project supervision was lightly documented for the project in Guatemala, while a flawed delivery of very expensive software in Ecuador was not documented, despite the waste of money involved. 13. In Colombia, the project deviated from the basic objectives of both the Bank's strategy for the country and the government, which were to reduce violence and poverty, while increasing access to justice-a critical element identified as a cause of violence. In xv addition, project supervision was unfocused. Bank performance in the context of this project is rated unsatisfactory. Borrower Performance 14. In both the Ecuador and the Guatemala project, preparation was done through a lengthy and participatory process of discussion with stakeholders. This was a process that was supported both by the government and the judiciary. The interest of the government and of the judiciary in the objectives of their respective projects was maintained throughout the execution. Also, there was adequate dissemination and continuation of the essence of the project after it closed. In Colombia, by contrast, the government and its executing agency showed little interest in the project. Not even the design was discussed with all the relevant stakeholders. Important procedural reforms to the civil justice were carried out during execution with no communication between the project, the government, the legislature or the parties that were promoting those reforms. Borrower performance is rated satisfactory for Ecuador and Guatemala, and unsatisfactory for Colombia. Lessons and Recommendations 15. I This PPAR suggests three broad lessons (see chapter 7). The first is that it is important to adapt the objectives and design of the project to local law, conditions, and peculiarities. The second lesson highlights the need for flexibility in overall approach in the conception of the project. In addition, it is essential to solicit local knowledge as well as input from specific beneficiaries in the process of selecting project objectives, design, and implementation arrangements. The third lesson is that it is unwise to rely exclusively on by-products of case management software to generate the data needed for adequate monitoring and evaluation. 16. The key recommendation for initiatives in judicial reform is that more resources should be invested in identifjing local experts to give them a leading role in the design, execution, and evaluation of projects. This will facilitate greater adaptation to the specific conditions of each country, and ultimately ensure ownership and sustainability. Daniela Gressani Acting Director-General Evaluation 1 1. Background 1.1 In the last two decades, most countries in Latin America have made efforts to renew or modernize their judiciaries. Nearly one billion dollars have been invested.2 For the World Bank ?the core of a judicial reform program typically consists of measures to strengthen the judicial branch of government and related en ti tie^."^ These measures aim to: (i) make the judicial branch independent or strengthen its independence; (ii) speed the processing of cases; (iii) increase access to dispute resolution mechanisms; and iv) professionalize the bench and bar! 1.2 Judicial reform has been deemed necessary for both democracy and economic development in Latin America.? For the World Bank, judicial reform ?is part of a larger effort to make the legal systems in developing countries and transition economies more market friendly., Judicial reform projects sponsored by the World Bank aim solely at enhancing a nation?s economic performance.,y6 1.3 The rationale for World Bank support for legal and judicial reform as an ingredient to promote economic growth and reduce poverty is relatively clear. One of the critical lessons of the East Asian financial crisis and the collapse of some Eastern European transition economies in the 1990s is that without the rule of law economic growth and poverty reduction can be neither sustainable nor equitable. Furthermore ?the rule of law promotes economic growth and reduces poverty by providing opportunity, empowerment, and security through laws and legal institutions.? And, to engender investment and jobs, laws and legal institutions must provide an environment conducive to economic activity. This requires the entire legal sector to function effectively, transparently, and with due process. First and foremost, the judiciary must be independent, impartial, and effe~tive.~ 1.4 It is beyond the scope of this assessment to provide a full review of what the development literature says about these causal links. It is worth noting, however, that there is no consensus around these ideas. Almost everywhere in Latin America there persists a marked ideological and political polarization with regard to the priorities, or the impact, of the judicial system. This debate is associated with two major intellectual movements that have influenced the reform processes. On the one hand, there is the already mentioned economic approach to legal and judicial reform, that emphasizes the stabilizing and market fostering qualities of the rule of law. On the other there is the neoconstitutionalist approach? that places emphasis on equality and the expansion of civil, political and social rights. The relationship between these two movements has been a complex one, but not always conflicting. Although DeShazo and Vargas (2006). Such as public prosecutor, public defender offices, bar associations, and law schools. Messick (1999). Messick (1999,2002). DeShazo and Vargas (2006). A critical view of this assumption appears in Carothers (2006). Messick (1999,2002). ? Bank (2003a). World * This is the term proposed by Rodriguez (2009). 2 there have been some sharp controversies, the contemporar Latin American legal order admits both neoliberal trends and the neoconstitutionalists. ry 1.5 The World Bank has often been considered the leader of the so called neoliberul economic vision of law." Furthermore, the design of World Bank projects has not always sought and integrated input from the many legal think tanks neither the non-governmental organizations (NGOs) interested in judicial matters, nor the "global community of courts," in particular the constitutional courts that have grown significantly since the 1990s.'' 1.6 Another distinction has been relevant for judicial reform, and has central relevance in this evaluation. The top down approach-macro-projects, designed and coordinated at national level-as opposed to bottom up programs, based on municipal governments, local organizations, and even individual judges committed to change and innovation. The top down approach, which seems predominant in the Bank's judicial reform's efforts, has evolved into a homogeneous model that tends to be promoted by international experts with scant understanding of a client country's Law and peculiarities, contrary to local knowledge. 1.7 In all the countries where reforms were undertaken, the approaches selected have been affected by the political context or supervening crisis.12 Also, there have been huge regional differences in terms of success, even within a single country. It will become clear in this assessment that some of the economic, political, and social situations particular to Ecuador, Guatemala, and Colombia affected the performance of the reform projects and the judiciary. Even within the countries themselves, local peculiarities were also determining factors. 1.8 In Guatemala, almost 40 years of armed conflict ended in 1996 with the signature of the peace accords. Military confrontation had a serious effect on economic growth, social services, and infrastructure. Guatemala had to reconstruct its economy, and build a state that better reconciled the claims of the opposing groups involved in the conflict. Although the nature of violence changed, it continued to be one of the most serious problems in that country. Almost 40 percent of the population of Guatemala belongs to some 24 indigenous groups, who retain their own language, customs, community organizations, and authorities. These were the groups most affected by the armed conflict. Recognizing this, the peace accords aimed to restore the rule of law, strengthen respect for human rights, and create a more inclusive economic and social system. Judicial reform was included as an important component, and the sector's reconstruction was considered to be essential to promote post- conflict reconstruction, social stability, and economic growth. The Bank's judicial reform program in Guatemala was hlly integrated into this national effort to reform, or rather to rebuild the judiciary after the conflict. There was the additional challenge posed by traditional law practices of the indigenous communities, and the Bank project covered such aspects. One example of this is the Colombian constitution of 1991, where clearly-defined territories have been drawn for each of the two movements. The cost of this kind of arrangement has been internal consistency. l o Dezalay and Garth (2002), De Sousa Santos and Garcia (2001), Santos (2006) sees this orthodox economic view only as characteristic of the first wave of "structural adjustment" reforms in the early 199Os, with the later Comprehensive Development Framework (CDF) being established came out precisely as a response to the critiques of the neoliberal economic policies. He also challenges the assumption of consensus within the Bank about the strategy and programs for development. According to Rodriguez (2009) the European financial support, especially from Germany, has been crucial in the consolidation of this network. l2 DeShazo and Vargas (2006). 3 1.9 Despite levels of violence which have been similar to those of Guatemala, and sometimes worse, the Colombian situation cannot be considered a post-conflict scenario. Instead of a civil war, what has happened in Colombia is a confrontation among different illegal armed organizations and between them and the army. Major institutional changes, such as a new constitution-which was also the result of peace agreements with one of the guerrilla groups-had been introduced ten years before the project. The 1991 Constitution brought important changes to the judicial system. The main areas of this broad reform program were13 the protection of fundamental rights (Accidn de Tutela), alternative dispute resolution (ADR) mechanisms, the introduction of the adversarial criminal justice system, changes in military justice and in the "public order," jurisdiction and the creation of the Consejo Superior de la Judicatura (Higher Council of the Judiciary). The Bank-supported project did not deal with any of these issues. 1.10 Lower levels of violence notwithstanding, Ecuador has suffered greater economic and political instability. Instead of an internal armed conflict there was a long border dispute with Peru and, recently, problems around the border with Colombia. The incidence of the indigenous population (25 percent) is lower than in Guatemala (48 percent) and well above that in Colombia (1.7 per~ent).'~ Judicial reform efforts began in Ecuador in the early 1990s as part of a state modernization agenda. The Bank had a leading role, both in the assessment of the judiciary" and in advising on the design of the national judicial reform program.16 1.I 1 Common to all three countries is the extreme geographic heterogeneity, not only in terms of a large gap in living standards between urban and rural areas, but also in terms of significant economic, social, and cultural differences across regions. This high heterogeneity translates into regional differences in terms of legal disputes and their resolution. Thus, the idea of judicial reform designed and implemented in a uniform way at the national level becomes risky. High levels of poverty make the issue of access to justice highly relevant. In addition, in all three countries, some regions show strong influence of drug trafficking and violence, leading to criminal justice having a priority in country-owned judicial reform efforts. 13 See Fuentes (2006). l4 Chisaguano (2006). World Bank (1994). l6 Projusticia (1998). 5 2. Framework for the evaluation^^ 2.1 Several years after completion of the projects, there is still no consensus about the relevant theoretical model for judicial reform. Impact measurement and evaluation '* methodologies are still relatively underdeveloped. World Bank documents dated 2003 provide a sketch of a judicial reform model. The legal and judicial reform strategy is based on three pillars: (i) an independent, impartial, and effective judiciary; (ii) an ap ropriate legal framework that provides enforceable rights to all; and (iii) access to justice. E 2.2 This framework, however, was not explicitly stated when the three projects covered in this assessment were designed. At the time-1996 for Ecuador, 1998 for Guatemala, and 2001 for Colombia+mphasis was placed on to effectiveness of the judiciary and, in two of the projects, access to justice. Table 1: Nature of Bank Support in the Judicial Reform Projects Country Effectiveness of Judiciary Access Ecuador I . I . Guatemala I . I . Colombia I . NA Source: Staff Appraisal Report (SAR)for Ecuador; Project Appraisal Document (PAR) for Guatemala and Colombia. 2.3 To enhance effectiveness, Bank support was mostly aimed at judicial administrative reform, including software development. A significant part of the budget (19 percent in Ecuador, 33 percent in Guatemala, and 35 percent in Colombia) was invested in physical construction (or remodeling) of infrastructure. Construction of better infrastructure was considered a way to improve access to justice or (mostly in Colombia) to improve effectiveness. Alternative dispute resolution (ADR) mechanisms were thought to facilitate access but also to increase overall effectiveness of the judiciary, integrated broadly. l7 See Preface. Owen (2006). l9 World Bank (2003a), p. 2. 7 3. Objectives and Components 3.1 The objectives of the projects as stated in the respective loan agreements are outlined below. 3.2 Ecuador. "The overarching goal of the Bank project would be to improve the capacity of the judicial system by strengthening the administration of justice. Specifically, the proposed project aims at: increasing efficiency, effectiveness and transparency in the judicial process by improving case administration procedures; improving the infrastructure, expanding the use of alternative dispute resolution mechanisms within the court system; improving the access to justice by the public and women in particular; and improving court reform and research and legal education."20 3.3 Guatemala. "The development objective of the project is to create a more effective, accessible and credible judicial system that would foster public trust and confidence in it and improve consistency and equity in the application of law."21 3.4 Colombia. "The proposed Project constitutes the learning phase of a long-term initiative undertaken by the Government of Colombia, under the leadership of the [Superior Council of the Judicature] SCJ, aimed at improving the rendering by the Judiciary of judicial conflict resolution services. It is expected that the proposed Project will allow the testing of a participatory and comprehensive organizational change strategy to bring about change in the courts' operations which ensure people swifter, fairer and more transparent conflict resolution services.9y22 3.5 Table 2 summarizes the components and costs of the three projects. Table 2: Components and Costs of Projects Ecuador Guatemala Colombia (A) Case Administration and (A) Institutional Capacity of the (A) Change Strategies (US$5.4m) Information Support (US$4.5m) Judiciary (US$l7.9m) (B) Appropriate Policy and Support (B) Alternative Dispute Resolution (B) Anticorruption Support Environment (US$O.Sm) (US 1.4m) (US$2.5m) (C) Project Management, Monitoring (C) Program for Law and Justice (C) Access to Justice (US$21.3m) and Evaluation (US$0.8m)25 (US$3.5m) (D) Communications, (D) Infrastructure (US$2.8m) Modernization, and Management (E) PCU and Fund Management23 (us$s.~~) 24 US$1.3m) Total estimated cost: US$14.31 million Total estimated cost: US$49.7 million Total estimated cost: USS6.66 million Total actual cost: US$12.12 million Total actual cost: US$35.6 million Total actual cost: US$5.27 million The Bank financed US$10.7 million, The Bank financed US$33.0 million, The Bank financed US$3.9 million, 88.3% of the total actual cost 92.7% of total actual cost 73.6% of total actual cost Source: SAR - Ecuador; PAD - Guatemala and Colombia. "World Bank (1996), p. 21. '' World Bank (1998), p. 2. 22 World Bank (2001), p. 2 . 23 World Bank (1996), p. 26. 24 World Bank (1998), p. 7. 25 World Bank (2001), p. 16. 8 3.6 In Ecuador, the Case Administration and Information Support component (A) was conceived to help address the overall goal of improving the management of cases. The Alternative Dispute Resolution (ADR) component (B) sought to reduce the backlogs and costs of litigation and also to improve fairness and effectiveness through mediation. The Program for Law and Justice component (C) was intended to provide a mechanism to research and implement activities needed to prepare a judicial reform program throughout the country. The Infrastructure component (D) was a way to begin improving the infkastructure of the courts by developing a national strategy for courthouse facility development, by remodeling a building purchased by the judicial branch, and by creating pilot decentralized facilities in Guayaquil and Quito. Finally, the Project Coordination Unit (PCU) and Fund Management component (E) paid for project coordination and implementation, and it supported the establishment of a permanent PCU, Projusticia. 3.7 In Guatemala, the role of the Institutional Capacity of the Judiciary component (A) was to improve management procedures. The Anticorruption Support component (B) sought prevention and control of corruption. The Access to Justice component (C) was to help reach various goals. The first was to improve knowledge for policy making. Second, was to pilot- test the use of mediation mechanisms. Third, was to improve the performance of justices of the peace (JPs) in rural areas. And fourth, was to expand the judicial infkastructure. The Communications, Modernization, and Management component (D) sought to improve communications, to support the Judiciary Branch Modernization Commission, and to strengthen the PCU. 3.8 In Colombia, the Change Strategies component (A) was intended to generate "notable improvements in the expediency and quality in the provision of conflict resolution The services and in the productivity of judicial resource use.y926 Appropriate Policy and Support Environment component (B) looked for the development of an appropriate policy and support environment for the proposed change strategy. Finally, component (C) dealt with Project Management and Monitoring and Evaluation. 3.9 In Ecuador, the credit was disbursed fully, and there were minor revisions in project component^.^^ The delay for the implementation of the project was negligible. In Guatemala, there were no revisions to the components. However, the closing date was extended three times, mainly because of delays in construction dates. In Colombia, the project was not restructured and no changes were made to the design. Project implementation was subject to "sporadic delays in procurement and disbursement."28 Implementation took twelve months more than expected. Some minor adjustments were made in the allocation of funds. 26 World Bank (2001), p.11. 27 World Bank (2003b), p. 3. 28 World Bank (2007), pp. 8-9. 9 4. Relevance of Objectives and Design Relevance of Objectives 4.1 The relevance of project objectives was assessed using two criteria: (i) consistency with the Bank's country assistance strategy; and (ii) consistency with government priorities and policies. 4.2 In Ecuador, the project was appropriately framed within an effort to consolidate markets, modernize the public sector, and make justice more accessible and effective. The objectives were consistent with both the Bank strategy for the country and government priorities. In the country assistance strategy (CAS) for 1996, the poor performance ofjustice was recognized to be an obstacle to progress. A careful study of the Ecuadorian judicial system had identified major constraints to the proper workings ofjustice and access to those services by the most vulnerable groups. However, modernization of the public sector and the judiciary had been an explicit policy in Ecuador since the early 1990s. Project objectives remained relevant through the closing year of the project: "awareness has grown in the legal and judicial community about the need for an effective, efficient, and transparent legal and judicial system."29 4.3 In Guatemala, a national program for judicial reform was undertaken as follow-up of the peace accords, which ended nearly 40 years of armed conflict. The Bank-supported project was part of this global program to reform the judiciary. Because of the link with the peace accords and the ensuing reforms, the project likely had more political than juridical or technical significance. It was a reconstruction, and in some rural areas the creation, of a judicial system in a post-conflict scenario. The objectives announced for justice- effectiveness, accessibility, and credibility-were adopted by the project, and were relevant to both the situation and for the Bank`s strategy to contribute to the reconstruction. Another essential component of the peace accords was a strengthening of local power, taking account of the characteristics and cultural practices of the local community. The objectives of the project were consistent with the strengthening of local authorities and inclusion of indigenous communities. As stated in the 2005 CAS, objectives remained consistent with the government development priorities as of project closing.30 4.4 It should be noted that the Guatemalan project deliberately and correctly omitted certain objectives. What has typically featured in Bank initiatives for judicial reform, the need to strengthen markets, was not mentioned in this case. In an ideologically fragmented environment such as Guatemala, that decision facilitated the acceptance of the project. 4.5 By the late nineties, when the Colombia project was designed, two overarching development objectives were poverty reduction and the search for peace, both were at the center of the government's development plan and the Bank's country assistance ~trategy.~' The Bank was complementing government programs aimed at reducing violence, especially through support for a proactive social policy and investment in high-conflict areas. It was believed that the failure of the judiciary to provide an adequate system for conflict resolution *' World Bank (2003c), p. 33. 30 World Bank (2005), p. 15. 31 Government of Colombia (1999) Article 1 and World Bank (2005a), p. 8. 10 contributed to violence. Access to justice was identified as a key issue by the government and was consistent with country assistance strategy priorities. However, the project did not directly address the problem of access. It concentrated on issues related to judicial performance and the reduction of the backlog in the civil circuit courts. For various reasons, this concentration reduced the relevance of the project objectives. First, the project appraisal document itself points out that in 1999 the congestion in the civil courts had been reduced by more than 30 percent, and that there was no clear reason why this reduction should have occurred. Secondly, by that time it had become clear that the largest part of the burden on the Colombian civil courts was financial institutions' actions to recover bad loans.32 Thirdly, the deteriorating macroeconomic situation and, in particular, the emerging financial crisis suggested that there was soon to be an avalanche of new court cases. 4.6 At the time the project came to an end, the critical problems indicated in the previous CAS and overlooked by the project-violence and poverty-were still much alive. Further, the most serious expression of these two phenomena, internally displaced persons (Zos desplazados) headed the agenda of concerns for the Bank and for the government. Nevertheless, the idea of a causal link between access to justice and violence was abandoned by the project. The problems of the non-criminal justice system were focused on issues such as "procedures related to credit and financial Interestingly enough, these issues, particularly access to justice, loom large in a new Bank-supported judicial project in Colombia.34 Relevance of Design 4.7 In terms of design of the project, the evaluation assessed: (i) whether the strategy underpinning the project was based on a sound analysis of the country context, (ii) whether the project design indicated a clear results fiamework linking proposed activities to desired objectives; (iii) to what extent exogenous factors were identified and addressed; and (iv) to what extent the performance monitoring and evaluation system was consistent with project objectives. 4.8 Overall, the evaluation finds that the three projects failed in terms of (ii) and (iv). Even when objectives were realistic, the lack of a clear results fiamework underlying the projects led to weak linkages between the objectives and the proposed activities. These links were not always explicit or evident in the preliminary project documents. The clearest illustration of this flaw relates to construction and remodeling. These components received an important allocation of funds in the three projects, but the discussion regarding expected outcome or impact is missing in the program documents. In all three projects, the proposed set of actions did not typically result from a logical chain linking objectives and activities. The links between actions and objectives appeared almost intuitive. 4.9 The monitoring and evaluation (M&E) system in all three projects was ill-conceived. It did not adequately reflect project objectives and failed to take proper account of the financial weight of the components. The M&E system accorded excessive relevance to a limited set of indicators based on court statistics that took it for granted that the software components would yield optimal results and, as a result, were inadequate to detect flaws in ~~ 32 Rodriguez (2001). 33 World Bank (2005a), p. 42. 34 World Bank (2009). 11 the software or of the other components when the software did not operate as well as expected. 4.10 In both the Ecuador and Guatemala projects, a portion of the k n d s was left to be allocated using a participatory process that defined certain specific actions. In these cases, there was a clear tension between grass-roots participation in the design and the internal coherence of the project. 4.1 1 In Ecuador, the strategy underlying the project was based on a detailed prior assessment carried out by the Bank. The staff appraisal report (SAR) identified institutional and political risks to the project. The financial crisis, which ultimately had an impact on project implementation, was not raised, although this would clearly have been difficult to anticipate. The design had two good features. First, the project tried to combine standard elements for facilitating economic growth, such as increasing effectiveness, with components that directly promote access and enhance the rights of the most vulnerable groups, such as legal aid for poor women. Second, the project had limited scope, with objectives that were realistic, and analytical work was undertaken by the Bank that fed into the design of the project. There were also some limitations. First, the links between the objectives and some of the activities were weak. Second, the design emphasized the importance of administration, organization, and software to reduce processing times, whereas the judicial assessment had identified issues of legal procedure as the main reason for delays. In particular, insufficient attention was paid to the general application of oral proceedings as these are meant to work under Ecuadoran Law. Third, the design overlooked regional differences in the workings of the system. Moreover, the project chose as first pilot tests for the new model courts the cities with the worst problems and greatest congestion, instead of going for "low-hanging-fruit" cities where the learning curve would have been less steep. 4.12 In Guatemala, the final design was the result of exhaustive consultations with all relevant stakeholders. As the ICR put it, "Project preparation was exemplary." There were some limitations on project design that arose as a result of the political, negotiated nature of the peace accords as well as the participatory nature of the process for defining objectives in a multiethnic and multicultural society with different traditional legal systems. There was a trade-off between democratic participation to satisfy the demands of different groups previously in armed conflict with one another and overall cohesion and consistency of the framework underlying the accords. In addition, exogenous political risks of the project were anticipated and addressed. In particular, despite the weighty presence of the criminal jurisdiction in the overall program for the modernization of justice, the Bank was only marginally involved in this area. Even knowing that the Bank's support could be criticized on the these ground~,~' decision remained to play a small role in the area of criminal justice. 4.13 For Colombia, the strategic underpinnings of the project and some crucial assumptions were not based on a sound and systematic assessment of the country context at that time. There was no evidence, for example, on the tacitly-assumed link between performance of the civil courts and poverty alleviation, or violence reduction, two key ~~ elements that would have anchored the project within the country's p r i ~ r i t i e s .From the perspective of poverty reduction, the relevance of Bank support for enhancing the speed of 35 World Bank (1998), p. 15. 36 As noted in paragraph 4.5 above, poverty and violence where the two dominating concerns of the Government's development plan and the Bank's country assistance strategy. 12 process in civil cases, mostly for debt recovery, that were initiated by financial conglomerates against households that were in arrears on their mortgages because of the crisis, was limited at best. Even from an exclusively economic viewpoint, it can be argued that the project was wrongly focused. From the mid-1990s there was a large volume of literature on the costs of violence, which demonstrated the enormous economic impact of the armed conflict. Taking it as given that the Bank could do little directly with regard to the conflict, this evaluation was unable to locate within the literature and ESW of that time any authoritative references highlighting the operation of the civil courts or the non-performing loans in the financial system as basic obstacles to growth in Colombia. A hrther factor that appears to have been overlooked in assessing the country context and the significant exogenous factors relates to the cursory manner in which project design took account of the impact of the financial crisis in the late 1990s on the courts, as well as the decisions by the Constitutional Court or the government's other efforts to address the judicial backlog.37 4.14 Based on the foregoing discussion, the assessment has determined ratings as follows: For Ecuador, this assessment rates the relevance of the project's objectives as Substantial. For Guatemala, this assessment rates the relevance of the project's objectives as High. For Colombia, this assessment rates the relevance of the project's objectives as Negligible. 37 Rather than address these contextual issues directly, the PAD noted that "A key technical issue dealt during the process of project preparation was the room provided by the existing legislative framework to undertake the proposed changes in case management. After thorough analysis it was concluded that the current legislative framework has been systematically misapplied by judges that rather than active directors of proceedings have chosen to assume a wait and see attitude until the stage of deciding the case. ... In fact, the project can be portrayed (sic) as an attempt to set right the use of the law." World Bank (2001), p. 20. 13 5. Achievement of Objectives (Efficacy) 5.1 This section examines, for each country, the realized outcomes in relation to the objectives established in the projects. The section ends with a discussion of the software development component, common to the three projects, which showed similar issues in all cases in terms of efficacy. Ecuador 5.2 Project objectives in Ecuador were: 0 Increased efficiency, effectiveness, and transparency in the judicial process; 0 Expanded use of alternative dispute resolution (ADR) mechanisms within the court system; 0 Improved access to justice by the public, in general, and women, in particular; and 0 Foster initiatives on court reform, legal research, and education. EFFECTIVENESS TRANSPARENCY JUDICIAL EFFICIENCY, AND IN THE PROCESS 5.3 One of the sub-objectives of the reform in Ecuador was to reduce the workload of pending cases and the length of time taken to clear them. The basic assumption for the corresponding component was that "case management and information technology is significantly correlated to the time needed to resolve a case."38 One tool for improving effectiveness was the Corporative Court Model (CCM) in which management tasks are separated from judicial activities. This component, which began with a pilot phase in courts in Quito and Cuenca, was extended to courts in Guayaquil in a second phase. 5.4 In terms of effectiveness, the project showed different degrees of success in different places. In the city of Cuenca, the CCM was adopted in civil cases and extended to other jurisdictions. Several of the judges believe that rather than being followers of some foreign model, they were the true managers and promoters of this initiative, which separates judicial from administrative functions. The impact of the CCM in the reduction of pending cases is not conclusive (see para 5.8) but at least the data for such an evaluation is available in this city. In addition, the CCM is considered to be useful in rationalizing management, human resources allocation and even the quality of judicial decisions. Another pioneering improvement in effectiveness, also from Cuenca, is that some legal shortcuts were found to adopt oral procedures in criminal cases. 5.5 In Quito, for no apparent reason, the judges were opposed to the implementation of the CCM, and even to the related remodeling. In Guayaquil a quite peculiar and hybrid court model was implemented. In both cities, the data to evaluate the impact of the project on pending cases is not publicly available. 5.6 In terms of effectiveness, both the ICR and the Ecuadorian Government - Contraloria General del Estado (CGE) made favorable evaluations that require a few comments. First, they do not allow for a long-term or even medium-term perspective on the outcomes. Second, these evaluations do not distinguish between the courts in which the CCM was implemented, and those in which it was not. 38 Staff Appraisal Report p. 11'. 14 5.7 An outlook over a longer period, based on judicial data from Cuenca, the most reliable in Ecuador, and the only city for which information is publicly available, shows that the impact of the project in terms of efficiency was slight and, above all, tran~itory.~' In effect, even before the execution of the project in Cuenca properly began, one performance indicator-the proportion of cases completed in relation to new cases-had already reached levels observed today, close to 90 percent. Most of the supposed greater efficiency was seen, for all civil courts in Cuenca, between 2000 and 2004. In 2000, the number of cases completed was almost three times the number of new cases. In the following years, the ratio is almost two to one, and for 2005 it is close to the target of 100 percent (Figure 1). These five years of good performance by the judiciary may be attributed to the reform. However, it also had to do with the process of clearing up the backlog of dormant case files with the help of law students, which began in Cuenca in 2000. This program also contributed very significantly to the increased efficiency.40 Figure 1. Cuenca, Ecuador: Ratio (outh) of Civil Court Cases Source httD //wfuncioniudicial-azuav POV ec/unnciDal htm 5.8 Further, neither of these evaluations-ICR or CGE-mention a continuous and significant fall in the number of new cases entering the civil jurisdiction. This exogenous ,trend was not only significant-almost 40 percent between 1998 and 2001-but was also bound to affect the performance of the courts. 5.9 It is important to take account of the link between the financial crisis and dollarization of the Ecuadorian economy on the one hand, and the new incoming cases to the civil courts and the capacity to process them on the other. Financial crises are typically associated with a marked increase in the volume of cases entering the civil court system, and Ecuador was no exception. There is no available information regarding the impact of this crisis on the administration of justice, but there is relevant evidence that the link was there. One stakeholder interviewed 41 estimates that, nationwide, the crisis generated 40 to 50 thousand 39 All data are from http:l/www.fUncionjudicial-azuay.gov.ec/principal.htm. 40 CGE (2001) p. 19. 4' The stakeholder asked to stay anonymous. 15 extra cases for the judicial system. That amount is close to the number of new cases arriving in a single year in a large city such as Guayaquil. A civil judge in Cuenca estimates that the workload increased by 20 percent as a consequence of the crisis. The matter was not only one of quantity, but also of the monetary value of the claims involved, and many of the cases are still being processed. Several of the enforcement proceedings promoted by a financial institution, the Corporucidn Finuncieru Nucionul in Guayaquil, were worth close to US$10 million, having to be seen by higher level civil courts. After being solved on a first . instance, some of those cases ended up back in the courts in appeals, exceptions, or even claims for damages. Thus, an important part of the unexplained decline in the number of new cases arriving at the civil courts with the change of century, and the consequent increase in the efficiency of the courts, seems to be related to the unwinding of Ecuador's financial crisis. 5.10 A good example of the local idiosyncrasies in Ecuador, and of the lack of a uniform model of judicial reform, is the physical design for the CCM in Guayaquil. Implicit in the CCM is an architectural design in which the spaces of the administrative personnel and the judge are kept separate. In that city one way to enhance transparency in the judicial process was through physical changes to the layout of courthouses. The judge's chambers adjoin a corridor, from which they are separated by large panes of glass, which allow anyone passing to observe all that is happening inside the court. "This guarantees the transparency of process," says the Council of Judicature representative in Guayaquil. For him, this architectural detail has been fundamental in the eradication of what he describes as frequent "arrangements" and cases of corruption, which, according to him, happened before this reform. Convinced that judicial transparency is today founded in part on physical visibility, one of the pending ideas is the installation of video cameras in the courts. Its originality notwithstanding, this hypothesis concerning the underlying drivers of corruption and what is needed to control it, based on physical transparency in the courts, is as difficult to verify as it is to try to convince the representative in question of its possible limitations. Regardless, the hypothesis of visual surveillance has been put into practice in Guayaquil for both the judges and the rest of the staff. In a hybrid version of the CCM, staff is still being assigned to a single court, each working for a single judge, but they all work in the same physical space, with each in effect looking over each other's shoulder. This "transparency by design" hypothesis has been applied only in Guayaquil. RESOLUTION DISPUTE ALTERNATIVE MECHANISMS 5.1 1 The promotion of mediation and ADR was an important objective in the project in Ecuador.42Although much progress remains to be made, indications are that achievements have been significant. Today, there are some 120 mediation attached to a wide range of organizations (superintendents, universities, chambers of commerce, NGOs, foundations, lawyers' associations, and professional association^).^^ Many of them are not part of the formal judicial branch. The demand for establishing new centers, which depend on approval by the judiciary, has grown steadily due to the greater trust that people have in decisions that have judicial backing. The workload of the centers varies, as is the nature of 42The decision to use an ADR instead of a court is typically made by the parties to a contract at the time they agree on that contract. The ADR usually becomes a first instance, which, in any case, can be appealed before the courts. 43 http://redesalternativas.com.ar/noticias_ver.php?id= 1. 15 44 http:llenlace-masc-ecuador.orgiazulmasc/archivos/institut.html. 16 the cases they deal with. In Cuenca, the mediation center attached to the judiciary currently handles some 7,000 cases a year. That amount is five times the number served in Guayaquil, and twice that of Quito. ?The people of this region (Cuenca) are more open to dialogue, readier to sit down and talk, and arrive at a sound agreement.? This observation would apply to the users as well as to the judges and lawyers. In Cuenca, the courts send so many cases to the centers that they are now reaching their capacity limit. In Quito and Guayaquil, by contrast, the courts continue to mistrust the mediation centers. While in Quito only 5 percent of cases had been sent in by the ordinary courts, in Cuenca, for several years, the number of cases was split quite equally between those referred by the courts and those brought voluntarily. Today, the proportion of court-referred cases is larger. In recent years, the number of cases handled by the Quito Mediation Center has grown (Figure 2). Although there is no baseline to which the results attained can be compared, this trend should be considered a successful achievement of the project. Figure 2. Number of ADR Cases: Quito, Ecuador 5.12 The mediation centers have brought relief to congestion and delays in the ordinary courts. For Family Law disputes in Quito, for example, the excessive workload has meant that court hearings are being scheduled more than a year in advance. This has meant an increase in the number of requests for mediation in the area of family and childhood, which has doubled in two years.45 In Cuenca, people prefer the mediation centers because they settle disputes in two weeks, while courts do so in eight months. As a result, the centers are receiving requests for mediation even from Ecuadorians that have migrated to Spain. 5.13 Judges, lawyers, and court staff are still cautious about mediation. Aside from cultural considerations, there may be peculiarities of Ecuadoran Procedural Law that remain to be addressed properly. This said, one litigation lawyer in Guayaquil, who charges on a per- case-solved basis, thinks mediation may be good business because it is quicker and therefore allows a larger number of cases to be accepted. Hence, despite the lack of aggregate information and the difficulties that mediation still faces, this evaluation concludes that the mediation centers are now on firm ground. ~ ?Data from the Mediation Center in Quito. 17 IMPROVING THE ACCESS JUSTICE WOMEN TO BY 5.14 Legal aid for poor women is the star component, not only in the Ecuadorian project but perhaps in all the initiatives fop legal and judicial reform initiated by the Bank in Latin America. It is an intervention that, despite its small size, has received much attention within the Bank and outside of it. It is not easy to find a program for judicial reform that has been so exhaustively and rigorously e ~ a l u a t e d . ~ ~ 5.15 The project directly targeted poor women who were victims of domestic violence or who were seeking child support. Local NGOs competed to provide legal aid and other types of support in specialized centers. AAer the project was completed, a detailed evaluation, with field work, interviews, and focus groups, all done with the assistance of local NGOs, showed that "women who used the legal aid centers are better off legally, economically, and subjectively, as reflected in qualitative and quantitative measures. Participation in the legal clinics increases the probability of receiving child support payments, decreases the incidence of domestic violence after separation, and is associated with a more positive outlook toward In the judicial sy~tem."~' addition, these achievements were consistent with specific objectives of the Bank: "reduction of poverty, empowerment of women, and the promotion of education."48 5.16 The program not only exceeded initial expectations with regard to the number of users and seryices offered but it also improved access to justice for poor women in a sustainable manner. In fact, in one of the regions where the program was implemented, there is recent evidence of an eventual long-term impact beyond the period of execution of the project. A survey made in 2008, representative of the population of the cantdn (province) of Cuenca, finds that "the great majority of women consulted (98 percent) are aware of the existence of these [services]" (Women and Family Commissioners). When the women were asked whether, in the face of a potentially violent situation, they would go to the CMF or not, "92.4 percent said that they would, 63.5 percent considered that these [services] had helped to reduce ill treatment by their partners.'" In other words, the seed sowed and financially supported by the project has borne fiuit. The positive impact was not limited to program users, but extended to nearly the entire female population of Cuenca. Unfortunately, however, the financial situation of the NGOs committed to providing legal aid for poor women is now rather fragile. EDUCATION IMPROVING COURT REFORM AND RESEARCH AND LEGAL 5.17 The purpose of the Fund for Law and Justice was to promote the participation of civil society and judicial operators in the process of judicial reform, with small programs that would facilitate the access to justice, and help deepen the.diagnosis. The Fund for Civil Society financed 22 projects, with an average value of $40,000 each, and a maximum of $84,000 each." 5.18 One undoubtedly successfd achievement in Ecuador was to have channeled the execution of the project through Projusticia, an entity which has distinguished itself for its 46 See Rodriguez (2000), World Bank (2003), Owen and Portillo (2003), and Rubio (2008). 47 World Bank (2006). 48 World Bank (2000) "Millennium Development Goals" cited by Owen and Portillo (2003). 49 Camacho and Hernandez (2009). 50CGE(2001), p. 41-42. 18 continuity and professionalism. Projusticia has been a well-accepted organization, which maintains fluent communication with a range of actors: the judges, the Council of the Judicature, and, on the other side, the Executive, the academic world, NGOs, and litigants. As a technical entity, it has been an effective buffer from the impact of political instability in the formulation of projects and programs for the judicial system. 5.19 The early alliance of Projusticia with the Asociacidn Ecuatoriana de Facultades de Jurisprudencia (Association of Law Schools) led to an improvement in the research on judicial issues and also to several changes in legal ed~cation.~' 5.20 In short, for Ecuador, although the main objective (reduction of case load) of the largest component appears to have only been achieved in Cuenca, the objective of increasing access to justice through alternate dispute resolution mechanisms and through the smaller components was fully achieved. Overall, the efficacy of the project is assessed to be substantial. Guatemala 5.21 Project objectives in Guatemala were to improve the judicial system's: 0 effectiveness; 0 accessibility; 0 credibility; and 0 consistency and equity in the application of the Law. EFFECTIVENESS 5.22 Although one of the objectives of the project in Guatemala was to create a more effective judicial system, no specific targets were set. The most commonly used indicators for effectiveness-delay, clearance and congestion rates-were not tracked, but the project documents, quite realistically, never pretended that the project would do 5.23 For the period 2003-2006, ASIES (2006) estimates a disposition rate of 10 percent and points out that the rate fell from 13 percent to 7 percent during that period. This study does not specify, however, that between 1999 and 2002 the rate was much higher in civil and labor courts, and much lower in criminal courts (Figure 3). The high variation in the number of incoming and outgoing cases is not mentioned, let alone explained. For aggregate efficacy indicators, the only conclusion that can be drawn seems to be that reliable judicial statistics are still not available. 'I Estrella (2001). '' The clearance rate is the ratio of the number of cases disposed of during the period to the number of cases filed during that same period. The disposition rate is the average speed of the resolution of a case. The delay rate is the average time taken for the resolution of a case. Finally the congestion rate is a measure of how the stock unresolved case evolves in a court system. 19 Figure 3. Guatemala Congestion Rates, 1998 - 2005 Source: Data from CENADOJ taken from ASIES (2003,2006) 5.24 In terms of delays, there is no systematic analysis done so far. Again, no performance indicator was proposed in the project appraisal document. With the adoption of oral proceedings in family and labor courts, there are some signs of a reduction in the time from filling to disposition in both areas.53 In some civil trials, a smaller number of routines have also led to a reduction in the time needed to solve a case.54 However, in a 2007 survey of users, only 18 percent of them thought that cases took less time to complete than ten years before.55 5.25 Although designed more as mechanisms to improve access to justice, one may consider that the higher coverage achieved by both Juzgados de Puz (Peace Courts) and mediation centers has meant an advance in terms of effectiveness-some cases that never reached the system are now settled by the judiciary. 5.26 From the standpoint of effectiveness, it is worth noting a small Guatemalan project component, the Antecedentes Penales. For this program, an in-depth assessment has not been made, but it could and should be done. In any event, a quick review of available information suggests that it has made a difference. 5.27 As part of the court support services, the Unidad de Antecedentes Penales (UCP, Office of Criminal Records) is the office that issues the legal situation certificate that citizens need to produce when applying for a job, a passport, a visa, a residence permit, or when they want to work abroad. There are no accurate estimates of the costs incurred by users in terms of time lost from work and travel to Guatemala City before the program was in place. Informal consultations suggest that this process could last up to a week, implying considerable costs for the users. An additional ill was the bad perceptions of justice associated with a slow, cumbersome and ineffective procedure just to certify that someone was free of legal problems. With the reform, the UCP is currently serving more than 600,000 53 UMOJ (2009), p 66. 54 UMOJ (2009), p 67. "Novoa and Hasuike (2007). 20 users a year.56 As trivial as this may seem, the rationalization, streamlining, and modernization of the delivery of a certificate required by law to over half a million people each year is no less important as a priority than to have a payroll registry for the judiciary, or a functional system of judicial ~tatistics.~' Also, for many Guatemalans, obtaining this certificate is the first contact with the justice system. It is conceivable that, as such, it will be a determinant of the perception and image that people will have of the judiciary. ACCESS JUSTICE TO 5.28 The expansion of the judicial system in recent years has been considerable. The number of courts in Guatemala increased from 367 in 1987, to 583 in 2006. In national terms, there were 3.9 courts per 100,000 people in 1987; in 2006 the number climbed to 4.6 courts for every 100,000. This increased coverage occurred not only in a generalized way, but also with a significant regional re-composition in favor of the departments (provinces) which were least well served by the justice system (Figure 4). Figure 4. Guatemala: Number of Courts (per 100,000 people) by Department, 1987 and 2006 5.29 In 1997, the Department of Guatemala with less than 23 percent of the population was home to almost 30 percent of the courts. By 2006, its share in the total number of courts was equivalent to that of its population. In 1987, the ratio of the number of courts per capita in the best-attended department (El Progreso) and the worst (Totonicapcin)was 1 to 5. In 2006, the ratio between the two extremes was only 1 to 3. The calculation of the GIN1 coefficient-an indicator of economic inequality- for the provincial distribution of the %MOJ (2009). 57 World Bank (2008), p. 12-13. 21 courts indicates it was halved. The dynamics were similar for the Peace C o u r t s t h e number increased from 229 in 1987 to 365 in 2006. There was also an increase in per capita terms, and a favorable regional redistribution. Furthermore, the provinces in which coverage most increased were those that had the largest proportion of indigenous communities. 5.30 Special note should be taken of the fact that the new buildings have been accepted by such a diverse population. This acceptance is not accidental because clear efforts were made to adapt them to the ethnic and cultural peculiarities of the various regions. The architect responsible for supervising the building designs had previous experience in micro planning, regionalization, and decentralization. He was trained to take a local view of problems and solutions. He had also worked with communities in the Peace Secretariat, on matters of mini- management, and had sat at the negotiating tables: "The community way of thinking was not alien to me." The land where the courts were built was allotted by the municipalities. This process allowed discussions of schemes, designs, and plans with the local authorities. In Palencia, Dutch funds were used to make the first model court. This initial design, although assisted by an external consultant, had important local inputs. With this experience, evaluation began from the contact of "the architectural part with the ethnic part." The same exercise then took place also with the courts in PetCn and afterwards in the provincial complex of Huehuetenango. In parallel, progress was made on the design for Quetzaltenango. The court buildings in Guatemala, therefore, were not uniform; "they have very few common characteristics, because they are a response to their different sites." The critical question was "how to ensure that the buildings would be accepted by the local people. [The answer was] not to produce a foreign building. We had to avoid rejection by the communities. Coverage is not only providing a service, but also ensuring that it is accepted." 5.3 1 Success was evident in this dimension. Some excerpts from the 2002 visit report by Gerald Thacker, an international expert on judicial infrastructure, are telling.58 "The projects in Huehuetenango and Quetzaltenango and in PetCn are extremely well-designed, and contain many features that should be duplicated in other similar projects. The quality of the projects I saw in Guatemala warrants their use as models by the World Bank for other countries initiating similar projects (for example, the Philippines). The projects I saw are so well done that a fuller documentation of them would serve as a wonderful resource for similar projects in other countries. In that regard, I will discuss with faculty at the Georgia Institute of Technology, College of Architecture the possibility of documenting representative projects from the Guatemalan Although the comments are positive, it is regrettable that Thacker seemed to be missing the relevance and significance of the regional adaptation effort made by Guatemala. 5.32 . Promotion of ADR mechanisms was a key element in the design of the project in terms of access. As part of the diversification of judicial services and the reorganization of the Peace Courts, the project included the design, creation, and operation of an ADR unit, and a pilot program for mediation in civil, commercial, family and labor matters in the lower courts of Guatemala City.6o The increase in the number of mediation centers has been steady, particularly since 2005. Furthermore, the increase has been more important in the provinces, 58 http:/huildcourts.com/bio.aspx. 59 Thacker, Gerad (2002). 6o World Bank (1998), p 6. 22 such as Alta Verapaz, Quetzaltenango, and SuchitepCquez,with a high proportion of indigenous groups.61 5.33 Between January and December 2008, these centers handled some 15,000 cases, two thirds of which were voluntary (compared with 17.5 percent of cases referred by the ordinary courts, and 13.7 percent of cases derived). By area of law, the highest percentage was civil cases (57 percent), followed by family (23 percent), criminal (16 percent), commercial (3 percent), and labor (1 percent). 5.34 The interviews with the judges in one of the remotest zones of Guatemala, PetCn, suggest that the initiative of the mobile courts can be considered a success for two reasons. For one thing, the initiative lasted longer than the term of the project, and was adopted by the judicial system to be extended to regions other than those where the two pilot tests were conducted (Guatemala and Quetzaltenango).62 For another, judges of the remotest areas perceive this as an ideal-and perhaps the only-solution, to the problem of access to justice and the large territories, which are geographically and economically remote. 5.35 It seems clear that the mobile courts have facilitated access to justice. Between May 2003, when they opened, and December 2007, the two mobile courts in the pilot exercise served 5,697 users, that is, an average of 610 users per mobile court, per year. Although the number appears low, it exceeds the 4 12 persons served on average in each of the 73 mediation centers in 2007. In both cases, these are the initial years of new forms of access to justice, which will undoubtedly require more time to consolidate. 5.36 The mobile courts also contributed to the objective of decentralizing the administration of justice. Both for access and for decentralization, these are a less expensive alternative than the construction of Peace Court buildings in each locality. Furthermore, they can be used to estimate the demand for justice in certain places, and hence, as a mechanism of ?trial and error,? facilitating the planning of infrastructure investment. In the two mobile courts in the pilot scheme, the regional differences in types of cases and in users served between 2003 and 2007 were ~ u b s t a n t i a l . ~ ~ JUDICIAL A CREDIBLE SYSTEM 5.37 The suggested key performance indicator for credibility in the judiciary was an ?increase in user confidence.? No baseline value was set for this indicator. Several surveys have been undertaken but they are not always comparable. Between 1997 and 2003, the ?perception of the performance of the judiciary has not changed. Its situation in terms of external image remains weak in the 5.38 Another survey was carried out in 2007 among users ofthe judicial system,65asking about their confidence. In a range from 0 ?I do not trust? to 10 ?I trust very much? the average rating for the Orgunismo Judicial (the Judiciary) was 5. About half of the users 61 UMOJ (2009), p. 91. 62 In 2007, 10 additional mobile courts were put into operation. UMOJ (2009), p 94. http:liwww.oj .gob.gt/index.php/juzgados-moviles. 63 UMOJ (2009), p. 94. 64 Arag6n y Asociados (2003), p. 17. 65 Novoa and Hasuike (2007). 23 (46 percent) reported medium levels of confidence (4 to 6 in the scale), and those who reported high confidence more than doubled those who had low confidence: 37.9 percent of ratings were between 7 and 10 versus 15.5 percent between 0 and 3 in the scale. Even with no previous benchmark for comparisons these results look acceptable in terms of credibility. 5.39 In the same 2007 survey, a question to internal and external u s e d 6 supports the idea of a favorable perception of the performance of justice throughout the last ten years. In both groups a significant majority believed that the judiciary was working better than it did 10 years before (Figure 5). Source Novoa and Hasuike (2007) 5.40 It is not easy to determine the source of this perception of improvement in justice. In the same survey, external users mentioned particular aspects of justice that had improved. First, there is physical infrastructure: 9 1 percent of those interviewed considered this aspect of the system to be better than before. Next in improvement, in the survey, was the way the courts are equipped (80 percent), the system of criminal records (antecedentespenale~) (79 percent), and the coverage of the courts (73 percent). 5.41 One observation should be borne in mind when interpreting the public opinion surveys on Guatemalan justice. The lowest indicators of confidence correspond to the institutions that are most directly involved in criminal affairs, such as the National Civil Police, or the Ministerio de la G o b e r n ~ c i d nresponsible for the penitentiary system. In the ,~~ 2007 survey, both institutions got an average rating of 3 out of 10. Thus, part of the mistrust which is still evident in relation to the judicial system seems to be related to the still weak performance of the criminal justice institutions. It could be argued that the noncriminal justice system, in which the Bank's project focused, is better perceived among users than the judiciary as a whole. 5.42 The two-fold impression from these surveys that: (i) credibility in the judiciary is higher today than a decade ago; and (ii) the criminal jurisdiction is still the most problematic, was supported by almost every interview done in Guatemala. 66Theformer include judges and their staff. The latter include lawyers, nonjudicial authorities, and members of civil society. 67 Novoa and Hasuike (2007). 24 5.43 Without knowing that this was one of the programs of the World Bank project, a Justice of the Peace in Dolores, PetCn, mentioned A D y with the Judiciary (Un diu con la a Justicia) as an example of a good initiative. This education program, included in the project as part of the component ?improved social communications,? was later adopted by the judiciary, and introduced nationwide with the support of the judges and the Ministry of Education. It is addressed to those under 15, enrolled in public primary schools, who take part in a contest for school performance, to be allowed to approach the judiciary. The prize for the best students in each promotion consists of two visits, one to the Provincial Tribunals, and the other to the capital, where they are received by a plenary meeting of the Supreme Court of Justice. At the request of the educators, the coverage of the rogram has been extended. Some 300 municipalities, and 250,000 children, took part! The Justice of the Peace interviewed was clearly an enthusiastic participant in the project, and he clearly saw that the program was a good way to enhance, for hture citizens, credibility in the judiciary. 5.44 The evaluation of the impact of legal education programs for the young is still very new, even in developed countries. Despite this observation, there are some advantage^.^' So the early contact of the young with the judicial system which the program promotes seems favorable, and a target population of close to a quarter of a million young people from each cohort is a very significant number in terms of improving credibility. IMPROVE CONSISTENCY AND EQUITY THE APPLICATION THE LAW IN OF 5.45 Policy on traditional justice systems was a mechanism designed in the project to improve consistency and equity in the application of the law. In the peace accords, it was recognized that discrimination against the indigenous peoples with regard to the provision of the services of justice had two dimensions: the difficulties of access to state justice, and the lack of recognition of their own systems of justice.70 5.46 A QuichC indigenous mediator, agreed to be interviewed in Chichicastenango on Good Friday, which was not a working day. She is 3.0 years old and studied social work. She was selected by contest among 20 participants, most of them lawyers, based on merit, and has plans to do graduate study in mediation in Argentina or Spain. It is not easy to determine exactly what components of the project, or of the whole judicial reform program, contributed to engaging her interest. Different kinds of impediments had to be overcome: physical obstacles, such as the construction of the courts; budget limitations, to be able to pay her; human capital and recruitment methods (that she would be interested in becoming a mediator, and would pass the examinations on her own merits). There were also cultural obstacles, such as the mediating parties accepting that an indigenous woman could put an end to a dispute, or that her husband would think it acceptable for her to work. However, ignorance about the possible contribution made by each factor, or the impossibility of measuring the contribution of each of them, does not negate the importance of the fact that she was recruited by the judiciary; that she is motivated, that she resolves some 500 cases a year, 80 percent of which are in the Quiche language, and that she wishes to do graduate studies for continued improvement, and as she herself says, ?bringing in more people all the time.? Even if this were an isolated case, it would be unwise not to consider it as a significant achievement of the reform in Guatemala. UMOJ (2009), pp. 115-1 16. 69 See, for example, Wintersteiger (2008). O ? FDPL FMM (2004), p. 18. 25 5.47 This mediator is not an exception. Of the 82 mediators currently serving, 56 percent are women, and 63 percent come from the indigenous communities. Indigenous women represent 28 percent of the total number of mediators. Almost two-thirds of centers are under the responsibility of indigenous mediators.71 5.48 The coordination of official justice and indigenous customary law has not been an easy task. For some, there are still legal-even constitutional-obstacles. There are also practical problems related to language, even though progress has been made in the appointment of indigenous judges.72 5.49 Although they are still uncommon, cases in which Guatemalan judges accept decisions made under indigenous law are beginning to appear. In 2005, the Supreme Court of Justice recognized, in a case of robbery, that a person tried by the indigenous system could not then be tried a second time by the official system. ?The School of Judicial Studies has made a great effort in training a group of more than some 600 individuals, including judges, clerks, and officers, who are indigenous, bilingual, and members of the community. The training has been conducted mainly in the Peace Courts. Judges have been sensitized and have assumed a different attitude to indigenous law.?73 5.50 Overall, efficacy of objectives for Guatemala is considered to be substantial. There has been an advance in the recognition of indigenous justice. The increased availability of courts in remote parts of the country is a first step in increasing access to justice. In particular, the mediation centers have been a mechanism for approach and coordination between the two systems ofjustice: the official one and the indigenous customary law. This is not only due to the ethnic composition of mediators but also that of the users, which today is similar to that of the general population in general. Certain coincidences in procedure-in mediation and customary law-and a greater sensitization of judicial operators have also helped to close the gap between the two legal systems.74 Colombia IMPROVING THE RENDERING OF JUDICIAL CONFLICT RESOLUTION SERVICES 5.5 1 The Colombia project had only one objective, which was to improve the rendering of judicial resolution services. The output indicators stated in the PAD focused on judicial performance and efficiency: (i) a reduction in processing time for case disposition, (ii) increased number of cases disposed per judge, and (iii) increased satisfaction of users. 5.52 The achievements of the Colombian project differed from city to city. Positive results were concentrated in Bucaramanga, the city that showed the highest efficiency indicators during the project (completed caseshew cases), exceeding 200 percent in 2003 .7s Other indicators show that, at the country level, the clearance rate increased from 15 percent in January 2001 to 120 percent in June 2006, while the total backlog decreased from 393,324 71 Data from Unidad RAC. 72 FDPL FMM (2004), p. 27. 73 FDPL FMM (2004), pp. 3&3 1 i 74 FDPL FMM (2004), p. 30. 7s CSJ (2006), p.55. 26 cases to 73,849 cases and the resolution rate increased from 62.6 percent to 81.26 percent during the same period. IEG found no statistically significant difference between developments in the aggregate and in the courts covered by the project.76 5.53 Aside from that distinction, there are two major issues with the data and what it reflects. The first one is the quality of the data itself. Among Colombian scholars, the government, and other stakeholders, there is deep mistrust in the judicial statistics of Colombia, even to the point of questioning the CSJ's Annual Report to Congress.77 As pointed out by one scholar, "the figures on new cases and cases cleared in the Report to Congress for 2001-2002 do not coincide with the data included in the Reports in 2003- 2004."78 5.54 Perhaps more importantly, the second issue, is that the output indicators selected for the M&E under the project were drastically affected by factors exogenous to the project itself. These external shocks and developments, which are behind the broad trends indicated above, were not taken into account in the design, implementation, or evaluation of the project by the Bank. 5.55 Although Colombia's financial crisis was less damaging then Ecuador's, it nonetheless had a significant impact on the performance of the judiciary. In the second half of the 1990s Colombia faced a financial crisis and an unprecedented economic downturn, due for the most part to very high real interest rates induced by substantial fiscal imbalances as well as to a breakdown of supervision of publicly owned banks. The downturn and the high real interest rates led to historical peaks in non-performing loans, especially mortgages. 5.56 Furthermore, a few Constitutional Court judgments completely altered the regulatory framework of the savings and loans system and the financing of real estate and housing. Various government measures adopted for debtor relief were insufficient, and debtors took advantage of the leeway offered by the Constitutional Court and embarked in an active process of "seeking individual solutions through the courts."79Thus the weakness of the civil courts was made all the more evident by the conjunction of the numerous processes initiated by the mortgage lenders, and the intense activism by the debtors, who were ready to dispute all sorts of issues before the courts. 5.57 The ensuing huge congestion in the civil courts led to the reform the Code of Civil Procedure (CPC) in 2003. The reform concentrated on procedural changes to the juicio ejecutivo, the special process used in debt collections.80In addition, the government moved to increase the number of civil court judges from 809 in 2001 to 854 in 2004, an increase that was later reversed as the number of new cases declined. 76 The clearance partial rate (judgmentshew cases) is three percentage points higher in the project courts, but the 95 percent confidence interval goes from -3.1 to +9.2 percentage points. In addition, the clearance total rate (judgmentsltotal case load) shows a 1.7 percentage point weaker performance in the project courts, with a 95percent confidence interval that goes from -9.5 to +6.0 percentage points. Data taken from World Bank (2009). l7 Consejo Nacional de Politica Econdmica y Social (2008). p 33. 78 Vizcaya (2008), p. 128. 79 CuCllar (2006), p. 39. 80 Law 794 of 2003 Diario Oficial No.45.058 - January 9,2003. http:llwww.superservicios.gov.coibasedoc/leyes.shtml?x=55OO5. 27 5.58 The impact that the financial crisis and higher judicial activism had on the workload of the civil courts, along with legislative and administrative emergency measures to solve it, were not even mentioned in internal evaluations of the Bank's project. In Said and Florez (2005), for example, for the peak of 1999, when new cases in the civil courts increased from 70,000 to 100,000 in a single year, no explanation is offered. The fall to less than 60,000 in the following year was attributed to "a transfer of competency from the civil circuit courts to the municipal courts, which face an additional demand that is generating congestion."" Oddly, the document went on to make the point that `'this phenomenon did not affect the project data, because the project began to be executed in 2002." What is even more disturbing is that the drastic increase and subsequent decrease in completions was totally attributed to the project. "The impact in terms of decongestion during 2003 was very significant. Almost 105,000 cases were completed, and the stock was reduced by 30,000 case files. The evolution of the stock of cases shows a positive impact from the project in the only variable it controls (completion)."82 No reference is made to the drastic measures taken in 2003, and 2004, such as the change in the Code of Civil Procedure and the creation by the government of special decongestion courts. 5.59 In the various implementation reports, the emphasis was placed on the congestion rate and the clearance rate. There was a clear effort to ensure that these indicators assess the project according to international parameters. The tacit assumptions in these reports were that the project was being executed under constant demand and that the legal framework remained the same and legal procedures had not been changed. The monitoring of the project then mistakenly pointed out that the efficiency gains in the Circuit Courts are the result of a combination of a new management model, articulated around a modern case-tracking system (Justicia XYI), physical infrastructure remodeling, upgrading to modern computer equipment, suitable training, and teamwork incentives. The environment and the legal system in which the program was being executed were thereby not adequately accounted for. 5.60 The omission of exogenous factors was repeated in the ICR. It compares of the basic aggregate statistics between the peak of the crisis and those of the period after the emergency measures, again disregarding the environment and the changes in the legal system. "The clearance rate increased from 15 percent in January 2001 to 120 percent in June 2006. The backlog decreased from 393,324 cases to 73,849 cases during the same period. Overall, the backlo of the 73 project courts by end-2004 amounted to only 19 percent of the backlog in 2000." 8 5.61 The scant importance assigned in the project documents to the impact of the financial crisis on the judicial system is disquieting, given that previous work by the Bank (2002) had already pointed out that factor.84 5.62 In 2005 the Consejo Superior de la Judicatura (CSJ)-the executing agency for the project-commissioned an opinion survey in 2005. The survey covered users-litigant lawyers, plaintiffs and defendants-of the civil circuit courts in the five cities that took part in the program. No baseline values were established at the beginning of the project, and this particular survey does not allow a comparison with the courts that were not part of the 81 Said and F16rez (2005). 82 Said and F16rez (2005). 83 World Bank (2007),p. 12 and p. 50. 84 Said and Varela (2002),p. 35. 28 project, nor with other cities. Only one of the questions in the survey allows the possibility of a comparison with the situation of the courts before, or without the project. In the form given to the litigant lawyers to fill, there is a question about the time it takes to clear cases.85 The litigant lawyers represent a large chunk of the cases handled, and represent a qualified group to the extent that they are experts that work with the higher (circuit) level courts. Their replies to the questionnaire were not positive for the project. Only 16.4 percent of them perceived that civil circuit courts covered by the project had improved in the sense that the time required to complete a case was now shorter than the "usual time." By contrast, 27 percent consider that the time was now longer, and 56 percent thought that the time had not changed.86Balancing these responses, if anything, the project seems to have worsened things rather than improved them. 5.63 In 2007, the Universidad Externado de Colombia, at the request of the government, did a detailed evaluation of the pr~ject.~'Based on judicial statistics provided by the CSJ, this evaluation found that the impact of the project was negligible and, when positive, it had been transitory. Furthermore, it found little systematic difference between the performance of pilot courts and the other courts. With the project, congestion was reduced only in Medellin, and remained stable or was higher in the rest of the cities. Interestingly enough, the congestion indicator was lower in the courts of the cities not included in the project. 5.64 As part of this same evaluation, another survey was done in 2007, canvassing 98 users in the five cities where the project was executed. More than half of those surveyed (55.1 percent) indicated that they had no idea that a project for improving the civil courts had Users been im~lernented.~~ perceived changes in the way that services were being offered, with better attention to customers.g0 In addition, almost half (47 percent) considered that the speed with which services were being provided by the courts was eitherfair or deficient." 5.65 External analysts interested in the impact of the financial crisis and regulatory changes on the performance of the civil courts completely discount or disregard the eventual impact of the Bank project. Some of those who worked closely with the CSJ at the time were not even aware of the Bank's efforts to promote the effectiveness of civil courts. On the other hand, they attribute the sharp reduction in the backlog ofjuicios ejecutivos involving mortgages--from a maximum of 123,000 in 2002 to about half (68,000) in 2005-to two factors: (i) the reforms to the Code of Civil Procedure introduced in early 2003; and (ii) 45 decongestion courts created in March 2004, which operated with the urpose of dealing with this type of case in the three largest cities until the end of that year.f2 5.66 Overall, efficacy of objectives in Colombia is considered to be negligible. CSJ (2006), p. 92. 86 CSJ (2006), p. 76 This evaluation was financed by a Japan Policy and Human Resources Development (PHRD) Grant requested by the authorities following the closure of the project under evaluation. World Bank (2009). 88Vizcaya(2008), p. 125. 89 Vizcaya (2008), p. 88. 5~ Vizcaya (2008), p. 90. 9* Vizcaya (2008), p. 92. 92 Cuellar (2006), pp. 395-396. 29 Software Development 5.67 Software development for case management deserves special attention because it was a common feature in all three countries, and because there were major issues with it in all cases. These issues, which would have not come to the surface easily without this in-depth evaluation, point out to an unfortunate lack of protocols and standards in the projects regarding the measurement of the output and outcome, as well as the costs, of the components related to software and computer systems. 5.68 In all three countries, the software development component was the least satisfactory. The projects brought in external-to the judiciary and to the country-"experts" to develop expensive software programs, which were far from custom made or user friendly, and that needed extensive adaption to local procedures and to the Law of the countries. 5.69 In Guatemala, various judges were of the opinion that the previous and home-grown i l - e f c see , system- W l y P r e t Box l-outperformed the international experts' software, originally designed for criminal justice cases using the Law, procedures and mores of a different country. Initially these judges were very reluctant to adopt the new and, for them, inferior platform. Their main concerns were: (i) ease of use; (ii) consistency with Guatemalan Law and formal and informal procedures; and (iii) the loss of very valuable files in the migration to the new system. Box 1. Willy Perfect In the field visit to Guatemala, in a meeting with judges, talking about the case-management software, one of them candidly said: "Willy-Perfect worked better". Asked about Willy-Perfect, an interesting story came out. Years before the international experts working with the project came in, Willy Ochoa, an amateur computer expert and at that time a court clerk (secretary), had developed a set of Word- Perfect macros simply to do his job more efficiently. This plain, unsophisticated, but quite useful tool was rapidly adopted and adapted by fellow workers and even by some judges, who were beginning to use computers. It soon became known as "Willy-Perfect." It was a good example of what is now known as "open architecture" software: every user made suggestions or changes and Willy, working on weekends on his own time, made the necessary adjustments for sharing both innovations and information. When the Bank project began, Willy Ochoa, quite modestly, wrote a letter to the international experts. He simply wanted to be part of the team in charge of the development of the new, high-tech, software. His request, however, was rejected and he was never brought into the software development team. When the experts' software was finally implemented, Willy Ochoa himself had to work on the "migration" of information to the new system and data base. Source: Interviews with Willy Ochoa and withjudges in Guatemala City il-efc, 5.70 The Colombian version of W l y P r e t officially and emphatically denied by a magistrate of the CSJ,93was a set of templates and macros developed by a jud e from Itagiii that, according to many experts, became the core of the Justicia XYI softwaref4 In a twist that is puzzling about CSJ, within that institution there is no agreement on what was the real 93 Personal communication to the evaluator from a former magistrate of the CSJ. 94 Interview No.44. Justicia XXI is a software commissioned by the CSJ, with its own resources and at a cost of US$3 million, between 1996 and 2000. 30 output from the software component of the Bank's project. For some it was an overhaul of Justicia =I, for others it was mostly training users of that software. 95 5.71 In Ecuador, even among the technical staff of the judiciaryg6there is no agreement on whether the current software is an upgrade of the product supplied by DPK, an international consulting firm, or whether it is based on a native system that was developed by a student working on his college dissertation in consultation with some judges. In any event, the current platform is the result of needed continuous upgrades and adaptations by a group of Ecuadoran programmers, and can be considered mostly native at this stage. There are many advantages of this new, native, software, which can be grouped in two. For one thing costs- salaries and royalties. DPK's software was written in Xnear/workflow, which entailed a considerable yearly fee. The new, native, software is developed in DELPHI and pays no annual royalties. The differences in final price tag of each version are huge: DPK software's cost was in the range of US$1 and US$2 million, with annual royalties of around US$200,000. According to the information supplied by different programmers to this evaluation, the development of the new platform was done with cost of about US$50,000 in salaries, and no royalties are being paid. For another, the new software is really custom made and adapted to Ecuadoran Law, procedures and mores, because it is the result of long-term, day-to-day, interaction between users and developers. The learning and training processes provided and continue to provide the opportunity to easily incorporate suggestions for improving the interface. Accordingly, improvements have been, and in the future can be, incremental. In addition, these improvements bring in comments and suggestions from different cities. Last but not least, the new, native, software is fully adapted to Ecuadoran Law and has been adjusted smoothly to take into account changes in legislation. 95 Management indicates that most project resources were allocated to providing the hardware for that software to operate (including an integrated network) and strengthening human resource skills relevant for the court and case management applications of the software. 96 For example, Rupert0 Amaguai, Jefe de Sistemas of the Consejo de la Judicatura, himself part of the team that developed the new platform. 31 6. Ratings Outcome 6.1 This review rates the projects' outcome as follows: Ecuador, Satisfactory, Guatemala, Satisfactory and Colombia, Unsatisfactory. Table 3. Ecuador: Efficacy by Objective and Outcome Rating Objective Relevance Efficacy Outcome Effectiveness and transparency Modest Expanding ADR Substantial Improving Access ... .. ....................____________._______...............................-..I substantial ............................ ........... ....... Fostering initiatives Substantial Summary Rating substantial substantial Satisfactory Table 4. Guatemala: Efficacy by Objective and Outcome Rating Objective Relevance Efficacy Outcome Effectiveness .... . Modest Accessibility Substantial Credibility Substantial Consistency and equity ............. .... .............._.. High Summary Rating High Substantial Satisfactory Table 5. Colombia: Efficacy by Objective and Outcome Rating Objective Relevance Efficacy Outcome Improving j udicial services Negligible Summary Rating Negligible Negligible Unsatisfactory Risk to Development Outcome 6.2 Sustainability of the projects depends on the degree of institutionalization and the way in which the initiatives proposed have been adopted by the judiciaries involved. In this regard, the results were good in two of the countries. 6.3 In Ecuador, several years after the project closed, Projusticia is still recognized in the media as "the body responsible for the modernization of the justice system,"97and the entity responsible for coordinating the agenda between the Judiciary, the Executive, and Congress. The consolidation of this coordination work to support the judicial system was expressed in 97 "Projusticia busca rnodernizar la Corte." Hoy, May 21, 2007. www.hoy.com.ec. 32 the creation of the Ministry of Justice at the end of 2007.98 Both the first Minister of Justice and the present one are former directors of P r o j u ~ t i c i a . ~ ~ 6.4 In Guatemala, the Judiciary assimilated the project well in institutional terms. This undoubtedly has to do with the magnitude of the reforms. In every instance during the visit of the team for this evaluation, comments gave the impression that the project's legacy was something which was still alive. Despite its small staff, there is a perception of continuity between the project coordination and the current Unidad de Modernizacidn del Organismo Judicial (UMOJ-Unit for the Modernization of the Judiciary. All files and documents from the project are available, and the person currently responsible, Napoleon Guix, diligently responded to all questions. Maria del Carmen Ortiz, sub-Coordinator for six years in UMOJ, is currently the General Manager of the Judiciary. Another indication of sustainability of the project is that some of the actions undertaken continue autonomously, and in some cases, without being identified as having been initiated by the Bank. No factors have been detected which might endanger the main achievements of the project: the consolation of the judiciary, greater access, and integration with indigenous justice. The emphasis on the construction of infrastructure, and the fact that more than adequate quality standards have been achieved, is also positive for sustainability. 6.5 Unlike Ecuador or Guatemala, where it was relatively easy to find an organization or a person to be the guardian of the institutional memory of the project, in Colombia this task was much more elusive. The CSJ seems to be concerned with other matters, and there is no great interest in speaking about or discussing something considered as remote as the project. Reflecting this way of seeing things, the main promoter of the project disapproves of the custom of evaluating the project two years after it closes. loo Even before the project ended, CSJ seemed to be engaged in rather different endeavors. Moreover, the government itself obtained a Japan Policy and Human Resources Development (PHRD) Grant to develop its own strategy for continuing improvements in the justice system, including by broadening access to it. These resources were also used to finance the evaluation of the project carried out by Universidad Externado de Colombia and to create a new set of baseline data for a new policy agenda. 6.6 Two sections of the careful evaluation made at the end of a project by Universidad Externado de Colombia corroborate these observations. The first is the impression among some in the judiciary that the project had not even been formally closed and what was to follow: "The way in which the process concluded was inadequate, there was a gap, people were up in the air, no one knew what had been achieved, what was missing, what had happened, what was going to The second is a remark that the remaining files of the project are deficient, and the recommendation that when similar programs are undertaken in the future, an effort should be made to construct and consolidate an institutional memory.'02 The project did not leave its mark, even in the CSJ where, as a matter of principle, people should be aware of what was done. A disconcerting and surprising finding '* Decreto Ejecutivo No. 749, November 15,2007. 99 "Ministerio de Justicia cobra forma." Hoy, November 6,2007. loo Entrevista 41. lo' Vizcaya (ZOOS), p. 85. Io2Vizcaya(2008), p. 16. 33 was that neither the associations of lawyers--experts in procedure or litigation, lo3 nor two well-respected research centers on judicial matters lo4 were aware of the existence of the project. It is thus regrettable that there will be little institutional or academic memory of what went wrong and why in a case where learning from an unsatisfactory outcome would be fruitful. 6.7 Based on these considerations, this report rates the risk to development outcome as presented in Table 6. Table 6. Risk to Development Outcome Rating Ecuador Guatemala Colombia Rating Modest Negligible Significant Monitoring and Evaluation: Design 6.8 In terms of M&E design, the evaluation assessed the extent to which the project was designed to collect appropriate data given objectives and given existing and available data. 6.9 To measure success of the monitoring framework, the appraisal documents of the projects proposed looking at the following key performance indicators: Table 7. Suggested Key Performance Indicators Ecuador Guatemala Colombia Time from filing to disposition. Percentage increase in user Significant reduction in confidence. processing time for case Number of dispositions. disposition in participating Percentage increase in Judiciary courts over a two-year period Cost to process cases to Branch coverage and access (compared to baseline). conclusion. lo5 outside of Guatemala City. Increased number of cases Number of corruption-related disposed per judge per year complaints received and (above baseline). resolved. Increasing satisfaction of users of participating court^.'^' Policy on traditional justice systems developed and tested. Number of subprojects executed through the participation program.lo6 Instituto Colombiano de Derecho Procesal, Colegio Nacional de Abogados Litigantes, Legal Office of COVINOC. 104 Cijus & Dejusticia. lo' WorldBank(1996),Annex 15,p. 119. (1998), Annex 1, p. 1. 106WorldBank lo7 World Bank (2001), pp. 2-3. 34 The specification of baselines and target values for overall indicators varied among projects. It was quite precise for Ecuador. In Colombia, specification was more flexible. In Guatemala, some broad ideas were stated about how performance indicators could be measured (surveys, progress reports, Supreme Court documents, focus groups) but no baselines or target values were specified. 6.10 Other performance indicators were proposed in the appraisal documents. Both in Ecuador and Guatemala, each project subcomponent had its own performance indicator. Most of these indicators were limited to the delivery of an output. 6.1 1 The M&E design of the three projects, and especially in Ecuador and Colombia, shows a quantitative bias without a previous assessment of availability or quality of information needed to calculate the indicators. 6.12 Key performance indicators were mostly based on judicial statistics or opinion surveys. For the former, there is a conflict of interest arising from the fact that the organization responsible for compiling the judicial statistics is simultaneously being evaluated with these figures. This conflict of interest was ignored in the design of the M&E. 6.13 Opinion surveys, when carried out by the same agency whose performance is to be assessed, are hardly reliable. The second limitation of opinion surveys relates to how representative the chosen sample is. Knowledge of the users of justice is tenuous, and knowledge of potential beneficiaries even more so. Perhaps the only qualified group would be trial lawyers, yet their opinions may be biased by their particular interests. Under these conditions, the design of a sample framework that will properly represent the population affected by the reforms is not an easy matter. These two problems were not discussed in the M&E design. 6.14 Finally, public's perception of justice is heavily influenced by the facts that judicial systems are too complex and heterogeneous, and that the performance of its various branches depends on the actions of a number of related state agencies. For instance, the performance of the police or other State law enforcement agencies, both in a way alien to the judicial system, has a bearing on the image of the criminal justice system. As was the case in Guatemala, the public's perception of justice may be adversely affected by the performance of these alien institutions. Monitoring and Evaluation: Implementation and Utilization 6.15 In terms M&E implementation, the evaluation assessed the extent to which appropriate data was actually collected using adequate collection methods. For M&E utilization, the evaluation was looked at the extent to which appropriate data was used to inform decision-making and resource allocation. 6.16 In the three projects, it was assumed that the task of compiling judicial statistics could be delegated to computer software being promoted with the project, which was expected to produce them as an automatic byproduct of file management. The results obtained shed a dim light on the availability, quality and impartiality of that information. With a few exceptions, judicial statistics to date are either nonexistent, or are not reliable, or are not transparent and made public, or are not vetted, discussed or analyzed in an independently rigorous manner. 35 6.17 Baselines values for key performance indicators were not always established, even during supervision. When opinion surveys were done at different moments of project implementation, they were not always comparable. 6.18 The so-called pilot programs were not always true pilots because there was no controlled experiment on a small scale, executed in advance to check the feasibility or to improve the relevance of a preliminary design. For the projects in Ecuador and Colombia, the term "pilot" was used simply to refer to many courts in which simultaneously, not sequentially, and in different places, the project was executed. 6.19 Physical infrastructure and support for computer systems are recurrent components in the judicial reform projects of the Bank. When a significant portion of the budget is invested in physical construction, it is surprising that the specifications, unit costs and characteristics of these works receive such little attention. No detailed procedures were drawn up to receive them, and the evaluation design was implicitly restricted to an attempt at measuring their eventual impact through surveying user opinion. The same point can be made regarding the funds invested in software, for which no effort was made to determine what the outcome should be, or how to evaluate its cost. 6.20 The use ofjudicial statistics for evaluation was not complemented with a detailed analysis of the circumstances affecting the performance of the courts. As noted above, in Ecuador and Colombia the chosen indicators were substantially affected by factors exogenous to the projects. Nevertheless, the evaluation completely disregarded these exogenous factors that were bringing significant changes to the civil courts covered by the project. Moreover, major developments within the courts themselves appear not to have been accounted for in Colombia-in 2007, about 81 percent of the cases were judged to be inactive, that is, without any reaction from the parties to the lawsuit in the last year.lo8 Surprisingly, this dead load does not seem to have been identified when monitoring and evaluating the project. 6.2 1 Survey-based indicators also showed their limitations. In Colombia, for example, one of the surveys was done directly by the Consejo Superior de la Judicatura (CSJ, with no baseline or control group. The survey could not, therefore, provide the basis for an evaluation. And the only reliable answers of the survey, those of the trial lawyers, were left aside when summarizing the results. 6.22 The problems of opacity, poor quality, conflict of interest, or simply nonexistent judicial statistics, or bias or limitations in opinion surveys, did not receive the attention they warranted in these projects. The inability to calculate reliable indicators based on judicial statistics was evident in all of the projects. Instead of facing this stark reality, all three projects ignored the difficulties evidently on the assumption that these would resolve themselves, say by fixing some technical obstacle. 6.23 The interviews for this evaluation suggest that the demand for judicial statistics is small or almost nonexistent. Also, that the incentive to generate them is weak. For the judges, and for the support staff in the courts, they are no more than a nuisance. Among the judges of the five Colombian cities where the project was executed, there was a consensus that the least-used part of the management software was that of the statistical template^."^ log Consejo Nacional de Politica Econ6mica y Social, (2008). 109 Vizcaya (2008), p. 103. 36 Furthermore, a number of judges said that they found it rather unreliable, and almost irrelevant, because they were easy to manipulate. 6.24 Only in Cuenca, Ecuador, are judicial statistics available to the public."' Despite the good quality of computer programs in that city, the collection of information has taken place manually by its main user, the local delegate of the Council of the Judicature. Once a year, since the project began, he visits all courts in Cuenca to determine, with every clerk, how many new cases came in during the previous year, how many were cleared, how many are active, and how many can be cleared out of the statistics. 6.25 Also, only in that city a few indicators based on those statistics are used as management tools. The Cuenca delegate of the Council of the Judicature trusts his data charts because he gathers the data himself. Furthermore, he maintains permanent contact with judges and administrative personnel, and therefore he is aware of all the eventualities that could affect the functioning of the courts, and the major changes in his data charts. For the annual evaluation of the results of his jurisdiction, these statistical figures are only one element that for the most part is just a complement to the detailed personal dialogue that he maintains with each of the judges. The officer admittedly does not automatically use the statistics to support decisions. They are only one more element in a participatory process on the allocation of resources. 6.26 Based on these considerations, this report rates M&E as follows: Table 8. Monitoring and Evaluation Rating Ecuador Guatemala Colombia Rating Modest Modest Modest Bank Performance 6.27 For the rating of bank performance, this evaluation assessed the extent to which services provided by the Bank (i) ensured quality at entry of the operation and (ii) supported effective implementation through appropriate supervision. 6.28 In terms of ensuring quality-at-entry, in Ecuador the project responded both to the Bank's strategy and to the county's judicial policy. Also, there was a detailed assessment of the judiciary before the project was carried out by the Bank. The design of the project was based on this assessment. 6.29 In terms of significance in the quality of supervision, one black spot in this project was related to the software contract with the firm DPK. The company delivered a very expensive software that needed a great deal of upgrades and a complete adaptation to Ecuadoran Law, procedures and mores. Not surprisingly, Ecuador ended up replacing DPK's program with a home-grown platform, as discussed above. This evaluation notes that this replacement highlights the fact that the project itself had already generated enough internal capacity to replace the faulty software, and, to the Bank's credit, it supported the adoption of the new, home-grown platform. Surprisingly there is no discussion of the poor performance 'lo http://www.funcionjudicial-azuay.gov.ec/principal.htm. 37 of the consultancy firm in the Bank documents, despite the substantial waste of resources and of the potential lesson to be learned. 6.30 Another weak point in the project supervision in Ecuador was that there was no recognition of the impact of the economic and financial crisis on the performance of the courts, nor of the differences in procedures and mores between the different regions. 6.3 1 In Guatemala, the project design and objectives matched the Bank's strategy for the country and the government's overall reform program. The fact that the Bank's project supported only a small part of an ambitious plan to modernize the judiciary, gives the impression that too many objectives were set for the project and that they were too diverse." The plan for the modernization of the judiciary was a byproduct of the peace accords and only a part of the undertakings stemming from them, a process was a political act that had broad participation, as well as international contributions and supervision. Taking this political dimension into account, it is difficult to imagine how the Bank project could have been more pointed on objectives and indicators. 6.32 It should be noted that the Guatemala project correctly omitted certain objectives that show up in other Bank legal and judicial reform projects. For example, there were no components aimed at strengthening markets or creating a legal environment favorable to private business and foreign investment. In an ideologically fragmented environment such as the one in Guatemala at the time of the project, the fact that the Bank demonstrated such flexibility led to greater country ownership of the project. 6.33 Despite the weighty presence of the criminal jurisdiction in the overall program for the modernization of justice, the Bank was only marginally involved in this area, which has been one of the most difficult and criticized. Given the detailed scrutiny of judicial matters in the country, the simple fact that the project bypassed, almost unnoticed, much of the criticisms of the judicial system in Guatemala is a real achievement. 6.34 In Colombia, the project deviated from the basic objectives of both the Bank's strategy for the country and the government, which were to reduce violence, poverty, and the linkage between the two. The idea that poor resolution of conflict was one of the causes of violence, which in the Bank's documents was understood to be critical in the most underprivileged areas, was reshaped at preparation to concentrate just on civil courts. With no justification, the problem of access to justice was left aside, despite being a critical element in the diagnosis of violence. 6.35 No effort was made during either preparation or execution to identify the potential beneficiaries of the project. Had this been done, it would have been clear from the beginning that the big financial conglomerates were the group that would gain the most from the project. 6.36 In the supervision of the project, the focus on unreliable aggregate statistical .indicators translated into a lack of attention to external economic factors and important legal changes that ended up playing a key role in determining the performance of civil courts. As noted by IEG in the ICR Review, the project "did not develop clear output and outcome indicators, leaving it to be done during project implementation. Even to the extent it did, the distinctions between outputs and outcomes were blurred." Moreover, following the closing of 'I1 World Bank (2008), p. 37. 38 the project, the government had to use a Japan PHRD Grant to finance the development of a new set of baseline data for a follow-up project.'" 6.37 Based on these considerations, this report rates the Bank performance laid out in Table 9. Table 9. Bank Performance Rating Ecuador Guatemala Colombia Ensuring Quality at Entry -......-...-.....I " Satisfactory " Satisfactory Unsatisfactory Quality of Supervision Moderately Satisfactory Moderately Satisfactory Unsatisfactory Overall Rating Moderately Satisfactory Moderately Satisfactory Unsatisfactory Borrower Performance 6.38 In terms of borrower performance, this evaluation assessed the extent to which the government and implementing agency (i) ensured quality of preparation and implementation, and (ii) complied with covenants and agreements towards the achievement of development outcomes. 6.39 In both Ecuador and Guatemala project preparation was done through a lengthy and participatory process of discussion with stakeholders. This was a process that was supported both by the governments and the judiciaries. There is no evidence of a similar process in Colombia, even in a smaller scale. 6.40 In Ecuador and in Guatemala, the interest of the governments and of the judiciaries in the objectives of their respective projects was maintained throughout the execution. Also, there was adequate dissemination and continuation of the essence of the projects after these closed. 6.41 In Colombia, by contrast, the government and its executing agency showed little interest in the project. Not even the design was discussed with all the relevant stakeholders; no such discussion was restricted to governmental agencies and institutions. Important procedural reforms to the civil justice were carried out during the project's execution with no communication between the project staff, the government, the legislature, or the parties that were promoting those reforms. Funds were disbursed slowly and, as a result, the project took longer to execute. The dissatisfaction with the project among judges in Medellin was such that in the final evaluation they collectively refused to answer the survey forms."3 6.42 Based on these considerations, this report rates the borrower performance as follows: Table 10. Borrower Performance Rating Ecuador Guatemala Colombia Rating Satisfactory Satis factory Unsatisfactory 'I2 Corporaci6n Excelencia en la Justicia (2009) and World Bank (2009). Vizcaya (2008), p. 81. 39 7. Lessons 7.1 Three clusters of lessons can be identified based on this review. These cannot, of course, be presented as definitive, but the implementation experience of the three projects provides sufficient elements for them to be put forward with reasonable confidence as to their significance and broader applicability. 7.2 Adaptation of Project Objectives and Design to Local Law, Conditions and Peculiarities. The importance of adapting project design to country conditions is a common lesson from other project contexts, but it is important that in judicial reform projects the adaptation extends to the full range of factors that may significantly impact implementation. Thus, anchoring the projects in a thorough understanding of the country and its conditions becomes a priority. In the case of the projects under consideration here, for instance, the possible effect of economic downturns and upturns on the workings of the justice system was virtually ignored, despite being very significant and notwithstanding the assumed link between judicial reform and economic development. Financial crisis in Ecuador and Colombia had a considerable impact on the performance of the judiciary in the course of implementation of the projects. Economic conditions determine the demand for judicial services, so it is important to take them into account in both the design and evaluation of projects. 7.3 The interpretation of "local" also needs to drill down well below the legal framework and the national level. In the projects considered here, implementation was influenced to a large extent by conditions specific to the distinct regions or localities of the client countries that were being targeted. What worked in Cuenca or Bucaramanga failed in Quito and Bogoth. Knowledge of the local environment continues to be sufficiently scant such that there can be little certainty, even expost, about the reasons for the variability in results from city to city. Both the design of projects and prior analytic assessment efforts need to keep in mind variations in conditions and peculiarities across regions and localities, and appropriate flexibility needs to be woven into the fabric of project design. For instance, what emerges from the experience of the projects under consideration here in architecture and construction of infrastructure, as well as in software development, but likely extends to other areas as well, is that local conditions require adequately customized designs, and they must be of local, not imported, origin and of manageable scale and complexity. In sum, successful judicial reform seems to be less about global technology, and more about local expertise, executed by people with intimate knowledge of the judicial system and Law and associated procedures. 7.4 Flexibility in Overall Approach, and Solicitation of Local Knowledge and Speci3c Benejciary Input, for Selection of Project Objectives, Design and Implementation Arrangements. While it is clear that identifying the right instruments, and the right agencies (among the multitude of other factors affecting project outcomes) ultimately has a determining influence on the success of Bank projects, what has been less clear is the "what" and the "how to." What appears to have been decisive in the case of the projects assessed here is the willingness to avoid constraining the overall approach to project conception and design. For instance, results of the projects appear to have been better in those components that, beyond economic considerations, privileged equality and the expansion of civil and social rights. Similarly, bottom-up or grass-roots approaches appear to have worked better than top-down approaches because they related more effectively to local mores and drew on local knowledge. To generalize, project design tends to be undermined when the Bank lacks 40 in-depth knowledge concerning the judicial system, the country's Law, legal procedures, customs, and mores. 7.5 In particular, components designed with the input of lawyers knowledgeable about the client country's Law and legal system and with contributions from local legal think tanks and NGOs typically proved more effective than components lacking this feature. Successful interventions-whether involving software internal to the judiciary or legal aid for poor women in Ecuador, architecture or antecedentes penales in Guatemala-were more likely to be based on detailed assessments of local needs and peculiarities. Beyond this, however, they were also more likely to be based on specific identification of, and close interaction with, the beneficiaries of proposed actions. A generalization of this latter point would suggest that it is better to try to satisfy the concrete needs of a known, specific group of people, as well as to seek a clear understanding of the conflicts that have impacted or will likely impact them and how the country's legal system might resolve them, than to design and execute a project the notional beneficiary of which is a more abstract entity, such as "the public," "the private sector," or "public opinion." 7.6 In sum, experience points strongly to the need to devote more resources in the design stage to finding the local experts-the Willys-to give them an appropriate role in the design, execution, and evaluation of the projects. This also appears important in order to sow the seeds of two critical properties of a project, which foreign experts seldom bring (at least by themselves-wnership and sustainability. Engagement of such local expertise from the outset would almost certainly result in diagnoses and proposals concerning project coverage that go beyond what happens mechanically in the courts, if only based on the fact that the local experts live and work in that environment and know the Law of the country. Diagnoses and proposals will better integrate local peculiarities and identify specific beneficiaries and conflict types affecting them. 7.7 Limitations of By-products of Case Management Software for M&E. One lesson that emerges clearly from the projects under assessment here is that useful and reliable judicial statistics do not tend to be automatically generated as a by-product of case management software. Indeed, such software tends to be a costly product with very few users. There are pitfalls in focusing attention on idealized indicators that are rarely available to enable adequate project M&E-this will tend, for instance, to detract from more rudimentary, yet often equally enlightening, tasks such as monitoring outputs. Equally, there are pitfalls in focusing exclusively, or even predominantly, on aggregate performance indicators-there may, for example, be a temptation to "think bigger" in the design of reform, at the expense of more modest initiatives that might otherwise result in very concrete gains in the administration of justice for specific categories of local users. 41 References Aragon and Asociados. 1997. "Estudio de Opinion Publica sobre la Administracion de Justicia en Guatemala.'' In Diagnostic0 del Organismo Judicial. Anexo 11. Guatemala. 2000. .2003. "Informe Comparativo de Estudios Cuantitativos de Opinion Publica sobre la Administracion de Justicia en Guatemala a Nivel Nacional". Informe a la Unidad de Modemizacion del Organismo Judicial. ASIES .2003. "Proceso de Fortalecimiento del Sistema de Justicia: avances y debilidades. " (Julio 2003 - Junio 2006). Asociacion de Investigacion y Estudios Sociales. .2006. "Proceso de Modemizacion y Fortalecimiento del Sistema de Justicia: Avances y Dificultades . " (Enero 2002-Junio 2003). Asociacion de Investigacion y Estudios Sociales. http://www.asies.org.gt/justiciamJFORME%ICO~02OARCHIVO.p df Camacho, Gloria and Kattya Hemhdez. 2009. Derechos, Violencia, Comisarias de la Mujer y Acceso a la Justicia. Percepciones de las mujeres del Cantdn Cuenca - Ecuador. Centro de Planificacion y Estudios Sociales. CEPLAES. Carothers, Thomas. 2006. Promoting the Rule of Law Abroad: In Search of Knowledge. Washington: Carnegie Endowment for International Peace. CGE. 2001. Auditoria de Gestidn a1 Programa Nacional de Apoyo a la Reforma Administrativa y Modernizacidn del Sector Judicial (Projusticia). Quito, Ecuador: Contraloria General del Estado. Chisaguano, Silverio. 2006. Lapoblacidn indigena del Ecuador. Andisis de Estadisticas Socio- domgrrijkas. Quito: Instituto Nacional de Estadistica y Censos. ind httD://www.inec.eov,ec/web/~uest/publicaciones/estudios/soc/pob ecu Consejo Nacional de Politica Economica y Social, Republica de Colombia. 2008. Concept0 Favorable a la Nacion para Contratar Emprestitos Extemos con la Banca Multilateral hasta por US$62.000.000, o Su Equivalente en Otras Monedas, Para Financial un Proyecto de Fortalecimiento a 10s Servicios de Justicia. No. 3559, Bogoth. http://www.dnp.gov,co/PortalWeb/Portals/O/archivos/documentos/Subd~eccion/Conpes/3 559.pdf Corporacion Excelencia en la Justicia. 2009. Linea de Base de la Congestion - Jurisdiccibn Ordinaria. Bogoth. http://www.cej .org.co/memorias-eventos/l7 14-la-cej-en-el-foro-quincenal-del-instito- colombiano-de-derecho-procesal CSJ. 2006. Encuesta sobre la Satisfaccidn del Usuario con el Servicio de Justicia en 10s Juzgados Civiles del Circuit0 de Bogotri, Cali, Medellin, Barranquilla y Bucaramanga. Bogoth: Consejo Superior de la Judicatura. Mayo. Cuellar, Maria Mercedes. 2006. L A la Vivienda Quien la Ronda? Situacidn y Perspectivas de lapolitica de vivienda en Colombia. Bogoth: ICAV, Universidad Externado de Colombia. Delazay, Yves and Bryant Garth. 2002. La internacionalizacidn de las luchaspor elpoder. La competencia entre abogados y economistas por transformar 10s Estados latinoamericanos. Bogoth: ILSA, Universidad Nacional. DeShazo, Peter and Juan Enrique Vargas. 2006. "Judicial Reform in Latin America. An Assessment." Policy Papers on the Americas, Volume XVII, Study 2. CSIS American Program. latin http://www.csis.org/medidcsis/~ubs/O609 iudicial reform.udf De Sousa Santos, Boaventura and Mauricio Garcia. 2001. (eds.) El Caleidoscopio de las Justicias en Colombia. Andisis Socio-Juridico. Bogoth: Siglo del Hombre Editores. Estrella Vintimilla, Pablo. 2001. (ed.) La Educacidn Legal en el Ecuador. Quito: EAFJ. FDPL FMM. 2004. "Justicia de Paz y Derecho Indigena: Propuesta de Coordination." Memoria del Seminario Taller. Fundacion Para el Debido Proceso Legal - Fundacion Myrna Mack. httu://www.dslf.ordudoads/1184703719.udf 42 Fuentes, Alfredo. 2006. "Reforma judicial en Colombia: Progresos en 15 aiios." Trabajo presentado en la conferencia "Judicial Reform in Latin America: An Assessment. CSIS, Washington, Junio 7 de 2006. http://www.csis.orp/media/csis/events/060607 judicial fientes.pdf Government of Colombia. 1999 "PlanNacional de Desarrollo `CambioPara Construir la Paz. ' " Law 588 of 1999. http://www.bvsde:paho.org/bvsacd/acodal42/ley.pdf Messick, Richard. 1999. "Judicial Reform and Economic Development: A Survey of the Issues". The World Bank Research Observer 14(1): 117-36. .2002. "Judicial Reform: The Why, the What, and the How." In Strategies for Modernizing the Judicial Sector in the Arab World. Marrakech, Morocco. http://www.undp- pogar.org/publications/judiciary/messick/reform.pdf Novoa, Maria and Takao Hasuike. 2007. "Analisis Cualitativo y Cuantitativo sobre la Percepci6n de 10s Usuarios Internos y Externos del Servicio de Justicia en relaci6n a1 proceso de Modernizacion Judicial en Guatemala." Mimeo. Owen, Bruce. 2006. "Economic Approaches to Project Impact Measurement and Evaluation Methods for Law and Justice Reform: Toward a Preliminary Taxonomy of LJ Projects". Washington, D.C.: World Bank. Mimeo. Owen, Bruce and Jorge Portillo. 2003. "Legal Reform, Externalities and Economic Development: Measuring the Impact of Legal Aid on Poor Women in Ecuador." Stanford, SIEPR Discussion Paper No. 02 02-32. PROJUSTICIA. 1998. Modernizacidn y Fortalecimiento de la Administracidn de Justicia. Quito: Corte Suprema de Justicia. Unidad de Coordinaci6n del Programa Nacional de Apoyo a la Reforma de la Administraci6n de Justicia. Rodriguez, Cesar. 200 1. "La justicia civil y de familia'' en de Sousa Santos y Garcia (200 1) pp. 547 a 6 14. .2009. La Globalizacidn del Estado de Derecho. Bogota: Universidad de Los Andes. Colecci6n Estudios CIJUS. Rodriguez, Marcela. 2000. Empowering Women: An Assessment of Legal Aid under Ecuador's Judicial Reform Project. Working Paper Report No. 32140. Washington, D.C.: The World Bank. httD://go.worldbank.ordOMTCFOI5Hl Rubio, Mauricio. 2008. "Evaluacidn de las Leyes: Lecciones de la Criminologia." Revista de Economia Institucional 10(19) :131-160. hrcD://www.economiainstitucional.com/pdf7No 19/mrubio 19.pdf Said, Javier and Oscar F16rez. 2005. "Evaluaci6n de Impacto." Mimeo. Said, Javier and David Varela. 2002. "Modernization of the Itagui Court System: A Management and Leadership Case Study." Working Paper Report No. 25080. Washington, D.C.: The World Bank. http://go.worldbank.orp/3MJ8LZKG70 Santos, Alvaro. 2006."The World Bank's Uses of the "Rule of Law" Promise in Economic Development." In Trubek, David and Alvaro Santos. The New Law and Economic Development. Cambridge University Press. 253-300. Sen, Amartya. 2009. The Idea of Justice. London: Allen Lane. Thacker, Gerad. 2002. "Report of trip to Guatemala, May 2 1-25,2002.'' Unpublished internal document. UMOJ. 2009. "Informe Final. Proyecto BIRF 4401-GU Reforma Judicial Guatemala." Unidad de Modernizacion del Organism0 Judicial. Uribe, Jose Dario and Hernando Vargas. 2002. Financial Reform, Crisis and Consolidation in Colombia. Borradores Semanales de Economia No. 204. Bogota: Banco de la Republica. http://www.banrep.gov.co/docum/~~orra204.pdf Vizcaya, Carlos. 2008. Proyecto de Mejoramiento de la Resolucidn de Confictos Judiciales. Evaluacidn de Resultados. Informe No 2 Ajustado.. Bogota: Universidad Externado de Colombia. Facultad de Administracion de Empresas, Direction de Desarrollo Gerencial. Agosto. 43 World Bank. 1994. Ecuador Judicial Sector Assessment. Report No. 12777-EC. Public Sector Modernization and Private Sector Development Division. LAC Region. Washington, D.C.: The World Bank. . 1996. StaffAppraisal Report: Ecuador Judicial Reform Project. Report No. 15385. Washington, D.C. : The World Bank. . 1998. Project Appraisal Document on a Proposed Loan in the Amount of US$33 Million to the Republic of Guatemalafor a Judicial Reform Project. Report No. 18404. Washington, D.C. : The World Bank. .200 1. Project Appraisal Document on a Proposed Learning and Innovation Loan in the Amount of US$5 Million to the Republic of Colombiafor a Judicial Conflict Resolution Improvement Project. Report No. 23 184. http://imagebank. worldbank.org/servletiWDSContentServer/IW3P/IB/200 1 1/12/17/000094946~0 120404004484/Rendered/PDF/rnultiOpage.pdf .2003 Impact of Legal Aid: Ecuador. Working Paper. Report No. 26915. Legal Vice Presidency. Washington, D.C. : The World Bank. http://go,worldbank.org/OJMOW48ZOO . 2003a. Legal and Judicial Reform: Strategic Directions. Working Paper. Report No. 26916. Legal Vice Presidency. Washington, D.C.: The World Bank. .2003b. Implementation Completion Report on a Loan in the Amount of US$lO. Million to the 7 Republic ofEcuadorfor a Judicial Reform Project. Report No. 26259. Washington, D.C.: The World Bank. httD://imaaebank.worldbank.ordservletlWDSContentServer/IW3P/IB/2003/07/08/000012009 20 030708 152143Rendered/PDF/262590ECOICR.pdf .2003c. Report and Recommendation on a Country Assistance Strategy for the Republic of Ecuador. Report No. 25817. Washington, D.C.: The World Bank. . 2005. Country Assistance Strategy for the Republic of Guatemala. Report No. 3 1776. Washington, D.C.: The World Bank. .2005a. Country Assistance Strategy Progress Report for the Republic of Colombiafor the Period FY03-FY07. Report No. 32999. Washington, D.C.: The World Bank. .2006. "Developing a Methodology to Measure the Impact of Law and Justice." Mimeo. .2007. Implementation Completion and Results Report: Learning and Innovation Loan in the Amount of US$5 Million to the Republic of Colombiafor a Judicial Conflict Resolution Improvement Project. Report No. ICR0000539. Washington, D.C.: The World Bank. h~://ima~ebank.worldbank.or~servletiWDSContentServer/IW3P/IB/2007/08/17/0003 10607 20 0708 17120843/Rendered/PDF/ICR5390ouo.pdf .2008. Implementation Completion and Results Report on a Loan in the Amount of US$33 Million to the Republic of Guatemalafor a Judicial Reform Project. Report No. ICR0000623. Washington, D.C.: The World Bank. http://imagebank.worldbank.org/servletiWDSContentServer/IW3P/IB/1998/ 11/17/000178830-98 11170355168l/Rendered/PDF/multiOpage.pdf .2009. Project Appraisal Document on a ProposedAdaptable Program Loan in the Amount of US$20 million to the Republic of Colombiafor a Justice Services Strengthening Project- Phase I (APLI) in Support of the First Phase of the Justice Services Strengthening Program. Report No. 47338. Washington, D.C.: The World Bank. http://imagebank.worldbank. org/servletlWDSContentServer/IW3PIIB/2009/ 12/01/0003 34955-20 091201014438/Rendered/PDF/473380PADOP083 101OfficialOUseOonlyl.pdf Wintersteiger, Lisa. 2008. "Young People's Experience of Law-related Events: The role of Public Legal Education." Public Legal Education Network. httu://www.lawcentres.org.uWuuloadsNoung Peoples Experience and PLE.pdf 45 ANNEXA Annex A. Basic Data Sheet JUDICIAL ECUADOR: REFORM PROJECT (LOAN 4066) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % o f estimate current estimate appraisal estimate Total project costs 14.31 12.12 84 Loan amount 10.70 10.70 100 Cofinancing 3.61 Proiect Dates Original Actual ~ ~ Initiating memorandum 0812411994 0812411994 Negotiations 0911811995 0911811995 Board approval 0711811996 07118/ 1996 Signing 0911911996 0911911996 Effectiveness 0 110111997 0312011997 Closing date 0613012002 1113012002 Staff Inputs (staff weeks) Stage Staff Cost in US$ IdentificatiodPreparation AppraisalNegotiation 270,883 Supervision 503,195 ICR Total 774,078 46 ANNEXA Mission Data I I No. of Persons and Specialty (eg. 2 Stage of Project Cycle Performance Rating Economists, I FMS, etc. MONTHNEAR COUNT 1 SPECIALTY DEVELOPMENT IMPLEMENTATION PROGRESS OBJECTIVE Identification/ Preparation April 3, 1995 11 Task Manager (1) Judicial Sector Specialist (3) Legal Specialist (3) Operations Analyst (2) Procurement (1) Financial Management (1) July 10, 1995 5 Task Manager (1) Judicial Sector Specialist (2) Legal Specialist (1) Operations Analyst (1) AppraisaVNegotiation - January 22,1996 6 Judicial Sector Specialist (3) Legal Specialist (1) I Supervision July 3, 1997 Task Manager (1) S S . ............ " .......... " November 13, 1998 Task Manager (1) S S March 12, 1999 5 Task Team Leader (1) S S Judge (1) GeneraVCivil Society (1) Infiastructure (1) Information System (1) July 2, 1999 2 Task Manager (1) S S Informatics (1) November 16, 1999 4 Task Manager (1) S S Legal Aspects (1) Infrastructure (1) Judicial Aspects (1) April 20,2000 3 Task Manager (1) S Legal Aspects (1) InfiastructureProcure (1) November 24,2000 3 Team Leader (1) S V.P. and General Counsel (1) Consultant (1) 47 No. of Persons and Specialty (eg. 2 Performance Rating Stage of Project Cycle Economists, 1 FMS, etc. November 24,2000 6 Task Manager (1) 5 HS Procurement (1) Research Assistant (1) Gender Specialist (1) CT Management Specialist (1) Judge/Judicial Reform (1) .,..."...................... ..... ..... .... I November 16,2001 5 Task Manager (1) 5 HS Sr. Counsel (1) Case/Court Management (1) Law and Justice ProglGender (1) JudgeMediation (1) ......... ......, ~ ........ May 8,2002 3 Task Team Leader (1) 5 HS Judge Sector Specialist (1) Operations Support (1) February 10,2003 8 Task Team Leader (1); 5 HS Judge Sector Specialist (4); Operations Analyst (1); Procurement (1); FMS (1) 48 JUDICIAL GUATEMALA: REFORM PROJECT (LOAN 4401) Kev Proiect Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 49.7 35.6 71 Loan amount 33.0 33.0 100 Cofinancing -Donors 49.3 34.4 70 Project Dates Original Actual Initiating memorandum 12/19/1997 Negotiations 06/30/1998 Board approval 10/22/1998 Signing 01/22/1999 Effectiveness 04/22/1999 0412211999 Closing date 06/30/2004 0613012007 49 ANNEXA Staff Inputs (staff weeks) Staff Time and Cost Stage No. of Staff Weeks US$ ((000)s Lending FY97 134.36 ................................................................................................................................................................................................................................................................1 FY98 200.68 ................................................................................................................................................................................................................................................................1 FY99 68.39 FY00 ........................................................ 8 ............ 37.45 .......... FYOl 5 19.25 Total 13 460.13 Supervision/ICR FY98 1.71 FY99 44.20 ................................................................................................................................................................................................................................................................1 FYOO 18 90.82 FYOl .......................................................... 9 48.30 ............ FY02 10 132.89 FY03 10 121.66 FY04 11 95.77 FY05 15 72.48 FY06 17 90.83 FY07 17 83.24 FY08 6.29 Total 107 788.19 50 ANNEX A Task Team Members I I I Responsibility/ Names Title unit specialty (FROM TASK LENDING TEAM PROGRAM DOCUMENT) IN Waleed Haider Malik Public Sector Management Specialist and Task Manager ILCSPR I Ian Bannon John Underwood Lead Economist and Sector Leader I Program Manager Lcc2c ILCSPR I1 Jose Roberto Lopez-Calix I Resident Representative ILCCGT I Geoffrey Shepherd Lead Specialist LCSPR Public Sector Robert Lacey LASLG Sally Zeijlon Principal Country Officer ILCC2C I Robert Crown William Partidge LCSES Douglas Allen Webb Legal Advisor LEGPS Sherif Hassan LEGOP Andrew Vorkink I ILEGVP I Rudy Van Puymbroeck I Chief Counsel ILEGLA I Caroline Moser I Lead Specialist I LCSES I Governance Maria Correia Lead Specialist LCSPR Gender Roberto Maclean LEGPS Armando Araujo EOPR] Roberto Panzardi I Public Sector Specialist I LCSPR I Public Sector Albert0 Ninio I Project Counsel ILEGLA I Genero Alarcbn, Reynaldo Pastor, David Varela Ronald Myers Sr. Public Sector Management ILEGLA I Spec. Mario Del Carril ILCRVP I Monica Echeverria I ILCRVP I Michael Fowler LOAEL Jaime Roman, Emilio Rodriquez LCOPR Mauricio Mathov ISG Maria Elena Anderson, Marina LCSPR Vasilara, Adriana Weisman, Ninia Ohman, Sharon Isaac, Clarisabell COSS, Veena Mayani, William Mayville, Richard Messick, Shirley Matzen, Maria Gonzalez 51 ANNEXA Responsibility/ Names Title unit Specialty Cora Shaw, Sehlton Davis, Juan LCSES Martinez, Annika Tornquist Maria Kuraishi ECA Judy Rivers, Barbara Walker, Marlene Sims, Gerald Carter Sandra Alborta, Elizabeth Greene, lLC ZC II LCSPR Hazel Vargas Jesse Casaus, Edgardo Buscaglia, Consultants LCCGT I Fulvio Carbonaro, Linn Hammergren, Eduardo Freudenthal, Tanja Utunen Mario Marroquin, Lubio Garcia, LCGGT Anabela Lucas, Celeste Peralta, Claudia Hernhndez, Gabriela Gonzalez, Alejandra Santacruz (FROM TASKTEAM ICR) SUPERVISION/ICR IN Shelton H. Davis, Nina-Christina Ohman Alexandra M. Habershon Consultants I Consultant ILCSPS II IECSSD I Roberto 0.Panzardi I Sr. Public Sector Mgmt. Spec. I LCSPS I I Karla Lopez Flores Program Assistant LCSPS Anthony Wanis-St. John Consultant LCSPS Linn Hammergren Sr. Public Sector Mgmt. Spec. LCSPS Richard Moore, Laura Louza, Consultants LCSPS Andrew Blandford Keisgner Alfaro, Monica Lehnhoff Procurement Analyst LCSPT Antonio Leonard0 Blasco Financial Management Specialist LCSFM Sr. David Bernstein Public Sector Mgmt. Spec. ECSPE ICR Reviewer Eduardo Somensatto Lead Economist LCSPR ICR Reviewer Robert Varenik Consultant LCSPR ICR Reviewer 52 ANNEXA COLOMBIA: JUDICIAL RESOLUTION CONFLICT PROJECT IMPROVEMENT (LOAN7081-CO) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 6.66 5.27 79 Loan amount 5 .O 3.9 78 Cofinancing-Government 1.7 1.4 84 Cancellation 1.1 22 Project Dates Original Actual Initiating memorandum 03/29/2000 Negotiations 08/05/2001 08/05/200 1 Board approval 11/08/2001 Signing 12/04/2001 Effectiveness 01/30/2002 01/30/2002 Closing date 0613OI2005 06/30/2006 53 ANNEXA Staff Inputs (staff weeks) Staff Time and Cost ~~ Stage No. of Staff Weeks US$(6000)s Lending FY98 4.06 FY99 3.06 FYOO 13 58.02 FYOl ............ 6.......... . 11.22 8 ............................................................................................. FY 03 ................................................................................................................................................................................................................................................................= 0.00 FY 04 ................................................................................................................................................................................................................................................................= 0.00 FY05 0.00 .................. FY 06 ............. 0.00 ............................................................ FY07 0.00 Total 27 124.74 SupervisiodICR FY98 0.00 FY99 0.00 FYOO ................................................................................................................................................................................................................................................................= 0.00 FYOl ................................................................................................................................................................................................................................................................= 0.00 FY 02 9 ................................................................................................................................................................................................................................................................= 58.37 FY 03 8 ................................................................................................................................................................................................................................................................= 70.04 FY04 11 91.67 .................. FY05 9 ........... 73.89 FY06 19 127.46 FY07 2 72.49 Total 58 493.92 54 ANNEXA Task Team Members Responsibility/ Names Title nt Ui Specialty (FROM TASK LENDING TEAM PROGRAM DOCUMENT) IN Felipe She, Lead Country Officer LCC7C Team Leader ..................................................................................................................... Luiz Gazoni Senior Procurement Advisor LCSPT Procurement Jairo Arboleda Lead Social Development LCSEO Participatory Strategy Specialist ................................................................................................................................................................................................................................................................5 Issam Abousleiman Principal Invest. Officermebt BCFBD Disbursement Officer Capital Luis M. Schwarz Senior Financial Management LCSFM Financial Management Specialist Specialist Waleed Malik Lead Public Sector Management LCSPS Peer Reviewer Richard Messick Senior Public Sector Specialist PRMPS Peer Reviewer Pilar Mengod Executive Assistant LCCVE Team Assistant SUPERVISION (FROM TASKTEAMMEMBERS ALL ARCHIVED ISRS) IN Feliue Saez Lead Countrv Officer LCC7C Team Leader David Varela Senior Public Management LCCVE Team Leader Specialist Javier Said Consultant LCSPS Judicial Sector Specialist ................................................................................................................................................................................................................................................................5 Orlando Abello Consultant LCSPS Judicial Sector Specialist Rodolfo Aldea Moscoso Consultant LCSPS Judicial Sector Specialist Javier Madalengoitia Consultant LCSPS Operations Specialist Angel Cardenas Consultant LCCVE Operations Specialist Juan Pablo Molina Consultant LCSPS Operations Specialist 55 ANNEXB Annex B: List of People IntewiewedMet in Respective Countries Guatemala Abreu, Anabela, WB - Guatemala Aceiia, Maria del Carmen, Former Minister Education Barrios, Victor Manuel, Juez Civil Berduo, Elia Maria del Carmen, Juez 1 de Familia Bonilla, Maria Isabel, CIEN Ciceres, Edy, Juez Civil - San Benito Peten Cacil ICO, Martin, Juez de Paz - San Luis Canteo, Marco, IECCP Castellanos, Alvaro Litigante - U Landivar Castillo, Bremly Jamely, Juez de Paz - Flores Peten Choc Pop, Inocente, Alcalde - San Luis Colmenares, Carmen, Maria, ASIES Escobar, Claudia, Juez Civil Mixto Fernandez, Cynthia, ASIES Fernandez, Maria Cristina, Juez Civil Game,, Patricia, Juez Penal Guix, Napoleh, Coordinador UMOJ Hernandez, Luis Mariano, Unidad RAC Herrera, Marvin, Juez Penal Hus, Fernando, Consultorio Legal - Antigua Hus, Hugo, WB - Guatemala Ical, Neftali Feniciano, Juez de Paz Comunitario - San Luis Lavarreda, Jorge, CIEN Mendizhbal, Carlos Enrique, Unidad de Antecedentes Penales Morales, Juan Luis, Arquitecto - UMOJ Morales, Manuela, Mediadora - Chichicastenango Muiioz Murai, Jose Dario, Juez de Paz - Dolores Peten Ocaheta, Claudia Lisbeth, Juez de Paz - San Benito Peten Ordoiiez, Oscar Samuel, Secretario Juzgado 1 de Familia Ortiz, Maria del Carmen, UMOJ Quisquinai, Oscar, Pastoral Social - Santa Elena Rivera, Orfa CENADOJ Rodriguez, Ana Maria, Juez Penal Sandoval, Angel Giovani, Juez de Paz - S Francisco Peten Soel Contreras, Roni, Juez de Paz - Santa Ana Peten Stein, Ricardo, Former Minister Villatoro Lbpez, Julio Cesar, Juez de Paz - La Libertad Peten Yoc, Amelia, CENADOJ ANNEXB 56 Colombia Ahumada, Diana, Asobancaria Alba, Santiago, Consejo Superior de la Judicatura - CSJ Arbelaez, Lucia, Ex-magistrada CSJ Barrera, Antonio, CENDOJ - CSJ Borrero, Gloria, Corp Excelencia en la Justicia - CEJ Caro, Diana Centro Sew Administrativos - CSJ Cespedes, 'Jairo, Colegio Nacional Abogados Litigantes Durhn, Fabio, Econometria Lasprilla, Martha, Asobancaria Lizarazu Bitar, Myriam Juez 44 Civil Circuit0 Bogot6 Lbpez, Carlos, CENDOJ - CSJ Meza, Martha, Directora Cobros - Covinoc Olivera, Mauricio, Fedesarrollo Palacio, Yamile, SEI-Consultores Parra Quijano, Jairo, Inst Col de Derecho Procesal Pedroza, Mbnica, CEJ Peralta, Marco Director Juridic0 - Covinoc Polo, Joaquin, Ex-coordinador Ramos, Ana Maria, CEJ Rivera, Jairo, Director Posgrados U Externado Rodriguez, Cesar, CIWS - U de 10s Andes Saenz, Claudia, Covinoc , Useda, Carlos, UDAE - CSJ VBlez, Angela Maria Ex-coordinadora Vizcaya, Carlos, Evaluator - U Externado 57 B ANNEX Ecuador Aguas, Karolina, Projusticia Amaguai, Ruperto Director Informatica - CJ Arbito, NCstor, Ministro de Justicia Arias, Alicia Ex-Projusticia Arias, Miguel Juez Penal - Cuenca Armijos, Douglas, Sibdirector Adm Financier0 CJ Guayaquil Artieda, Michelle, Ministerio de Justicia Bordes, Beatriz, Fundacidn Maria Guare - Guayaquil Buenaiio, Luis, Ministerio de Justicia Cardenas, Sofia, Corp Mujer a Mujer - Comisaria Cevallos, Victor Hugo Colegio Abogados Litigantes Pichincha Coloma, Geovanni, Programador - CJ Guayaquil CortCs Lugo, Fernando, Litigante - Guayaquil Estrella, Margarita, Centro Mediaci6n Quito Fierro, Napoledn, Centro Mediacidn Quito Frank, Jonas WB - Ecuador Jaramillo, Ana Lucia, Projusticia Luzarral Hurtado, Jorge, Juez 28 Civi - Guayaquil Macias, Ginger Jefe Informatica - CJ Guayaquil Mansilla, Sara, CEPAM Medina, Belfor, Programador - Riobamba Melo, Verdnica, Centro Mediacidn Quito Mina Cifuentes, Segundo, Litigante - Guayaquil Piedra, Teresa, Corporacidn Mujer a Mujer - Cuenca Portilla, Maria Mercedes, Juez 5" Civil - Quito Prieto Cabrera, Leonidas Juez 29 Civil - Guayaquil Rivera, Racl, Arquitecto - Edificio Juzgados Santacruz, Martha, Directora Centro de Mediacidn Cuenca Santos de Macias, Rocio, Juez Garantias Penales - Guayaquil Segovia, Fabricio, Juez 1" Civil - Quito Ugalde, CCsar, Juez 1' Civil Cuenca Urgiles, Gonzalo, Funci6n Judicial - Guayaquil Vintinilla, Jaime, CIDES Ycaza, Maria Fernanda, CIDES - Litigante