Document of The World Bank Report No: ICR0000623 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-44010) ON A LOAN IN THE AMOUNT OF US$33.00 MILLION EQUIVALENT TO THE REPUBLIC OF GUATEMALA FOR A JUDICIAL REFORM PROJECT March 10, 2008 Poverty Reduction and Economic Management Unit Central America Country Management Unit Latin American and the Caribbean Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective July 26, 2007) Currency Unit = Guatemalan Quetzal (Q$) US$1 = Q$7.7981 Q$1 = US$0.12824 FISCAL YEAR January 1 to December 31 FISCAL YEAR ABBREVIATIONS AND ACRONYMS ADR Alternative dispute resolution AECI Agencia Española de Cooperación Internacional, Spanish assistance agency Antecedentes Penales Office registering criminal records of citizens and providing certificates as proof that no such records exist ASIES Guatemalan research institute, Asociación de Investigaciones y Estudios Sociales Auxiliar judicial Courtroom support staff BB (World) Bank budget Cabecera de distrito Central city of judicial district CAS Country assistance strategy CENADOJ Centro Nacional de Análisis y Documentación Judicial, document center for Guatemalan Judiciary CIT Centro de Informática y Telecomunicaciones, informatics and telecommunications center for Guatemalan judiciary Consejo de la Carrera Judicial Career Council, responsible for overseeing Judicial appointment, supervision, evaluation, and complaints against judges CMU Country management unit Departamento de Office of Judiciary responsible for registering lawyers and Protocolos notaries and documents pertaining to their work Departamento de Part of Consejo de la Carrera Judicial, responsible for in situ Supervision de inspections of judicial offices and investigating complaints Tribunales Dirección de Servicios Judicial office responsible for receiving and distributing in- Tribunalicios coming cases, recording actions in criminal cases, holding files for active criminal cases, and running the notification service EU European Union Falta grave Serious administrative infraction causing written reprimand, leave without pay, or recommended dismissal FUNCEDE Fundación Centroamericano de Desarrollo, Guatemalan research institute Gestión Penal Office within Dirección de Servicios Tribunalicios responsible for handling criminal cases IADB Inter American Development Bank ICR Implementation completion report IEG Independent Evaluation Group IFMS Integrated financial management system IRIS Integrated Records and Information Service ISR Implementation status report Juicio ejecutivo Summary debt collection proceedings Junta de Disciplina Part of Consejo de la Carrera Judicial, responsible for holding hearings to make decisions and recommendations on claims of judicial malfeasance Juzgado de turno 24-hour court established to make early determinations as to whether suspects detained by police should remain in jail, or be released on bail or on their own recognizance. In some variations may also review police and prosecutors' requests for searches, seizures and similar actions LCSPS Latin American and Caribbean Public Sector Group M&E Monitoring and evaluation MINUGUA (Observer) Mission of United Nations in Guatemala NGO Nongovernmental organization OJ Organismo Judicial Organismo Judicial Guatemalan term for Judicial Branch of government PAD Project appraisal document PCU Project Coordinating Unit PDO Project development objective PHRD Policy and Human Resources Development (Fund) PROFED Programa de Fortalecimiento del Estado de Derecho, UNDP program to provide support to Guatemalan justice sector Protocolos Used as a short form of Departamento de Protocolos Secretaria de Planning office for judiciary Planificacion y Desarrollo Instituciona Servicios Departamento de Servicios Tribunalicios Tribunalicios SGT Sistema de Gestión de Tribunales, automated case management system SIAF Sistema Integral de Administración Financiera, Guatemala's IFMS SPDI Secretaria de Planificación y Desarrollo Institucional STC Short-term consultant Supervisión Departamento de Supervisión de Tribunales TF Trust Fund Título ejecutivo Document establishing existence of a liquid debt required to use summary debt collection proceedings (juicio ejecutivo) rather than longer, ordinary proceedings TOR Terms of reference TTL Task team leader UMOJ Unidad de Modernización del Organismo Judicial, PCU for WB Project UNAP Unidad de Antecedentes Penales (see Antecedentes Penales) UNDP United Nations Development Program UNOPS United Nations Office of Project Services URNG Unidad Revolucionario Nacional Guatemalteco, rebel forces during Guatemalan civil war USAID United States Agency for International Development WB World Bank Vice President: Pamela Cox Country Director: Laura Frigenti Sector Manager: Nick P. Manning Project Team Leader: Linn A. Hammergren ICR Team Leader: Linn A. Hammergren Guatemala Judicial Reform Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 9 3. Assessment of Outcomes.......................................................................................... 21 4. Assessment of Risk to Development Outcome......................................................... 36 5. Assessment of Bank and Borrower Performance ..................................................... 37 6. Lessons Learned ....................................................................................................... 42 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 45 Annex 1. Project Costs and Financing.......................................................................... 47 Annex 2. Outputs by Component ................................................................................. 48 Annex 3. Economic and Financial Analysis................................................................. 58 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 59 Annex 5. Beneficiary Survey Results........................................................................... 61 Annex 6. Stakeholder Workshop Report and Results................................................... 67 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR..................... 68 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders....................... 72 Annex 9. List of Supporting Documents ...................................................................... 73 A. Basic Information GT JUDICIAL Country: Guatemala Project Name: REFORM Project ID: P047039 L/C/TF Number(s): IBRD-44010 ICR Date: 10/23/2007 ICR Type: Core ICR REPUBLIC OF Lending Instrument: SIL Borrower: GUATEMALA Original Total USD 33.0M Disbursed Amount: USD 33.0M Commitment: Environmental Category: C Implementing Agencies: Judiciary Branch Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 12/19/1997 Effectiveness: 04/22/1999 04/22/1999 Appraisal: 06/30/1998 Restructuring(s): Approval: 10/22/1998 Mid-term Review: 10/06/2003 Closing: 06/30/2004 06/30/2007 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Unsatisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Unsatisfactory Government: Moderately Satisfactory Quality of Supervision:Moderately Implementing Unsatisfactory Agency/Agencies: Moderately Satisfactory Overall Bank Moderately Overall Borrower Performance: Unsatisfactory Performance: Moderately Satisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Performance Indicators (if any) Rating Potential Problem Project No Quality at Entry None at any time (Yes/No): (QEA): Problem Project at any Quality of No None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Law and justice 100 100 Theme Code (Primary/Secondary) Law reform Primary Primary Legal services Primary Primary Other accountability/anti-corruption Secondary Secondary E. Bank Staff Positions At ICR At Approval Vice President: Pamela Cox Shahid Javed Burki Country Director: Laura Frigenti D-M Dowsett-Coirolo Sector Manager: Nicholas Paul Manning Guillermo Perry Project Team Leader: Linn A. Hammergren Waleed Haider Malik ICR Team Leader: Linn A. Hammergren ICR Primary Author: Linn A. Hammergren F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The development objective of the project is to create a more effective, accessible and credible judicial system that would foster public trust and confidence and improve consistency and equity in the application of law. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Revised Achieved at approval Target Completion or documents) Values Target Years Indicator 1 : Significant increase in user confidence Increase in user 2003 survey shows Value 89% perceive quantitative or perceive administration ofmeasured Approx. 88% of users confidence as by the inadequate. Final Qualitative) justice as inadequate. stakeholder partial survey surveys and suggests confidence workshops. same or lower Date achieved 10/20/1998 04/22/2004 07/27/2007 Comments Comparable survey not done at end of project; information taken from two (incl. % partial surveys, one project funded, one funded by UMOJ. Last survey does achievement) suggest some expectations of future improvements, but the grade for this indicator, based on real change, is 20 % Indicator 2 : Significant increase in Judiciary Branch coverage and access outside Guatemala City Justices of Peace courts constructed/rehab. All municipalities Value and operational in have justice of the quantitative or Operations are centralizedthe majority of 331 peace, 2 regional Qualitative) in Guatemala City. municipalities, 2 regional- and 2 and 1 departmental departmental center constructed centers constructed. Date achieved 10/20/1998 04/22/2004 06/29/2007 Comments (incl. % 95% A joint effort by all donors and Organismo Judicial; WB does not construct achievement) second departmental center. Indicator 3 : Number of corruption related complaints received and resolved through the Anti- Corruption Commission 561 complaints Value against judges, 149 quantitative or 0 -- no complaints system No numerical admitted, and 28 Qualitative) target find faults. Prior year 602, 121, and 80. Date achieved 10/20/1998 04/22/2004 11/30/2006 Comments AC Commission does not hear complaints, thus judicial disciplinary bodies (incl. % substituted. Lack of target impedes evaluation. 50% because of high percentage achievement) of complaints that are not admitted or not heard. `` Policy on traditional justice systems developed and tested Value Policy No policy quantitative or No policy Qualitative) Date achieved 10/20/1998 04/22/2004 06/29/2007 Comments Not accomplished. Really outside scope of project., but will give 50 % because (incl. % all modernization plans have listed a series of activities which, if carried out, achievement) might produce a policy Indicator 5 : Number of subprojects executed through the participation program Value Unclear because of quantitative or 0 No target -- just uncertainty as to Qualitative) says "number of" what events should be counted Date achieved 10/20/1998 04/22/2004 06/29/2007 Comments Lack of target impedes evaluation ­ possibly 50 %, but this requires a (incl. % reinterpretation of what is counted. The activities mentioned in the PAD were achievement) not carried out and the main "participatory" subproject was a series of workshops on lynching, financed and directed by the PCU. (b) Intermediate Outcome Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Achieved at approval Revised Completion or documents) Target Values Target Years Indicator 1 : New management and organizational model designed, discussed, approved, and tested Model Value Model designed, (administrative (quantitative No model discussed, reengineering) or Qualitative) approved, and designed and tested carried out by end of 2001 Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % Never tested. 80% completion achievement) Indicator 2 : Judiciary Branch administratively restructured Value Restructuring done (quantitative No restructuring Restructuring by end of 2001 -- or Qualitative) completed few changes since Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 100% achievement) Indicator 3 : Standards for case flow established Value (quantitative No standards Standards set No standards set or Qualitative) Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 30% on basis of pilot experiments and other activities that might help develop achievement) standards Indicator 4 : Human resource development program (including training) established and tested Value HR Office created, (quantitative No program Program exists career system and or Qualitative) training for judges Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 60 % completed ­ 100 % for first instance judges, but auxiliary and achievement) administrative staff lack career, evaluation, and systematic training program. Indicator 5 : Inventory of cases prepared and judicial statistics developed Only partial Value inventory; no (quantitative No inventory or statistics Inventory and change to statistics or Qualitative) statistics system from what was already being used. Date achieved 04/22/1999 04/22/1999 06/30/2007 Comments (incl. % Inventory only for cases filed post-2002 in 5 cabeceras de distrito; statistics still achievement) inaccurate and incomplete. No new system developed 30% Indicator 6 : Court delay reduction program in Guatemala City and experimentally tested Partial program Value designed for (quantitative No program Program created appellate and few or Qualitative) and tested family and labor courts, but neither tested nor adopted Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments Experimental programs for appellate and other courts only focus on courtroom (incl. % paper flow. Broader delay reduction program (involving actions outside achievement) courtrooms) only contracted in 2005 and never produced results acceptable to UMOJ. 60% Indicator 7 : Judicial information system designed, established and experimentally tested in Guatemala City and two pilot jurisdictions Still using old manual system at Value national level, SGT (quantitative No system System designed established in or Qualitative) and tested Guatemala City with statistical capacity, but no statistics yet Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments Listed as 15 % complete in the 2004 ISR, this activity only advanced in the (incl. % project's final year. On basis of experimental development of SGT (which does achievement) not yet use its MIS capacity) in Guatemala City and Quetzaltenango, 75% Indicator 8 : Judicial career law approved and applied Value Career law Law approved (quantitative No career law approved and 10/99 and goes into or Qualitative) applied effect 12/99 -- is in application Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 100% although career only covers first instance judges. Appellate and Supreme achievement) Court justices excluded. Indicator 9 : Anti-corruption commission established and tested Value (quantitative No commission Commission Commission or Qualitative) created and tested created Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % Commission created, but no "testing" and general impression that it does little. achievement) 100% for creation Indicator 10 : Ethics code prepared and approved Value (quantitative No code Code prepared Code prepared and or Qualitative) and approved approved in 2000 Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 100% achievement) Indicator 11 : Department of Supervision of Courts reorganized Value Reorganized and (quantitative Unreorganized Reorganized provided with or Qualitative) equipment, TA, remodeled office Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 100 %. However, needs more equipment, more TA, better methodologies and achievement) probably decentralization if to do an adequate job Indicator 12 : System to register lawyers assistants developed and tested Departamento de Value Protocolos (quantitative No system system developed automated, and or Qualitative) and tested processing times greatly reduced Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % Great success in terms of speed of registration. Impact on problem of achievement) unregistered litigants or quality of legal services unknown. 100% Indicator 13 : Review of incentive systems for judges Value No review ever (quantitative Incentives undiagnosed Incentives done ­ salaries or Qualitative) reviewed were raised to "improve incentives" but were politically negotiated. Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 0 for lack of any review or study achievement) Indicator 14 : Expansion of oral procedures in other areas of law Draft of civil Value procedures code (quantitative Only criminal Expansion to prepared with oral or Qualitative)proceedings are oral other areas hearings provided; some experiments in family courts Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % Informally justice of peace courts may use more oral hearings, but no formal achievement) policy or legal change. 30% Indicator 15 : Policy on traditional conflict resolution mechanisms prepared, discussed, and approved Value (quantitative No policy Policy No policy or Qualitative) Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments Beyond the power of the project to accomplish, but judicial modernization plans (incl. % list several actions, which if taken, may produce a policy. Still lack studies on achievement) traditional systems essential to policy development. 50 % Indicator 16 : Formal and alternative dispute resolution expanded Value Both expanded -- All municipalities (quantitative Not all municipalities no numerical have JPs and there or Qualitative)have courts; no ADR target are 70 ADR centers Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % Successful and popular, although new units are underutilized. 100% achievement) Indicator 17 : Program of new role for justices of the peace operational Value JPs do more (quantitative No new role New role conciliation and or Qualitative) refer clients to ADR Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % Never clear what new role was, but on assumption it includes conciliation, 50% achievement) Indicator 18 : Number of judges increased in rural areas Value 365 JPs, at least one (quantitative 229 justice of the peace No numerical in every or Qualitative) value municipality and 119 in rural areas Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 100% through joint effort of all donors and OJ; WB contributes by building JP achievement) courthouses, equipping them and training judges Indicator 19 : Two regional centers, two departmental centers, and JP courts constructed or remodeled 4 centers and 2 regional centers, Value one departmental (quantitative --- undetermined center, 6 JP courts or Qualitative) number of JP courts built and 11 remodeled Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % Successful 95% achievement) Indicator 20 : Number of participation subprojects provided and implemented Numerous training Value sessions at local (quantitative none Unspecified level, 2 programs or Qualitative) with school children, anti- lynching workshops Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments 50% for new activities, but those enumerated in PAD were not done and (incl. % emphasis on community and NGO development of programs disappeared. achievement) Instead major project was the workshops on lynching, with participation, but run by UMOJ Indicator 21 : Long-term social communication and information sharing strategy developed and operational Value No strategy, but (quantitative No strategy Strategy many or Qualitative) communication activities Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 60 % achieved taken into account list of activities included in current achievement) modernization plan Indicator 22 : Channels of internal and external communication and coordination developed and made operational Value More channels -- Many types of (quantitative Few channels communication or Qualitative) number? developed Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 100% achievement) Indicator 23 : Office of Public Affairs with multilingual capabilities established to prepare media releases on implementation progress Office created and Value multilingual press (quantitative No office Office created and releases done, or Qualitative) functioning however much still implemented by PCU Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 60% because of limited institutionalization within the office, as opposed to the achievement) PCU Indicator 24 : Modernization Commission trained in change management and coordination Value Commission Commission exists (quantitative No commission exists and is and meets, but little or Qualitative) trained training Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 60% Technically members did participate in some courses on change achievement) management although quality and impact are unclear. Indicator 25 : PCU functions and responsibilities clearly established PCU takes over many functions that Value Functions and belong to OJ (quantitative PCU small and functions responsibilities offices; either rules or Qualitative) not developed clearly established were not clear or they were not followed Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 70%. Excessive mission creep by PCU. It needs to return to coordination and achievement) support for implementation but let the OJ do more Indicator 26 : Adequate quality assurance, monitoring and evaluation capacity developed Value No base line specified by M&E and quality (quantitative functions probably did Both capacities assurance or Qualitative) not exist developed developed for project deliverables Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 30% --to the extent capacity exists, only in PCU, but it does not monitor and achievement) evaluate impacts, only deliverables Indicator 27 : PCU FM and reporting system automated Value FM automated; (quantitative No system FM and reporting reporting system or Qualitative) system automated appears to be only partially automated Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments 80%. Judging by number of reports requested by ICR team (e.g on training) that (incl. % had to be developed manually, automation system seems incomplete achievement) Indicator 28 : Annual modernization report published Value (quantitative No report Annual reports Annual reports or Qualitative) published Date achieved 04/22/1999 04/22/2004 06/30/2007 Comments (incl. % 100% achievement) G. Ratings of Project Performance in ISRs Actual No. Date ISR Archived DO IP Disbursements (USD millions) 1 01/04/1999 Satisfactory Satisfactory 0.00 2 06/15/1999 Satisfactory Satisfactory 1.32 3 12/08/1999 Satisfactory Satisfactory 1.32 4 06/22/2000 Satisfactory Satisfactory 3.14 5 12/19/2000 Satisfactory Satisfactory 3.20 6 06/28/2001 Satisfactory Satisfactory 4.37 7 12/21/2001 Satisfactory Satisfactory 7.41 8 06/27/2002 Satisfactory Satisfactory 9.38 9 11/21/2002 Satisfactory Satisfactory 12.37 10 12/19/2002 Satisfactory Satisfactory 13.52 11 06/19/2003 Satisfactory Satisfactory 14.87 12 12/05/2003 Satisfactory Satisfactory 17.69 13 06/10/2004 Satisfactory Satisfactory 19.23 14 12/21/2004 Satisfactory Satisfactory 19.86 15 05/24/2005 Satisfactory Satisfactory 21.34 16 01/04/2006 Satisfactory Satisfactory 22.84 17 10/18/2006 Satisfactory Satisfactory 26.26 18 01/05/2007 Satisfactory Satisfactory 32.34 H. Restructuring (if any) Not Applicable I. Disbursement Profile . 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Guatemala's population (at the time of project approval 11.2 million) comprises a mosaic of cultures, ethnicities, languages, socioeconomic realities and geographies. About forty percent are members of one of 24 linguistically differentiated indigenous groups, each further retaining, to a greater or lesser extent, its own communal organization, values, and authority structures.1 As of the late 1990s, sixty percent of the population and the vast majority of indigenous citizens resided in rural areas. Extreme poverty was (and still is) concentrated there as well, especially among women. Per capita income was $1,500 in 1997, but Guatemala had the third highest degree of income inequality (exceeded only by Brazil and Pakistan) among 44 low-to-middle income countries worldwide.2 As the project was being developed, Guatemala was in transition from 36 years of violent internal conflict, often described as a civil war. The war began in 1960 and formally ended with the December 1996 signing of the last set of Peace Accords between the Government and the rebel forces (known as the Guatemalan National Revolutionary Unit, or URNG by their initials in Spanish). The Accords were part of a larger process to negotiate an end to conflicts in several Central American countries, all of which arose from disputes among competing political forces, but were fed by severe levels of inequality within the affected nations. Guatemala's conflict began earlier than the others and had especially serious consequences for economic growth, social capital and physical infrastructure. GNP per capital fell by an annual average of 2 percent between 1976 and 1986 and rose by only 0.9 percent annually between 1987 and 1997. While the war took its toll on all citizens, indigenous groups were particularly affected by the concentration of fighting and the most brutal human rights abuses in rural areas, voluntary or involuntary displacement from their communities, and the recruitment of young men (and some women) by the army or the guerrillas (Hendrix, 2000; Goldman, 2007). The war's termination did not end the social disorder or the animosities it had fed; in the late 1990s, sixty-nine percent of the population still perceived violence as a major threat. Violence remains a problem to this day, a product both of the war's disruptions and a subsequent increase in ordinary and organized crime. At the national level there are currently 47 homicides per 100,000 inhabitants, second only to Colombia in the region. The figure rises to 110 per 100,000 in the capital city. (The Economist, September 8, 1 The forty percent figure is taken from the PAD, Annex 2a. However, recent IADB figures (cited in Van Cott, 2007; 128) show the indigenous population as 60 percent. As both numbers originate in government statistics, the difference may be due to changes in census methodologies and also a greater tendency of citizens to identify as indigenous. 2 Statistics in this and the next paragraph are taken from the PAD. 1 2007, p. 40) Violence and citizen security are seen as the most important issues for Guatemala as evidenced in the presidential campaign of 2007. Beyond setting a formal end to the conflict, the Peace Accords established a long-term development plan for the country to restore the rule of law, strengthen the respect for human rights, and create a more inclusive economic and social system. Sections in the Accords recognized judicial reform as an important component, reflecting a consensus among civil society, governmental and political authorities, and members of the justice sector itself that the sector's reconstruction was essential to promote post-conflict reconstruction, social stability, and economic growth. The Accords (which, it should be noted, were not legally binding) also incorporated recognition of traditional forms of dispute resolution. Moreover, Guatemala is a signatory to the International Labor Organization Agreement Number 169 mandating inter alia a respect for indigenous culture and requiring that indigenous peoples be consulted on any change of legislation affecting their interests. Although a 1999 referendum on the constitutionalization of certain aspects of the Accords (including the recognition of traditional dispute resolution) did not pass, the 1986 constitution, a series of amendments enacted in 2000 and 2001, and infra- constitutional law did introduce important changes to the justice sector's legal framework. These included the creation of a Constitutional Court, a Judicial Council and career, a budgetary earmark for the Judiciary, and the strengthening of human rights guarantees. A law regulating the council and career was enacted in 1999 (13 years after both were incorporated in the Constitution), along with a second law establishing a civil service career for the judiciary's administrative employees. The 1992 passage of a new criminal procedures code (entering effect in 1994) also promised improvements in the handling of criminal cases, giving an augmented role to the Public Ministry (prosecution) and a public defense agency. Equally importantly, local and donor interest in the theme provided considerable information on justice system workings, which fed into the Peace Accords commitments and the present project. As these studies indicated, the most common criticisms of the courts and the sector as a whole were: lack of public confidence, limited access to formal state mechanisms and lack of recognition of traditional ones, lack of sensitivity to customs and traditions of indigenous groups, corruption and politicization, lengthy delays and large backlogs in the state system, and lack of transparency and predictability in its outcomes. In a 1997 poll financed during Project preparation (reported in Aragón y Asociados, 2003) 88 percent of respondents rated court performance as inadequate, 26 percent believed the courts were corrupt, and 79 percent noted the high costs of litigation. Lack of confidence in the system has produced another highly undesirable side-effect, a tendency for communities to take the law into their own hands, lynching those suspected of crimes (Mendoza A, 2007). During project preparation, a series of workshops with judges (including 32 percent of the existing bench), indigenous groups, the Commission formed under the Peace Accords to oversee sector strengthening, NGOs, and other stakeholders emphasized these and 2 other deficiencies. One result was a five-year plan for sector modernization, the judicial part of which (Comisión de Modernización del Organismo Judicial - UMOJ, 1997) became the framework within which donor cooperation was to fit. Bank project preparation financed the drafting of the judicial section and of an associated implementation plan. As had occurred earlier with El Salvador (whose Peace Accords were signed in 1992), foreign assistance flooded into the country to support reconstruction, causing some observers to fear that the justice sector in particular would not be able to use the possibly $300 million promised by the donor community.3 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) The development objective of the project was to create a more effective, accessible and credible judicial system that would foster public trust and confidence and improve consistency and equity in the application of law. Key indicators were: · Percent increase in user confidence. · Percent increase in Judiciary Branch coverage and access outside Guatemala City. · Number of corruption related complaints received and resolved through the Anti- Corruption Commission. · Policy on traditional justice systems developed and tested. · Number of subprojects executed through the participation program. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification The PDO was not revised. However, starting in mid-2005, the indicators were altered in the reporting form (ISR 15). Only two PDO indicators were tracked (credibility and expanded presence of the courts) in subsequent ISRs. There is no evidence that the indicators were formally changed, and thus, the ICR remains based on those in the PAD. The three indicators "eliminated" were clearly the weakest and thus no great loss, but as noted by two reviewers of an early ICR draft, there never was an indicator for efficacy (except insofar as that might be part of "user confidence"). Developing such an indicator would, however, have been difficult and costly as the usual candidates are things like backlog reduction, time to resolution of cases, clearance rates or average workloads, all of which require accurate and relatively advanced statistical systems or, alternatively, random sampling of courtroom caseloads. Very few such systems exist anywhere in the LAC region, and Guatemala's is not among them. 1.4 Main Beneficiaries 3Only about a third of this was offered to the courts, not all of it arrived, and what did come sometimes took its time. A proposed $30 million EU project for example, only materialized several years later and at roughly one third the initial amount. 3 The primary target groups as described in the PAD were "marginalized segments of society including women, children, small business and other groups in the interior of the country." It was intended that these groups would benefit from a more effective, accessible and transparent judicial system and from downstream impacts on promoting social peace and economic development, strengthening the rule of law and democratic institutions, improving citizen security, heightening respect for cultural and human rights, and facilitating the implementation of the Peace Accords. However, all Guatemalan citizens would also benefit from these developments and especially from their anticipated impact on reducing violence and encouraging economic growth. Direct if secondary beneficiaries of the project were members of the judiciary, whose work and working conditions were to be enhanced by the provision of infrastructure, equipment, training, and rationalized procedures and processes. 1.5 Original Components (as approved): It bears noting that this project was developed without a rigorous logframe, possibly because Bank standards were more relaxed at the time. Thus the descriptions of the components are essentially a list of major activities (inputs), and these same activities also constitute the indicators of progress. There are a few output and virtually no outcome or impacts indicators for the components. As discussed further below, this posed some problems for evaluation of project success. Also, as the project was designed within the judiciary's larger modernization plan, it was intended from the start that various donors and the judiciary itself would contribute to advancing the same goals. This had some impact on how activities were later interpreted (e.g. the IADB ended up doing most infrastructure construction in the most removed areas, meaning that the WB focused its three large construction projects in secondary cities). It also meant that the project was designed, and thus its accomplishments must be interpreted, in terms of the collective efforts mobilized to advance the plan. As is clear from the PAD and other early documentation (Aide Memoire, January-February 1998), it was never intended that the operation advance the PDO on its own, but rather that it would contribute to a common effort to achieve the larger goals. Among the donor operations undertaken during the life of the project, the following bear mentioning: IADB $25.0 million Support to sector reform including courts, Public Ministry, Ministry of Government, and Public Defense) ­ to construct justice of the peace courts in outlying areas and integrated judicial centers, combining police, prosecutors, judges, and defenders. Also provided training, public education, equipment, and support to the coordinating entity for sector modernization USAID $18.5 million, in Support (largely technical assistance) to two large grants implementation of criminal justice procedural code, plus additional working with courts, defense, and prosecutors. smaller programs Complements IADB judicial centers by training staff within them. Also included ADR and community dispute resolution mechanisms. AECI (Spain) Various grants, To support public defense, judicial notifications and 4 roughly $2 intake system, indigenous defenders. Spanish million government was also major actors in police reform Sweden Various grants Largely support to UNDP PROFED program roughly $6 (strengthening of rule of law) in the areas of million administrative reorganization and reengineering. Norway Various grants, Support to Judiciary, Public Ministry, and $3 million indigenous defense, including construction of justice of the peace courts Netherlands $1 million Justice of the peace courtroom construction Finland $1 million Human rights education European Initially 30 Program to reinforce judicial education and judicial Union Euros, currently school. 12 million The quantities are approximate and are taken from various sources. Data on the Nordic countries, Spain and the Netherlands only cover activities up to 2003. In the early years, the UNDP coordinated some donor activities through its PROFED program aimed at "strengthening the rule of law." Part of this effort, financed by the Swedish assistance agency, involved the administrative reforms in the judiciary also supported by the WB project From 1994 to 2004 the UN also had an observer mission in Guatemala, MINUGUA, which monitored compliance with the Peace Accords as well as human rights abuses, and in the justice area, provided, with its own staff, technical assistance to Public Defense, the Public Ministry, the police, and judicial training. The only operations approximating the size of that of the World Bank were the IADB multi-sector loan (including police and prosecution as well as the courts), the two USAID programs, and the promised EU operation which has only recently materialized after years of negotiations at somewhat less than its original dimensions. Although in theory all worked off the sector modernization plan, it is apparent that donors were guided by what they considered important, often using items figuring in the plan's long list as a justification, not as a guide. 1.5.A Institutional Capacity Improvement ($10.7 million WB, $17.9 million total): Entailed administrative restructuring of the Judicial Branch; modernization of administrative systems at central and regional levels to improve management; setting standards for managing caseflow; rationalizing the distribution of caseloads; expanding auxiliary service centers; designing, creating and testing a judicial management system; permanent training of judicial and non-judicial staff, and support for the establishment of a judicial career and a civil service framework (for auxiliary and administrative staff). Key Indicators: · New management and organizational model, designed, discussed, approved, and tested. · Judiciary Branch administratively restructured. 5 · Standards for managing case flow established. · Human resources development program (including training) established and tested. · Judicial career law approved and applied. · Inventory of cases prepared and judicial statistics developed. · Court delay reduction program in Guatemala City established and experimentally tested. · Judicial Information system designed, established and experimentally tested in Guatemala City courts and two pilot jurisdictions. 1.5.B Transparency ($2.3 million WB, $2.5 million total): Introduction of anti-corruption measures and extent of openness and user confidence, including establishment of anti- corruption commission in Guatemala City; preparation and testing of ethical code for judges and non-judicial staff, and expeditious resolution of complaints. Key Indicators · Anti-corruption commission established and tested. · Ethics code prepared and approved. · Department of Supervision of Courts reorganized. · System to register lawyers assistants developed and tested. · Review of incentive systems for judges. · Expansion of oral procedures in other areas of law. 1.5.C. Access to Justice: ($12.5 million WB, 21.3 million total): Expansion and diversification of judicial services, including collection of information on practices and needs of current marginalized and non-users; development of performance indicators for two departmental centers and creation of these centers and of two regional centers; construction and rehabilitation of justice of the peace courts, training of judges in local cultures and customs, creation of alternative dispute resolutions services, and introduction of subjects to promote studies, workshops, and citizen and donor participation. Key Indicators: · Policy on traditional conflict resolution mechanisms, prepared, discussed and approved · Formal and alternative dispute resolution mechanisms expanded · Number of judges training in local cultures and customs · Program of the new role for justices of the peace operational · Number of judges increased in rural areas. · Two regional centers (centros regionales), two departmental centers (complejos departamentales) constructed and justice of the peace courts constructed or rehabilitated and operational · Number of participation subprojects provided and implemented 6 1.5.D Improvement of Communications and Strengthening of Modernization Commission and PCU ($5.2 million WB, $5.7 million total): Included training in project implementation and coordination matters; clarification of functions and responsibilities; development of adequate monitoring and evaluation capacity; appointment of specialists for programming, quality assurance and carrying out specialized studies; creating channels for inter-institutional coordination at central level and in specified regions; creation of an Office of Public Affairs for the development and implementation of a social communication strategy with a multi-lingual focus and to prepare press releases, radio addresses and TV spots on progress in the modernization process and other activities related to the judicial branch. Key Indicators · Long term social communication and information sharing strategy developed and operational. · Channels of internal and external communication and coordination developed and made operational. · Office of Public Affairs (with multilingual capabilities) established to prepare press releases, radio addresses, and TV spots on progress in implementation · Modernization Commission trained in change management and coordination matters. · PCU functions and responsibilities clearly established. · Adequate quality assurance, monitoring and evaluation capacity developed. · PCU financial management and reporting system automated. · Annual modernization report published. 1.6 Revised Components There were no revisions to components. However, as with the PDO indicators, those for the components were altered in ISR 15 and after. From mid-2005 onwards, only two outcome indicators were tracked ("consultation, workshop results, user survey results," and "formal and alternative dispute resolution mechanisms expanded"). They are virtually identical to the PDO indicators, and once more there is no explanation or further documentation of the change. Thus, the ICR's Annex 2 and Data Sheet still work with the PAD indicators and component activities. Many of these activities were also indicators in the initial list and moreover figure in Schedule 2 to the loan agreement. 1.7 Other significant changes Among the principal changes in the project were the three extensions of its closing date (to June 2005; December 2006, and June 2007). The major reasons for the extensions were delays in procurements, and especially in construction projects. Construction in turn was delayed because of difficulties in acquiring land from the municipalities where the new courts were to be built, and in the case of the Escuintla center, the discovery that the land was unsuitable for the planned construction. (Unfortunately, this only occurred once construction began, as the firm hired directly by the PCU to test the soil had 7 certified its suitability.) This required significant additional investments to prepare the site and added another year to the time required to complete the project. A second significant change involved two reallocations of funding among the major cost categories, with funds moved from consulting services and training to goods and works, and a final reallocation of funding during the grace period, as shown below: Expenditure Original amt, June 2003 November December category US $millions reallocation, 2006 2007 (1998) US $millions reallocation, reallocation US $millions US $millions Goods 2.4 6.0 8.85 8.26 Works 11.0 13.7 12.72 11.92 Consulting services 14.0 10.3 9.21 10.57 Training 2.7 1.8 1.4 1.4 Administrative 0.6 0.6 0.5 0.5 expenses Fee 0.33 0.32 0.32 0.32 Unallocated 1.97 0.3 0 0 Total 33.0 33.0 33.0 33.0 The first reallocation was done at the request of the Court and justified by the initial underestimates of the equipment and construction costs. As regards equipment, it also appears that the Court's own requirements changed, now including the installation of equipment "in all the courtrooms of the country." The second reallocation again responded to a request from the Court, and added more monies for equipment (in part because of the requirements of the new case tracking system, but also to ensure expenditure of the balance of unused funds) while cutting slightly the works budget from its previous high.4 Although both requests from the Borrower claimed a lesser need for technical assistance funds, some activities requiring consultant services undertaken toward the end of the project were severely under budgeted, causing procurement delays, cancellation of at least one procurement, and disputes over deliverables. The final reallocation was in essence a bookkeeping exercise, to move unused funds into the category of consulting services. The reallocations also affected the distribution of expenditures among and within project components. For example, by the end of 2006, the Transparency Component had only expended $185,000 of the $2.3 million of Bank funds allocated in the PAD, and the Social Communication subcomponent of Component D accounted for only about one- tenth of the $3 million initially budgeted. It bears noting that in his March, 2003 4It is worth noting that by the time of the second reallocation, it was evident that one of the departmental centers would not be built. 8 communication to the Bank, the UMOJ (PCU) coordinator indicated the need for adjusting downward the amounts programmed for these components ­ to $1.2 million for Transparency and $1.5 million for Social Communication. The Bank's response made no reference to this suggestion. A third change involved the disappearance of the donor "co-financers" from the counterpart budget. In 2005, the external auditors stopped mentioning "co-financiers'" counterpart (which they had never tracked, but only listed as estimated total contributions over the life of project), and the official counterpart budget was limited to the $3.6 million to be provided by the government. Although the auditors refer to a formal change, insisting it was introduced to ensure adequate amounts were available for each of the four components ("con el objetivo de disponer de suficientes fondos en las categorías correspondientes," Lara, Aranky, Ramos, and Associates, 2006), the ICR team could find no documentation to this effect (which, we stress, does not mean it does not exist). In any event, co-financing was always problematic, as except for WB-UNDP cooperation on a reengineering exercise in 1999-2001, other agencies, even those operating through or in coordination with UMOJ, programmed and negotiated their operations separately. There were also slight changes in the implementation arrangements, significant only because of the problems that underlay them. UMOJ, the project coordinating unit, was ultimately responsible for project implementation (overseen by a Supreme Court Justice serving as the project Coordinator). However, responsibility for financial administration and for large procurement actions had been transferred to UNDP and UNOPs, respectively, a practice not uncommon in LAC. These arrangements did not change, but the division of labor among UNDP, UNOPS, and UMOJ experienced some slight alterations, if not in the formal agreements governing these relationships then in their interpretation by the parties. Initially UNOPS was to handle infrastructure and equipment purchases, and large service consultancies, doing all procurement and signing and managing the contracts. UNDP handled disbursements and payments and was to provide advice on smaller service contracts, except for those of UMOJ staff (which the UMOJ did on its own). UNDP did not sign or manage any contracts. However, it could not pay vendors without UMOJ's prior agreement. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry The strengths and weaknesses of preparation, design, and quality at entry are as follows: Correct choice of PDO, but overly ambitious statement of aims: The three-part PDO accurately reflects the issues Guatemalan citizens identified as most important (and was also consistent with the commitments of the Peace Accords). However, in five or even eight years, substantial advances in all three areas were unlikely. Furthermore, the Peace Accords marked a symbolic beginning to the process of improved justice and access to justice, not the end, and any project initiated in the context of post-1996 Guatemala would require a gradual and cautious program of interventions. While the project could 9 have done more in pressing for outcome improvements, it might have stated its aims more modestly (and thus not set itself up for failure when judged against its goals). This was admittedly not part of the WB ethos in earlier years (when such overstatements of aims were not only allowed, but encouraged) and would have required a more sophisticated discussion of the necessary change sequence (and thus of the need to establish an organizational base for future performance improvements) as well as a greater emphasis on critical steps needed to accomplish it. Lengthy preparation: While often considered problematic within the WB, this was in effect an asset and helped the project surmount the challenges of having to start implementation with a new set of counterparts (the new Supreme Court, see below). The opportunity to involve significant numbers of stakeholders in the development of a sector and judicial modernization plan and to develop work plans for activities like the administrative reengineering (in cooperation with the UNDP's PROFED) created an impetus and expectations to push initial activities ahead. It was also helpful to have done the preparation with a project implementation unit that would also take over the operations and thus was pretty much on board with their contents. Whether intentional or not, the several-year preparation provides an important lesson to those undertaking institution building activities in difficult environments; so long as the extra time is used, as it was here, to expand the operation's constituency by involving them in planning and diagnostic work, it can pay off with a smoother transition into execution. Participatory methodology for project preparation and design: This is both a strength and weakness. During the preparation phase, the Bank (via a PHRD) funded a series of workshops and other events and the eventual production of the Judiciary's Modernization Plan (on which the project's content was based). Project documents estimate that roughly one third of the seated bench and their assistants as well as other local stakeholders and donors participated in these activities. The consequence was a modernization plan which reflected the judiciary's own analysis of its needs and problems and also incorporated perspectives of a wider community of real and potential users. Another grant funded the development of a detailed implementation scheme for the judicial plan, using the services of a firm with experience in designing and implementing USAID projects. The positive side of these practices is that the project's content for the most part coincided with local perceptions of needs. The weakness is the likelihood that local perceptions were based on incomplete knowledge and excessive faith in practices which international experience had already put in question. The World Bank, and other donors, have still not reached any conclusions as to how to deal with this conundrum, and the project at most can be criticized for not recognizing the problem and for thus assuming that local buy-in would guarantee successful achievement of its objectives. 10 Internal and NGO criticisms of the first WB project in Venezuela5 had already forced modifications to its initial reliance on infrastructure and equipment to induce institutional change (Lawyers Committee, 1996). Four years before the PAD was approved, USAID published a strategic framework (Blair and Hansen, 1994) relegating "institutional strengthening" composed of training, buildings, and equipment to a last step in its own programs, following constituency building, legal and structural change, and access expansion. Nonetheless, Guatemala was no exception to another general rule ­ that when judges (or for that matter members of any organization) are asked what is needed to improve their performance, they will generally mention higher salaries, better equipment, and better facilities (infrastructure). Significantly, one of the last studies financed under the project (FUNCEDES, 2006) also noted that while Guatemalan judges again cited infrastructure and equipment as their basic needs, system users complained about insufficient attention to reducing corruption, delays, and unpredictable judgments. As this last comment suggests, there are two related problems arising in the participatory approach as practiced in Guatemala. First, in identifying problems it is better to rely more on users than on system operators. The latter equate problems with what they don't have; the former are a more reliable source of information on real performance failings (what the latter don't do). Second, however well local participants of either type identify performance failings, they may be less capable of designing solutions. Here more experienced technical assistance may be useful to ensure that users' problems do not end up with providers' solutions ­ the former complain about delay or corruption, but faute de mieux, end up endorsing the latter's suggestion that higher salaries, better equipment, and modern buildings are the answer. In short, participation is important, but it has its limits as a guide to project design and is best supplemented with external expertise. 6 Incorporation of lessons learned: As noted above, the project was prepared after the pre- approval redesign of the Venezuela operation (to shift infrastructure to the Venezuelan counterpart contribution), but did not appear to incorporate the lessons learned there or in earlier USAID projects (Blair and Hansen, 1994). Its preparation coincided with an NGO report on the Venezuela operation, sharply criticizing the failure to focus on, or take serious steps to deal with three major problems ­ arbitrariness, corruption and delays ­ and to address the extensive political interference that accounts for them (Lawyers Committee, 1996). Although the Guatemala PAD does mention delay and corruption, they, along with political interference, probably did not receive the attention they deserved in design and implementation. Obviously these are sensitive topics, and the PAD authors are to be congratulated for even mentioning them at a time when the Bank was still tip-toeing around the themes. However, at that time, and especially during 5 Venezuela Judicial Infrastructure Development Project (3414-VE), 1992-2003. 6As another example, the Hondurans are still fighting to introduce a Judicial Council that will handle administrative and governance functions for the courts, despite unsuccessful experience with this model elsewhere in the region. The Guatemalans are to be congratulated for not taking this route, but it is another example of why external expertise and experience are important. 11 implementation (by which point the Bank was on an anti-corruption course) there might have been ways of ensuring they were more adequately addressed. The PAD's incorporation of other recommendations from the Venezuela critique (use of participation and development of a long-term plan) seems to miss the larger point ­ design by committee is only a first stage in a participatory approach, and as the Lawyers Committee aptly notes, "a comprehensive plan means more than identifying the range of problems plaguing a judicial system; its real importance lies in developing and sequencing reform initiatives in a logical progression" (p. 123). The project would also have benefited from other lessons already emerging in 19987 ­ the inefficacy of ethics codes in fighting corruption, the need to design training to address problems and not just a desire to be trained, the difference between access to courts and access to justice, and the need to push for institutional change on several coordinated fronts. The project did benefit, even after it went into effect, from the Bank team's familiarity with numerous innovative practices introduced in other Bank and donor projects, but perhaps overlooked a more basic lesson that institutional change is slow, that the incorporation of multiple goals and activities can overextend energies, and that where counterparts do not internalize the underlying performance goals, projects can simply produce much action and little improvement. While the preparation team did consult with donors already working in Guatemala, it is not evident that they used their lessons on the difficulties of operating in this environment. Longer discussions with USAID and MINUGUA/UNDP would have usefully identified the political obstacles and institutional weaknesses they had faced. Weak links between PDO and activities included in components: The PAD and logframe did include activities important to advancing the overall objective of promoting more accessible, efficient, and credible justice. However, these appear as a list of actions to be taken without any noticeable prioritization, sequencing, or necessary connections among them. Most of them appear as unusual endeavors, and the PAD does not emphasize any need to support their continued development once completed. In fact, one major conclusion is that the absence of follow-up in implementation marred the potential for success in several areas; human resources is a prime example. Some activities only introduce a study or a restructuring with no further reference to necessary follow-up. Certain activities as mentioned in the PAD or interpreted by the counterpart, although receiving considerable emphasis, were clearly less important to the PDO. Examples include the automation of Antecedentes Penales (issuing the certification of no criminal record required by those seeking employment) and of the Departamento de Protocolos (essentially a registry of lawyers, notaries, and documents related to their work). While both activities were successful (and can be considered quick wins), their impact on the overall quality of sector performance is minimal. They clearly take second place to creating a full registry of human resources (more or less complete but only in its payroll 7Worth mentioning here are one WB publication (Hammergren and Messick, 1998); Hammergren, 1998 a and b; and several works that appeared slightly later but reflected ideas already emerging in the late 1990s. See for example, Popkin, 2000; Prillaman, 2000; and articles in Méndez et al, 1999. 12 module) or a functional system for tracking judicial workloads and outputs (still nonexistent). The PAD resembled a recipe listing all necessary ingredients, but omitting information as to quantities or order and manner of their incorporation. Emphasis on input rather than outcome/impact indicators and resultant poor M and E framework: The indicators are poorly designed and offer little if any means of tracking performance improvements. As discussed in some detail below, those for the PDO are difficult to interpret, and if rigorously applied to test project advances, would produce a very low grade. Nearly all indicators represent either inputs or implementation benchmarks. This not only posed a problem for the ICR but also seems to have diverted counterpart attention from the end goals. It tended to encourage an emphasis on implementing as many actions as possible, with little distinction as to relative importance or preferred sequencing. As the indicators did not include any measures of judicial performance (actual delay reduction, number of targeted users attracted to decentralized units) this also affected the monitoring and evaluation component and may largely explain the failure to develop one for performance improvement. Lack of baseline data and failure to prioritize their creation in early stages of project: During preparation and at project initiation, the Guatemala judiciary lacked reliable statistics for tracking input, output, or timeliness of decisions.8 Thus, while the PAD contains references to excessive delays and backlogs and to a general failure to keep up with rising demand, there was no empirical base for these observations. The problem might have been addressed by prioritizing, in the PAD and implementation plan, those activities related to performance tracking so as to provide baseline data for monitoring and evaluation and as a first step in strengthening institutional capacity. Unfortunately, few actions were taken in the early years and what is produced in the way of statistics is still regarded as incomplete and unreliable. One exception may be Guatemala City where Gestión Penal, the intake center for all criminal cases in the courts there, now tracks (as the result of a USAID project) all new criminal cases, the stages in their processing, and their closure. However, interviews in Gestión Penal and in the existing statistical office did not indicate that the latter used the former's database to supplement or test its own manual system or to do further analysis of case processing times. (There are also reported problems with Gestión Penal's statistics, but nothing has been done to identify and correct them.) Inadequate attention to structural problems affecting counterpart leadership: While the PAD makes mention of the risk that the new Supreme Court (entering in November, 1999) might have less commitment to the project goals than its predecessor, it neglects some important details: the fact that the Supreme Court and entire appellate bench are fully replaced (with a potential for reelection that is infrequently used) every five years, and that administrative directors and higher ranking officials lack tenure, serving at the pleasure of the Court President (changed every year with no chance for reelection). Thus, 8The modernization plan does make reference to a "preliminary" inventory, but it was done only in the Guatemala courts, and seems never to have been used after that. It was not provided to the ICR team. 13 the risk and its repercussions were considerably downplayed, as subsequent events bore out. One further unanticipated consequence was the dominant role assumed by the PCU in setting policy and the failure to develop judicial bodies that might do so, even within each court's five-year tenure. In short, while all three Supreme Courts with which the project worked seemed content with the project, none of them developed real leadership or ownership nor did they try to encourage the strengthening of administrative offices or arrangements, apart from the PCU, that might have taken over some broader policy development functions. One such arrangement might have logically focused on the Secretaría de Planificación y Desarrollo Institucional; yet, following the reengineering of the early 2000s, it remained relatively isolated from the project and any further strengthening of its operations occurred through its own efforts, not as a result of project interventions. 2.2 Implementation Frequent changes of key actors on counterpart side: As explained above, this was an unavoidable but highly predictable consequence of the Guatemalan legal framework, and it made both judicial and administrative ownership and leadership of the project problematic. The Supreme Court entering in 1999, soon after the project became effective, is generally regarded as having had less interest in reform than either its predecessor or its successor, and as noted in a previous section, it may be largely the impetus created through the lengthy preparation period that kept it more or less on course. However, in 2003, and despite the recommendations of the mid-term evaluation, this same court determined it needed little more technical assistance and thus requested a reallocation of funds to equipment purchases. Beyond their impact on maintaining inter-temporal agreements and strategies, the frequent turnovers create a number of additional perverse incentives as regards leaders' willingness to press for difficult changes. For Supreme Court and appellate justices this would require a highly altruistic approach with little personal pay-off. Even the Court President, according to knowledgeable observers, has to use considerable political capital (lobbying the rest of the justices) to push for any change in standard operating procedures. For administrators it risks angering those who could push for their removal. Those responsible for oversight and discipline of judges also noted a "what-goes-around-comes-around" outlook; steps taken against any individual might cost them considerably when they returned to their ordinary positions (as judges) or even in their current posts. Uneven pattern of disbursements and substantive emphasis: This is a likely consequence of changes in judicial leadership and fluctuations in their own interest in different project components. As regards new initiatives, the project was most active during preparation and its early years. This was when the reengineering and restructuring of administrative offices took place, as well as the provision of equipment and some technical assistance to many of them. It also saw a potentially useful, if truncated effort to work on speeding up courtroom processing of cases and the introduction of an Alternative Dispute Resolution (ADR) program. As most of this was consultant services, disbursements lagged. Disbursements accelerated in the last years (including a record $10m in 2006), but this was largely due to the emphasis on completing construction and equipment purchases. Further attention to improving judicial (courtroom) performance was delayed until the 14 very end, and what was done there (i.e. the case inventory and related study on delay and its causes) has still not paid off in any concrete impacts.9 Some of it may never be used owing to the disputes between the contractor and the UMOJ. Innovative introduction of practices from elsewhere in the region: the Bank TT and the TTL in particular made good use of examples gleaned from around the region (or the world) to introduce new initiatives into the project. Although the ICR team, along with some outside observers (Svensson, 2007), have doubts about their long-term value added, they did maintain interest in and attention to the project. This should not be discounted, especially given the usually boring nature of the more classic reform ventures. The pursue of quick wins as well as publicity for innovations are important tactical additions to an operation whose normal activities may more often resemble watching grass grow. . Use of Guatemalan technical assistance: This was a conscious decision by the PCU and for the most part was a plus for the project. Not all technical assistance came from Guatemala. However, of the two major exceptions, the firm developing the case management software and the organization conducting the inventory and delay reduction study, the first is considered a success and the second a failure. On the plus side, use of Guatemalan experts ensured that funds went further, expanded the number of active stakeholders in the process, and arguably formed a group of local experts able to apply external disciplines to court processes. One excellent example was the exercise in accelerating processing times in the appellate courts of Guatemala City. The locally hired expert did a casebook perfect job, was able to spend sufficient time to ensure counterpart internalization of the goals and processes, and was later used to teach the methodology to justices of the peace. Unfortunately, as soon as the process was completed, the appellate bench was replaced and the new members did not approve the plan. Project records and interviews with key personnel (including the consultant) do not indicate whether the recommendations were less formally adopted and if so, whether they produced the predicted reductions in delays. Those reported by the PCU for the appellate courts (Novoa, 2006) are based on estimates of time to be saved and some piloted efforts, not eventual results. The ICR team was unable to get information from the PCU as to any impact of the seminars with the justices of the peace on real courtroom practices, or in fact ascertain whether there was any follow-up to encourage adoption of the consultant's recommendations. The down side of a reliance on local expertise is the lesser opportunity for adopting knowledge developed elsewhere. The PCU seemed to recognize this in its use of an international firm to build the case management software. The experience demonstrates both the advantages of outside consultants (delivery of a state-of-the-art product, benefiting from testing in other settings) and the disadvantages (insufficient consultation 9This is largely because the inventory was incomplete and its results inconclusive. However, the contractor's lack of understanding of or loss of interest in the delay reduction portion meant that what was delivered has been rejected by UMOJ. It is the ICR team's conclusion that what the contractor was asked to do would not have provided the necessary information for a complete delay reduction program. 15 with local users and thus continuing complaints that the end product still lacks certain capabilities required by local law and organization; these are however being addressed). The mid-term technical supervision is still another example of the benefits of well chosen external expertise; comments made by consultants selected for the supervision make it evident that local empiricism had overlooked some opportunities and problems. Still on a whole, the policy had mostly positive effects, and the real complaint related to technical assistance, whatever its source, is the failure to continue to provide it to entities that had been reorganized and then left to develop on their own. A review of project documents in IRIS also suggests that while BB funds were used prior to 2003 to bring in international consultants to work on some of these areas (most notably ADR, see Cornish, 2002 and Wanis, 2003), the process stopped after that for reasons that remain unclear. Development of a strong PCU with considerable continuity in its staff. This was a plus for the project and in large part an explanation of the advances it made. At its height the PCU had 43-48 staff members (upper number varies according to the source), of whom a large number were professionals. The quality of implementation was more than satisfactory at least as regards pushing ahead the majority of procurements outlined in the PAD. How the PCU continued to keep staff, despite the changes in the larger administrative system, remains unclear but it experienced relatively less turnover. Although of much reduced size now, many of the members with whom the ICR team interacted had been there for some time. Also a few PCU members (the subcoordinator and one of the informatics specialists, as well as the specialist in judicial education for civil society, who was, however, first moved to UMOJ from her position in the judicial communications office) have now moved to the judicial administration. Ideally, more of this should have occurred. The downside is the PCU's role in policy planning, selection of activities for emphasis, and direct implementation of too many. It appears that none of the key staff was especially concerned with judicial as opposed to administrative performance, which may explain the lesser attention to factors affecting the former. There also are indications of the PCU's excessive involvement in activities ­ especially organization of workshops and short training events and publications ­ that should have been directed by judicial offices (the Judicial School, the Communications Department, Planning, Finance and so on). Ideally, the project should have funded these activities, provided technical assistance where needed to improve quality, but let the relevant offices develop and fit them into their own programs. Instead, the PCU was even accused of further weakening operating units by hiring key staff (Moore, 2007). This hypothesized over-involvement of the PCU deserves more attention; in the time allocated for fieldwork, the ICR team could not explore it sufficiently. It bears noting, nonetheless, that of the $5.2 million of Bank financing initially allocated to component D (Communications, Support for Modernization Commission and PCU), by the end of 2006, $3.9 million had gone to "strengthening the PCU," and that the other two items had received $350,700 and $320,800 each. Failure of counterpart to act on, or WB to press for implementation of recommendations in the mid-term evaluation: The mid-term supervision, conducted in October 2002 16 contains a long list of excellent recommendations as regards the initiation of neglected activities, follow-through on others, establishment of a monitoring and evaluation system, and certain adjustments to most project components. The ICR team was unable to go over the recommendations in detail with PCU staff, but its review of the project timeline and subsequent activities, suggests that many recommendations were not taken and others were acted on only toward the end of the project. Significantly, despite the need for additional technical assistance and training implicit in the recommendations, the first reallocation, shifting funds from these two categories to goods and works, followed shortly. Clearly this was the Court's preference; the question is how or whether the PCU or the WB could have convinced it otherwise. Coordination with other donors: There was a good deal of coordination realized through the PCU, especially because of its management of IADB resources and agreements to coordinate infrastructure with other donors implementing their programs separately. While overall placement of new facilities was sometimes less than rationale, this at least avoided duplication of efforts. Other good examples of PCU-initiated coordination include the combined efforts of the WB and PROFED projects in the 2000 reengineering exercise. Still, coordination could have been much better, especially with donors that did not work through the PCU. Both the new EU operation and the on-going USAID projects are two examples, and relationships of the latter with the PCU seemed to be especially volatile. Whether coordinating formally or not, donors often promoted different and not entirely compatible models for the activities they financed (see Carothers, 1999, p 15, specifically on earlier experience in Guatemala). The Dirección de Servicios Tribunalicios, (intake center for new cases and also responsible for notification services) in Guatemala City had its civil side assisted by the Spanish aid agency, and its criminal side by USAID, resulting in two separate sets of practices and procedures. USAID also supported development of software for case registration, leading to some conflicts (apparently now resolved) with the WB financed case-management software. Moreover, USAID recently (October 2, 2007) presented a model for reorganizing first instance and peace criminal courts that appears not to match the assumptions underlying the WB software design and is also inconsistent with an early model developed, but not implemented, by the WB project. Coordination among judicial branch offices and between them and the PCU: Coordination here was even less notable, and seemed to have slacked off once the initial restructuring and reengineering activities were completed (by the early 2000s). The ICR team found several examples of redundant practices within the administration (for example four offices collecting their own judicial statistics), failures to use or even be aware of the potential to combine forces or utilize each other's databases, and lack of knowledge of relevant studies done by the PCU (for example the case inventory which should have interested several offices). Various administrators interviewed referred to their lack of knowledge of what the PCU was doing (Moore, 2007). For a project that placed an early emphasis on administrative restructuring and rationalization, this is not a good sign. 17 Failure to implement key studies early enough to utilize them in the project: Because the project was only intended to last for five years, it may have made sense to limit some activities to conducting studies. Still even within the initial timeframe, had these studies (on traditional law, on delay, on human resources, the case inventory) been initiated early on, they could have had an impact on the design and implementation of related activities. Failure to follow-up on early activities so as to encourage further progress: As suggested above, and partly as a consequence of the initial short timeframe for the project, the PAD rarely mentioned any follow-up on restructurings, studies, or activities (delay reduction exercises) intended to improve administrative and judicial performance. As a result, such follow-up rarely occurred, meaning that pilot efforts were not replicated, restructurings were never tested and readjusted, and continuing assistance to reengineered entities was rare (Moore, 2007). In some cases this was critical because the initial activities needed readjustment; in others it was equally necessary because more assistance would have helped them perform their new tasks. As one example, while the project supported the creation and equipping of the Consejo de la Carrera Judicial and its various units, the ICR team found no record of further attention to improving their operations. This attention is vital as they are experiencing many problems, ranging from still inadequate equipment (e.g. vehicles) to legal and methodological constraints. A second clear example was the failure to provide appropriate training for the Director of Human Resources. Another, already cited above, involved the exercise on reducing courtroom delays conducted by a local consultant with appellate and justice of the peace courts. Although the 3 volume report was provided to the PCU, the latter could not provide evidence of even informal adoption of his suggestions. Finally, PROFED (UNDP) during the 1999-2001 reengineering worked on improving the internal organization of family courts, using two pilot courts in Guatemala City. However, this activity also lacked follow-up. Excessive reliance on a new case management software to fix the problems not resolved earlier: As regards many of the criticisms made above (lack of attention to delay reduction, failure to follow up on exercises to facilitate courtroom processing of cases), UMOJ now holds that replication of the new management software will provide the answer. Judging by comments made as early as 1996 (Lawyers Committee), the lessons of other projects, and Microsoft's own experts (Roberto Adelardi De Diego, in presentation to V Seminario Internacional de Gestión Judicial, Lima, October 10-11, 2007), this faith is misplaced. The software is an important tool and can save judges and their staff time, but it cannot overcome problems originating in legal obstacles, procedural inefficiency, leadership's failure to monitor and resolve performance weaknesses, or parties' efforts to increase delays and exert irregular influences on judges and staff. Ideally, most of these problems should be addressed before the software roll- out. The expectation that the software will make that unnecessary raises two additional risks: that the software will incorporate them in its own design and that it will further postpone any effort to resolve them. Use of UNOPS to manage infrastructure and equipment contracts: As regards equipment purchases, there were no problems reported, but in the case of infrastructure, problems 18 and conflicts were frequent. Neither the PCU nor UNOPS regards the arrangement as satisfactory, and both have suggested that it be discontinued. Whatever the immediate source of the problems, their resolution has been impeded by the contract designs, which give UNOPS responsibility for the contract, but UMOJ the final determination as to whether the products are satisfactory and what payments will be made. (However, this detail is not reflected in the contracts signed by UNOPS with the construction firms). Negative results included delays in finishing projects, cost overruns (which in a few cases UNOPS had to pay), and dissatisfaction with the final results. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization There really was no monitoring and evaluation system for this project; what existed focused almost exclusively on tracking the delivery of inputs, disbursements, and payments. At least through 2001 (UMOJ, 1999, 2000, 2001a),10 the PCU provided annual reports tracking progress in the "outcome" indicators for the project. However, estimates of completion of activities seem somewhat questionable.11 No system was established to measure the impact of individual activities in improving access, efficiency, and credibility. As support for this statement, readers are referred to the 2006 table on CAS inputs (N.A. 2006) prepared by a consultant and in which many of the boxes listing PDO and component indicators are left blank, or reported as "in progress." For others, the author seems to have taken a fairly liberal interpretation of what they represented. The ICR team asked the PCU to update the chart, but is still awaiting the response. The several activities that were intended to set baseline data and measure progress were either done late or not at all. The last user survey was not done, and what was substituted does not allow comparison with the earlier two surveys. Failure to implement a reliable judicial statistics system means that efficacy could not be measured, and measures of access are limited to the number of new courts and judgeships created. The external consultant (Novoa, 2006) hired by UMOJ to collect information as an input to the ICR was unable to find good indicators of impacts and from what was available largely concluded, as does this ICR, that except for physical access, impact on system performance appears minimal. 2.4 Safeguard and Fiduciary Compliance The team reviewed a sample of the financial and procurement post reviews as well as all the external audits, none of which reported significant problems. Those that were reported have been or are being resolved. The PCU adopted an automated system to track its expenses; unfortunately this was not the one used by the judiciary's budget office (SIAF) and thus information supplied to the latter on the project is entered into the judicial system only as aggregate expenditures (Moore, 2007). 10These were the only reports provided by UMOJ. 11For example, there is the question of how the percentage of participatory activities completed could be calculated when no target number was ever set. 19 One issue not noted until after project closure involves the inadequacy of the contractual arrangements among the UMOJ, UNDP, UNOPS, and the contractors the latter managed. Aside from, but perhaps contributing to, the increasingly difficult relationships among the parties, the "nest of contracts" was poorly designed, leaving UNOPS wholly responsible for a contract but allowing UMOJ the final say as to whether the contractor complied with requirements (final say, because UMOJ authorizes UNDP to make the payments). It should also be noted that UMOJ has had problems with contracts it has managed on its own, and that its tendency to reject products with little indication of the necessary improvements, or to provide what the contractors described as insufficient support for their activities, should be addressed. The project is ending with two contract disputes totaling roughly $150,00 each and the grace period is being extended for two additional months in the hopes that some definitive resolution can be reached. As none of the parties seems inclined to repeat the former arrangements in a new project, UMOJ will probably manage contracts on its own, but to do that satisfactorily it will need to improve its manner of handling disputes over deliverables. Based on reports from the affected contractors, it also appears that they (and UNOPS) erred in not documenting their purported agreements to certain changes. If this is true, UMOJ may benefit from their oversights, but in the future, it will be important for UMOJ to insist on the documentation of any such changes so as to avoid any repetition of these conflicts. These and other contractors also complained about the poor quality of the UMOJ terms of reference, and continuing disputes (generally won by UMOJ, but only because it had the final say) as to what they really meant. The complaints were frequent enough to suggest a real problem and the need to address it in any future project. 2.5 Post-completion Operation/Next Phase At the writing of the ICR, the project was still operating in its grace period. Operation and maintenance of constructions and equipment have been assumed by the judiciary. The ICR team did receive some complaints on equipment maintenance during its interviews ­ a few administrative and judicial staff members noted they had been without functioning equipment for some time. The WB intends a follow-on project scheduled for board approval in July, 2008. In the meantime the IADB has finished negotiating a new loan with the sector, of which a portion will again be managed by UMOJ. It is anticipated that the IADB loan will take over financing of the latter at least until the WB project comes on line. The two banks and the judiciary will then have to decide how its funding will be handled. The initial WB plan was for a repeater project, with board approval in December, 2007. It appears that the plan has changed, not only as involves the dates, but also as involves the content of the operation, which will be a follow-on, not a repeater. The CMU has expressed to the judiciary its lesser interest in financing infrastructure, and its interest in financing equipment only as it contributes to the furtherance of the institutional change and performance improvement agenda. It is thus envisioned that the next operation will focus on using the organizational base created by its predecessor to push forward the PDO by further strengthening judicial oversight, planning, and implementation capacity as regards administrative and judicial operations, advancing the adoption and use of the case management software, encouraging steps to break down the barriers to access apart from physical proximity of services, and improving the entire human resource management system. With regard to any future 20 project implementation unit, it is imperative that the Bank encourage greater integration with existing administrative and judicial units, and in particular with the Secretaría de Planificación y Desarrollo Institucional. 3. Assessment of Outcomes As a preface to this discussion, a short note on "attribution" seems necessary. We know this is stressed by IEG, and we understand its significance. However, we emphasize that in an institutional development project, and especially one where many donors, not to mention the national government, are operating simultaneously, attribution of outcomes is extremely difficult, if not impossible to determine. Projects are not controlled experiments and the key question ­ would things have occurred without the project? ­ can rarely be answered. There are exceptions ­ especially as regards inputs (where a project did something no one else was supporting) ­ but outcomes and impacts are another matter. What a project usually does is increase the chances of changes in the quality of organizational performance, and even there attribution is an inference, not a quantifiable fact. In any event, in the present project, the evaluation is working in one of the more difficult contexts ­ many donors and local agencies working on similar activities, poor records and memories as to who did what, and many shared endeavors, in a highly volatile and difficult national environment. Since the WB project was supporting a PCU that also managed IADB funds and coordinated with several other donors, there is the additional question of how much credit should go for actions financed by others through this process. (While one could give the WB credit for the IADB activities the PCU managed, it strikes us that absent the WB, the IADB would doubtless have paid for its own PCU ­ hence attribution of IADB results is a bit of a stretch.) In dealing with the issue of attribution, the ICR team has followed a logic based more on common sense than anything more scientific. Thus, while the project's presence may have marginally increased the chances of certain key legislation (e.g. the career law) being enacted, we are not giving it credit there. It does however get credit for what it did to strengthen the agencies that applied the law and to provide the equipment and infrastructure they required to do so ­ and is criticized for what it did not do, did too late, or initiated without sufficient follow-up. 3.1 Relevance of Objectives, Design and Implementation The PDO ­ creating a more efficient, accessible, and credible judicial system ­ was and remains highly relevant to national, global, and Bank assistance strategies. Improving the justice system and increasing its accessibility to indigenous citizens were highlighted in the Peace Accords and the overall development plan they included for the post-conflict period. Improving access to justice and the quality of that delivered has only increased importance as a global priority over the past decade. The project and its PDO were fully consistent with the CAS emphasis on advancing the post-war social reconstruction process and modernizing the public sector. Moreover the CAS (18036-GU) in effect at the time of project preparation and appraisal specifically emphasizes judicial reform as part of its support to the Peace Accord commitments. 21 Project design was another matter. This point and those on implementation have been mentioned in other sections of this ICR, but they are summarized here and referred to where otherwise critical. Project design evinced four major flaws: · The components lack adequate definitions, never rising above a collection of activities presumed to advance some common objective, and conflate lines of action (e.g. strengthening of administrative systems and of judicial performance) that might better be treated separately. · Indicators are poorly chosen, inadequately defined, and in a majority of cases refer to inputs not outcomes or impacts. In several cases, their interpretation is ambiguous (e.g. what value to assign to the number of complaints about corruption registered, or to the percentage investigated and processed). · The causal linkages between activities within each component and the development objectives are not sufficiently clear. This is "the recipe without the quantities or order and manner of incorporation." · While project preparation included preparation of a Gantt chart, detailing the intended order of implementation down to the week or day, the chart had little impact on the order in which activities were initiated and completed. In the end the Gantt chart was overly-complex; it would have been better to develop a less detailed plan emphasizing prioritization and sequencing of lines of action (along the lines recommended by the Lawyers Committee in 1996). As a consequence, implementation appears to have followed a path determined by the PCU's strengths and perspectives, altered only by last minute efforts to attend to areas ignored until that time and to spend remaining funds. Activities that should logically have been initiated early on (a nation-wide inventory of cases, development of a reliable statistical system for tracking court performance, development of a human resources program to rationalize assignment and retraining of personnel) came relatively late, and several key activities (development of programs for delay reduction, the judicial information system, and of standards for managing caseloads) are now deemed dependent on the installation of a sophisticated case management software. As software design was only completed in mid-2006, and the system has so far been implemented in only 32 trial courts, it probably will not provide these results for several more years. Moreover activities initiated and completed early on have usually not received any follow-up ­ and to the extent they have prospered seem to have done so on the basis of the talents of the officials in charge. Noteworthy here are the unit within the Judicial Council responsible for evaluating judges, the budget office, the planning office, the human resource office, and Servicios Tribunalicios, responsible for receiving, assigning, and tracking new cases, managing an archive of active criminal case files, and running the notification services for all Guatemala City courts. All of these departments received some initial assistance (especially during the reengineering exercise of 2000) and equipment, but by their own reports, had little further contact with the project or the PCU. It is worth noting that the mid-term supervision (World Bank, 2002) called attention to the problem of follow-up and of the areas still not touched, but it does not appear that its recommendations had much impact. This is especially true as regards those relating to: 22 · Further changes in the management of administrative staff ­ development of profiles and their use to reassign, retrain, or encourage voluntary retirements of staff not fitting the profile. · Integration of the PCU's activities with those of the judiciary's existing administrative structure to advance internalization of the program, its methods, and its goals. · Evaluations of the Department of Human Resources, Judicial School, ADR program, and of the training sponsored by the project. · Development of a judicial statistics system and implementation of the various activities intended to produce it. · Realization of the study on "particularidades" of rural and urban communities (only completed in late 2006). · Realization of the studies and activities associated in the PAD with the civil society participation program. · Development of a system of monitoring and control of the Judicial Modernization Program. While recognizing the many advances made under the program, we believe project design impeded further progress toward achievement of the PDO; allowing implementation of an arguably excessive number of activities and placing insufficient attention on the need for follow-up of actions initially completed. Indicators stressing outcomes and impacts rather than inputs might have encouraged better implementation practices and a longer term emphasis on producing real improvements in performance. As none of the evaluation team is extensively familiar with Bank projects developed in the late 1990s, they recognize that they may be setting the bar too high and have tried to take this into account in later rating sections. However, regardless of how this specific project is or should be graded, the point is that these are the standards against which contemporary judicial reform or any institutional development efforts should now be judged and in terms of which they should be designed. 3.2 Achievement of Project Development Objectives The achievement of the PDO ­ creation of more effective, accessible, and credible judicial system performance ­ is rated as moderately unsatisfactory. This rating is based less on visible improvements in the three areas than on the progress in putting in place elements that might, over the longer run, allow their achievement. However, of the five initial performance indicators, only one (which remained post-2005), "percent increase in Judiciary Branch coverage and access outside Guatemala City" ­ can be considered as adequately met and it is the result of the collective efforts of many donors and of the government (Organismo Judicial, 2005). Even so, much depends on how one interprets access. The project, and related donor activities, some of them implemented by the same PCU, indisputably augmented physical access to court and related services throughout Guatemala by constructing and equipping new justice of the peace courts, departmental and regional centers, and mediation offices throughout the country, and initiating a program of mobile courtrooms (so far only in Guatemala City and Quetzaltenango). 23 Moreover, the judiciary ensured that these facilities were staffed, and there was a serious, if not fully successful effort, to ensure judges and staff were fluent in local languages and/or were provided with interpreters Still, as almost every knowledgeable observer of the project and the court's broader modernization program has noted, access to courts is not the same as access to justice, and the latter depends both on attacking other barriers (including costs, lack of confidence, and insufficient familiarity with services) and on raising the quality of the services provided. Of the other PDO indicators, the most relevant to the question of quality is the "percent increase in user confidence." This indicator also remained after 2005. The project funded two nation-wide surveys in 1997 and 2003. Like other opinion polls conducted over this period (CEJA, 2003, and 2005), they indicate no significant change in the initial low levels of confidence, although they do show that all citizens, and rural citizens in particular, perceived justice as less costly by 2003 (Aragón y Asociados, 2003). Unfortunately, a final complementary survey scheduled to coincide with the project's end was not conducted.12 Instead two consultants interviewed court staff, lawyers, and community authorities in the central districts of Guatemala City, Quetzaltenango, Huehuetenango, Escuintla, and El Petén. As they note, open ended interviews brought little mention of improvements, but when interviewees were asked specific questions, they did note some advances in such things as infrastructure, number of courtrooms, automation, training of judges and staff, and attention to violence against women. However, in areas like attention to the public, delays, transparency and accountability the great majority saw no improvement or even some worsening of the situation. Another survey done in 2006 (FUNCEDES) and tapping reactions of inhabitants of rural communities, found respondents complaining about a lack of progress on delays, corruption, and attention to users. As neither of the last surveys is directly comparable to those done earlier, the best that can be said is that the results on this indicator are inconclusive. Unfortunately, the third indicator (dropped in the last ISRs) ­ number of corruption related complaints received and resolved through the Anti-corruption Commission ­ is poorly stated. The Commission (which was formed after the project began) does not receive or investigate complaints, but instead operates as a coordination and policy- recommendation body for the various organizations in the justice sector. It was supposed to monitor complaints received by other bodies, and their follow-up, but so far has not taken up this task. Given that the misstatement was probably unavoidable, a substitute indicator ­ complaints heard and resolved by the judiciary's civil service and judicial disciplinary boards can be used. Results of Disciplinary Processes, Staff and Judges (2000-2006)* Year Complaints Not Verbal Written Suspension Recommend Transfer 12 PCU staff explained that this was not done because of the timing of national elections and the likelihood that the results would be prejudiced by the political environment. As the elections were held in September, 2007, the explanation is not entirely convincing ­ surveys could have been done a year earlier. 24 admitted reprimand reprimand firing to other organiz. 2006 Staff 854 431 6 49 159 22 Judges 561 412 1 2 24 1 2005 Staff 1127 650 38 48 153 27 Judges 602 481 3 17 50 10 2004 Staff 832 524 7 46 203 14 Judges 553 404 14 18 27 0 2003 Staff 914 307 165 78 252 45 373 Judges 600 448 0 0 34 9 2002 Staff 694 182 19 71 170 44 346 Judges 529 392 9 18 34 5 2001 Staff 912 307 81 161 134 37 373 Judges 896 696 1 5 24 2 2000 Staff 787 204 27 102 101 22 491 Judges NA NA NA NA NA NA NA Total 6120 2605 343 555 1172 211 1583 *Own elaboration with data from Novoa (2006) based on statistics from Human Resource Department and Junta de Disciplina. Note that Juntas de Disciplina change mid-year. Thus the date given is for the end of the period in which each junta operates. Interpretation of the figures is difficult and the PAD gives no indication of what was intended. The fact that there is a disciplinary system and that it is being used is important, but how well it is working and with what results on performance are impossible to evaluate. In interviews we listened to many complaints about the fairness and accuracy of the system, and it also is apparent (as demonstrated by the number of admitted complaints, especially against judges, with no further processing) that there are many logistical problems especially as regards complainants, defendants, and witnesses located outside of the capital city. The fourth indicator (also dropped in mid-2005) ­ policy on traditional justice systems developed and tested ­ shows no concrete progress, and is also an area where the project's activities were minimal. A key study, on the "particular needs of rural and urban populations" was only finished in 2006, and moreover did not assess traditional mechanisms in any but the most superficial fashion (FUNCEDES, 2006 which only lists "nonjudicial authorities," without explaining how they operate, and briefly discusses the conflicts usually referred to each and any interactions with the formal system) for the 33 communities it surveys. It does not directly address the question of how to incorporate them. The PCU's answer to questions about advances here was to emphasize the role of justices of the peace, of alternative dispute resolution, and of community courts. However, the development of a policy arguably was outside the areas the project could realistically tackle ­ depending on political authorities and political discussions at other 25 levels.13 Thus, while the project made little or no progress here, the fault largely lies with an unrealistic setting of goals. As regards the fifth indicator (dropped mid-2005) ­ number of subprojects executed through the participation program ­ neither the evaluators nor the PCU staff was able to determine what was intended. In the PAD, the civil society participation program is described as encompassing "subprojects (totaling up to $50,000 for each one) through the promotion of partnerships with donors, non-governmental organizations and other participants, for developing broader public participation in access to justice programs such as free legal assistance for the poor, access to justice for young people and women, analysis of costs of litigation, and other related activities that will promote social reconstruction and trust in post war conflict areas." This suggests the initial intent was the creation of a small grants program, like that introduced in the World Bank Ecuador judicial reform operation. However, the mid-term evaluation equated advances in this indicator with centrally organized workshops on cultural sensitization of judicial operators (of which 14 had been held by October, 2002) and to prevent lynchings (in which 600 community leaders participated) as well as with a second stage program on the latter theme intended to involve mayors and justices of the peace. An UMOJ (2003) publication on citizen participation stresses other, centrally financed and directed activities ­ two programs with school children, various publications, and the mobile courtrooms. In the evaluation team's meetings with the PCU, none of the latter activities was mentioned. In any event, and whatever is officially included here, the indicator, like the previous one, does not measure impacts nor suggest how to interpret the results (See also Novoa, 2006 on this point). Absent further information, advances on this indicator must be considered as unmet. On the basis of a review of the five indicators (or even the two retained in the post-2005 ISRs), a grade of moderately unsatisfactory might be considered as too high. However, we believe the problem is less with the project's advances than with its design. The PDO indicators were poorly selected, poorly explained, and to some extent, never really understood by the counterpart ­ not to mention by the evaluation team. Giving them equal importance in grading the PDO achievement, while technically defensible, thus strikes us as an unacceptable formula for evaluating progress. Instead, we briefly review the three elements of the PDO, their relationship to elements of the four components, and indicators (taken from other sources) of measurable advances. A more detailed discussion of each component and progress on its indicators is included in Annex II. Access Improved, Especially for Marginalized Groups: 13This point is also made in ASIES (2007; 72) in reaction to a new Supreme Court proposal to promote work in this area. Moreover, and in seeming contradiction to constitutional provisions and the official (if unimplemented) government policy, the Court suggested it would go forward with a study on the desirability of recognizing indigenous law. 26 Less discussion is included here, as much of this is covered above. Over the project life, it is indisputable that physical access to courts greatly increased. There are now 365 justices of the peace as opposed to 229 in 1997, and at least one located in every municipality; 70 mediation centers; and two mobile courtrooms with 10 more ready to deploy. This cannot be attributed only to the World Bank, but it directly contributed with the construction of 6 justice of the peace courts in Petén, one of the departments with least services, the remodeling of 11 others, the mobile courtrooms, the construction of 16 mediation centers in the interior and the central office in Guatemala, and the construction of one departmental and two regional centers. While the three justice centers are located in (or near, Escuintla) secondary cities, they are intended to provide more and better access to people there and to encourage the decentralization of system administration to the department and regional level, thereby contributing, if indirectly, to improved judicial performance. The project also equipped what it constructed or remodeled, and provided equipment to other facilities, to ensure staff had the resources necessary to do their jobs. Questions about the quality of justice (which also impact access defined more broadly) are addressed in the next sections. Here it should be stressed that the project sponsored other activities to improve access in this broader sense. These include numerous workshops to sensitize judges and staff to the needs of indigenous populations, 14 publication of material in indigenous languages, and the hiring/training of interpreters. We know these activities occurred, but absent any effort to evaluate impact, cannot attempt to assess what it was. Much the same can be said about the proliferation, sponsored by the Bank, other donors, and the judiciary itself, of courtrooms and judges. Statistics on ADR services exist, and except in more populated areas, they appear underutilized, suggesting the need for more work on user needs and how to meet them more adequately, the reasons for non-use and follow-up to determine whether agreements are respected by the parties.15 Despite not entirely overwhelming workloads, those in urban centers are beginning to complain about congestion, indicating the need for attention to their own management systems and prioritization of cases. The judiciary has made some effort to employ more staff and judges fluent in indigenous languages (although predominantly in the mediation centers, not always as part of a systematic plan, and sometimes still not trusted by local inhabitants). The extent of the WB contribution to this policy is unclear. As of April 2006 (ASIES, 2006; 69) there were 749 judicial personnel fluent in indigenous languages, but of these only 56 were judges, and 33 interpreters. (There were 10 additional interpreters but their other language was not an indigenous one). A further problem is that the locations where 14We are tempted to say "countless" as the list provided by UMOJ defies quantification, but adequately demonstrates that considerable training was done. 15Statistics supplied by the central ADR office for several months in 2007 indicate that the 70 centers process an average of about 10 cases monthly, although some do as many as 30 mediations. The center keeps fairly good statistics and does some basic analysis ­ however, it seems to have not yet developed a policy to respond to the apparently insufficient demand (even in the busiest centers). 27 bilingual judges and support staff are most needed are also the least desirable for those hoping to advance in their careers. Thus, there is a reported frequent rotation of individuals initially assigned there, and in any event we received no information on where these bilingual personnel were located. Finally, while sensitization courses and multi-lingual publications exist, their utility in resolving the problem deserves questioning ­ the courses because training, absent incentives to apply it, is notoriously ineffectual, and the publications because of the high levels of illiteracy especially among indigenous groups. Nonetheless, project efforts served to keep the theme alive whatever their direct impacts on fixing the problem. Access or the potential for access has clearly improved over the last 10 years, but there is still a long way to go. The World Bank project has played a part in the process, but without more research, it is impossible to say 1) how much and 2) whether it could have done more. We believe the answer to the second, and intrinsically more important question is probably "yes" and that the failure to work on items related to the indigenous issues is one indication. Still further progress on the access variable could have been facilitated by advances on the other two, both of which involve the quality of justice delivered and thus affect citizens' inclination to use the services now more physically accessible. Efficacy of Judicial Services Improved The project, as interpreted by UMOJ and the Judicial Branch, addressed this issue through a focus on improving the structure, resources, and operating procedures of three main parts of the judicial organization: · The system-wide administrative apparatus, whose more efficient workings would provide better support to the courts and rationalize resource use · What the judiciary considers "auxiliary offices," units responsible for oversight, discipline and training of judges and for helping them process their cases through the provision of services like intake, notification, and temporary archiving of case files · The judicial offices themselves and the personnel working in them (both judges and support staff) While indicators could be developed to assess improvements in each one, they were not introduced (and for the first two subsystems pose certain conceptual difficulties). The indicators most commonly used for overall institutional efficacy (and for judicial offices alone), delay and clearance and congestion rates, were not tracked either, although as discussed in the final part of this section, what evidence exists does not suggest great advances. However, such changes take time, and depend on a good strategy. Thus, before reviewing overall results, activities and progress in each of the three types of strengthening are examined. Attention to administrative and auxiliary offices came first, beginning as a restructuring effort even before the project went into effect. Most changes were effected in 1999, 28 although there were further readjustments as documented in Novoa (2006). Some of these seem to be purely arbitrary and not all were improvements. A major problem not resolved by any of the restructurings, was the placement of the auxiliary offices (which in all versions continued to report independently to the President of the Court) and coordination of their activities (Louza, 2007). A second stage began in 2000, combining the efforts of UNDP (through its PROFED and financing from a variety of donors) and the WB/UMOJ. This emphasized reengineering of internal processes, the development of manuals, and the equipping and some remodeling of the restructured agencies. At that point the process, except for the design of a series of software programs for several of the administrative agencies, stopped, and therein lies the major flaw. From then on, most offices either stayed as they were or developed on their own, with no more technical assistance, and little communication with the others. ICR consultants also observed (Moore, 2007) that manuals were not used, had not been updated, and appeared in several cases, not to exist. The results are a rationalized structure, but one in which the internal operations of the parts and their ability to coordinate with each other are still inadequate. Some offices (Servicios Tribunalicios) have advanced because of assistance from other donors (most notably USAID for the Gestión Penal section). Others advanced on their own, but slowly and without benefit of national or international technical assistance that might have accelerated the process. There are evident problems, which the members of the agencies noted to us, but seemingly have no power to overcome. For example: · The statistics office has moved back and forth between Planning and CENADOJ (an automated document service). It arguably belongs with the former which should be a primary user of its output and which might be better able to insist on the provision of adequate statistics from individual courtrooms and to force improvements in a very rudimentary system with such unreliable results that CENADOJ has so far only published those for criminal justice corresponding to 2002 (Louza, 2007; Moore, 2007). Instead Planning tries to collect its own statistics on workflow (used to help decide where additional judges and auxiliary staff are needed), but lacking technical assistance, tends to organize them chaotically ­ much double or triple recording of the same action, and a failure to distinguish between those initiated by the parties and by the judge. · The Departamento de Supervisión de Tribunales, which contrary to some project reports, was not created by the reorganization, although now linked to the Judicial Career Council, is physically located in another building. Given its responsibility for investigating complaints against judges, it would make more sense to place it with the latter. Moreover, it, like the office of judicial evaluation (also part of the Council) lacks sufficient vehicles to do its investigative and preventive (normal supervisions) work. · Supervisión and the Junta de Disciplina (another part of the Council, responsible for reviewing complaints about judges and making recommendations on disciplinary actions) should both be decentralized The former now has a branch office in Quetzaltenango, but it is already overburdened, and the Junta's efficacy 29 in adjudicating cases is greatly hindered by its inability (for practical reasons) to operate outside Guatemala City. · The Judicial School, which is also part of the auxiliary offices, seems to do fairly well with the orientation course, but its in-service training are not systematic, and there are numerous complaints about its requiring judges, on very short notice, to attend courses they have already taken, repeatedly.16 Obviously, a better record system kept by the Council with judicial profiles and a real plan for in-service training would help, but so far no one has promoted that. · Human Resources now has a database on civil service employees (those in administrative and auxiliary offices, as well as courtroom staff) but it serves largely to track attendance and calculate pay. There is still no evaluation program, and no expanded database that would allow the tracking of any sort of career path. Moreover, neither it, nor the Council, has done anything to systematize the situation of courtroom support staff (auxiliares judiciales), a group seeming to fall between the two stools, officially civil service, but actually aspirants to judgeships. · Although the judiciary installed an IFMS, it was the system developed with Bank funds for the executive branch (SIAF). The project took no part in its installation, nor in training in its use, and it is still not linked to the other administrative software (some of it developed on very rudimentary databases like Access) funded by the project. The Director of Financial Administration has requested assistance in improving the system, but so far has received no response from the PCU (Moore, 2007). . There is no doubt that the restructuring and reengineering represent improvements in what was a wholly chaotic prior situation (Louza, 2007). However, the few measures of impacts offered by UMOJ and its external consultant (Novoa 2006) are fairly peripheral ­ documented decreases in times to register lawyers and notaries or to receive certificates of no criminal record, and the increases in fees collected by both services. Even the claim (Novoa 2006) that administrative costs have been reduced seems doubtful, given that total administrative staff increased by 78 percent between 2002 and 2006 (ASIES, 2006; 64). The restructuring was positive; it just ended too soon. Lack of counterpart interest may be one explanation, but it also bears mention that the project design only demanded that it be planned, applied, and "tested." The decentralized administrative centers, while potentially another source of improvements, are for the most part not yet functional (Moore, 2007), and the same questions could be raised as to their further, effective development. In summary, the real improvements in performance achieved by 16 As an anecdotal illustration, the course mentioned most frequently was "intellectual property." Someone explained to us that the course was sponsored by the American Chamber of Commerce because of its interest in influencing judicial thinking on this theme. (We were also told that banks financed courses on debt collection to sway judges toward accepting credit card debt as "títulos ejecutivos" thus allowing it to be processed through the much faster juicio ejecutivo.) Because the "invitations" to the course are backed by a blanket decree signed by the Chief Justice, nonattendance constitutes a "falta grave" (refusal to follow an order from the Justice). 30 the administrative and auxiliary restructuring could get a moderately unsatisfactory to moderately satisfactory, but only on the basis of a comparison with the status quo ante and a qualitative assessment of their potential future impacts. It is also disturbing that the long-time PCU "subcoordinator" contends that the judiciary's ordinary administrative offices are still incapable of managing this type of operation. Attention to the third element, strengthening actual judicial performance came later and ended even more abruptly. Slightly later attention is not entirely illogical, assuming the work in the other two areas lays a crucial base, for example in establishing a system for monitoring judicial performance. Unfortunately, this did not happen. As noted, an early (2001) effort by PROFED to improve case processing in pilot family courts was not picked up by the project, despite formal cooperation between the two donors. In 2002 and 2003, there was an interesting and promising effort, largely conducted through one local consultant, to help courts reduce internal processing times ­ by rationalizing internal workflow and eliminating steps not required by law. Unfortunately, the process ended with the change of courts ­ for reasons which we again do not understand (new priorities, failure of the WB and the UMOJ to lobby with the newcomers?). Moreover, the failure to develop a basic set of valid statistics on court caseloads and some further attempt to measure processing times means that impact would have been difficult to determine. The project only addressed the latter at mid term (2004) and due to problems with procurement, the study was only completed at the end of 2006. Moreover, because of its design (the TOR provided by UMOJ) it did not provide adequate information either on delays or their causes.17 The contractor never produced a satisfactory delay reduction program, but it is doubtful it could have done so using the information it was asked to collect or in the time allowed. Some of the activities conducted with the auxiliary offices (centralization of notification for the Guatemala City courts, the work of the Judicial Council's Supervision Department, Evaluation Unit, and Disciplinary Board) might have had some impact on reducing delays and raising productivity, but were hampered by the absence of good statistics and adequate methodologies. The project could have worked on both, but did not. Higher salaries (introduced independently of the project, but equated by UMOJ with an improved incentive system) and the stricter entry standards introduced by the judicial 17 The inventory covered only cases entered after 2002 in five cabeceras de distrito (courts in the principal city of each district). Thus, it did not include all active cases, even in those courts, and moreover was an inadequate basis for calculating delays inasmuch as the starting date was too recent. Not surprisingly (in fact logically) the results indicated that delays in reaching a firm judgment averaged under three years for all types of cases. Neither the PCU nor the contractor seemed to notice that most cases had not been resolved, thus suggesting either 1) that cases not resolved within three years would never be resolved or 2) that most cases took more time to reach a firm judgment. It bears mentioning that by 2005 LCSPS had conducted studies using casefile analysis to explore delays and related problems in five other countries of the region. The first of these, done in Mexico (World Bank, 2002), faced similar problems and resolved them with a second sample of adjudicated cases to determine times to resolution. Unfortunately, the research team, whose members could have provided advice, was not contacted as the Guatemala study was being designed, nor, so far as we could determine, was the Mexico report made available to the Borrower. 31 career law may also have produced improvements in efficacy and efficiency, but if that happened, little credit goes to the project, and in any event, it cannot be measured. In short, although several items under component A (institutional strengthening) and one under component B (Transparency as it involves incentives) might have impacted efficiency and efficacy, these received short shrift from the project. Both efforts and progress here are unsatisfactory. Two additional activities promoted by the project, the automated case management system and ADR, might improve efficacy, but neither has advanced far enough to do so. In theory, and apart from its access benefits, widespread use of ADR should reduce court congestion and thus delay by diverting conflicts to this alternative forum. Unfortunately, even in the most utilized centers, the amount of demand attended is still too low to have this effect. Moreover, there is considerable debate among the experts as to whether ADR really reduces court congestion or simply attracts cases that would not have gone to court. Nonetheless, the innovation appears popular with users, and the FUNCEDE (2006) study indicated considerable demand for more mediation centers among inhabitants of small municipalities and rural areas. As for the new case management system, the team specialist evaluated it as of good quality, with certain functions not available in similar systems developed elsewhere, but also with problems, still being addressed, as regards ease of use (Louza, 2007). Although UMOJ claims that efficacy and efficiency will be greatly improved with its nation-wide application, this will easily take five years and will require a considerable expansion in the Judiciary's Informatics and Telecommunications Center (CIT) staff now assigned to the task, as well as further technical assistance to help with modifications to the programs and to ensure the source codes are adequately protected against manipulation. Its impact will also depend on quality control of data entry; adequate statistical analysis, and the conversion of the results into new policies. For that to happen, internal analytic capacity will have to be created (preferably in SPDI, not CIT ­ analysis is best not left to software engineers), and judicial leadership will have to be convinced it is important. Several judiciaries in the region have already adopted similar software with limited impact on efficacy and efficiency because of the absence of these other elements. Use of statistics to enhance judicial management is clearly an emerging global trend, but in Latin America there are few countries (Chile, possibly parts of Argentina and Brazil, Costa Rica but with some limitations) that have moved to adopt it. As many more have case management software, its presence is clearly only a facilitating factor, not a solution. In the end, neither the judiciary's own statistics, nor those offered by the UMOJ study (Novoa, 2006) or external observers (ASIES, 2006) suggest an increase in the productivity of the courts. Production has risen, but largely because of the addition of more judges. Demand has grown still more rapidly. ASIES (2006:65) estimates a case clearance or disposition rate (cases resolved over those entering in one year) of 10 percent for the entire period. This is extremely low, especially considering the apparently light average workload. Novoa (2006; 119) finds an increase in congestion (cases resolved over total active cases) from 2000 to 2004; in light of the bad statistics on backlog and the likelihood it is far underestimated, the real rate may be still higher. 32 However, part of the problem may arise in legal impediments to judges closing inactive cases ex oficio; this means that absent the necessary request from the parties, much of the backlog may be composed of abandoned cases. The inventory and sample might have explored this, but did not. There is no good analysis on average delays. The case inventory (NCSC, 2006) found that most cases in most materials had not reached resolution within the three-year period covered. The FUNCEDE (2006) study, which only asked those interviewed for their perceptions on delays, found most considered them excessive, although predominately within the one to three year range, and reported no recent improvement in reducing them. The interviews did not ask about resolution rates (what percentage of cases were resolved). Credibility of Judiciary Improved Most of the relevant activities are in component two (Transparency), but some of those in component one (as regards delay), component three (especially sensitization of judges and development of a plan for recognizing traditional dispute resolution), and component four (those involving communication polices) are also relevant. Here the focus is on components 2 and 4, in part because they are not treated above, and in part because they together seem to get at the two sides of the problem ­ the lack of credibility because of real corruption and because of the judiciary's failure to "make its case" as regards efforts to improve the situation. Going by the polls (including those conducted by other agencies, CEJA 2003 and 2005), the combined effect of the two strategies has yet to be felt. The 2003 and 2007 surveys did note some improvements as regards: respondents' familiarity with the judicial modernization program, their estimation of cost as a barrier, and in the 2007 survey, in training of judges and staff. However, perceptions of corruption, poor attention to users, and delay remained fairly constant. The question then is whether the project selected the most appropriate path for making a difference ­ the connections between the PDO and the activities emphasized ­ and furthermore, whether implementation followed the plan. Again, the absence of statistics for monitoring performance is critical, not as a measure of impact but as a means of achieving it. Knowing you are being watched may be the most fundamental incentive for doing a better job, more important than raises, ethics and self- esteem courses, or even skills training. Statistics would also have helped the Department of Supervision, the Evaluation Unit, the Junta de Disciplina, and Human Resources evaluate performance, identify problems, and propose and monitor improvements. The initial creation and/or restructuring of these offices, the early technical assistance and equipment provided to them, and programs to inform system users and operators of their functions were also important, but more work in all these areas was needed, and for the most part was not done. The evaluation program for judges continues to receive criticism as does the rather truncated career it oversees ­ composed of only two levels, justices of the peace and judges of first instance. However, administrative and courtroom support staff still lack evaluation programs, another flaw that needs to be addressed. In short, while the project did take steps in promoting the creation of a human resource management program, it stopped early and far short of what was needed. In this light, the 33 many complaints of those interviewed in the FUNCEDES study about court personnel not seeming motivated to do their work well are easy to explain. As for the core issue in credibility, corruption, the statistics from the two disciplinary units indicate several problems: · The large number of complaints dismissed without a hearing suggest either that users do not understand what constitutes abuse or that the units simply are not working. · The small number of disciplinary actions taken is indicative of other problems: inadequate investigation (for lack of skills, incentives, or resources); the logistical obstacles posed for both defendants and complainants who do not reside in the capital city; complainants' fears of retaliation; or certain procedural obstacles (flawed regulatory and legal framework). · Finally, it bears noting that the most the disciplinary units can do is recommend dismissal of those they find to have committed major infractions. Their recommendations are not binding and it is up to the Supreme Court or the Chief Justice to take the final decision. They do not always follow the recommendation. · The same is true of the evaluation unit's recommendations as to judicial appointments and terminations, and in the latter case, those with low evaluations have been relatively successful in filing an "amparo"18 against the unit's findings. Obviously, many of the obstacles here, because of their political and legal nature, are beyond what the project could have overcome. However, it is significant that it was the Transparency or Anti-corruption component where expenditures were farthest below (less than 10 percent) what was budgeted in the PAD. 3.3 Efficiency No economic analysis was done for the project and the financial analysis emphasized only the estimated total costs ($165 million) of the judiciary's modernization plan, the availability of other donor funding to complement the WB contribution, and the steps taken to ensure adequate administration of project funds. As is usual in these projects, the economic section refers to the positive economic impacts of reliable judicial enforcement, reduced corruption, and improved crime control. However, given the impossibility of determining the project's impact on any of these variables, an ex-post economic analysis is not feasible. Annex 3 contests the PAD's contention that this type of analysis could not be done, and in fact argues for conducting one, even absent the necessary statistical inputs. As has been suggested in the text above, the ICR team has some doubts about the cost-effectiveness of the activities emphasized, but again this is a matter for further debate and largely in the abstract. Team members were not alone in expressing doubts. A former Chief Justice from Sweden, did a short evaluation of the project in June, 2007 and 18An amparo is the standard constitutional protest of a violation of basic rights. 34 likewise questioned the longer-term value of certain activities with a large "public relations" component (Svensson, 2007). 3.4 Justification of Overall Outcome Rating Rating: Moderately unsatisfactory We are rating the outcome moderately unsatisfactory not against what the project proposed to do (not achieved) but rather against what it really did. The project does not get a satisfactory or moderately satisfactory because we believe the Borrower and/or the Bank team could have overcome or circumvented some design flaws, by paying attention to the mid-term recommendations, placing more emphasis on real improvements in performance, instead of attempting to spend all the assigned funds with equipment purchases in the final year (and here there are also concerns as to whether these purchases coincide with the requirements of the case management system now to be installed). The project could have promoted other elements (like the abortive efforts to accelerate case processing, a better inventory and delay reduction program, or the enormous amount of work needed in human resource management). However, it did the constructions planned (or most of them ­ in the end one departmental center was omitted), provided essential equipment, promoted, on its own and with PROFED, some major rationalization of internal structures, and experimented with important (and so far as could be determined, unique) exercises in simplifying court procedures. All of this provides a base for further advances, and its importance should not be discounted simply because the project design was too ambitious in terms of promises. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development Except for some workshops and publications on gender, the project had no special activities directed at these themes. However, it's overall emphasis on extending access to marginalized groups and especially to indigenous populations was sufficient to cover them, and adding still more activities to an already over-ambitious design would not have been advisable. Whether enough was done to further the larger goal can be debated, but the underlying justification is sound ­ increasing access to court and alternative services for the poor, indigenous populations would provide them with the means to resolve their problems and defend their rights. Women were not selected out as a targeted subcategory for special treatment, but the project did keep track of gender and indigenous participation in all of its activities ­ courses, workshops, and ADR services. It does not appear that analysis of these data produced any readjustments to activities. This is a serious shortcoming given that Guatemala has been ranked extremely low in protection of women and very high in violence against women. (Prensa Libre, October 9, 2007) However, our own quick review of the data did not reveal anything that demanded immediate attention. (b) Institutional Change/Strengthening 35 As this was an institutional development project, these themes have already been dealt with exhaustively. As repeatedly noted, WB and other donor efforts effected considerable advances in imposing a more rational organization for the administrative and auxiliary structures. However, more work is required to improve coordination of the parts and the functions carried out by each of them. The PCU erred in not transferring more programming and implementation to the judiciary itself. This should be an aim of any future operation ­ with a goal of phasing out or eliminating the PCU over the next five years. While other donors have focused more attention on improving the activities of individual units within the larger organization, the WB and UNDP took the lead in the 2000 macro-reengineering which might not have occurred without their combined efforts. (c) Other Unintended Outcomes and Impacts (positive or negative) The project, less by its design than how it evolved, seems to have encouraged the judges' interest in buildings and equipment, and strengthened their initial inclination to equate these with improved performance. This is unfortunate and may pose problems for the design and negotiation of the next project. If anything the project amply demonstrates that buildings and equipment are not magic bullets, and that while the public recognizes their presence, there existence is not necessarily directly correlated with overall improvements in performance. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Time constraints and limited financing for this ICR did not allow a beneficiary survey or stakeholder workshops. However, as noted above, a survey sponsored by UMOJ and conducted after the project had closed (Novoa and Tacao, 2007) did indicate that some stakeholders, when presented with direct questions, did perceive a few improvements in certain aspects of court performance. The evaluation team reached the same conclusions in its interviews, but could not attribute the results to the project in particular. This is because there were many other donor projects and additional changes on-going at the same time. For example as reported in Novoa and Tacao, knowledgeable users did not note any improvement in delays and congestion, but did mention a better quality of judges because of the introduction of the career law, improved systems for selecting and evaluating lower level judges (justices of the peace, first instance and trial judges), better infrastructure and equipment, more first instance courts, and such innovations as the mobile courts, ADR, and automation of Antecedentes Penales and Protocolos. While the project cannot take credit for the career law, it did help set up the recruitment and evaluation systems. The earlier, 2003 survey also identified two positive changes: greater user knowledge of the judicial structure and its modernization plan, and a marked reduction (20 percentage points) in the belief that justice was costly. More information on the survey results is given in Annex 6. 4. Assessment of Risk to Development Outcome Rating: Moderate The project, along with activities undertaken by other donors and by the government, has set a base for future advances in the PDO. The judiciary is now better structured and 36 better equipped to produce the outcomes the project promised and has made some advances in achieving them. There are still problems in the administrative structure and its internal operations, in the legal framework regulating it and especially the aspects regarding human resources (both judges and administrative staff), and in the judiciary's overall ability to monitor its own performance and that of its individual members. These are all areas where further emphasis is critical to avoid backsliding or, more likely, preservation of the current glass-half-full status. Citizen interest in achievement of the goals stated in the PDO has not weakened. The fact that their dissatisfaction with judicial performance has not diminished, while perhaps underestimating the progress made, is thus a positive factor in mitigating risks to the development outcome. It will, however, be important that future Bank work and that of other donors now concentrate on using the base to push forward with the underlying objectives, finetuning where necessary, but focusing specifically on producing visible improvements to the quality of justice delivered to the citizenry. The project itself has introduced an additional risk by encouraging judicial personnel to define improvements in terms of infrastructure and equipment; while outsiders do recognize these "improvements," they clearly are looking for something more and this will have to be stressed in any new operation. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately unsatisfactory. Project preparation was exemplary and deserves to be replicated in other such efforts. Although the ICR team did not review in detail how the time was used, the three or so years devoted to preparing the ground appeared to have been well spent. The final design was still an impediment to successful implementation, but it seems terribly unfair to judge a project designed in the late 1990s against criteria in effect ten years later (which would have lowered the grade). However, even assessed against the standards of the time, there was room for improvements, building on, but not limiting itself to the results of the participatory events.19 Presentation of components and activities is overly vague and somewhat confusing; for the most part the component definition is simply a list of activities to be supported. An emphasis on inputs rather than outcomes and impacts and a failure to specify definitive causal links between activities and overall objectives probably contributed to many problems (lack of follow-up on key activities, delays in implementation of some, failure to take any actions in others). The PAD might have emphasized the predictable change of topmost authorities every five years, and with them many administrative directors. In the section on risks, there is only a mention of the moderate chances of the new Supreme Court's lower commitment to reform, with the suggestion that this would be resolved through discussions during the mid-term review, on-going workshops with stakeholders, and monitoring and evaluation 19One assumes, after all, that this is the Bank's role as a "knowledge bank," helping countries improve their proposals for resolving problems their stakeholders identify. 37 (an activity that remained underdeveloped). As the Court changed in 1999, a mid-term review in 2002 would be a little late to make corrections. (b) Quality of Supervision Rating: Moderately unsatisfactory. The mid-term supervision was excellent, as were earlier BB and TF-financed evaluations of some components. Unfortunately few of the recommendations provided therein were put into practice for reasons the ICR team does not understand. It clearly would have been helpful to continue with and use this type of external assessment; a lesson to be adopted in future projects. Although the initial two funding reallocations responded to counterpart requests, the movement of so much money into the goods category, absent efforts to ensure all components were receiving adequate attention, seems questionable. This it should be noted is a problem not related to the TT alone, but also involved higher WB management. The first reallocation in particular should have provided an opportunity to push for advances in the neglected areas and especially those highlighted in the mid-term supervision. The lesson here is that any such action requires far more attention from management and not just from the task team. The project did a good job on attracting positive public attention, in Guatemala and in the Bank ­ for example, the communications component and the inaugurations of the various construction projects, as well as activities like the initial two mobile courts, two programs with school children ("A Day with Justice," and "Judge for a Day"), and a Market Place prize winning activity featuring workshops to raise awareness about new justice of the peace courts in 36 rural municipalities. Again, these are models other projects might follow as they clearly facilitate the way for the harder actions. However, as regards the latter, there are some concerns. Interviews with Bank staff in Guatemala indicated they had expressed concern to the team as to advances in PDO achievement (i.e. concrete results in the quality of services) but received no satisfactory answer. Public relations, in short are an often overlooked responsibility of task teams, but they cannot replace advances in more substantive areas. If most projects err in neglecting the former, this one may have counted too much on their power to move other areas. Project reporting was deficient. ISRs were pretty much boilerplate (often repeating the same language in 2 or 3 periods) and rarely signaled emerging issues and problems. They never discussed major changes, like the reallocations, and introduced other modifications (reduction in indicators tracked, elimination of "co-financing" counterpart) with no explanation whatsoever. Ratings never deviated from S (Satisfactory), and although the mid-term supervision was mentioned, there was no discussion of its recommendations or a plan to comply with them. ISRs also sometimes gave credit to the project for things clearly done by other donors ­ public defense mentioned in ISR 10 where the project did some training, but most of the work was done by MINUGUA, UNDP, and USAID ­ or announced progress in areas (a case recording system) which simply overstates what was actually done. Although the Bank project team was entirely located in Washington, it seemed reluctant to involve local staff in supervision, at one point insisting that all procurement documents 38 be sent first to the purview of the project team before being forwarded to the local procurement office (interviews Guatemala, September, 2007) For a project and country of this complexity, this policy seems highly unwise. It also bears noting that there were at least two specialists in LCSPS with substantial knowledge of Guatemala and of regional judicial reform problems, and that neither one was asked to provide any support to the project. Instead, for internal support, the project relied largely on very junior trust- fund and other short-term consultants. Finally, although there were no major problems to be addressed in financial management and procurements, Bank staff did not react soon enough to the undesirable expenditures patterns. The disproportionate amount of funds spent on the PCU, the lower amounts spent on the rest of component D and component B, and the lag in spending on consultant services might have been identified earlier, allowing more effective attention to the neglected components and eliminating the need for a second transfer of a large portion of funds to equipment purchases in the final year. Again, responsibility here cannot be placed solely on the TT; management should have recognized the problems and requested further discussions. (c) Justification of Rating for Overall Bank Performance Rating: Moderately unsatisfactory For the reasons stated above. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory The Government's most important contribution to project advances was the enactment of critical legislation and constitutional reforms ­ the Judicial Council and Career law, a law covering civil service judicial employees ­ and increases in the court budget and salaries. Without these changes, many activities contemplated in the PAD would have been difficult if not impossible to implement. Nonetheless, further commitment to the project goals, either from the Government as a whole or from the judiciary, was never visible. The tendency towards an over optimistic interpretation of many of the advances made (not so much in the WB project as in the overall judicial reform) is widely commented within Guatemala. Despite the above steps have yet to be taken either by the Government or the judiciary to make the necessary corrections. ­ for a start, in improving the administrative problems, creating a career for administrative employees, creating a performance monitoring (statistics) system, or improving the evaluation and complaints processing mechanisms. The WB could have pushed, but this would have been easier if the Borrower had expressed interest. It also bear noting that certain aspects of the 1986 Constitution as regards selection and term lengths of the Court and appellate bench did pose impediments to creating strong judicial leadership. The periodic full renovation of both, the continued politicization of the appointment process, the annual election of the Court president, and the latter's ability to place and remove many administrative officers at will obstructed continuity in judicial policy making and implementation. Procedures for supervision and evaluation of judges, their support staff, and administrative personnel are underdeveloped (there are 39 none for the latter two categories) and remain highly controversial. Initial support to the responsible units should have been continued as these are areas where technical assistance would have been extremely useful. According to several observers (Moore, 2007), the present Court divided responsibility for the departments among the 13 justices, a practice justified as increasing attention to each, but also reputedly augmenting the incidence of patronage in appointments, promotions, and transfers of judicial and administrative staff. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory On the plus side, the creation of a strong PCU and considerable continuity in its staffing (including the retention of the same "subcoordinator" until early 2007 ­ when she was appointed to the position of General Manager for the Judiciary) were important factors in maintaining and eventually accelerating project implementation. The decision to have the PCU implement judicial portions of the IADB sectoral project and to coordinate with other donors in the construction and placement of infrastructure was also critical. The PCU's financial management and handling of procurement was validated by a series of Bank audits and supervisions. Nonetheless, the project is ending with two major contract disputes, to which both inadequate Bank supervision and PCU failure to push for a solution (as opposed to simply berating the contractor for poor performance), have contributed. On the minus side, the PCU extended its role beyond implementation to virtual ownership of the project. The current coordinator, a Magistrate assigned to oversee implementation, described his role as more of an administrative role instead of one requiring a more direct involvement in policy decisions. One consultant working on a high priority activity reported he had never been able to meet with the Court presidents and that his meetings with the "coordinator" were perfunctory at best. (However, see Moore, 2007 for a more positive view.) The coordinator forms part of a judicial commission (composed of four magistrates, the judiciary's general manager, and the subcoordinator) which meets monthly to review project progress, but again gives no indication of a more pro-active role. (The CMU did report occasional visits by committee members or other justices to lobby for the project, but that is different from pro-active management.) In its interviews with administrative and judicial staff, the ICR team also repeatedly heard references to the PCU's "disclosure policy" and many who did not complain simply noted that their contact with the project had stopped after the early period (reorganization, restructuring, and the delivery of equipment and training.) Only a few units ­ CENADOJ, the Judicial School, and the CIT -- expressed positive views about continuous support from the PCU. Their perceptions appear accurate, but in the case of the first two, this support may have confused rather than clarified their roles and programs. CENADOJ appears to have taken over some functions belonging to the Communications Office, while the multitude of short courses co-sponsored by the PCU did not encourage the School to develop a consistent program of in-service training. In many cases, short courses and workshops were spur of the moment interventions, seemingly unconnected. 40 The current contract disputes are not the only ones in which the PCU has been involved. At least one contract was terminated after initial implementation, with the contractor charging inadequate support from the PCU and the PCU charging that the contractor was not producing anything.20 Several contractors complained about similar lack of support as well as about ambiguous terms of reference and the PCU's post-hoc, unilateral interpretations as to what they meant. As regards the relationship with UNOPS, which appears to have worsened since 2004,21 the PCU claims UNOPS changed its TORs needlessly, while UNOPS holds that the TORS submitted by the PCU were incomplete, inconsistent, and of poor quality. The ICR team had no time (nor the necessary expertise) to investigate the competing claims. However, a team member did examine the TORS for one contract still under dispute (the inventory and delay reduction "study"), which was wholly managed by the PCU, and found it very poorly designed (see sections above). The software contract required the production of a thesaurus, an element that disappeared from such activities over fifteen years ago, once open search engines were developed. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately satisfactory Although many of the impediments to government and implementing agency performance in particular originated in a legal framework, political context, and certain design flaws that were beyond their control, borrower performance is rated as moderately satisfactory. The ICR team believes the Bank and the Borrower might have worked together to identify and adopt more satisfactory responses to these obstacles. It is debatable where the initiative should have started, but either one could have taken it and the other could have joined in ­ unfortunately, neither one did so, and thus the grades assigned here split the responsibility. The Court, the official counterpart, seemed quite content to let the PCU makes its decisions for it, and showed no signs of questioning some of its more doubtful actions ­ absorbing work that should have gone to the judiciary's administrative offices, expanding its size to encompass the major share of the fourth project component's budget, and entering into conflicts with contractors where the Court might have examined the issues more closely. Over the entire period, its most decisive actions were to request reallocations of funds to reduce technical services and increase equipment purchases. Arguably the Bank had a responsibility for seeking the Court's more active involvement, but just as one understands why Court members might not want to rock the boat, it is also understandable why the Bank might find engaging an essential passive client more than it wished to take on. 20 This was the first firm contracted for the case management software. The PCU first mentioned the cancellation to the team, but other interviewees who claimed to know the situation, provided the counterarguments. 21 Reports that PCU officials had gone to New York to request the removal of a UNOPS official were confirmed by both UNOPS and the PCU subcoordinator. Since the trips required no objections, the travel to New York, if not its purpose, is also verified by project records. 41 6. Lessons Learned Design and preparation Although so obvious as to hardly bear mention, without counterpart buy-in, a judicial or institutional reform effort will not get far. It is the corollary to this statement that is more important, buy-in means not just accepting the donor contributions, but rather internalizing and actively pursuing the change goals behind them. As a corollary to this corollary, it is also important that the Borrower understand and commit to the overall project goals and PDO and component indicators. In one of our meetings with the former and current PCU subcoordinators (the official actually in charge of the PCU), the ICR team reviewed the indicators with them and it was evident that there was a lack of clarity on several. Participatory design is good, but can lead to a project "designed by a committee." Highly qualified international assistance is also important, especially for countries only starting on the judicial reform route, and thus likely to repeat the errors of the past. Projects composed of a list of activities run the risk of turning implementation into a check-list and moreover of doing the more attractive rather than the most important. An appraisal document needs to outline a strategy, major lines of action, and sequencing of intermediate steps, not as a detailed Gantt Chart, but rather as a logical narrative that can guide implementation. Great caution should be exercised in specifying activities other than those (e.g. creation of a statistical system to track court performance) constituting important benchmarks. Indicators are important and should be chosen carefully on the basis of several criteria: closeness of relationship to the development objectives being tracked; likely availability of data; likelihood of measurable change within the project life; and ease of interpretation. Nearly all the indicators used in this project fail on one or more of those categories. Given the nature of institutional change as a three step process ­ inputs to induce the change, changes in internal behaviors, and finally improvements in outcomes and impacts ­ a five year project may not be able to complete the entire process, and moreover, successful passage through steps one and two can over the short run, negatively affect the third (the most classic example being the increase in reported crime elicited by a better functioning criminal justice system). Project design must thus anticipate both the slow passage from one step to another and the possibility of short-term negative results on the outcome/impact indicators. This may argue for focusing on the intermediate step (behavioral change) or a more sophisticated (but explicit) interpretation of the anticipated changes in the impact variables. Implementation 42 If baseline data and other means of tracking performance and project achievements do not exist when a project goes into effect, they need to be created early on. This is critical to establishing concrete performance targets, focusing implementers on achieving them, establishing a monitoring and evaluation system, and allowing the project to demonstrate advances. A PCU is a double-edged sword and where possible, one project goal should be its eventual phasing out and elimination. It is always tempting for the WB to rely on the PCU in the interest of faster implementation, but from the institutional development standpoint care should be taken that the PCU does not expand its control over the operation and take on activities better transferred to the, admittedly, less efficient normal organizational offices. Given the WB's own operating procedures and the difficulty of constant monitoring of the myriad events that can affect implementation, use of local staff to maintain a permanent dialogue with the target agency (and not just the PCU) is highly recommended. USAID's ability to affect a series of targeted, but highly important changes (introduction of 24-hour centers, juzgados de turno, to conduct the initial hearings for detainees in criminal cases; redesign of the evidence storage facilities; introduction of a new model for first instance criminal courts; transfer of underworked justices of the peace from central Guatemala to the nearby Villanueva; and multi-disciplinary approaches in provincial criminal justice centers) was conditioned by having a full-time, highly experienced contractor on the ground who could convince the judicial authorities to take these steps. If the WB does not have local staff qualified to do this, thought might be given to including an experienced consultant on the supervision team and having that person (or persons) make frequent trips to discuss the content of implementation with the counterparts and other stakeholders. A local NGO or consultant might also be used, although this could lead to other problems in a politically charged environment. Transferring substantial amounts of unused funds to massive equipment purchases, while understandable in light of the WB emphasis on disbursements, is a second or even third best choice. The large unexpended balance precipitating this action was predictable early on, and could have been handled by encouraging attention elsewhere (for example, in providing technical and even material assistance to the various disciplinary and evaluation bodies in the human resources office and Judicial Council, or to following up on the many recommendations from the mid-term evaluation). In the end, a final equipment purchase might still have occurred, but the amount could have been reduced and more progress made in more basic areas of institutional strengthening. Even with joint implementation agencies and formal donor committees, donor coordination remains problematic. This is not simply a question of personal relationships and organizational operating procedures, but also of the models underlying each one's work. Although rarely mentioned, the construction programs managed by the PCU for the WB and IADB involved different models ­ the WB centers were for housing court services; those for the IADB emphasized multi-organizational teams working on criminal justice. A more productive way of identifying and discussing such differences needs to be found. 43 In collegial organizations and especially those with frequent turnovers, organizational leadership is especially problematic, but unless it can be developed, PCU mission creep is likely and difficult areas are likely to be avoided (as even a strong PCU may be reluctant to enter them). "Institutional strengthening" does not end with a new organization chart, equipment, and manuals ­ it must be followed up throughout the project (and provisions for follow-up should be stressed in project design). Evaluation and after Where many actors are involved in promoting changes, attribution of results to any one will be extremely difficult, and may only be identifiable at the input level. From the standpoint of designing better operations, however, who did what may be far less important than what did and did not work to induce fundamental change. Bank records (e.g. what is in IRIS) are a very incomplete source of information on what happened. Unfortunately, the Bank encourages this with its own incentive system ­ reporting anything as unsatisfactory or identifying emerging problems is only likely to cause a TTL grief except in those rare cases where the authorities insist on this. ISRs by their very format do not encourage a good recounting of progress and problems. The ICR team did not find them to be a reliable source of information on past events and suggests three types of changes: a better review by management; some changes to the format to encourage the inclusion of more information; and some changes in the incentive systems to encourage greater realism in reporting. A follow-up operation is desirable, but only if it can focus less on material assistance and more on institutional change. However, this will be difficult to negotiate and design. Ideally, the design process should be participatory, but something short of the mass meetings used for the first project. Emphasis should be on the quality of debate and attention to real problems of the type identified here and in other treatments (some listed in Annex 9) of Guatemala's justice system. If a PCU is used in a follow-on project, its gradual elimination should be part of the plan. Amounts spent on "PCU strengthening" should be controlled, and in no case should the PCU be used to implement activities corresponding to existing judicial agencies. Guatemala now faces, and any follow-up project should address a dilemma not uncommon in the region (or for that matter in many more developed countries). This is the implicit trade-off between increased efficiency and increased access in situations where a significant portion of the population is widely dispersed in isolated rural areas. Doubling or tripling the number of judges and courtrooms (or adding substantially more mediation centers), as several of those interviewed suggested, would increase overall inefficiency and raise the courts' share of the national budget above what appears to be a reasonable 2 percent. Moreover, it would require similar measures for police, prosecutors, and public defenders, although of all four groups only the police (with 150 44 police per 100,000 inhabitants, as opposed to the recommended 300) appears radically understaffed. Further redistribution of existing personnel might help, but in the end, other alternatives will have to be explored. Aside from efficiency considerations, having a large number of underworked personnel in areas where they cannot be adequately supervised is an invitation to other types of problems. As demonstrated by the arguments about doubling or tripling the number of judges, it appears that local skills in reform planning and exposure to lessons from international experience are weak and need to be reinforced. Budgetary implications and constraints require more attention; implicit notions about "normal" workloads are poorly developed; legal obstacles are taken as givens (rather than something that can be changed); and there is an excessive faith in silver bullets (automation, "oralization" of procedures, ethics codes, and training). This is an area where the WB has and should exercise a comparative advantage as regards more sophisticated analysis and program design, not to impose solutions, but to help counterparts make their choices intelligently. Contracts with third party implementers (e.g. UNDP and UNOPS) should be written with great care to ensure the lines of responsibility are clearly stated. If one supervises and the other approves, then this should be stated in all contracts so as not to replicate the current catch-22. The same consideration applies to contracts providing technical or other assistance to judicial units other than the PCU (a preferred arrangement). The PCU may sign and supervise the contract, but the latter should stipulate the role of the beneficiary agency in evaluating deliverables. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies See Annex 7 (b) Cofinanciers Technically, there were no co-financers. Monies from the UNDP, the IADB and various bilateral development agencies were initially calculated as part of the counterpart for the WB operation, but they were never tracked in the audits, and as of 2005, any attempt to do so was halted. While their use was in theory guided by the sector and judicial modernization plans, donor funds were separately programmed and negotiated with the Court and other sector agencies. Although the PCU directly managed $12.95 million of IADB funds and coordinated with two other donors providing infrastructure, the only true example of co-financing occurred during the 1999-2001 reengineering when the WB project and UNDP's PROFED (which managed funds from a variety of donors) cooperated in implementing the effort. Despite the modernization plans, the creation of a donor coordination committee, UMOJ's more direct role vis-à-vis four donors (including the WB), and the joint reengineering exercise, coordination among donors/lenders/and other stakeholders appeared problematic. This was not new ­ see Hendrix (2000) and Carothers (2001) on the conflicts between USAID and MINUGUA over the former's "justice centers" and the proposals of each for reengineering the Public Ministry and police. In interviews with other donors, references to these conflicts were sometimes 45 made, but in general they were neutral on the WB project (professional courtesy?) and offered no comments for attribution. (c) Other partners and stakeholders Interviews were held with three local NGOs (ASIES, the Comunidad Jurídica del Occidente de la República de Guatemala, and the Observatorio Judicial) and documents produced by them (ASIES, 2004 and 2006; Juárez, 2007; Ramírez n.d. and 2007) and others (Mendoza, Pérez and López, 2000) were reviewed. Not unexpectedly, most were critical of the progress made in improving court performance, although not targeting the WB project in particular. In fact more criticisms were directed at the legal framework, continuing political interference in judicial selection and operations, and the failure to control corruption. Interviews with judicial staff are referenced in the text. 46 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Estimate (USD Percentage of Components Appraisal Estimate (USD millions)* millions) Appraisal Strengthen Institutional Capacity of Judicial Branch 10.7 11 103% Provide Anti-Corruption Support 2.3 0.2 9% Strengthen Access to Justice 12.5 15.4 123% Communications Support for Judiciary Branch Modernization 5.2 4.9 94% Total Baseline Cost 30.7 31.5 103% Physical Contingencies Price Contingencies Total Project Costs Project Preparation Fund Front-end fee IBRD 0.3 0.3 100% Unallocated 2 0 Total Financing Required 33 31.8 96.36% * Appraisal Estimate includes Bank Financing only (b) Financing Appraisal Actual/Latest Estimate Estimate Percentage Source of Funds Type of Cofinancing (USD millions) (USD millions) of Appraisal Borrower 3.6 2.6 72% US: Agency for International Development (USAID) None 3.6 0 Inter-American Development Bank None 6.7 0 International Bank for Reconstruction and Development 33 31.8 96.36% SWEDEN: Swedish Intl. Dev. Cooperation Agency (SIDA) None 2 0 0 UN Development Programme None 0.4 0 0 Total 49.3 34.4 69.78% 47 Annex 2. Outputs by Component The following chart tracks outputs by component and subcomponent, following the outline in the PAD. Numbers in parentheses under the activities column refer to output indicators listed in the PAD's Annex 1. As explained above, the "revised" output indicators were not used absent any evidence that they were formally adopted and because they are wholly inadequate for tracking component advances. A general summary of progress in each component is given at the beginning of each section. A. Institutional Capacity Improvement: As noted in the text, work under the project greatly improved the status quo ante. However, the reorganization and reengineering were completed early, but never tested or refined. Many offices need further support and coordination among them is insufficient. Information systems were developed, but are not integrated and of uneven quality. SGT (case management system) was only recently implemented in 32 courts and roll-out remains in question. Construction and equipment for the most part have been delivered as promised, although many key offices could use additional TA and more equipment. Subcomponents Activities Results and Project Comments Contribution Design and implement Phase I: design of new Model for general It is unclear whether new managerial and model based on review restructuring the alternatives organizational model of alternatives for developed on basis of referred to in the PAD organizing PHRD-funded study; were developed and jurisdictional and model for tested; in any event, administrative areas to reengineering of the restructuring and redefine administrative internal organization reengineering, once procedures, processes and structure completed in the first and functions in urban developed in 1999-2001 phase, were not and rural areas in coordination in systematically revisited including: 1) strategies UNDP/PROFED. to test results; for optimum resource Series of subsequent subsequent changes in utilization and minor restructurings placement of offices management change; done since. appear to be the result 2) proposals for of ad hoc decisions by decentralizing the Court (and administrative occasionally by the functions; 3) Chief Administrator) development of performance standards; and 4) proposals to improve organizational arrangements of courts at all levels, including their supervision (1.1). Phase II: Restructuring done in Administrative offices Implementation of new 1999; reengineering in still exist as isolated model, training of staff 1999-2001. As part of islands with in change management reengineering, central insufficient 48 and implementation administrative and coordination among (1.2) auxiliary offices them. Manuals provided with nonexistent or not equipment, remodeling used. Complaints of of infrastructure, lack of support from manuals, and technical or contact with project assistance to develop frequent. some new processes. Auxiliary offices With few exceptions (Judicial Council and (informatics, judicial its parts; Servicios school, CENADOJ, Tribunalicios; UNAP, and later Communication; Communication) no CENADOJ, Judicial further work done. School, ADR Center, No work on and SPDI) lack deconcentration done sufficient coordination. under project (pending Continuing support to termination of regional CENADOJ and School and departmental is a mixed blessing (has centers); no not strengthened their performance standards planning capacity and developed; no signs of has confused some work on optimum functions). Servicios resource use (except Tribunalicios gets most for development of assistance from other POAs for yearly donors. UMOJ budget exercise); absorbed much of limited work with communication court and courtroom function (and some organization, and staff) for a good part of pilots not replicated or project. Consequently supported afterwards. services provided to courts needs considerable improvement. Work on jurisdictional organization limited and not consolidated. Improve Administrative SPDI created and Progress in this Administrative modernization through provided with TA and element is relatively Capacity, HR 1) the planning and equipment, but advanced, but SPDI Management, and institutional planning function needs further Strengthen Judicial development function largely absorbed by strengthening and a Career Initiative (SPDI); 2) UMOJ; IT office more active role in strengthening office strengthened and planning (with retreat for IT, 3) developing master plan created; of UMOJ from that and implementing a regional and role); SIAF needs master plan for IT departmental centers further strengthening including design and built and equipped, and addition of other training in integrated but unclear how much modules (HR, judicial and of administrative inventory, etc) once management functions they have they are also information system taken on. Network improved. SGT will (1.8); and 4) connections for 18 of require considerable establishing regional 22 departments. work to ensure 49 and departmental Administrative adequate roll-out and centers with information system use. Now that regional automated still not integrated, and and departmental administrative SIAF executed by help centers are constructed functions, electronic from Finance. Other and equipped, links to the center, and modules of varying attention will be training, logistical quality and still not required to strengthen support and connected. Case their administrative infrastructure (really management system systems. 3.7) (SGT) designed and implemented in 32 courts ­ not linked to administrative system. Development of HR Project supports All judicial council policy and judicial implementation of units (evaluation, career (1.4, 1.5) judicial career and discipline, supervision) through 1) application civil service laws by require more and evaluation of equipping offices and equipment and TA to standards for providing technical improve recruitment, selection, assistance. Judicial methodologies. Also training, and selection has standards legal framework for performance of judges, although there are careers and their support, and many complaints operation is flawed and administrative staff; 2) about adequacy. HR requires amendment. implementing and classification study There is really no evaluating automated never done; no judicial career, just HR management evaluation system for tenure for 5 years with system; 3) support to civil service staff; HR reappointment for judicial school in information system lower judges based on comprehensive created but limited to evaluations. training programs; 4) payroll. Support to For civil service, comprehensive, judicial school position classification, integrated training (remodeling, evaluation system, and programs for regional equipment, some TA career paths are administrative and and financing of many needed as is a complete support staff; and 5) short courses). automated registry salary review for Training for regional (more than payroll). courts. and support staff done, These salaries require but unclear how review. integrated and School's program comprehensive (or needs improvement for effective). No sign of in-service training. salary study, but Needs assessments and UMOJ did comment evaluations are on proposed new necessary. In-service salary for judges. courses require better link to judges' career status and needs. Improve National inventory of Inventory done too late National inventory still Administrative cases to develop delay and is incomplete; no needed and best Capacity, HR reduction program delay reduction combined with case Management, and (1.6), prepare an program, no purging exercise and Strengthen Judicial integrated statistical performance introduction of 50 Career Initiative system (1.8); and indicators. Statistical automated or manual develop performance system is not reliable. system for registering indicators and assess all new cases. results (1.3). Improve Operation of Revision and Exercises conducted Rather than relying on courts simplification of for Guatemala City case management procedures in justice of appellate courts, pilot system (SGT) to fix the peace, first first instance and this (which it will not), instance, and appeals justice of the peace recommendations from courts in Guatemala courts. However, pilot work should be city to reduce backlogs recommendations not reexamined and steps and increase officially adopted. taken to adopt them. transparency (1.7) Further work on simplifying courtroom procedures and identifying needed legal change is also required (and again, the case management system will not do this). Review judicial Except for actions in Still required. processes as support prior point, this was for legislative and not done. administrative reform proposals and review through workshops, internal discussion, training Increase number of Servicios Tribunalicios These centers only judicial support has or will have make sense for centers for notification branch offices in the courthouses with and case distribution. new regional and multiple judges ­ departmental centers. improvement of However, logistically services for single or their reach will be few judges centers also limited. need to be worked out. Carry out related Not done More studies may be studies for required and a better strengthening and means for identifying evaluating court research needs should operations be created and institutionalized (in SPDI?). Develop judicial Existing systems are At present four offices information systems not reliable even are collecting their for Guatemala City within these targeted own statistics although and two pilot locations areas. Data are being only the office in (1.8). collected (an CENADOJ has this improvement over the responsibility. The 51 past) but they are statistics area in incomplete, and also CENADOJ's office not analyzed. SGT will should be moved back generate statistics, but to planning, and better only once rolled out manual and automated and if quality control collection of data maintained. enforced. Analytic capabilities also need strengthening as a basis for setting productivity targets and detecting problems beyond the low productivity of individual judges B. Transparency: major achievements were ethics code and training; support to Junta de Disciplina and disciplinary office for civil service employees, support for Departamento de Supervision, automation of Registro de Protocolos, and increased salaries (however this was not a project achievement). Several activities were not executed, and those achieved did not receive follow-up assistance to strengthen their impact. Total amount spent on component about one-tenth of that indicated in PAD. Subcomponents Activities Results and Project Comments Contribution Preventive measures Preparation and Code drafted, enacted No evaluation of implementation of an and publicized with impacts, which do not ethics code (2.2). project support. appear significant. Carry out study on Not done. Unclear what impact applicability of would have been. international treaties (IACC) for judiciary. Development of Judicial salaries were Clearly salaries are various mechanisms raised (but not by less important than (salaries, norms, project); reliable several other measures sanctions, statistics, statistics still missing, that were not taken ­ publication of publication of for example the judgments) to deter decisions is partial; statistical system, corruption (2.5) and investigative better investigation. systems appear relatively ineffectual Carry out studies on Except for pilot Not only were pilots introduction of oral experiments, this was not replicated ­ they procedures in not done. seem to have had no preparation for draft input to code drafting law (2.6) exercise. Carry out training on Series of courses held Absent better control anti-corruption on ethics code. mechanisms, impact of measures. courses appears minimal (and was not evaluated). Control measures Creation of an Anti- Done, but not by Commission (an corruption project. Project intersectoral body) Commission (2.1) supports Junta de appears to have had 52 Disciplina (for judges) minimal impact. and disciplinary office Impact of disciplinary (for civil service), both units impeded by created in insufficient resources, restructuring and legal framework further assisted during (which could be reengineering exercise improved) and insufficient technical assistance. Departamento de Reorganization done, Needs more logistical Supervisión de equipment and TA support, decentralized Tribunales reorganized provided, but project offices, improved legal and streamlining of support ends post- framework, and better procedures, including 2002. coordination with creation of complaints Junta de Disciplina. office (2.3). Monitoring of pending Not done except Given their cases. (Unclear from insofar as disciplinary undecipherable PAD whether these are offices keep their own statistics and the many disciplinary or all cases registries complaints about the ­ 2.1?) disciplinary systems, further study and monitoring are needed. Development of Not done World be useful and internal system for should be pursued as disclosure of assets of one more element to judges and staff fight corruption. Development of a Project funds Automation has system to register automation of Registry reduced registration lawyers' assistants to of Protocolos in times, but impact on address problem of Guatemala City. related problems unauthorized litigants (corruption, litigation accessing courts (2.4) by unregistered lawyers, etc) is unknown. Need to evaluate, study additional problems and develop alternative measures to limit lawyer abuses. Expansion of oral Draft civil procedures While the impact of procedures in other code supported (but orality in increasing (non-criminal) areas of not enacted) and pilot transparency or law (2.6) projects done with reducing delays can be family and labor overstated, this would courts, but not still be a useful step, replicated. especially in combination with other measures Studies on corruption Not done Both are necessary and improvement of elements in complaints system understanding and thus combating corruption 53 C. Strengthening of Access to Justice: major accomplishments were provision of infrastructure and equipping of what was built, introduction of mobile courts, and promotion of ADR, via infrastructure, equipment, training, and technical assistance. Work on reorganizing justice of the peace courts and promoting indigenous law was minimal, and civil society program appears to have been reconceptualized in the course of execution. Subcomponents Activities Results and Project Comments Contribution Diversification of Studies on rural and One study covering 33 Study had to be done judicial services and urban communities to communities was too rapidly to meet reorganization of assist in preparation of completed in last year PAD criteria ­ much justice of the peace policies to improve of project and thus more is needed on courts. access (3.1) with no impact on reasons for non-use implementation. even of popular services (ADR) and on types of services that would best meet needs of marginalized populations. Design and creation of Completed with 16 Program is popular an ADR coordination centers built; training but underutilized. No unit, pilot programs of mediators; evaluation done, and and their replication, programs with justices more work needed to and training and of the peace and public understand limitations dissemination to promote use. on use, and, for the few campaign. (3.2) centers with considerable work, to help develop a strategy to respond to the demand. Reorganization of Number expanded, but Although there are justice-of-the-peace no new model now justice of the courts to respond to developed. Instead peace courts in every community needs (3.4). the project has focused municipality (thanks to - on two elements: 1) the WB project, other training justices of the donors, and the peace in conciliation government), their and the use of ADR services attract many services, and 2) complaints and their publicizing their efficiency is generally presence through a low. Much more large number of information is needed workshops. on their operations, Presumably the weaknesses, and use. massive equipment purchase made at the end of the project will also benefit these units. Streamlining of Except for what is Complaints from coverage, provision of noted above this was FUNCEDES survey conciliation and other not done. Training in and others suggest 54 skills, preparation of "local customs" much need to be done regulations to better appears to be a general to improve the define role and profile course in performance of these of justices of the peace "sensitization." The officials. (3.3; 3.4, 3.5). judicial career laws have specified that justices of the peace must be lawyers and have regularized entry into the position (as well as providing evaluations every five years), but aside from this regulation of role has not improved Automation of Not included in PAD, Great benefit for poor Antecedentes Penales but a very successful (although they still activity. Times to must pay); impact on receive certificate other services minimal. greatly reduced In the longer run, Guatemala and many other countries need to eliminate this requirement. Expansion of judicial Construction and All accomplished by Construction took infrastructure services remodeling of project except for more time and was courthouses in the second departmental more costly than interior of the country center. Infrastructure anticipated, but ­ 2 regional centers, 2 plan developed early ­ buildings are occupied. departmental centers, has not been 6 justice of the peace readjusted courts, and development of second-stage master plan for infrastructure (3.6; 3.7) Program of mobile This was not included More of this could be courts. in the PAD but has done as part of a larger been very successful. plan to determine how Two are now in to meet the demands of operation and 10 more a very dispersed ready for deployment population in the most efficient fashion. However, where there are courts, there need to be police, prosecutors, and defenders ­ this problem cannot be fixed by the judiciary alone. Program of Civil Subprojects through 450 workshops to Contrary to the PAD Society Participation the promotion of prevent lynchings held, description, these were partnerships with 2 programs to expose all centrally initiated donors, NGOS, and school children to and organized, other participants to justice developed, although local 55 develop broader various workshops on authorities were asked participation in access justices of the peace, to participate. No to justice efforts (3.8) gender, and evaluation of impact, community practices and remains unclear as to what was really intended. D. Support for social communication, Judiciary Modernization Commission and PCU: The major part of funding here (nearly 90 percent) went to the PCU which as a consequence may have taken on or kept functions better transferred to the OJ offices. These functions include two figuring in this component ­ social communication and reform planning and oversight. Nonetheless, certain activities, and indicators, were not advanced ­ most notably, creation of a monitoring and evaluation system (even within the PCU) and of a program to gather and publish court data and surveys. Subcomponents Activities Results and Project Comments Contribution Social Strengthening of social Creation of Dirección Another example of Communications communications unit de Comunicación, and UMOJ mission creep? program in Judiciary (4.3) alter several shifts of Would be preferable to placement, its return focus communications to the Secretaría de la in Dirección, including Presidencia; staffing those for project to increased from 3-4 to ensure 12 in 2002 with 3 institutionalization of internal divisions. functions Post-2003 development not assisted by Project Development and There does not appear PCU carries on much implementation of to be a long-term plan; of communication long-term Communications Unit work for life of communications has little contact with project; also strategy (4.1) UMOJ since 2003. encourages CENADOJ's involvement in same rather than development of own priorities and activities. Development of Project (UMOJ) has As much of this is done internal and been active in by UMOJ, not clear institutional promoting novel kinds whether internal communications of internal and capacity has been mechanisms (4.2) external enhanced or who (what communications office) will carry on (website, publications, once project ends. radio spots, meetings, programs with schools and students) Office of Public Affairs Publications, radio Much activity with Multilingual spots, and events done especially by UMOJ, communications in local languages. but again, installed capacity, including However, installed capacity and strategy publications and capacity and strategy not evident. 56 reports (4.3) not evident. Development of Relevant actions The institutionalization program to gather and include CENADOJ of a program for publish data on court statistics (few monitoring and performance, surveys, published), several disseminating data on and civil education surveys done by performance and its efforts (4.2?) Project, and numerous direction and use by education activities. OJ leadership are However, there is no critical, but have not single program and occurred. many activities are ad hoc. Establishment of Communications Divisions of labor with public information division created within UMOJ not clear ­ center the Departamento (ex- UMOJ does much of Gerencia) de this for project on its Comunicación. own. Support for Training and Annual reports Commission remains Judiciary's development support; published, but training weak and dependent Modernization annual progress report and development of on UMOJ for planning Commission (4.8), exchange of best commission not visible and execution. No practices, and training independent in change management supervisory capacity (4.4) developed Strengthening PCU Provision of long and PCU grows in size over Question as to whether short term TA in period to maximum of PCU grew too much specialized areas to 48 employees. TA and so took on improve project provided by hiring activities performance, specialists to work corresponding to including (4.5) clear within unit and so to permanent judicial delineation of engage directly in offices. With some functions; (4.6) oversight and exceptions (CIT, Social adequate quality implementation of Communications) not assurance, monitoring project activities, thus clear that knowledge and evaluation; and clouding division of was transferred to (4.7) automated functions with OJ judiciary to enhance its financial management units; automated ability to implement on and reporting system financial system its own implemented (but not linked to SIAF); no monitoring and evaluation system except for tracking implementation of activities and disbursements. 57 Annex 3. Economic and Financial Analysis (including assumptions in the analysis) An economic and financial analysis was not done for this project, and admittedly, given the lack of basic statistics would have been hard to do. However, contrary to the assertions of the PAD something could have been produced in each area, even if largely based on illustrative statistics. The reasons for doing this are largely educational and inspirational: by demonstrating, even if in the abstract, the potential financial and economic benefits of the activities, stakeholders may come to understand a number of issues better and may be better prepared to take policy decisions. The ICR does not grade the project (or the Bank's role) any lower for not having done what virtually is never done for a judicial reform operation. It simply suggests that the contention that this is impossible (made by many others) be reexamined. For example, on the financial side, one could have at least calculated the returns of a potential doubling or tripling of judicial output against the amount invested plus operating costs versus what would be required to place two or three times the number of judges and courthouses. While the judicial plan also increased the numbers of judges and courthouses in the interests of serving isolated areas, it clearly could eliminate some and has already moved others to areas with insufficient service. In the end, Guatemala probably does not need more judges ­ it simply needs to increase the productivity of those in place and better distribute them. The analysis could have included several alternative scenarios for distribution ­ something the judiciary is going to have to do in any case On the economic side, even working with some rather dodgy data (World Bank, 2005) or methodologies (Sherwood et al's calculation of impacts on growth rates of a "better judicial system") some simple analytic exercises could have been done to demonstrate the theoretical impacts of a more effective, credible, and accessible justice system. It is suggested that this exercises be incorporated in the next preparation, offered with all the necessary caveats, but nonetheless placing the emphasis on the importance of improving outputs, and not just inputs. The principal advantage of both exercises would be to get the assumptions out on the table and encourage participants (on both the Bank and counterpart side) to reexamine them. Analysis should also include necessary inputs from other organizations ­ what happens if you increase the number of judges, but not prosecutors, police, and defenders? ­ opportunity costs, and the impact of different scenarios. 58 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialty Lending Supervision/ICR Shelton H. Davis Consultant LCSPS Alexandra M. Habershon Consultant ECSSD Nina-Christina Ohman Consultant LCSPS Roberto O. Panzardi Sr Public Sector Mgmt. Spec. LCSPS Karla Lopez Flores Program Assistant LCSPS Anthony Wanis-St.John Consultant LCSPS Linn Hammergren Sr. Public Sector Mgmt Spec LCSPS Richard Moore Consultant LCSPS Laura Louza Consultant LCSPS Andrew Blandford Consultant LCSPS Keisgner Alfaro Procurement Analyst LCSPT Monica Lehnhoff Procurement Analyst LCSPT Antonio Leonardo Blasco Financial Management SpecialistSr. LCSFM David Bernstein Public Sector Mgmt Spec. ECSPE ICR Reviewer Eduardo Somensatto Lead Economist LCSPR ICR Reviewer Robert Varenik Consultant ICR Reviewer (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle No. of staff weeks USD Thousands (including travel and consultant costs) Lending FY97 134.36 FY98 200.68 FY99 68.39 FY00 8 37.45 FY01 5 19.25 FY02 0.00 FY03 0.00 FY04 0.00 FY05 0.00 FY06 0.00 59 FY07 0.00 FY08 0.00 Total: 13 460.13 Supervision/ICR FY97 0.00 FY98 1.71 FY99 44.20 FY00 18 90.82 FY01 9 48.30 FY02 10 132.89 FY03 10 121.66 FY04 11 95.77 FY05 15 72.48 FY06 17 90.83 FY07 17 83.24 FY08 6.29 Total: 107 788.19 60 Annex 5. Beneficiary Survey Results The project supported three surveys, two by the same firm using comparable samples and methodologies (Aragón y Asociados, 2003), and a third, conducted by UMOJ, which we only learned of after the fieldwork (because of its mention in Novoa and Tacao, 2007). There was a forth survey (Novoa and Tacao, 2007) conducted after the project closed and apparently financed from non-project funds. It was done by two consultants hired by UMOJ who drew their sample from court staff and court users in four urban centers and one semi-rural area where the project had provided major infrastructure. Unfortunately, the change in methodology and samples (the first two drew from a universe of the entire national population; the last was geographically and functionally limited) means that the final survey is not directly comparable with the first two. The first two surveys were conducted in 1997 and 2003. The sample size for each was 1,000; they were drawn nationally and stratified to capture groups by gender, age, and geographic areas (department). A comparison of the two periods shows some loss of optimism. Questions about the state of the country in general and expectations for the future brought slightly more negative responses in 2003, with 11 percent judging the situation as improved in 1997 as opposed to 8 percent in 2003; and positive expectations for the future declining by one point (from 12 to 11 percent) over the six years. The most notable improvements tapped by the survey were perceptions of the cost of justice, as shown below: Responses to Question: "Is Justice Expensive?" Date Population Expensive Not expensive NR Total 79 10 11 Capital 80 10 10 1997 Urban 79 11 10 Rural 78 10 12 Total 51 46 3 2003 Capital 60 36 4 Urban 54 43 3 Rural 48 50 2 Source: Data from Aragón y Asociados, 2003. While by 2003, respondents showed some familiarity with the modernization program, and rated certain aspects (Antecedentes Penales and the increase in judges and other personnel) positively, scores for the overall performance of the judiciary dropped during the six years as shown below: Responses to Question: "How would you evaluate the work of the Organismo Judicial?" 61 Year Population Excel. V. Good So-so Bad V. Bd Terrible Good Total 0 NA 12 49 19 NA 10 1997 Central 2 NA 12 46 19 NA 16 Urban 0 NA 10 51 21 NA 11 Rural 0 NA 13 49 19 NA 7 Total 0 2 9 43 27 9 10 2003 Central 1 1 14 41 27 8 9 Urban 0 1 10 45 25 9 10 Rural 0 2 8 42 27 9 10 Source: Data from Aragón y Asociados, 2003. Note: the options given in 2003 were greater than in 1997. It is possible this affected the distribution of results, but impossible to tell in which direction. A final difference between the two sets of results is that the number of mentions of corruption as the principal problem affecting those seeking justice increased from 26 to 41 percent. In 2003, many more negative factors were mentioned, and some of them (lack of professional ethics 3 percent, lack of administrative control 3 percent) are also related to corruption. It is important to recognize that the question and the answers are not directed only at the judiciary and likely incorporate perceptions of corruption within the entire sector. In their conclusions the authors note the following: · Changes from 1997 to 2003 as involves the increase in the population's ability to identify the components of the justice system (Supreme Court, Public Ministry, Judges) and a lesser tendency to see justice as depending on the National President. · The perceived greater accessibility to courts and other sector organizations in 2003 · A continuing perception that justice is excessively influenced by groups (the rich, the political parties) and organizations external to the sector, but with some differences as to which are accorded most importance. · A continuing low evaluation of sector performance combined with high expectations for an improvement The survey conducted in 2007 used a sample of 376 respondents drawn from "internal and external users" of the courts within Guatemala City, Quetzaltenango, Huehuetenango, Escuintla and El Petén. On the internal users side (202 respondents), judges, secretaries, mediators, and administrative staff were interviewed. External users (174) included lawyers, community leaders and civil authorities. Nearly one-third of the respondents (103) came from Guatemala City. One can conceive of the two groups as the most direct, but not the targeted beneficiaries of the program, probably better informed of its contents but also less capable of measuring the impact on the poor, indigenous citizens. (As the authors note in their introduction, a full vision of citizen perceptions on the state of the judiciary and the advances made in the modernization program would require a sample of 62 all citizens, and not just those with the closest contact with the organization and its reform program, Novoa and Tacao, p. 3) Different questionnaires were used for external and internal users, making it still more difficult to summarize the results. Also the final report focuses more on external users, but some graphs do not specify whether one or the other or both groups are included (no N is cited). Among external users, corruption was still perceived (40.2 percent) as the major cause of low confidence in the judiciary. When asked about the principal areas in which services had improved, the following answers were given: External Users In your opinion, which of these aspects of the Organismo Judicial have improved, remained the same or worsened? (Answers are absolute numbers of responses) The NS/NC Better Same Worse Total Physical infrastructure (judicial centers) 3 158 12 1 174 Training of judges and staff 7 87 65 15 174 Provision of materials and computer equipment 9 139 22 4 174 Faster processing of cases 8 31 97 38 174 Communication and public information 7 48 97 22 174 Attention to the public 12 42 94 26 174 Transparency and accountabiity 18 28 94 34 174 Creation of more courtrooms 7 127 32 8 174 Processing of antecedentes penales 10 138 22 4 174 Judges' performance 7 45 96 26 174 Mobile courtrooms 29 109 25 11 174 Protocolo registry 25 92 55 2 174 Attention to violence against women 13 81 66 14 174 Source: Novoa and Tacoa, 2007, p. 20. Own translation Clearly, donor and government efforts to construct more courts, create more judgeships, and supply all with equipment had not gone unnoticed. Two activities sponsored by the WB project alone, the automation of Antecendentes Penales and Protocolos and the introduction of the mobile courts, were also recognized. However, areas relating to judicial performance (transparency, delay, treatment of the public) were generally seen as not having improved. In a separate set of questions on conciliations services (p. 25), perceptions of improvements were mixed. This may be due to continuing lawyer 63 resistance (ADR can be done without their services) with non-lawyer "users" perceiving them more positively. External users in their majority (73 percent) also did not recognize programs facilitating access by indigenous communities (p. 29) and believed (73 percent as well) that linguistic barriers still impeded access. The summary report focused less on the responses of internal users, but did include their evaluation of certain aspects of the reform program. One of these was the efficacy of the measures taken to reduce delay: .- Internal Users In efforts to reduce courtroom delays, how effective were the following measures? NS/NC Very Not Much Not at all Total Legal and normative reform 20 51 116 11 198 Human resource development 6 111 73 8 198 Technology 7 153 38 198 Strengthening of regional offices 21 127 45 5 198 Strengthening of communication and interinstitutional coordination 17 106 68 7 198 Implementation of ADR programs, 13 135 46 4 198 (mediación, conciliación) Source: Novoa and Tacoa (2007), p. 34. Own translation. Interestingly, although both groups recognize the reform programs (and especially those financed by the WB), the internal users seem to give more credit to their effects. When internal users are asked what aspects of judicial "performance" have improved or worsened, they coincide with external users in mentioning buildings, decentralization, and equipment, but also mention other factors which external users either ignore or do not consider important. Internal users 64 In your opinion, which of the following aspects of administrative efficiency have improved or worsened in the Organismo Judicial? NS/NC Better Same Worse Total Existence of strategic and operational plans 40 104 52 2 198 Organizational structure and functions 18 125 50 5 198 Position descriptions consistent with organizational needs and objectives 10 114 62 12 198 Development of regional-level administration 18 134 43 3 198 Linkages between areas and levels of work 23 104 69 2 198 Existence of a human resources management system 6 132 53 7 198 Skills updating of professional, administrative, and technical personnel 11 133 50 4 198 Improvements in information technology 3 160 33 2 198 Adequate physical space and equipment 6 137 46 9 198 Suffiicient budget 29 32 108 29 198 Strengthening of organizacional culture 15 88 85 10 198 Institutional coordination, communication, and information 9 117 64 8 198 Salary scale and incentives 9 54 99 36 198 Quality of judicial statistics 22 105 70 1 198 Source: Novoa and Tacoa, 2007, p. 38. Own translation External users are considerably less sanguine about program achievements, and 36 percent of them note the need for improvement in all of the following: simplification of procedures, information, treatment of users, judicial supervision, and the availability of free legal services. What this suggests (and the answers still do not tap the perceptions of the general public) is that to date the modernization programs have improved the situation of those working within the system (although still not as much as they would like), but that the results have not been translated into better services for external clients ­ certainly not if those who know best (the lawyers and local authorities) perceive corruption as having increased and see no impact on delay. Interestingly, interviews done by FUNCEDE and reported in its 2006 study note the same difference ­ when asked about what is needed, judges and staff mention more buildings, equipment and higher salaries while users emphasize the failure to resolve the same old problems of corruption, delay, and poor treatment. This puts into question one of the implicit (and sometimes explicit) assumptions of many WB projects ­ that better housed, remunerated, and 65 equipped judges will automatically perform better ­ and suggests the need for additional actions (a bit of stick with the carrot, or simply greater emphasis on improving services) to bring about the ultimate goals of a more credible, effective, and accessible (in more than the simple physical sense) justice system. 66 Annex 6. Stakeholder Workshop Report and Results While stakeholder workshops were held during project preparation, none were held at the close of the project. Thus there is nothing to report here. 67 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 68 69 70 71 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders There were no cofinanciers of the project. Other donors did commit to supporting portions of the judiciary's modernization plan and of the overall sector modernization plan. One (the IADB) shared the same PCU (UMOJ) contributing to its strengthening with the purchase of equipment, but not salaries, and two others (Norwegian and Dutch assistance agencies) coordinated closely with UMOJ in their construction of courtrooms. The UNDP not only assisted procurement activities for the current project, but also was an equal partner (managing its own funds and those of several other donors) in the reengineering program of the 1999-2001 period (list of contributions supplied by Josefina Coutiño, on file with authors). Other stakeholders include members of the judiciary and other sector institutions, NGOs, the private bar, and court users. Although interviews were held with several donors and with three NGOs on their activities and interactions with the WB project, no formal statements on the project were offered for citation. Interviews were supplemented with written materials and the results of surveys. In general, remarks were neutral to negative, but this was either affected by on-going conflicts with UMOJ (USAID, UNDP, UNOPS) or in the case of NGOS and outside users, so mixed with views on the judiciary and marred by a frequent inability to separate WB activities from those of the Court or of other donors that we do not believe they can be taken as evaluations of WB performance. When pressed (see Novoa and Tarao, 2007), court users and knowledgeable observers generally admitted some improvements to the judiciary ­ in terms of transparency, better organization, and more capable judges ­ but do not see any noticeable change in the quality of output. Rural inhabitants interviewed by FUNCEDE (2006), while quite critical of court performance, offered positive evaluations on the ADR programs. Many of these positive views can be attributed to changes the World Bank supported, but in which its direct involvement varied considerably. In interviews with judicial staff, as reported in the text, a few (CENODOJ, the Judicial School, and CIT) were very positive about the project and UMOJ, but others cited only early support to their activities, seemed less aware of recent developments, and in several cases reported a "policy of secrecy" on the part of UMOJ. We were led to understand through discussions with knowledgeable outsiders, that the current divisions in the Supreme Court (precipitated by the upcoming election of a new President) include disagreements on the project and especially UMOJ. We were unable to verify this observation. 72 Annex 9. List of Supporting Documents ICR Consultants' Reports (on file): Louza, Laura. 2007. "Evaluación sobre el proyecto de reforma judicial de Guatemala" Moore, Richard. 2007. "Report on Administrative Reforms of the Judicial Reform Project in Guatemala, IBRD 4401-GU Project Documents Note: In preparation for the field work and during the course of ICR preparation, the ICR team (Linn Hammergren, WB; Andrew Blandford, Consultant; Laura Louza, Consultant; and Richard Moore, Consultant) prepared a list of the 4,000 or so documents filed in IRIS and reviewed (sometimes only cursorily) about 10 percent which on the basis of their titles seemed potentially significant. A list of even those actually used in the report would take several pages. We are thus listing only those of particular importance, and where relevant have cited additional ones in the main text. World Bank. 1998. "Guatemala: Judicial Reform Project, Project Appraisal Document," September 30 World Bank. 2002. Proyecto de Modernización del Organismo Judicial Guatemala, Misión de Supervisión Técnica, Ayuda Memoria, October 7-12. PSRs, ISRs for period January 4, 1999 to October 18, 2006, series of 17 reports Aide Memoire, Programa de Modernización del Organismo Judicial (PMOJ), Misión de preparación, pre-evaluación Banco Mundial, January-February, 1998. Aide Memoire, Misión de Evaluación de la Administración Financiera, Unidad Coordinadora del Programa del Organismo Judicial, Guatemala, May 21, 1998. Lara, Aranky, Ramos y Asociados. 2006. "Información financiera complementaria periodos del año 1999 al 2006. Proyecto "Reforma Judicial" Préstamo No. BIRF-4401- GU." (Compilation of external audit reports provided by UMOJ). N.A. 2006. "CAS Goals Related to Justice," reported provided to team along with other documents forwarded by María Novoa. Authorship not available. Various authors and dates, post reviews of procurement, Loan IRDB 4401-GU. Proyect reports and studies: Andino Grotewold, Jorge Mario, 2001. "Diagnóstico Salas de la Corte de Apelaciones Torre Marfil," June (Forwarded to Waleed Malik by UMOJ on April 30, 2002) 73 ______. n.d. "Manual de Organización y Funcionamiento Juzgados de Primera Instancia de Trabajo" Aragón y Asociados. 2003. "Informe comparativo de estudios cuantitativos de opinión pública sobre la administración de justicia en Guatemala a nivel nacional, 1997-2003," Guatemala, December. Contrato de Servicios de Consultoría entre Organismo Judicial de la República de Guatemala y National Center for State Courts, December 19, 2005 Cornish, Mary. 2002. "Promoting a Culture of Dialogue and Diversity: Report to the World Bank on Technical Visit to the Guatemala Judicial Reform Project," June 12 FUNCEDE (Fundación Centroamericana de Desarrollo). 2006. "Informe final de la consultoría: Particularidades de las comunidades rurales y urbanas en el contexto de los juzgados," and reports on various subproducts, unpublished documents prepared under World Bank Project. National Center for State Courts. 2006. Series of reports, intermediate products and final report on project for case inventory, study of delay, and pilot project for delay reduction. Novoa, Maria. 2006. "Preparación insumos para la evaluación del proyecto BIRF 4401- GU, Reforma Judicial de Guatemala, December. ______. 2007. "Análisis cualitativo y cuantitativo sobre la percepción de los usuarios internos y externos del Servicio de Justicia," unpublished report prepared with project funding Svensson, Bo. 2007. "Judicial modernization projects in Guatemala and El Salvador," report prepared with project funds. On file with authors. Thacker, Gerald. 2002. "Report of Trip to Guatemala May 21-25." (Evaluation of infrastructure) UMOJ. 1999. Informe de Progreso: Reforma Judicial de Guatemala. ______. 2000. Informe de Progreso: Reforma Judicial de Guatemala ______. 2001a. Informe de Progreso: Reforma Judicial de Guatemala ______. 2001b. "Training and Cultural Sensitization Project for Justice Sector Personnel," June. ______. 2003a. Educando para vivir en paz 74 ______.. 2003b."Informe ejecutivo de agilización de procesos Judiciales, enero 2001- junio 2003". ______. 2003c. "Manual de Organización y Funcionamiento de Juzgados de Paz, Agilización de procesos Judiciales." Wanis-St John, Anthony. 2003. "Evaluation of World-Bank supported ADR in Guatemala." Guatemala, Organismo Judicial Castillo y Castillo, Carlos. 2007. Reports from Unidad de Resolución Alternativa de Conflictos (RAC), various months. CENADOJ. 2002. "Boletín Estadístico, Ramo Penal". Comisión Combate contra la Corrupción. 2003. "Primer Informe:: Por el cambio y la justicia". Comisión de Modernización del Organismo Judicial. 1997. Plan de Modernización del Organismo Judicial 1997-2002: Corte Suprema de Justicia y Organismo Judicial. Consejo de la Carrera Judicial. 2005. "Manual de Evaluación del Desempeño y Comportamiento Profesional de Jueces de Primera Instancia para su Ingreso a la Carrera Judicial." ______. 2005. "Manual de Evaluación del Desempeño y Comportamiento Profesional de Jueces de Paz para su Ingreso a la Carrera Judicial." ______. 2007. "Reglamento de Evaluación del Desempeño y Comportamiento Profesional de Jueces de Primera Instancia para su Ingreso a la Carrera Judicial." ______. 2007. "Reglamento de Evaluación del Desempeño y Comportamiento Profesional de Jueces de Paz para su Ingreso a la Carrera Judicial." "Ley de Servicio Civil del Organismo Judicial y su Reglamento General." "Ley de la Carrera Judicial y su Reglamento acuerdo N° 6-2000)". Organismo Judicial. 2005a. Guatemala: La nueva cara de la Justicia. ______. 2005b. "Propuesta de Modelo de Juzgado de Primera Instancia Penal". ______. 2006. Plan Quinquenal 2006-2010. Other documents: 75 Asociación de Investigación y Estudios Sociales (ASIES). 2004. Estudio sobre avances y dificultades en la implementación de la Ley de la Carrera Judicial, diciembre 1999- agosto, 2004. Guatemala: ASIES and Konrad Adenauer Stiftung. ______. 2006. Proceso de fortalecimiento del sistema de justicia: avances y debilidades Julio 2003-junio 2006. Guatemala: ASIES and Konrad Adenauer Stiftung. Azpuru, Dinora. 2006. Estudio de cultura democrática. Blair, Harry and Gary Hansen.1994. Weighing in on the Scales of Justice. Washington, D.C., USAID. Carothers, Thomas, "The Many Agendas of Rule of Law Reform in Latin America." In Rule of Law in Latin America: The International Promotion of Judicial Reform, eds. Pilar Domingo and Rachel Sieder, 17-46. London: Institute of Latin American Studies:4-16. Centro de Estudios de Justicia de las Américas (CEJA). 2003. Reporte sobre el Estado de la Justicia en las Américas, 2002-2003. Santiago, Chile. ______. 2005. Reporte sobre el Estado de la Justicia en las Américas, 2004-2005. Santiago, Chile. Goldman, Francisco. 2007. The Art of Political Murder: Who Killed the Bishop. New York: Grove Press Guatemala, Instancia Coordinadora para la Modernización del Sector Justicia (ICMSJ). 2006. "Plan estratégico Sectorial para la Transformación del Sector Justicia Penal, 2006- 2010. Hammergren, Linn. 2007. Envisioning Reform: Improving Judicial Performance in Latin American. University Park, Pa: Penn State University Press. ______.1998a. "Code Reform and Law Revision." USAID, CDIE ______. 1998b. "Fifteen Years of Judicial Reform in Latin America: Where We Are and Why We Haven't Made More Progress." Paper prepared for Seminar on Judicial Reform in Latin America, Corporación Excelencia en la Justicia, Bogota, Colombia, July, 1998. Hammergren, Linn and Richard Messick. 1998. "The Challenge of Judicial Reform," in Shahid Javed Burki and Guillermo Perry, Beyond the Washington Consensus. World Bank. Hendrix, Stephen,. 2002. "Guatemala" in Herbert Kritzer (ed), Legal Systems of the World: a Political, Social and Cultural Encyclopedia. Santa Barbara, Cal: ABC-CLIO, Vol. 3, pp. 613-622. 76 ______. 2000. "Guatemalan `Justice Centers': The Centerpiece for Advancing Transparency, Efficiciency, Due Process, and Access to Justice." American University International Law Review, 15:4, 813-867. Juárez Elías, Franklin Erick. 2007. "De la reforma normativa a la reforma de las practicas." Unpublished paper on file with authors. Lawyers Committee for Human Rights and the Venezuelan Program for Human Rights Education and Action. 1996. Halfway to Reform: The World Bank and the Venezuelan Justice System. New York. Malik, Waleed and others. 2005. "Guatemala: The Role of Judicial Modernization in Post Conflict Reconstruction and Social Reconciliation," Social Development Notes, Conflict Prevention and Reconstruction, World Bank, No 21, February. Méndez, Juan E., Guillermo O'Donnell, and Paulo Sergio Pinheiro, eds. 1999. The (Un) Rule of Law and the Underprivileged in Latin America. Notre Dame, Indiana: University of Notre Dame Press. Mendoza A., Carlos A. 2007. Ausencia del estado y violencia colectiva en tierras mayas. Guatemala: FLACSO. Mendoza G., Lissette Beatriz and Ricardo Mendoza Orantes. 2007. República de Guatemala: Constitución explicada. San Salvador: Editorial Juridica Salvadoreña. Pérez Ruiz, Yolanda and Eleazar López. 2000. "Guatemala: Independencia judicial," unpublished document prepared for USAID handbook on judicial independence, on file with authors. Popkin, Margaret. 2000. Peace Without Justice: Obstacles to Building the Rule of Law in El Salvador. Pittsburgh: The Pennsylvania State University Press. Prillaman, William C. 2000. The Judiciary and Democratic Decay in Latin America. Westport, Connecticut: Praeger Ramírez García, Luís .N.d. "La reforma de la justicia penal en Guatemala: Aportes para una evaluación," unpublished paper, on file with authors. ______. 2007. "Monitoreo del Desempeño del Sub-Sistema de Justicia Penal," paper presented at V Seminiario Internacional de Gestión Penal, Lima, October 10-11. Sherwood, Robert M., Geoffrey Shepherd and Celso Marcos de Souza. 1994. "Judicial Systems and Economic Performance." The Quarterly Review of Economics and Finance. 34, Special Issue: 101-116. 77 Van Cott, Donna Lee. 2007. "Latin America's Indigenous Peoples," Journal of Democracy, 18:4; 127-141. World Bank. 2002. The Juicio Ejecutivo Mercantil in the Federal District Courts of Mexico: A Study of the Uses and Users of Justice and Their Implications for Judicial Reform. Washington, D.C: The World Bank, Report No. 22635-ME. World Bank. 2005. Doing Business in 2005: Removing Obstacles to Growth. Washington D.C: The World Bank, the International Finance Corporation, and Oxford University Press. 78 Annex 10: List of individuals interviewed for ICR Judicial Personnel: Javier Oswaldo Alegría, Supervisión de Tribunales Vilma Álvarez, Secretaría de Planificación y Desarrollo Institucional Elia María Berdugo, Juez del Juzgado Primero de Familia de Guatemala Claudia Lorena Castañeda Castillo, Centro de Gestión Penal de Quetzaltenango Herbert Castillo, Director de Servicios Tribunalicios de Quetzaltenango Carlos Humberto Castillo y Castillo, Coordinador General de la Unidad Resolución Alternativa de Conflictos Luis Alfredo Cifuentes, Gerencia de Administración Brina Cordero, Recursos Humanos Carlos Gilberto Chacón Torrebiarte, SCJ Justice and current coordinator for PCU Dilia Estrada García, Asistente de la Secretaría del Consejo de la Carrera Judicial Mónica Paola Faillace de Zelada, Jefe del Centro de Gestión Penal de Quetzaltenango Ronald Estuardo Figueroa Hernández, Director of Centro de Informática y Telecomunicaciones (CIT) Jorge Isaías Figueroa Pérez, Junta de Disciplina Aída Franco, Directora de la Escuela de Estudios Jurídicos Ludin Noemí Franco, Comisionada del Juzgado Primero de Familia de Guatemala Rita Marissa García, Junta de Disciplina Verónica Herrera, Coordinadora de Educación a Distancia de la Escuela de Estudios Jurídicos Victor Hugo Herrera, Juez Penal de Turno, Guatemala María Leonor Hurtado, Oficinista del Juzgado Primero de Familia de Guatemala 79 Ulises Ixcot, Juez Primero de Paz de Quetzaltenango Byron de León, Especialista, Departamento de Comunicación Social, Comunicación y Divulgación Beatriz Ofelia de León Reyes de Barreda, Magistrado Vocal IV, SCJ Héctor Aníbal de León Velazco, Secretario de la Presidencia Mildred Luna de Espina, Departamento de Comunicacion Social, Coordinadora de Programa Educativo Héctor Emilio Méndez, Presidente, Junta Disciplina Carlos Enrique Mendizabal Arriola, Director de la Unidad de Antecedentes Penales Gladys Miranda de Valencia, Gerente de Recursos Humanos Lisette Nájera, Directora de Servicios Tribunalicios. María del Carmen Ortíz, former subcoordinator of PCU, currently Gerente General Mario Francisco Pérez, Jefe del Centro de Servicios Auxiliares de Quetzaltenango Edgar Pérez Bosque, Secretaría de Planificación y Desarrollo Institucional Isabel Prem, Supervisión de Tribunales Lucía E. Rivas Alvarado, Gerente de la Secretaría de Planificación y Desarrollo Institucional Orfa Julieta Rivera Acedo, Jefe de CENADOJ Olga Roch Muñoz, Secretaria del Juzgado Cuarto de Primera Instancia Civil y Mercantil de Guatemala Carlos Rojas Loarca, Mediador del Centro de Mediación de Quetzaltenango Carlos Ruiz, Tesorero Nora Torres, Gerencia de Administración Financiera Silvia Patricia Valdez, Juez Primero de Primera Instancia Civil y Mercantil de Guatemala 80 Santos Alejandro Villatorro, Coordinador de la Unidad de Evaluación del Desempeño de Jueces y Magistrados, Consejo de la Carrera Judicial Amelia Yoc, CENADOJ, Directora de Estadísticas Hector Zapata, Notificador del Juzgado Cuarto de Primera Instancia Civil y Mercantil de Guatemala PCU Personnel Napoleon Guix, Sub-coordinator, UMOJ Carmen Elena Enriquez, coordinator for institutional development (component A) Juan Luis Morales, coordinator for infrastructure Miguel Pichiyá, coordinator for case management software PCU Consultants: Eladio Alcázar Martín, Indra (firm competing for several contracts under the Project) Jorge Mario Andrino Grotewöld, Comisión Presidencial para la Reforma, Modernizacion y Fortalecimiento del Estado (COPRE) and ex-consultant to project Robert Bonnafon, National Center for State Courts William Kasbach, National Center for State Courts Carlos Guillermo Monsalvo Torres, Gerente Operaciones (ISI ­ software firm) Fernando Jordan Flores, case management software Jorge Obando, National Center for State Courts (judicial inventory study) Danilo Ovalle, Consultor (ISI ­software firm) World Bank Keisgner Alfaro, CMU2 Antonio Leonardo Blasco, CMU2 Lisa Bhansali, AFTPR, former LCSPS and former MINUGUA staff member 81 Waleska García Corzo, Economist, CMU2 Monica Lehnhoff, Procurement Analyst, CMU2 Neeta Sirur, Country Manager, Guatemala United Nations Development Program (and UNOPS) Esau Beltrán, UNOPS, Guatemala Josefina Coutiño, former judicial branch employee, former advisor to UNDP judicial reform programs (PROFED), and current consultant to USAID judicial reform project in the Dominican Republic Eduardo Tercero Muxi, UNOPS, Guatemala Sergio Caceres, UNOPS, Guatemala Ana Luisa Rivas de Douma, UNDP/Guatemala Other Multilateral and Bilateral Donors Alvaro Ferrandino, former head of Public Defense (Costa Rica), currently Chief of Party for Checchi/USAID judicial reform program in Guatemala Jose Garzón, Democracy Officer, USAID/Guatemala Mario Yano, IADB, Guatemala NGOS Franklin Erick Juárez Elías, Comunidad Jurídica de Occidente República de Guatemala Luis Ramírez, Observatorio Judicial Karen Wagner, Asociación de Investigación y Estudios Sociales (ASIES), Guatemala 82