ICRR 12909 Report Number : ICRR12909 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 05/27/2008 PROJ ID : P047039 Appraisal Actual Project Name : Gt Judicial Reform US$M ): Project Costs (US$M): 49.7 35.6 Country : Guatemala Loan/ US$M ): Loan /Credit (US$M): 33 33 Sector Board : PS Cofinancing (US$M ): US$M): 13.1 0 Sector (s): Law and justice (100%) Theme (s): Legal services (40% - P) Law reform (40% - P) Other accountability/anti-corr uption (20% - S) L/C Number : L4401 Board Approval Date : 10/22/1998 Partners involved : UNDP, Government Closing Date : 06/30/2004 06/30/2007 of Sweden Evaluator : Panel Reviewer : Group Manager : Group : Gita Gopal Rene I. Vandendries Ali Khadr IEGCR 2. Project Objectives and Components: a. Objectives: The development objective of the project was to create a more effective, accessible and credible judicial system that would foster public trust and confidence and improve consistency and equity in the application of law . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The specific components and their output indicators as stated in Annex 1 of the PAD were: Institutional Capacity Improvement (WB financing included: planned US$10.7m, actual US$11m): The aim of this 1.Institutional component is to improve justice administration through : (i) New management and organizational model designed, discussed, approved and tested . (ii) Judiciary Branch administratively restructured; (iii) Standards for managing case flow established; (iv) Human resources development program on the new project administration (including training) established and tested; (v) Judicial Career law approved and applied; (vi) Inventory of cases prepared and judicial statistics developed; (vii) Court delay reduction program in Guatemala City established and experimentally tested; and (viii) Judicial Information system designed, established, and experimentally tested in Guatemala City courts and two pilot locations. 2. Transparency (WB financing included: planned US$2.3m, actual US$0.2m): (i) Anti-corruption commission established and tested; (ii) Ethics code prepared and approved; (iii) Department of Supervision of Courts reorganized; (iv) System to register lawyers assistants developed and tested; (v) Incentive systems for judges reviewed; and (vi) Oral procedures in other areas of law expanded 3. Strengthening Access to Justice (WB financing included: planned US$12.5m, actual US$15.4m): (i) Policy on traditional conflict resolution mechanisms prepared, discussed and approved; (ii) Formal and alternative dispute resolution mechanisms expanded; (iii) Number of judges trained in local cultures and partnership; (iv) Program of the new role for judges of the peace operational; (v) Number of judges increased in rural areas; (vi) Infrastructure master plan prepared, approved,and implemented; (vii) Two regional centers, two social departmental centers constructed, and justice of office peace courts constructed or rehabilitated and operational; and (viii) Number of participation subprojects provided and implemented . 4. Social Communication and Strengthening of the Modernization Commission (WB financing included: planned US$5.2m, actual US$4.9m) This was mainly to finance the PCU and included expenses for training, communications, and for monitoring and evaluation : (i) Long-term social communication and information sharing strategy developed and operational; (ii) Channels of internal and external communication and coordination developed and made operational; (iii) Office of Public Affairs (with multilingual communication capabilities ) established to prepare press releases, radio addresses and TV spots on progress in the modernization process; (iv) Modernization Commission trained in change management and coordination matters; (v) PCU functions and responsibilities clearly established; and (vi) Adequate quality assurance monitoring and evaluation capacity developed; (vi) PCU Financial management and reporting system automated; and (vii) Annual modernization report published. ** Note: At the time of preparation of the ICR, the project had only disbursed US$ 31.8 (including a front-end fee (IBRD) of US$0.3m) but was later fully disbursed, although the review is unable to ascertain the categories to which the remaining US$1.2m were disbursed. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project's closing date was extended three times (to June 2005, December 2006, and June 2007) mainly because of delays in procurement, and especially in construction projects . The ICR notes that construction was delayed because of difficulties in acquiring land from the municipalities and the discovery in one case that the land was unsuitable for the planned construction .This required significant additional investments to prepare the site and added another year to the time required to complete the project . A second significant change noted by the ICR involved two reallocations of funding among the major cost categories, with funds moved from consulting services and training to goods and works, and a final reallocation of funding during the grace period . Funds were also reallocated for equipment at the request of the client, thereby some activities requiring consultant services undertaken toward the end of the project were severely under budgeted, causing procurement delays, cancellation of at least one procurement, and disputes over deliverables . Donor financing of US$13.1m did not materialize as planned. The ICR does not explain why the donor financing fell through.The official counterpart budget was US$ 3.6 million, but only US$2.6 million was spent as planned. 3. Relevance of Objectives & Design: Project objectives were highly relevant. Project objectives of creating a more efficient, accessible, and credible judicial system – were and remain highly relevant to national, global, and Bank assistance strategies . Improving the justice system and increasing its accessibility to indigenous citizens were highlighted in the Peace Accords in Guatemala and in the overall development plan for the post -conflict period. Project objectives were designed in a participatory manner enhancing borrower ownership . In 1997, with World Bank support, the Judiciary Branch Modernization commission (appointed by the Supreme Court of Justice ) carried out several consultations in which 32% of judges participated, as well as 100% of managerial personnel working in the administrative area . In addition, interviews were conducted with several sectors such as government, law schools and the legal community, the economic sector and the media . Opinions gathered from these consultations highlighted the fundamental problems in the judicial system and this fed into the preparation of the Modernization Plan for the Judicial Branch (1997-2002). Improving access to justice and the quality of justice has only increased in importance as a global priority over the past decade. Project objectives and design were developed with full consideration to the assistance provided by other development partners. The project and its PDOs were fully consistent with the CAS emphasis on advancing the post-war social reconstruction process and modernizing the public sector . Moreover the CAS in effect at the time of project preparation and appraisal specifically emphasized judicial reform as part of its support to the Peace Accord commitments. Project Design : The project was expected to finance part of the comprehensive Judicial Modernization Program that outlines an appropriate sequence and budget for each activity with performance monitoring indicators . The PAD notes that a parallel effort of the Bank took stock of donor assistance programs, which was mainly directed towards criminal justice reform, police reform, and human rights and legal /constitutional reform. The PAD included a results framework consisting of about 30 output indicators, three outcome indicators, and two impact indicators . While the definition of the indicators was poor and the PAD could have better described the analytical framework for Bank support, it made an attempt at the design stage to formulate some indicators . The project design also set aside funds for perception surveys - one common method of understanding progress in judicial reform . The PAD identified key development risks correctly, but this recognition was not adequately reflected in the project objectives or the design . The wide ranging and multiple objectives left the project overly ambitious for a five year period and severely challenged existing capacity in the country . Thus, the review finds that while the objectives were highly relevant, the ambitiousness and complexity of project design reduced the overall relevance of the project . 4. Achievement of Objectives (Efficacy): Overall, the development objective of the project was to create a more effective, accessible and credible judicial system that would foster public trust and confidence and improve consistency and equity in the application of law . The review rates the achievement of objectives as ' modest ', considering the progress in the two largest components (1 and 3), which contributed to an overall improvement and some institutional strengthening over the 'wholly chaotic prior situation" at the beginning of the project (ICR, pg. 30). Admittedly, it is difficult to establish the counterfactual, but the support was provided at the request of the government and the Bank support of US$ 33m was expected to finance only a small part of the US$ 165m modernization program, constituting the largest contribution to the program. A component-wise review of results is provided below : Strengthen the Institutional Capacity of the Judiciary Branch Judiciary ca reer law passed : The most important contributions were (i) the enactment of critical legislation and constitutional reforms – the Judicial Council and Career law, a law covering civil service judicial employees; and (ii) increases in the court budget and salaries . Without these changes, many activities contemplated in the PAD would have been difficult to implement . Thus, a firm basis for judicial reform was established. A transparent, competitive, and merit -based process of recruiting judges was instituted . A new Disciplinary Board now investigates and sanctions ethical breaches . Between 2000 and 2006, the Board received 6120 complaints (2605 of the complaints were not admitted ). Those admitted resulted in 898 verbal and written reprimands, 1172 suspensions, 211 firings, and 1583 transfers (numbers are taken from the ICR and do not fully add up). While it is true that there are no benchmarks to measure the results and that transparency needs to be improved (for example, why were so many requests turned down ), the project has contributed to the establishment of a functioning institution that can be strengthened over time, provided the Government is committed . A new case management system (verified during the ICR mission to be of good quality ) has been introduced although it needs strengthening on several counts . The Ministry of Justice opines that it will lead to efficacy and efficiency, but the ICR notes that this will involve significant capacity building to ensure data quality and relevant and user -friendly statistical analysis . Views of internal and external users reflects some improvement . While comparable surveys were not undertaken,a 2007 survey used a sample of 376 respondents drawn from “internal and external users� of the courts within Guatemala City, Quetzaltenango, Huehuetenango, Escuintla and El Pet én. Internal users (202 respondents) included: judges, secretaries, mediators, and administrative staff . External users (174) included: lawyers, community leaders and civil authorities . Areas relating to judicial performance (access to information, accountability and transparency, attention to the public, judge's performance ) remained the same according to the majority of those surveyed . However, the majority considered that mobile courtrooms and processing of antecedents penales had led to improvement in the functioning of the judiciary . Internal users also perceived that the human resource management system had improved . Overall, some critical outputs and intermediate outcomes related to institutional strengthening of the judiciary branch have been achieved . The efficacy of this component is considered to be substantial . Support for Anti -Corruption and Transparency Measures : A judicial code of ethics has been developed and published, and training workshops for judges were carried out. anti -corruption commission has been established in December 2002. 561 complaints against judges An anti- were received, out of which 149 were admitted. There were decreases in times to register lawyers and notaries or to receive certificates of no criminal record. Higher "salaries (introduced independently of the project, but equated by UMOJ with an improved incentive system) and the stricter entry standards introduced by the judicial career law may also have produced improvements in efficacy and efficiency ". Studies were carried out on introduction of oral procedures in preparation for a draft law and some pilots attempted. However, it is not clear that the outputs have led to measurable outcomes . The Worldwide Governance Indicators, however, show no improvement in perception indicators on different counts between 1999 and 2006. Another survey conducted for the World Development Report in 2005 on constraints to doing business found that 80 percent of 453 Guatemalan firms felt corruption was the major constraint (the highest rate among all 53 countries surveyed). It is also unclear whether the Commission is able to function in a credible or effective manner that has reduced perceptions on the prevalence of corruption. Thus, the outputs have not yet been translated into outcomes . The achievement of objectives is therefore considered to be modest for this component . Strengthened Access to Justic e: The project increased the physical access to court and related services throughout Guatemala by constructing and equipping new justice of the peace courts, departmental and regional centers, and mediation offices throughout the country, and initiating a program of mobile courtrooms (so far only in Guatemala City and Quetzaltenango ). The project directly contributed with the construction of 6 justice of the peace courts in Petén (as planned in the PAD), one of the two planned departments, the remodeling of 11 others, the mobile courtrooms, the construction of 16 mediation centers in the interior and the central office in Guatemala, and the construction of two regional centers . As a result of Bank and other donor support, all municipalities have Justices of Peace . Seventy Alternate Dispute Resolution Centers (ADRCs) ( the project financed 16) ADRCs ) (the 16 ) have been set up and are functioning . The ICR notes that statistics supplied by the central ADR office for several months in 2007 indicate that the 70 centers process an average of about 10 cases monthly, although some do as many as 30 mediations. The center keeps fairly good statistics and does some basic analysis – there is however a need for a strategy to enhance demand for these institutions . The ISRs note that a policy on traditional conflict resolution mechanisms was developed and tested . However, the ICR clarifies that this was beyond the power of the project to accomplish, but that the judicial modernization plans list several actions, which if taken, may constitute a policy . The ICR considers that there still is a lack of studies on traditional systems essential for policy development . This subcomponent was sma ll (US$200 200 ,000) US$ 200, 000 ) and expected to carry out subprojects (totaling up to about US$50,000 for each subproject) through the promotion of partnerships with donors, non -governmental organizations and other participants, for developing broader public participation in access to justice programs such as free legal assistance for the poor, access to justice for young people and women, analysis of costs of litigation and other related activities that will promote social reconstruction and trust in post war conflict areas. However, instead it seems to have financed 450 workshops to prevent lynchings.There is no impact evaluation of these activities, although some evidence in project -related documents suggests that less Guatemalans approve of lynching as a form of summary justice after the program. Two programs to expose school children to justice were also developed,and various workshops on justices of the peace, gender, and community practices organized . These were all centrally initiated and organized, although local authorities were asked to participate . Most importantly, under this sub-component, the project initiated a program of mobile courtrooms (only in Guatemala City and Quetzaltenango); ten more are ready to be initiated . Mobile courts can help to meet the justice demands of a dispersed population, especially indigenous peoples, and this program was appreciated by those surveyed. Surveys noted above suggest that access to courts and dispute resolution have been enhanced, particularly through the introduction of mobile court services and new court houses . However, physical access to justice does not mean access to justice . Although it is likely that the above steps will lead to improved justice, the contribution of this component to overall project outcomes is assessed to be modest . Social Communications, Modernization Commission and PCU : This component mostly financed PCU activities. A review of indicators #21-28 in the ICR indicates that progress was modest in this area . Most importantly, an annual modernization report was published, the PCU FM reporting systems were automated (80 percent), and channels of internal and external communication and coordination were developed and made operational (100%). 5. Efficiency (not applicable to DPLs): No information to assess efficiency . The ICR states that such an examination would have been possible but does not explain the framework or attempt to calculate efficiency retroactively at project closing . The total Bank budget for appraisal and supervision amounted to US$ 1.25m, constituting 3.7 percent of the total project costs . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: First, attribution of the results to the Bank is challenging given that there were many donors that contributed to such results. For example, justices of peace have been established in all 331 municipalities, while the Bank aimed to and mainly contributed only to the construction of 6 of these (see PAD). Thus, the Bank was one of the development partners that contributed to the overall progress of the program and the overall results must be seen in this light . In such instances, it becomes important to rebuild the causal chain between inputs, outputs, and outcomes . Based on the substantial relevance of this project for Guatemala, the review of efficacy as indicated through establishing the causal link, and based on a reading of the Borrower's ICR, this review upgrades the rating for project outcomes to moderately satisfactory. This upgrading also takes into consideration the very low starting points in Guatemala, and the achievement of some priority intermediate outcomes that have helped to strengthen basic legal institutions in the country (namely, establishing a legal framework for institutional strengthening of the judiciary, introducing a transparent, competitive, merits based recruitment system for two tiers of judges, the introduction of a case management system, the introduction and operationalization of the ADR system and the mobile court system, the processing of the antecedentes penales , and the establishment of the anti -corruption commission and the initiation of the work ). Finally, although there are problems with the manner in which the surveys were conducted, internal respondents perceive some progress in 5 of the 6 areas commented upon. External respondents were less positive but nevertheless felt that there was positive improvement in several of the 13 areas they were surveyed on (these include the initiation of the mobile courtrooms, the processing of antecedentes penales , the improvement in physical infrastructure, the registry protocols, and attention to violence against women ). These outputs and intermediate outcomes are likely to lead to the desired project outcomes . For the above reasons, the project outcome is rated as moderately satisfactory. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The CAS 2007 notes that several risks (political grid lock, societal tensions, natural disasters, and macroeconomic shocks) continue to exist. However, the ICR notes that "the next operation will focus on using the organizational base created by its predecessor to push forward the PDO by further strengthening judicial oversight, planning, and implementation capacity as regards administrative and judicial operations, advancing the adoption and use of the case management software, encouraging steps to break down the barriers to access, and improving the entire human resource management system ". Therefore, the review assesses the risks to development outcomes as modest. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at Entry was moderately satisfactory. Significant attention was paid to ensure borrower ownership (see relevance section above ). There was advance preparation with the Project Coordination Unit being set up and key administrative staff in place before project effectiveness . Procurement training was provided to the PCU staff with the GPN published in the Development Business of the United Nations . Procurement documentation was being prepared to initiate activities once the project became effective . The first year work plan was drafted before project effectiveness. The PAD had a results framework which stated key indicators . Although they were not the most optimum indicators and not well-defined, there was an attempt to measure outputs and go beyond outputs to outcomes and even impact. (it was not followed up during implementation, but this is rated in the next section .) The PAD suggests that several alternate options were considered in designing the project . Supervision Quality was unsatisfactory for the several reasons . First, focus on development impact was weak . Only two PDO indicators were tracked (credibility and expanded presence of the courts ) in subsequent ISRs. The Bank failed to implement key studies early enough to utilize them for the project and also failed to follow -up on early activities as to encourage further progress . Interviews with the PCU manager indicated that he or she was unclear about the stated indicators . Finally, although there was some donor collaboration, coordination with donors that did not work through the project PCU was low, with donors often promoting "different and not entirely compatible models for the activities they financed " (pg. 17). Second, supervision of Safeguard aspects was weak The ISRs provide little management or staff attention to indigenous People's issues, although IP weak. candor and quality of performance reporting is not evident consultants have been involved in supervision . Third,candor from a review of the ISRs and the disconnect between the last ISR rating and the ICR rating shows some lack of management attention to the project . As of 06/13/2007, the ISR reports for the project ( #1-#18, the last one prepared on 01/05/2007) assess that development outcomes and project implementation progress have been satisfactory. After three extensions of the closing date, the final ICR mission assesses outcomes as moderately unsatisfactory and Bank performance as 'unsatisfactory' . This significant disconnect between the previous 18 ISRs and the final ICR rating on outcome seems to reflect negligent attention to project implementation and progress in development outcomes . Overall, therefore, Bank performance is rated as moderately unsatisfactory . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Overall, the review finds Government performance was moderately satisfactory . One, it supported the passing of some critical laws. Two, it ensured that the PCU also implement the judicial portions of the IADB loan and coordinate donors on the construction of infrastructure - this was a sound decision. However, there were a few key shortcomings in Borrower performance . One, the Government could have also coordinated with donors that did not work through the PCU. The lack of such coordination led to inconsistencies in systems and processes introduced. Two, the land issue for construction seems to have been badly managed causing significant procurement delay and affecting overall outcomes in some areas . Three, government commitment to achieving the outcomes (as opposed to outputs) was not consistently evident . The PCU performance was considered to be moderately satisfactory based on the ICR's quantitative indicators . The PCU was well staffed (with continuity in staffing). The PCU published regular annual modernization reports . In general. the ICR notes that its handling of FM and procurement matters was validated by a series of Bank audits and supervision, although there were two contract disputes at closing and complaints of irregularities during project implementation. However, there were shortcomings . The implementing agency should have better revised its M&E framework. The Implementing Agency should have also better coordinated its activities . The ICR notes, for example, that it found several examples of redundant practices within the administration (for example four offices collecting their own judicial statistics ), failures to use or even be aware of the potential to combine forces or utilize each other ’s databases, and lack of knowledge of relevant studies done by the PCU (for example the case inventory which should have interested several offices ). Third, it focused more on administration rather than on policy areas, which was its main mandate . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: In addition to 3 impact indicators, Annex 1 of PAD had stated key outcome indicators as : (i) Percent increase in user confidence. (ii) Percent increase in Judiciary (iii) Branch coverage and access outside Guatemala City . (iv) Number of corruption related complaints received and resolved through the Anti -Corruption Commission. (v) Policy on traditional justice systems developed and tested . (vi) Number of subprojects executed through the participation program. It also mentions another 30 output indicators. Not all of them were well defined. No baseline indicators were provided, and these indicators were not measured through project implementation in a systematic fashion . The analytical framework is also unclear and the links between outputs, outcomes, and impact is not clearly enunciated in the PAD. Some surveys were undertaken as planned but these were not consistently undertaken, focused on outputs, and badly planned making assessment and comparison difficult at closing . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Indigenous Peoples were affected by the project . The PAD had noted that the project was designed consistent with the 'Identity and Rights of Indigenous Communities Accord which is consistent with the Bank's then OD 4.20 (currently OP 4.10 on Indigenous Peoples). It had stated that the modalities have been identified to alleviate these problems and beneficiary assessments will be carried out . Additionally, innovative projects were planned in predominantly indigenous regions . Although an IP specialist is found participating in missions and in the final ICR preparation, there is little or no information on whether this project impacted IP groups in any manner . There is only some limited evidence provided on the increase in people speaking IP languages in the court system and significant sensitization training for court personnel and judges on indigenous law . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The project was successful in Unsatisfactory Satisfactory generating a majority of the planned outputs that have led to the achievement of some key outcomes stated in the PAD. Above all, it helped to lay a firm basis for reform in a very difficult environment. Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: A results framework needs to be carefully formulated at project appraisal, and realistic and measurable indicators need to be included all within an analytical framework which explains the linkages between the outputs, outcomes, and final impact. It is very important for Bank management to then ensure that the task team focuses on ensuring that systems are in place for monitoring the indicators and assessing both development outcome and implementation progress. It is inadequate to only recognize political economy issues; it is equally important that such risks influence the formulation of more realistic project objectives and that indicators are included that can signal whether the risks are materializing. 14. Assessment Recommended? Yes No Why? Very important lessons can be learnt as to whether the several outputs generated by this Bank project were translated into the desired outcomes and the final ratings can be adjusted . The audit findings can feed into the larger IEG judicial reform evaluation. 15. Comments on Quality of ICR: The ICR quality is satisfactory with some qualifications . While the ICR must be commended for its candidness and detailed review, there are some issues of inconsistency and lack of evidence in a few instances . Additionally, the ICR could have better focused on the Bank project rather than focus on broader sectoral issues . Finally, there seems to be a disconnect between the numerical ratings for the indicators and the overall ICR ratings . Specifically: There is an inconsistency between its reporting on how indicators were met and the final rating (see also Government comments on subject.) In a few cases, the ICR provides opinions or judgements without adequate back -up data. For example, it states that a "change involved the disappearance of the donor co -financiers from the counterpart funds ", but does not explain the implications of such 'disappearance' or why this happened . Similarly it questions decisions by the task team (such as decision to prior review of contracts (pg 37) or not using two 'specialists in LCSPS with substantial knowledge of Guatemala' (pg 38)) without explaining why such prior review was unnecessary given its own suggestions that "inadequate Bank supervision " contributed to procurement problems or without explaining how the specialists would have better addressed the issues . Its strict interpretation of attribution is rather unusual for ICRs . This is a regular problem that all ICRs have to contend with and most address this issue by focusing on project objectives, inputs, outputs, and outcomes to establish the causal chain as stated above . The ICR does not do this in all cases . For example, the ICR notes that it does not give credit to the Bank support for legislation such as the 'career law' . It is unclear why this is so, when the PAD clearly notes that it will support this output . The ICR could have better addressed the attribution issue by linking the outputs it has identified to the desired outcomes . The ICR refers to standards against which 'contemporary judicial reform or any institutional development effort should now be judged' but fails to utilize the ICR to establish an helpful analytical framework for results assessment. However, overall, the ICR presented a candid and thoughtful summary of the results of Bank support and addresses all the key issues. a.Quality of ICR Rating : Satisfactory