Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) Report Number : ICRR0021325 1. Project Data Project ID Project Name P118028 AF: ARTF-2nd Judicial Reform Project Country Practice Area(Lead) Afghanistan Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-12533 01-Jun-2017 24,220,340.86 Bank Approval Date Closing Date (Actual) 21-May-2012 30-Nov-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 40,000,000.00 40,000,000.00 Revised Commitment 24,220,340.86 24,220,340.86 Actual 24,220,340.86 24,220,340.86 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Robert Mark Lacey Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components a. Objectives According to the Grant Agreement (p. 5), the project's objectives were "to increase access to and use of legal services." A June 2016 restructuring reduced the project's scope and changed outcome indicators and targets accordingly. This Review will therefore perform a split assessment. At the time of restructuring, US$ 13.47 million, or 54.6% of Bank financing, had been disbursed. Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 09-Jun-2016 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The project initially contained four components: 1. Partnership for Justice (appraisal, US$ 13.05 million; restructuring 0; actual 0). This component was to increase the scope and quality of legal services (dispute resolution, deed registration, and legal information) in selected provinces and districts, concentrating in geographic areas with promising economic activity, and therefore with demand for legal services from small and medium enterprises or from communities impacted by development projects. It was to improve collaboration and capacity of local justice institutions (JIs), engaging them and other legal service providers in community justice partnerships. Focus was to be placed on courts and Hoqooqs (units in the Ministry of Justice, MoJ, that mediate in civil disputes/cases). Technical assistance was to cover: (a) preparation of a 3-4 year provincial justice reform strategy; (b) building the capacity of state JIs to respond to local needs; (c) creation of a community justice partnership; and (d) promotion of users' involvement with legal institutions through information centers and new practices such as publishing court decisions, organizing public hearings, etc. At the June 2016 restructuring, this component was dropped due to challenges in the judiciary's engagement with community-based justice providers, institutional capacity limitations, a prolonged leadership gap within the JIs, and the security situation. Overall, only activities that were reasonably advanced or that were backed by political commitment to completion by closing were retained. From this component, operating costs for established provincial Legal Aid Offices were shifted to the second component, and some limited renovation and construction activities were shifted to the third component. 2. Legal Empowerment (appraisal, US$ 17.55 million; restructuring US$ 2.8 million; actual US$ 2.8 million). This component was to increase people's ability to participate in the public (state) redress or dispute resolution system. It contained two subcomponents: (a) Legal Aid was to strengthen and expand the state legal aid system by consolidating existing legal aid sub-systems and building policies, procedures, and capacities for an effective, fiscally sustainable, nationally coordinated state system of legal aid. Technical assistance and investment were to focus on MoJ (Legal Aid and Hoqooq Department) and an Independent Legal Aid Board. (b) Legal Outreach was to help the Afghan people acquire and incorporate legal knowledge into their choices and interactions with the government and private actors, improve their ability to monitor and influence legal service providers, and secure their support for Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) reforms. It was to strengthen the capacity of the MoJ, Supreme Court (SC), and Attorney General's Office (AGO) to plan, finance, and manage legal awareness activities, including development and dissemination of print and broadcast media materials. Initial implementation of this component was hampered by the complex political transition and lack of practical strategies for public outreach. At the June 2016 restructuring, the component was streamlined and scaled back into activities involving development of a legal aid regulatory framework and roadmap, support to legal aid providers, and establishment of a legal information call center. The Legal Outreach subcomponent was dropped. 3. Organization and Capacity of Justice Institutions (appraisal, US$ 47.4 million; restructuring US$ 18.4 million; actual US$ 18.4 million). This component was to improve the capacity of state JIs to manage and deliver legal services. It included two subcomponents: (a) Effective Organizational Design was to determine and encourage structural reform in the sector. It was to adjust courts' and prosecutorial offices' functions, jurisdictions, size, and territorial organization to current needs; incorporate independence and accountability safeguards into the organizational design, operations, and processes of courts and offices of public prosecution; and streamline procedural designs. Technical assistance was to include analytic work, periodic user surveys, consultations with stakeholders, and policy and law development, including the development of a Strategic Plan for MoJ. (b) Strengthening Capacity of Justice Institutions was to improve the capacity of state JIs to plan, finance, and manage resources, focusing on human capital (pay and grading reforms in SC and AGO, and development of human resources strategies and reforms for MoJ, SC, and AGO), case management (development of an automated court case management system for the SC), and physical infrastructure (development of a capital investment plan and infrastructure design standards, and construction of courts and offices). The complex political transition and resultant delay in political buy-in created challenges for initial implementation of this component. As of the June 2016 restructuring, the revised component streamlined activities to focus on those judged to be critically important to the functioning of JIs and feasibly completed by project closure, including development of a human resources management strategy for all JIs, training needs assessments and selected training focused primarily on judges, and some construction and information technology projects. Case management automation, strategic planning for MoJ, and most of the originally planned infrastructure activities were dropped. 4. Implementation Capacity (appraisal: US$ 7.5 million; restructuring US$ 3.8 million; actual US$ 3.8 million). This component was to facilitate project implementation, and in the longer term strengthen JIs so that they can carry out reforms and other activities without significant additional assistance. The June 2016 restructuring continued support for the project management infrastructure, consolidating separate Project Units in MoJ, the SC, and the AGO into a single Project Support Unit (PSU). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) Project cost and financing: The project was initially financed by a US$ 85.5 million Grant from the Afghanistan Reconstruction Trust Fund (ARTF). Funds were to be separated into three tranches: Tranche 1 (US$ 40 million), Tranche 2 (US$ 30.5 million), and Mid-Term Review (MTR) Conditional Funds (US$ 15 million). The latter two tranches were to be released only if MTR targets were met. The June 2016 restructuring, following the January 2016 MTR, cancelled US$ 60.5 million of the Grant, eliminating the first component and downsizing others as indicated in Section 2d. Of the remaining US$ 25 million, US$ 24.7 million was disbursed, with the difference due to currency fluctuations. No Recipient contribution was expected or made. Dates: The project was approved on May 21, 2012 and became effective on May 31, 2012. The MTR was held in January 2016, leading to a restructuring on June 9, 2016 that cancelled the majority of project financing, downsized activities, and revised the results framework, institutional arrangements, implementation schedule, and financing and procurement plans. A May 13, 2017 restructuring extended the project's closing date from June 1, 2017 to November 30, 2017, at which point the project closed. 3. Relevance of Objectives Rationale At the time of appraisal, Afghanistan was a fragile state where mechanisms to hold the government accountable for delivery of basic services were undermined by corruption and lack of capacity. Many complementary, competing, and conflicting spaces for rule setting and conflict resolution existed, with most disputes settled through community organizations whose relationship to the state was poorly defined. There was no mechanism to preclude recycling disputes through different forums, encouraging "forum shopping" and enabling conflicting decisions. Within the formal court system, most cases were appealed to the SC level, meaning that nine SC judges were responsible for issuing decisions on almost 80% of all cases; this level of appeals undermined the legitimacy and effectiveness of the lower courts. Development of a formalized, professionalized, and effective justice system at all levels, including accountability mechanisms through which citizens could pursue peaceful resolution of grievances and hold the state responsible for its actions, was a prerequisite for progress toward stability, economic growth, and poverty reduction. The project's objectives reflected these country conditions. The project was also relevant to national strategy and Bank strategy at the time of appraisal. It was intended to implement the government's National Priority Program in the justice sector, the 2012 "National Program for Law and Justice for All," whose objective was "to restore the faith of Afghans in the ability of the law to protect and defend their best interests as individuals and as a nation." Specific elements of that program included institutional strengthening, increased access, outreach to traditional justice systems, and support for other key government programs such as counter-narcotics, anti-corruption, and land tenure reforms. The project was also aligned with the Afghanistan National Development Strategy (2008-2013), which identified improved institutional capacity to deliver sustainable justice services, improved coordination and integration within the justice system, and improved quality of justice as main Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) goals. The Bank's Interim Strategy Note (FY 2012-2014) specifically included support for equitable service delivery, highlighting intervention in the justice sector. The Project's objectives somewhat miscalculated the political realities and capacity. The 2017-2020 Country Partnership Framework (CPF) calls for a more cautious and nuanced approach to justice sector reform (ICR, p. 15). The overall relevance of the project's objectives is assessed as Substantial. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Increase access to and use of legal services (original outcome targets) Rationale The project's original ambition envisioned broad support to conceptualize and implement a reform strategy for the overall justice sector. Project design combined supply-side (capacity building, human resource development, and physical infrastructure improvements to facilitate specialization and collaboration among providers; alignment of the structure, organization, processes, and capacities of JIs to the actual perceptions and need of users) and demand-side (increased information to users, and availability of formal dispute resolution services at minimal cost) interventions intended to combine to increase access to and use of formal legal services. These interventions were to contribute directly to increased access to, and use of, legal services by improving the available scope and quality of services, and indirectly by improving the productivity and accountability of legal service providers (which was to increase citizen confidence in the services, making people more likely to use the formal legal system rather than "forum shopping" through various community-based options). Outputs: Supply-side: • Two "Partnerships for Justice" workshops were held, but no provincial reform strategies or community justice partnerships were approved, not meeting the original target of four strategies/partnerships. • No collaboration frameworks between legal aid providers and enforcement agencies (courts, public prosecutors, police) were developed as originally envisaged. Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) • A review of the JI map for optimal scale (size and location) and specialization of the system was completed, though not with project support as originally planned, since the government decided to undertake this activity directly. • A human resource strategy for judges, prosecutors, legal aid providers, and Hoqooq officers was not approved, as originally envisaged, but a human resource strategy only for judges and prosecutors (the SC and AGO) was put in place, meeting the revised target. The European Union is using the this strategy as the starting point for its benchmarks-based budget support for the AGO. Pay and grading reform was introduced for administrative and operational staff in the AGO. • A training needs assessment and capacity development plan was implemented for all JIs. Training was carried out in Kabul and Egypt (through affiliations with Egypt and Turkey) for 158 judges. This did not meet the original target of 250 judges and court staff, 200 prosecutors, and 500 other staff, for a total of 950 persons trained, and it also did not meet the revised target of 165 judges trained. • A case management system was installed in the AGO, but not in the SC as originally envisaged. • 17 buildings were constructed or renovated for MoJ, SC, and AGO offices, including the MoJ headquarters building in Kabul. 21 new law libraries were established, and another 20 libraries improved, for lawyers, judges, prosecutors, and other staff at JIs. Internet access was established in the AGOs and improved in the SC. Demand-side: • One legal aid information center (a call center in Kabul) was established and made operational, not meeting the original target of four, but meeting the target revised at restructuring. The project supported public legal awareness radio broadcasts, covering topics including concerns specific to women. • The MoJ's Regulatory Framework and Roadmap for Legal Aid was put in place, meeting the original and revised targets. This framework, developed through contracts with the Asia Foundation and Open Society Foundation, facilitates government decision making on allocation of legal aid resources across the country based on need. The project provided salaries and operational support for legal aid providers and relevant consultants. Female staff were employed to provide consultations for female citizens. • No legal outreach units were established, not meeting the original target of seven. Outcomes: The project's original outcome indicators were broadly defined and lacked baselines and targets: "scope and quality of legal services increased"; "productivity of legal services providers improved"; and "accountability of legal service providers enhanced." Outcome indicators on productivity and accountability of legal service providers have no information. Legal outreach activities were cancelled. Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective Increase access to and use of legal services (revised outcome targets) Revised Rationale The theory of change under the revised outcome targets was tightened considerably to focus on the provision of legal aid services. The establishment of a legal aid information center, supported by physical infrastructure improvements and the development of state legal aid roadmap, were to make more services available to citizens at low or no cost, and the development of a human resources strategy for judges and prosecutors as well as training for judges was to enhance the system's capacity to process cases. Outputs are described above, under the discussion of achievement under the original outcome targets. Outcomes: At restructuring, the indicators related to accountability and productivity of legal service providers were dropped. The indicator measuring increased scope and quality of legal services was refined to "number of legal aid cases per month directly supported by the project." This indicator gauged direct project support for legal aid cases. This support involved 680 cases/month by project closure, exceeding the revised target of 650. In total, project-funded legal aid providers represented 22,065 clients in 21 different provinces. This indicator is considered to be an adequate measure of the objective to increase access to and use of legal services. Revised Rating Substantial PHOVRLEFFRATTBL Rationale Efficacy is rated Modest under the original outcome targets. The original outcome indicators were intended to measure the scope, quality, productivity, and accountability of legal services and providers, but they were poorly specified, and lack information on some areas. The project did not achieve its original, broad set of intended outcomes related to provincial justice reform and community justice partnerships, institutional reform and capacity development across the justice system at all levels, and legal aid and outreach strengthening. Efficacy is rated Substantial under the revised outcome targets. The scope of the revised project was narrowed primarily to increasing the availability of low-cost legal aid services. The project exceeded the target for the single Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) revised outcome indicator (number of cases covered monthly by project-supported legal aid), and this indicator is considered to be an adequate measure of access to and use of legal services. Overall Efficacy Rating Primary reason Modest Low achievement 5. Efficiency The EPP did not conduct a formal economic analysis due to "lack of data on the justice system, the long term nature of seeing results arise from investment in the justice sector and so forth" (p. 95). It qualitatively discussed sector benefits to accrue from reduced costs and increased productivity of the formal justice system, including facilitation of economic activity through improvements in the business environment. The ICR similarly did not attempt an economic analysis. The ICR (p. 20) highlighted some administrative efficiencies: the consolidation of project implementation units into a single office following restructuring; close supervision by a field-based task team leader (TTL), allowing quick resolution of issues related to construction and renovation; and completion of planned activities following restructuring, with no cost overruns. The project team later explained that relatively low construction costs were due, at least in part, to the government covering security costs. However, the ICR also pointed to sources of implementation inefficiency prior to restructuring (separate project units in each of the JIs, resulting in duplication of services and higher costs), and cost comparisons with similar construction projects financed by other donors relied only on anecdotal evidence. Due to incomplete evidence, Efficiency is rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) 6. Outcome The Project's objectives were substantially relevant. Efficacy was modest under the original objective and substantial under the revised objective. Efficiency is rated Modest due to incomplete information. These ratings combine for an Outcome rating of Moderately Unsatisfactory under the original targets and Moderately Satisfactory under the revised targets. Disbursements under the original targets was 54.75% and under the revised targets was 45.25%. Therefore the split rating is: (0.5475x3) + (0.4525x4) = 3.4525. This rounds down to 3 and overall outcome is rated as Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The project's primary intervention supported legal aid service providers. The ICR (p. 19) stated that it is unlikely that these providers will be able to continue services at the current level in the absence of external financing. Government financing is extremely limited, though the Ministry of Finance has allocated some resources to MoJ in the current fiscal year to fund some of the providers; the project team later stated that this funding is likely to continue into the next fiscal year as well. Similarly, many of the human resource management and institutional reforms are likely to stall or backslide without the technical advisers financed by the project. Overall, political commitment to the project's outcomes is dependent on the new administration that will come to power following presidential elections in 2019. However, the judicial learning system has been strengthened through the establishment of affiliation arrangements with training institutes in Egypt and Turkey. In addition, some of the construction and civil works projects have reduced JIs' expenditures on rent, possibly allowing them to focus ongoing budget requests on higher-priority needs. 8. Assessment of Bank Performance a. Quality-at-Entry Key lessons (EPP, pp. 6-7) were gleaned from previously attempted rule-of-law reforms in Afghanistan, including a predecessor ARTF-funded project that focused on the basic needs of the JIs (establishing implementation structures, building procurement capacity, assembling baseline data, and strengthening human capital): the need to involve the entire range of JIs, to condition release of resources on the achievement of specific performance targets, to pursue a design approach that considered institutions' individual needs for structural/organizational reform and absorptive capacity, and to rely on partner countries with an Islamic legal culture. Overall risk was assessed as High (EPP, pp. 50-57), stemming from general country-level risks (security, political instability, leadership turnover, and slow implementation of correlated reforms), project stakeholder risks (deeply entrenched legal pluralism and forum shopping, Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) uneven engagement across institutions, and limited citizen access to project information due to high illiteracy rates), implementing agency risks (weak capacity and low engagement of provincial and district- level authorities), and governance and project monitoring risks (environmental and social risks related to construction and rehabilitation, weak transparency and accountability in JIs, and weak supervision and monitoring mechanisms). Mitigation measures included the tranche financing mechanism, a participatory approach to provide space for consensus building, implementation of pay and grading reforms throughout the JIs, and a focus on building strong implementation structures at all levels. However, as the ICR noted (pp. 23, 27), there were key gaps in project design. The Bank team did not undertake the necessary political economy analysis to clarify the impact of the political transition on the judiciary, and also to understand the lack of political will for broader reform of the justice system, particularly at the sub-national level where community justice mechanisms were most strongly rooted. In particular, the timing of the transition was not taken appropriately into consideration. The project was therefore unrealistically ambitious; given weak capacity and the slow implementation progress of previous projects, a more incremental approach would have been appropriate. The project's results chain and outcome indicators were weakly specified (see Section 9a), limiting early warning signs of challenges during implementation. In addition, the ICR (p. 24) noted that the project was hindered by lack of English language competency among many Afghan judicial staff; appropriate design would have included provision for translation of documents. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision Prior to restructuring, the Bank team undertook regular implementation support missions, and issues were identified and frankly assessed in Implementation Status Reports. However, movement toward restructuring was insufficiently prompt. The MTR, initially scheduled for May 2014, was delayed for almost two years by prolonged absence of leadership in all three JIs, where interim appointments and post-election vacuums in 2014 impacted ownership and commitment; the project team later elaborated that the JIs did not have directors with constitutional authority to make decisions like restructuring. The security situation also impacted supervision, as there were provinces that the Bank team could not access at all; overall M&E implementation was also negatively affected by security concerns as well as limited capacity (see Section 9b). Following restructuring and assignment of an in-country TTL, the quality and impact of supervision improved, though access and field visits remained constrained. The sustainability of project achievements was inadequately addressed during implementation. Quality of Supervision Rating Moderately Unsatisfactory Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design M&E was to be primarily the responsibility of the PSU, which was in turn to work with the staff of the Project Units in the three JIs. The EPP (pp. 41-46) laid out a series of outcome and intermediate outcome indicators, but the outcome indicators were only vaguely specified ("scope and quality of legal services increased"; "productivity of legal service providers improved"; "accountability of legal service providers enhanced"), no baseline or target data were provided, and it is not clear how these broad indicators were intended to measure the objectives of increased access to and use of legal services. Many of the intermediate indicators captured progress in areas that could not be directly attributable only to the project. However, the output-oriented MTR targets (on which release of the second and third funding tranches were to be conditioned) were clearly delineated (EPP, p. 16). b. M&E Implementation At restructuring, indicators were revised and downsized to reflect changes in the design and scope of project components. For example, training targets were reduced and focused only on judges rather than all government legal sector practitioners (judges, prosecutors, and MOJ personnel). Progress was recorded in Implementation Status Reports and aides-memoire, but due to lack of access to field visits and even to the head offices of the JIs (which were primary targets of insurgent groups), the task team was unable regularly to collect detailed data and update progress. Periodic external reviews of the project were supposed to help to compensate for these gaps, but "evidence of external reviews is lacking" (ICR, p. 25). Bi-annual progress reports were prepared, but their format did not allow for tracking of project progress, and the PSU "insisted that reporting was not possible because of the absence of baseline data" (ICR, p. 25). In addition, as the ICR noted (p. 25), weak conceptual capacity at the policy-making level, limited technical capacity at the implementation level, the lack of an M&E strategy prevented the government from adequately assessing the project's achievements. Following restructuring, the assignment of a field-based TTL, as well as the scaling down of project-financed activities, improved the functioning of M&E, though M&E reports were still not timely. c. M&E Utilization The absence of regular and timely reporting meant that M&E data and analysis were not used to impact decision making in any significant manner. M&E Quality Rating Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) Modest 10. Other Issues a. Safeguards The project was classified as category B for environmental assessment purposes. As well as Environmental Assessment (OP/BP 4.01), the Involuntary Resettlement (OP/BP 4.12) safeguard policy was triggered. Construction activities, both new and rehabilitation of existing infrastructures, were planned for four provinces. An Environmental and Social Management Framework was developed and disclosed in April 2012, with Environmental and Social Management plans put in place for each site. A Resettlement Policy Framework was also developed and disclosed in April 2012 (Emergency Project Paper, p, 87). Land acquisition or relocation did not occur, as all construction work was carried out on available government land. Overall, "no complaint or non-compliance cases were recorded" (ICR, p. 26). The project team later confirmed that there was full compliance, to the best of its knowledge, with safeguard policies. b. Fiduciary Compliance The project complied with all fiduciary covenants (ICR, p. 26). Internal control arrangements were put in place, and adequate financial management, procurement, and disbursement systems were maintained. Quarterly interim unaudited financial reports were submitted to the Bank on time. The project's final audit, carried out by an internationally affiliated firm, was unqualified, but some weaknesses were noted: no consolidated fixed asset record for the overall project was kept; the records of fixed assets lacked traceability features; and the ceiling amount of the operational advance was not strictly followed by the project team. All procurement packages were completed according to the approved procurement plan, though there were extensive delays. A central procurement facilitation service was established to handle large-value procurements, and a Procurement Policy Unit embedded procurement specialists in the JIs to provide technical assistance, oversight, capacity building, and assistance for complex procurements. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) Reason for Ratings ICR IEG Disagreements/Comment The ICR's main text (p. 20, 54) Moderately Moderately finds outcome to be MU. Outcome Unsatisfactory Unsatisfactory There is no discrepancy in ratings. Preparation did not fully take into account political economy factors related to justice Moderately system reform, and those Bank Performance Unsatisfactory Satisfactory risks materialized. The project's results framework and M&E arrangements had major shortcomings. Quality of M&E Modest Modest --- Quality of ICR Modest --- 12. Lessons The ICR (pp. 28-29) offered several useful and insightful lessons, including the following (restated by IEG): Country strategy usefully guides a project's level of ambition as well as its specific objectives and activities only if that strategy is grounded in a realistic political economy and security assessment. In this case, a realistic appreciation of political and institutional constraints did not appear until the 2017-2020 Country Partnership Framework. Political economy analysis cannot rely solely on interaction with a relatively small group of reform-minded counterparts who many not have a realistic view of the reform landscape, especially at the sub-national level. In this case, the underlying political situation was inadequately considered in the original project design, and it later became clear that many counterparts did not fully understand the nature of the reforms envisaged by the project and/or were reluctant fully to express objections during the design phase. Incremental design can usefully pair basic infrastructure with institutional reform. In this case, some of the institutional actors were most interested in investments in infrastructure and equipment, and these were used as incentives to adopt human resources and pay/grading reforms. This approach could be more broadly and deliberately adopted. Concern about sustainability of recurrent costs cannot wait until a project's conclusion. In this case, in a situation where most of the country's population cannot afford legal assistance, project support for legal aid to indigent individuals cannot be sustained in the long run. This project might have considered a transition to a hybrid system combining the services of non-governmental organizations and private lawyers alongside the state-run legal aid system. Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: ARTF-2nd Judicial Reform Project (P118028) 13. Assessment Recommended? No 14. Comments on Quality of ICR The abstract was a useful overview of the project's storyline and challenges, and the main text offered a clear, evidence-based assessment of project achievements. The lessons were candid and well derived from project experience. However, the ICR's analysis was not consistent in the specification of the objectives being assessed (for example, p. 7 cites "increasing productivity of legal service providers" as the project objective), the amount of Bank financing that had been disbursed at the 2016 restructuring (stated as US$ 16.12 million, p. 2; US$ 11.12 million, pp. 16, 20; US$ 13.47 million, p. 54; the project team later explained that the different figures were system-generated because of advances in the designated account, but these inconsistencies could have been eliminated prior to finalization of the ICR), or recording of the project's Outcome rating (inconsistent between Moderately Satisfactory on the cover sheet but Moderately Unsatisfactory throughout the main text). No formal economic analysis was attempted, and construction cost comparisons relied only on anecdotal evidence. a. Quality of ICR Rating Modest Page 14 of 14