Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) Report Number : ICRR0021244 1. Project Data Project ID Project Name P126791 AG - Pub&Soc Sctr Transformation Country Practice Area(Lead) Antigua and Barbuda Social Protection & Labor L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-82690 31-Dec-2018 2,564,254.27 Bank Approval Date Closing Date (Actual) 05-Jun-2013 30-Sep-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 0.00 Revised Commitment 2,564,254.27 0.00 Actual 2,564,254.27 0.00 Prepared by Reviewed by ICR Review Coordinator Group Antonieta Romero- Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Follette 2. Project Objectives and Components a. Objectives The operation had three Project Development Objectives (PDOs), namely: “(i) strengthening [the Government of Antigua and Barbuda's (GOAB’s)] capacity in managing public policies and the public service; (ii) improving the efficiency of social protection spending through an integrated monitoring and targeting system; and (iii) improving the income and employability of the vulnerable population through temporary employment and training programs." Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project consisted of five components: Component 1. Build effective institutions for strategic management of government policies (appraisal estimate US$1.06 million, actual cost US$0.44 million). This component was to strengthen GOAB decision-making capacity to undertake planning coordination that facilitated the more effective alignment of resources with Government policy priorities. Activities to be financed included technical assistance, training, workshops, and consultations to public servants, as well as the development of websites. Component 2. Modernize human resource management (appraisal estimate US$2.47 million, actual cost US$0.40 million). This component was to provide technical assistance geared towards: (1) the establishment of an initial integrated public employment register, (2) the design of a single public service structure, and (3) the development of an effective Public Service Management System. Component 3. Improve the efficiency of social protection spending (appraisal estimate US$1.21 million, actual cost US$0). This component was meant to support the Ministry of Health, Social Transformation, and Consumer Affairs. Shortly after taking office in June 2014, the current administration expressed its wish to cancel this component, and consequently no activities were implemented. Component 4. Support Active Labor Market Programs (ALMPs) (US$4.10 million, actual cost US$ 0.88 million). This component was to support the Ministry of National Security and Labor (MNSL) in: (a) the establishment of a Temporary Employment Program (TEP) to help less experienced unemployed individuals find temporary employment and then enter or reenter the labor market, (b) establishment of a training program (TP) for carrying out subprojects, and (c) strengthening of the MNSL employment and counselling service, its capacity to operate a competence certification system, and its capacity to coordinate, monitor, and evaluate ALMPs. Activities envisaged were to increase employability: (i) training to beneficiaries of the TEP, (ii) training to participants in the TP, (iii) internship activities in subprojects to participants of the TP, and (iv) provision of equipment to the Labor Department. The project closed in September 2017 without implementing the TP or the activities to strengthen the MNSL. This explains the much-reduced actual financial cost when compared to the cost at appraisal. Component 5. Project management (appraisal estimate US$1.14 million, actual cost US$1.25 million, increased in 2017). This component was geared towards supporting project implementation through two activities. The first was financing the staff of a Project Management Unit (PMU) to be established within the Ministry of Finance (MEFEPA) at an initial stage and later be transferred to be a Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) function of the MEFEPA, which did not happen. The other activity involved financing management of the fiduciary and administrative aspects of the project, including the contracting of annual audits. Actual cost varied considerably from appraisal estimates in part because the GOAB did not absorb the PMU as planned, keeping it instead financed by the project and staffed by consultants (ICR para 46). The initial PMU was never transitioned over to PMU staff becoming civil service employees. In addition, cost also varied because the Bank had the PMU manage two additional grants not related to the project (ICR para 31). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The total project cost estimated at appraisal was US$10 million. The actual cost was US$2.56 million, due to partial or non-implementation of activities. An undisbursed amount resulted from the project cancellation in 2017 (upon request by the Borrower), as the Bank and a new administration could not agree on restructuring parameters of the project. According to the ICR, “a final restructuring, dated August 2016, only included Components 4 and 5. This restructuring was never processed, and in July 2017, the Government sent a formal request to cancel the loan entirely” (ICR, para 14). Consequently, the unwithdrawn loan balance in the amount of US$7.01 million was cancelled, as was the Project. Financing: The World Bank financed 100% of the project, committing the US$10 million of the loan. US$2.56 million was actually disbursed. Borrower contribution: No Borrower contribution was planned or made. Dates: There was one proposed Level 2 restructuring, dated August 2016, that would have reduced the project to only Components 4 and 5. That restructuring was not enacted, shortening the closing date to September 30, 2017, 15 months earlier than the original closing date of December 31, 2018, and resulting in the cancellation of the unwithdrawn loan balance of US$7.01 million. 3. Relevance of Objectives Rationale The project involved a complex set of public sector reforms touching the three areas reflected in the PDOs. At the time of appraisal, the PDOs were relevant and were in line with the priorities set in the World Bank Regional Partnership Strategy (RPS) covering FY2010–2014. Modernization of the public sector was one of the relevant three areas of engagement of the RPS (the other two being competitiveness and resilience). As of the time of this ICR Review, this modernization remains relevant under the current RPS for 2015– 2019. The PDOs were also consistent with the National Economic and Social Transformation Plan, as both sought to have strong social protection programs, capable of protecting the economically disadvantaged Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) and vulnerable (for example, crisis-affected) populations and to provide income and activation support to the unemployed (PAD, para 6). In 2014, national elections brought in a new government with a new set of priorities. Consequently, the project lost relevance as the vehicle to achieve the modernization of the public sector. Instead of strengthening “capacity in managing public policies and the public service” (PDO1), the new GOAB preferred to address public sector inefficiencies through initiatives such as e-government and improving productivity, both of which were activities outside of the project. In addition, improving the efficiency of social protection spending (PDO2) was no longer a priority for GOAB and therefore PDO2 became irrelevant. The new GOAB wanted to focus loan proceeds to support more employment and jobs (ICR, para 17 and 18). Only PDO3 “improving the income and employability of the vulnerable population through temporary employment and training programs” maintained some relevance given that the new GOAB remained tepid on the type of employment it wanted the project to support. As a result, only the temporary employment program under component 4 remained partially relevant, while components 1, 2, 3 (under PDOs 1 and 2) and the other activities of component 4 ceased to be relevant. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Strengthening Government capacity in managing public policies and the public service Rationale Strengthening the GOAB’s public policy management capacity was to happen through the creation of a policy unit under the Cabinet Secretary and a regulatory framework for policy coordination and strategic planning. With both outputs, the project intended to support the alignment of resources with Government policy priorities. For the modernization of human resource management, the project was to support the creation of an integrated public employment regime and the development of an effective public sector management system. PDO1 was not achieved because fewer activities than envisioned were implemented, and no targets were met. A policy unit was not created, and the project proposals did not get the Cabinet to approve any policies. The Cabinet did not pursue any of the project-proposed recommendations in the Regulatory Framework for Policy Coordination and Strategic Planning, the initial policy paper on public service modernization, or the functional reviews of key Government departments (ICR, paras 19-21). Outputs on the Regulatory Framework for Policy Coordination and Strategic Planning included: an initial Cabinet Manual outlining Cabinet procedures, templates, and the roles and responsibilities of all participants in the new system; training to staff of the Cabinet Secretariat and key ministries on the content of the manual and on a Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) new Cabinet process; a proposed regulatory framework for policy coordination and strategic planning; and training in policy-making to a select group of policy analysts from line ministries. Outputs on modernization of the public service that did not yield any change due to cancellation of activities included: data incorporated in the public employee register, a plan for staff optimization, and analysis of capacity-building needs to define organizational strengthening. Neither a training plan to meet these needs, nor legislation for a new Public Service Management System, took place. Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Improving the efficiency of social protection spending through an integrated monitoring and targeting system Rationale To achieve PDO2, the project was to support the creation of an integrated monitoring and targeting system, which was to help the Government enhance its capacity to design, implement, and monitor social policies and interventions. According to the ICR, PDO2 was not achieved because associated activities ceased to be a development priority for the GOAB, and actions to improve efficiency of social protection spending were dropped. There was no change in the number of civil servants covered by the Public Service Act, and the targeted 55 percent of the uncovered population of Antigua and Barbuda continued to remain outside these regulations and regime. Similarly, no action was taken to develop a unified targeting system and a single registry for beneficiaries (ICR, para 22). Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Improving the income and employability of the vulnerable population through temporary employment and training programs Rationale Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) PDO3 was to support active labor market programs by having qualified beneficiaries participate in a TEP and a TP. The project was also to support the strengthening of a one-stop employment center. Both interventions were to lead to increases in beneficiaries’ labor income and placement 40% higher than for those who did not enroll in these programs. The ICR (para 23) noted that the project partly achieved PDO3. The project sought to provide knowledge and skills to 300 beneficiaries through the TEP and the TP; the latter was not executed. The following outputs did materialize: (i) The TEP contributed to improving the income and employability of 136 low-income beneficiaries, i.e., 43% of the target (from 3 cohorts), of whom 39 were women (30% of the target). (ii) The project delivered the intended basic vocational training that included math and remedial English, as well as life-skills training, to 136 participants. (iii) 98 participants who completed the TEP also completed the six- month subprojects program envisaged by the project, of whom on average about half (56% of the first cohort, 40% of the second, and 66% of the third) responded that the knowledge gained from the program would be beneficial to their job search. Female beneficiaries who completed the TEP reported incomes 81% higher than those in the control group (ICR para 30). (iv) The One-Stop Employment Center strengthened after TEP staff were hired by the Center. The following outputs did not materialize: Cohort 4 TEP training and subprojects program, strengthening of the MNSL's capacity to operate the competence certification system, and strengthening of the MNSL's capacity to coordinate, monitor, and evaluate ALMPs. Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Rationale None of the PDOs were achieved due to the post-election GOAB’s lack of endorsement of the first two PDOs and limited commitment to the activities supporting PDO3 as stated in the loan agreement. In spite of some favorable outputs, the unresolved disagreement between the Bank and the GOAB, as well as new GOAB priorities not reflected in the PDOs, prevented the restructuring of the project, and hampered achievement of the original project development objectives. Overall Efficacy Rating Primary reason Negligible Low achievement 5. Efficiency The ICR reported one major shortcoming related to efficiency, specifically, the lack of continuity in the GOAB’s political will to support the reforms embodied by the project. In addition, GOAB's non-compliance Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) with the legal covenant to absorb the PMU into the civil service (starting in October 2015), and other factors, led to inefficient resource use. For the first two components of PDO1, no observable benefit was realized from the US$840,000 invested in preparing a regulatory framework for policy coordination and strategic planning, a draft policy paper on public service modernization; and the functional reviews of key Government departments. As noted above in section 4, these outputs did not foster the government's buy-in to undertake the reforms the project was set to support. The US$880,000 investment in the TEP enabled a total of 136 beneficiaries to participate in subprojects in three of the four planned rounds of TEP training. Of the 300 targeted, 98 beneficiaries completed the program (ICR para 27). The TEP’s overall cost per beneficiary (US$6,470/participant) was higher than estimated due to several factors unaccounted for during project preparation. A lower-than-estimated number of beneficiaries met the qualification requirements, and therefore cohorts were smaller than expected. To enhance the TEP’s efficiency, following recommendations from a 2015 evaluation of Cohort 1, the number of beneficiaries per subproject was reduced, the level of fees was increased, and funds were allocated for remedial life-skills training to mitigate/alleviate psycho-social issues of participants that were disrupting the program. Following additional recommendation from the 2016 evaluation of Cohort 2, the TEP also funded transportation and child care stipends to foster women's participation, as well as an increase in supervisors' remuneration. Finally, the planned training of Cohort 4 did not happen, leaving the remaining 202 planned beneficiaries without training (ICR Annex 7). The ICR also pointed out that management costs at closure rose to US$1.25 million, 9.64% higher than the appraisal estimate (US$1.14 million) and, according to the ICR (para 31), higher than World Bank standards. In addition, other elements of execution inefficiency were: no consistent government focal point for the operation (ICR, para 44), staffing issues with the PMU (para 46); and limited information sharing between government agencies (para 47). The ICR also noted that the overall project cost was much lower than anticipated because of the cancellation of many or all activities intended to achieve PDO1 and PDO2. However, project administration costs increased because, contrary to the original plan, the PMU's financing remained part of the project budget, and because the PMU also unexpectedly had to manage two unrelated World Bank grants. The information above provides support for this Review to conclude that the efficiency with which the project's resources were used was negligible. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives is rated Modest, as the government withdrew its commitment to two of the three objectives in 2014. Achievement of the first two objectives was negligible, as the GOAB did not pursue recommendations for public sector reform, and it stopped interventions to enhance the social protection system. The third objective to improve the income and employability of vulnerable populations was partly achieved, as employment-support training was implemented but fell short of targets. Efficacy is therefore rated Negligible. Efficiency is rated Negligible due to low value-for-money, inadequate cost-effectiveness, and project management cost overruns. These ratings are indicative of severe shortcomings in the project's preparation and implementation, consistent with an Outcome rating of Highly Unsatisfactory. a. Outcome Rating Highly Unsatisfactory 7. Risk to Development Outcome The ICR indicated that PDO1 and PDO2 are unlikely to be pursued going forward because the associated goals and activities are no longer priorities of the GOAB. In addition, the Government and the Bank have not engaged in dialogue to pursue further either public sector reform or enhancement of social protection systems. Scaling up the income and employability of the vulnerable population through the TEP is also unlikely. Even though temporary employment continues to be a priority for the GOAB, according to the ICR, “the Government has not continued supporting the TEP since the project’s cancellation. The Government’s own national employment program remains the main tool to provide support to unemployed citizens. None of the design features of the TEP have been incorporated into the national employment program as of the writing of the ICR” (para 66). It appears highly unlikely that the TEP supervision framework will be used for other projects. 8. Assessment of Bank Performance Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) a. Quality-at-Entry Bank specialists from the public and social protection sector teams prepared technically sound appraisal studies on the civil service and social safety net situation. To assure efficacy, they based the design of the project on Bank, Caribbean-specific, and international best practices about public sector and social protection efficiency enhancement (PAD para 53). The team used public sector management lessons from the Caribbean Center for Development Administration. Among the key lessons applied to strengthening social protection were good practices from the Dominican Republic's unique beneficiary system to address inefficiencies in targeting mechanisms (PAD paras 36-38). To support the labor market by alleviating unemployment of vulnerable populations, the project followed mainstream knowledge asserting positive impacts of temporary employment programs such as the TEP (studies by Ravallion [1998]; Jalan and Ravallion [1999]; and Galasso and Ravallion [2003]), which have proven to be effective mechanisms to provide participants a higher income than they would receive without such a program. The PAD noted that the project was the first loan of the Bank to the GOAB, and that there was no prior operational performance experience on which to assess implementation risks. Recognizing the weakness of GOAB’s systems to comply with Bank fiduciary and procurement requirements, implementation risk- mitigating arrangements were integrated into the structure of the project. These included technical assistance and close Bank collaboration and support to supplement the Borrower’s limited experience with Bank processes and the demands of the project’s multisectoral approach (PAD para 51). The appraisal team managed to secure strong champions at the highest level of government, including the Minister responsible for public service reforms, but not strong commitment at other levels of the civil service or society. More importantly, perhaps, exchanges with representatives of the opposition parties were limited when defining the project's goals (ICR para 41 and 42). The team did structure the project to address the risk of the Borrower failing to meet its commitments. Namely, “the Bank [was] prepared to monitor Government commitment during implementation and calibrate disbursements against results achieved by the Borrower” (PAD Annex 4). However, given the imminent national elections and the project's politically sensitive reforms, it is not clear from the PAD whether the Bank had a contingency plan regarding discontinuity of priorities in the event of a new government taking office and having different goals than those of the outgoing government. Such a situation materialized, rendering most of the project irrelevant. It should be noted that Antigua and Barbuda holds elections every five years (or when the Prime Minister calls them), and at the project’s approval stage in 2013, national elections were expected to take place the following year. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision Overall, the Bank project team performed adequately in monitoring initial activities and in implementing the recommendations highlighted by TEP surveys. Under PDO3, among the changes to the TEP, as noted before, were the inclusion of transportation and child care fees, and the proposed inclusion of additional Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) subprojects. When requested by the Borrower, the Bank’s performance was effective in providing timely clarifications and advice on fiduciary aspects, specifically related to procurement. After the 2014 national elections, when the new authorities showed no interest in pursuing recommendations in the regulatory framework reform and other initial studies under PDO1, or in pursuing PDO2 and requested a restructuring in June 2014 (ICR para 14), the Bank and government counterparts engaged in dialogue. They worked for over a year to restructure the project to satisfy the new Government's priorities while adhering to Bank policies. A restructuring paper, dated August 2016, canceled all components except for the part of Component 4 that financed the TEP. However, in view of the substantial changes requested by the GOAB, the lag in implementation, and the challenges of a lengthy Level 1 restructuring, during Bank-IMF Annual Meetings 2015 the Bank then proposed that the project be cancelled outright and a new project be developed. The Borrower responded to this proposal by formally requesting, in July 2017, that the entire loan be cancelled. This Review observes that Bank-country dialogue began soon after the new GOAB took office and requested revisions of the PDOs, indicating that it did not intended to pursue with the originally-agreed course of action (ICR paras 12 and 14). The ICR noted that: “The Bank could have done more to resolve the paralysis/impasse affecting the implementation and restructuring of the Project, particularly since this was the first lending operation in the country and the only vehicle for Bank dialogue" (ICR para 64). This Review concurs with the ICR, however, that the Bank had room to suggest termination of the project at an earlier date, i.e., right when the Borrower voiced its disagreement with the project’s design, rather than attempting an almost full remake of the project and waiting for the government to respond, putting the project “in problem status for over two years” (Restructuring Document). Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD described the M&E arrangements and a Results Framework with indicators to be monitored for each PDO component. The Cabinet Office and the Establishment Department were responsible for the M&E functions for Components 1 and 2 under PDO1, with support from the PMU and the Public Sector Transformation Unit. To offer efficient support to Active Labor Market Programs (PDO3), the project was to develop and administer end-of-activity surveys to monitor implementation and enhance the design of the TEP training activities. It was also to administer one tracking survey on the impact on the placement rate and income levels of beneficiaries of the employment programs. Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) However, there were shortcomings. The ICR noted that “the indicators for public sector transformation were difficult to measure and not well defined (that is, outcome indicator 3); [the project] trusted the M&E process for two components to tools that still had to be created, which is not best practice, and ultimately, the tools were never created” (ICR paras 48-49). This Review notes that indicator 1 in the Results Framework, i.e., % of policies approved by the Cabinet aligned with the medium-term strategic plan, presented an attribution issue, as there are various factors besides the project that could have affected government decision-making capacity or planning and coordination. b. M&E Implementation None of the M&E benchmarks or milestones were met due to the lack of commitment from the Borrower creating a slow pace or no implementation of activities. The ICR noted that the limited activities under PDO1 were carried out only during the first months of the project, and thus there was little or no activity to monitor, and there was similarly no activity to monitor toward PDO2. None of the planned progress in the development and implementation of an integrated monitoring and targeting system took place (ICR para 22). Concerning PDO3, as noted earlier, the team administered post-training assessments, and the recommended modifications were incorporated to enhance the TEP´s effectiveness. Also, the 2016 Mid-term Review noted strong demand by participants for subprojects originally not eligible under the original TEP, which were going to be added had the project continued (para 54). According to the ICR, the quality of the TEP evaluation reports was adequate and improved over time by including an analysis of data gathered by the Labor Department Technical Unit. Monitoring of project implementation was negatively affected by short-staffing of the PMU, which had difficulties in carrying out adequate consolidation of M&E data from the technical units and in preparing progress reports. Because it was not incorporated into the Government, the PMU faced challenges also to adequately coordinate data sharing. c. M&E Utilization The project's M&E arrangements were not fully used because of the limited or complete lack of project activities under PDO1 and PDO2. As stated in the PAD, for PDO3, in particular the implementation of the TEP, post-activity follow-up surveys were deliberately created and were able to capture suggestions from the participants to enhance the TEP’s design and effectiveness. The ICR noted that the Borrower would have carried out a second tracer study had the project not been cancelled (ICR paras 50-51). There were no data gathered to inform project management and decision making (ICR para 52), though the Mid-term Review highlighted parameters of the TEP that needed to be changed to reach the benefits desired by the new administration with the support of the project. M&E Quality Rating Modest Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) 10. Other Issues a. Safeguards The PAD indicated that the project did not trigger OP 4.12 Involuntary Resettlement. Complying with the PAD's prohibition (para 63), there was no land acquisition (para 63). There were no major environmental impacts expected from the originally approved subprojects under the TEP (PAD para 65), and so there was no need to include environmental safeguards. The project was cancelled before the Borrower’s request to add two types of subprojects with limited environmental impact took effect. b. Fiduciary Compliance The ICR noted that financial management was moderately satisfactory (para 55), primarily because the understaffing of the PMU affected financial management and procurement. There were no compliance issues, but there were delays as well as errors and incomplete information in the reports. "Challenges and delays in procurement stemmed from inexperience of the Borrower in conducting its first World Bank lending operation. [As anticipated at appraisal,] the World Bank provided substantial capacity building and training to the PMU staff and regularly accompanied the PMU during closing" (ICR para 56). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Highly Unsatisfactory Highly Unsatisfactory --- Bank Performance Unsatisfactory Unsatisfactory --- Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) The ICR identified a number of lessons emerging from the project's experience (section V, paras 67 to 71): 1. "Vanilla" operations are optimal when engaging with a first-time Borrower with limited institutional capacity. Breadth and complexity of reforms matter, and these must be matched with the capacity of the Borrower to undertake change. It is not advisable to engage with new Borrowers in major politically-sensitive reforms that involve complex multi-sector operations. Similarly, dialogue and implementation are more challenging when dealing with Borrowers that are unfamiliar with fiduciary, procurement, management and supervision requirements of Bank operations. 2 . "The chain is as strong as its weakest link." Change management at lower levels of government needs to be strong. In the face of public sector reforms, potential resentment between established civil service and non-established public employees, as in the case of Antigua and Barbuda, is something that needs to be thoroughly studied. At project preparation the degree of resistance to change cannot be overestimated, especially in view of upcoming elections and the likelihood of change of leadership and overarching priorities. 3. Instead of aiming for large overall, public sector reforms, in a context such as Antigua and Barbuda’s, a focus on technical aspects can tailor decisions to the Borrower context. In this case, difficult redistributions of authority in governance of the civil service, in particular from the Public Service Commission to a new entity responsible for civil service management, were unlikely to be adopted. 4. New Borrowers require intensive and continuous Bank engagement. This project confirmed that that first-time operations call for early, constant, and intensive support from appraisal throughout implementation of an operation. Lessons suggested by IEG: 1. Support for reform champions among stakeholders is key to moving forward a process of change. When the scope of reform requires new mindsets, culture and attitudes at all levels, from leadership to lower-level civil servants, anticipated outcomes are highly unlikely to be achieved in the absence of broad stakeholder ownership and commitment. The Bank should be ready to face political challenges, provide support for allies, and become involved in additional dialogue to help unlock impasses. 2. Political economy analysis is paramount during project preparation in an environment of imminent elections to understand, and perhaps even anticipate, potential discontinuity of interest in reforms initiated by an outgoing administration. 13. Assessment Recommended? Yes Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AG - Pub&Soc Sctr Transformation (P126791) Please explain Assessment of reform change. Lessons could emerge about factors that caused the Bank-borrower dialogue to stall. A political economy study. Lessons could also emerge about what needs to be understood and even anticipated in public sector modernization projects facing imminent elections and the possibility of uncommitted new leaders. 14. Comments on Quality of ICR The ICR was clearly written with an overall comprehensive description of the design, implementation, and results of the project. The ICR gave adequate explanation of results from partial activities under PDO1 and PDO3, as well as a description of the project's administration. The lessons were logical and adequately derived from the experience of project performance. The ICR highlighted particularly well the lessons' relevance to assessment of performance risk when dealing with new small Borrowers. There were, however, some shortcomings: (i) There was limited explanation on the dialogue around the restructuring of the project (e.g., details on the challenges, timeliness of response by either the Borrower or the Bank, influence of Bank champions to persuade the GOAB, etc.). (ii) The ICR did not elaborate on the Bank's rationale for launching the project right before elections versus waiting a few months, in view of the risk of the new GOAB discontinuing interest in a politically sensitive reform as pursued by the prior administration. a. Quality of ICR Rating Substantial Page 14 of 14