IEG Report Number: ICRR14895 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 03/23/2016 Country: Timor-Leste Project ID: P092484 Appraisal Actual Project Name: Planning And Project Costs (US$M): 37.27 31.75 Financial Management Capacity Building Program L/C Number: Loan/Credit (US$M): 7.00 7.48 Sector Board: Public Sector Cofinancing (US$M): 25.10 24.32 Governance Cofinanciers: Australia, EC, New Board Approval Date : 03/21/2006 Zealand, Ireland Closing Date: 07/10/2011 01/31/2014 Norway Sector(s): Central government administration (95%); Sub-national government administration (5%) Theme(s): Administrative and civil service reform (29%); Public expenditure; financial management and procurement (29%); Tax policy and administration (14%); Other accountability/anti-corruption (14%); Macroeconomic management (14%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Judyth L. Twigg Robert Mark Lacey Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Financing Agreement (p. 4), the project's objective was "to support the Recipient's Ministry of Planning and Finance, its agencies, line ministries and district entities involved in financial management to strengthen planning, budgeting, public expenditure management and revenue administration with emphasis on efficiency, effectiveness, accountability, integrity, service culture, and transparency." The Project Appraisal Document (PAD, p. 9) states the objective slightly differently: "sustainably strengthened planning, budgeting, public expenditure management and revenue administration for growth and poverty reduction, with emphasis on efficiency, effectiveness, accountability, integrity, service culture, and transparency." In keeping with IEG/OPCS guidelines for differing statements of a project's objectives, this Review will evaluate achievement of the objectives as expressed in the Financing Agreement. At a February 2010 restructuring, the project's objectives were revised to "strengthen capacity in the Ministry of Finance for prudent, effective and accountable planning and management of public finances to promote growth and poverty reduction" (Program Paper, p. 3). The revised objectives narrowed the scope and focus of the project, and therefore this Review will perform a split rating. At the time of restructuring, US$ 7.31 million of Bank funds and US$ 4 million of Recipient-Executed Trust Funds (RETF) (out of US$ 31.75 million actual total financing), or 35.6%, had been disbursed. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval: 02/19/2010 c. Components: The project originally contained five components: A. Public Expenditure Management (appraisal, US$ 15.46 million; actual at restructuring, US$ 2.17 million). This component was intended to strengthen the expenditure side of public financial management, including planning, budget formulation, and budget execution. Relevant agencies were to receive capacity building through two subcomponents, one focusing on skills, knowledge, systems, processes, attitudes, and behaviors, and the second on upgrading and integrating information technology (IT) systems into one single Financial Management Information System (FMIS). Activities were to cover a range of agencies, including not only key Ministry of Planning and Finance (MoPF) agencies, but also the Ministry of Agriculture, Fisheries, and Forestry; Ministry of Education and Culture; Ministry of Health; Ministry of Natural Resources, Minerals, and Energy Policy; and Ministry of Public Works, including at the district level. B. Revenue Administration and Macroeconomic Management (appraisal, US$ 9.61 million; actual at restructuring, US$ 2.15 million). This component was intended to strengthen the revenue side of public financial management and the macroeconomic management functions of the MoPF. Subcomponents focused on capacity building in the MoPF institutions involved in revenue collection (the East Timor Revenue Service (ETRS), Petroleum Tax Division, and Customs Service) and in economic policy, modeling, and forecasting (the Macro-Economy and Tax Policy Unit). C. Program-wide Activities (appraisal, US$ 7.05 million; actual at restructuring, US$ 0.08 million). This component was intended to enhance the cross-cutting skills and knowledge, systems and processes, and attitudes and behaviors that underpin a professional work ethic, integrity, civil service standards, communication and information flows, basic skills, and human resource functions. D. Support to Improve Governance and Management (appraisal, US$ 2.30 million; actual at restructuring, US$ US$ 1.82 million). This component was intended to ensure that the MoPF and its resources were managed in a strategic, transparent, and effective fashion, through training in leadership, management, and specific technical, supervisory, and regulatory skills for the Minister and Vice-Ministry of Planning and Finance and their offices. E. Program Implementation (appraisal, US$ 2.84 million; actual at restructuring, US$ US$ 0.92 million). This component was intended to support program administration through advisory support and training. At the February 2010 restructuring, the components were revised as follows to support only the restructured Ministry of Finance (MoF) and match its new organizational structure. Direct support to line ministries, autonomous agencies, and district entities was dropped, and the original indicative training plan was replaced with a more flexible professional development program. A. Strengthening Capacity in the General Directorate of State Finances (restructuring appraisal, US$ 7.20 million; actual, US$ 7.34 million), with support for the director general, budget directorate, treasury directorate, procurement directorate, asset management directorate, and expenditure review unit. B. Strengthening Capacity in the General Directorate of Revenue and Customs (restructuring appraisal, US$ 6.52 million; actual, US$ 6.29 million), with support for the director general, domestic revenue directorate, petroleum revenue directorate, and customs directorate. C. Strengthening Capacity in the General Directorate of Policy Analysis and Research (restructuring appraisal, US$ 2.01 million; actual, US$ 4.46 million), with support for the statistics directorate, macroeconomic directorate, petroleum fund directorate, and investment policy and analysis unit. D. Strengthening Capacity in the General Directorate of Corporate Services (restructuring appraisal, US$ 13.06 million; actual, US$ 13.64 million), with support for a flexible professional development program (providing scholarships and training courses to meet the evolving needs of the MoF), human resources, internal audit and legal services, information technology, external communications, strategic planning, and change management. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: Total project costs were estimated in the PAD at US$ 37.27 million; actual costs were US$ 31.75 million. According to the Operations Portal, differences between planned and actual costs were due to exchange rate gains and lags in filling consultant positions left vacant during the project's final extension. The project team also explained that US$ 37.27 million was an overestimate, and that the project proceeded throughout implementation within the envelope of available funds from the Bank and other donors. Financing: The project's original financing plan included a US$ 7.0 million International Development Association (IDA) Grant and co-financing of "around US$ 27.0 million" at the time of appraisal (p. 8). Australia, Ireland, Japan, and the UK were actively involved with the Government and the Bank during preparation; the PAD also listed the International Monetary Fund (IMF), Norway, New Zealand, Portugal, and Sweden as prospective development partners. The IMF and Japan were to participate through bilateral support, while the others were contemplating bilateral support or co-financing with the Bank. The project was designed as a five-year rolling program, following a "platform" approach with early steps tackling key bottlenecks and creating a foundation or "platform" for future developments, and outer years building on earlier achievements and lessons. It was explicitly recognized that capacity building of the nature and scope envisaged under the program would take years, if not decades, and therefore that support would likely be required beyond the five-year scope of the original project (PAD, p. 8). When eventually confirmed in 2010, the planned contribution from co-financiers (Australia, the European Commission, Ireland, Norway, and New Zealand) was through a Recipient-Executed Trust Fund (RETF) in the amount of US$ 25.1 million. US$ 7.48 million of the planned Bank financing was disbursed and US$ 0.1 million cancelled. Prior to the February 2010 restructuring, disbursements were primarily from the IDA Grant: US$ 7.14 million of the IDA Grant, and US$ 4 million from the RETF (Program Paper, January 28, 2010, pp, 1, 8). At closing, US$ 24.32 million of the RETF had been disbursed, with US$ 0.78 million cancelled. Additional resources of "about US$ 4.0 million" (ICR, p. 24) were provided by direct support from the IMF to the Treasury, Norwegian government petroleum resource management, and Japanese government support to Aid Effectiveness. Borrower Contribution: According to the PAD (p. 8), the Government was intended to contribute at least US 0.5 million. The ICR (p. 24) states that the Government provided in-kind contributions that "exceeded by far the original estimate," but to which it is difficult to assign a monetary value. Dates:  February 2010: A Level I restructuring amended the project development objectives to align the program with a new Ministry of Finance administrative structure and streamline its objectives, activities, and results framework.  May 2011: A Level II restructuring extended the closing date from July 10, 2011 to November 30, 2012 to allow completion of activities.  October 2012: A Level II restructuring again extended the closing date from November 30, 2012 to January 31, 2014, to maintain implementation of activities and bridge to a follow-on program financed by Australia. The total time extension was just over 42 months, an increase by 63% of the implementation time estimated at appraisal. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of the original objectives is rated Substantial . At appraisal, Timor-Leste was experiencing rapid expansion in petroleum sector revenues, putting considerable pressure on its public financial management system. Effective and efficient execution of a rapidly growing budget was seen as the country's principal potential source of growth in the short term (PAD, p. 2), but corruption, lack of skills and knowledge, and cumbersome and incomplete systems and processes were identified as key obstacles. The project's objectives were responsive to these country conditions. At appraisal, the objectives were also relevant to the World Bank Group's (WBG's) Country Assistance Strategy (2005), directly supporting its three pillars (delivering sustainable services, creating productive employment, and strengthening governance) and four principles of engagement (building institutional capacity, deepening results orientation, strengthening transparency and communication, and consolidating and extending strong partnerships established among development partners), with particular focus on the good governance pillar. The project was aligned with the Government's recognition of the public financial management system's key role in service delivery and poverty reduction, and with its integrated, three-pillar approach to institution building: skills and knowledge, systems and processes, and attitudes and behaviors (PAD, p. 4). At closing, the objectives were consistent with the government's current Strategic Development Plan 2011-2030, which focuses on institutions (including sound macroeconomic management, public financial management, and capacity development for state institutions) as one of its four pillars. The WBG's current Country Partnership Strategy (2013-2017) is constructed around this Strategic Development Plan. However, the original objectives were overly ambitious, targeting the Ministry of Public Finance and its ten directorates, five line ministries, and thirteen district entities across a wide range of outcomes. Relevance of the revised objectives is also rated Substantial . The scope was narrowed in terms of beneficiaries (only to the Ministry of Finance) and dimension (to planning and management of public finance), while the objectives remained substantially relevant to country conditions, government strategy, and Bank strategy. The inclusion of "growth and poverty reduction" in the statement of objectives, however, was ambitious, as the project's planned activities would not reasonably have been expected immediately to impact growth and poverty. The project team later explained that, in Timor-Leste, the government budget represents virtually all non-oil gross domestic product, and therefore a very strong link exists between enhancing capacity at the Ministry of Finance and economic growth/poverty reduction, even if that link is difficult to illustrate directly. b. Relevance of Design: Relevance of design under the original objectives is rated Modest . The project's planned activities were logically and plausibly linked to achievement of desired outcomes. It was explicitly recognized that short-term, off-site training programs had been ineffective in achieving capacity-building objectives in other projects, and therefore planned activities focused on mentoring, on-the-job training, and competency-based on-site capacity building. Because of extremely low capacity at the project's outset, activities were deliberately structured as "scaffolding" that could be shifted and realigned as needs were identified and additional donors became involved. There was an explicit attempt to balance technical assistance with capacity building, with early focus on the establishment of core public financial management systems and processes (so that expenditure plans could be executed and services delivered), and evolving development of on-the-job and formal training. However, despite efforts to match activities to capacity, this approach resulted in a too-wide and complex array of activities, targeted at too many different entities, that was unlikely to result in achievement of objectives. In particular, implementation arrangements and human/financial resources designated for line ministries and district entities appear to have been inadequate for achieving desired outcomes. Relevance of design under the revised objectives is rated Substantial . The 2010 restructuring mapped the components to a new organizational structure of the Ministry of Finance (MoF), reducing the complexity of project design and consolidating activities at the core central level of that single ministry. The detailed five-year training plan was revised to a more flexible Professional Development Program that could better adapt to evolving MoF needs. 4. Achievement of Objectives (Efficacy): Achievement of the original o bjective -- to support the Recipient 's Ministry of Planning and Finance , its agencies, line ministries and district entities involved in financial management to strengthen planning , budgeting, public expenditure management and revenue administration with emphasis on efficiency , effectiveness, accountability, integrity, service culture, and transparency -- is rated Modest, due to lack of evidence related to line ministries and district entities. Outputs: The project's primary contribution was the recruitment and oversight of a series of international and national advisors, mostly in long-term positions, as well a support for the Ministry of Finance’s (MoF's) reform strategy and 2011-2030 Strategic Plan. As a result of capacity developed, the number of international advisors decreased from 45 at project inception to 18 at closing. These inputs led to: Public expenditure management:  The Financial Management Information System (FMIS), procurement, and treasury systems and processes were simplified and strengthened, with an increased delegation of responsibilities to line ministries. FMIS upgrades are now handled by a dedicated staff. A Transparency Module allows real-time monitoring of spending and revenue mobilization.  MoF records were reconciled with bank accounts, Free Balance (a web-based transparency portal) was extended to line ministries, payroll was put onto Free Balance, and quarterly budget reports were published on the web. As a result of this increased transparency, public and parliamentary scrutiny of the budget process has been enhanced. A workshop held in May 2013 to set the fiscal envelope for the FY 14 budget was opened to the public for the first time.  Policy and research functions were consolidated into the Directorate General for State Finances. Revenue administration:  A detailed Customs diagnostic study was completed in 2012, providing the basis for reform and modernization of the Customs Directorate. The Customs Code was revised, and an Automated System for Customs Data was implemented. Core tax administration functions were stabilized, compliance levels were improved, petroleum tax audits were introduced, capital gains tax assessments were conducted, and customs clearance and post-clearance audit were strengthened. The average time taken to clear green line customs declarations was 30.1 hours in 2013 (no baseline available), exceeding the target of 47;4 hours. However, capacity gaps remain, mainly relating to downstream operational functions, understanding of tax rules, and rationalization of the customs legislation.  Use of the Standard Integrated Government Tax Administration System has improved, and a user support system was established. Governance and management:  The Ministry of Finance prepared and implemented an organization change strategy and Strategic Plan 2011--2030. An Organic Law for the MoF has been approved, and a resulting reorganization is under way (including Annual Action Plans). Capacity development for the MoF was implemented through a comprehensive Professional Development Program, resulting in the graduation of 59 civil servants with diploma and master's degrees, the establishment of a PFM Training Centre, overseas study visits, and the creation in 2011 of a Human Capital Development Fund.  The General Directorate of Policy Analysis and Research published analyses of national accounts from 2011-2012, sources of economic growth, revenue performance, and inflation drivers and prospects, prepared a fiscal framework and underlying long-term growth model, and laid the ground work for a Living Standards Survey and full poverty assessment. Training on macro modeling, analysis, and reporting was conducted. Information technology:  A stable operating platform was established to provide information technology support for government-wide applications and support services. Budget execution rates are now defined and included on-line. 90% of logged IT cases are now resolved within 24 hours. However, much of the support that was envisaged for corporate services (human resource management, internal audit, financial management, etc.) was delayed pending the outcome of an ongoing Ministry of Finance restructuring. Outcomes: The 2014 state budget was prepared almost exclusively by civil servants rather than consultants. Annual budgets have become smaller since 2012, aligning with spending capacity (which remains low in line ministries). 2014 budget documents are comprehensive, including relevant analysis, and are available to the public. The 2011-2014 budgets were submitted to parliament 45 days before the end of the fiscal year (with the exception of 2012, which experienced election-year delays that are accounted for by law); this met the target. The 2013 and 2014 budget documents received the maximum Public Expenditure and Financial Accountability (PEFA) score of A. The budget reflects government policy priorities, although outstanding issues remain with regard to the realism of capital budget expenditure projections. This achievement essentially meets the target. The 2013 PEFA assessment also showed clear strengthening of public financial management from 2007 to 2013, with the strongest gains in the 2010-2013 period in policy-based budgeting and tax administration (and smaller gains in internal controls, accounting and recording, external scrutiny, and budget credibility) (ICR, p. 43). Improvements were seen along 12 dimensions of the PEFA, with two dimensions showing decline and 14 no change. The ICR (p. 28) states that there have been improvements in the non-oil fiscal deficit, revenue out-turns, and ratio of domestic tax to GDP. Given project outputs, is it reasonable to conclude that the project made some contribution to these outcomes:  The non-oil fiscal deficit declined to 71% of non-oil GDP in 2010, meeting the target of less than 90% for that year, but then rose to 81% in 2011, 78% in 2012, and 98% in 2013, based on a downward revision of estimated 2013 non-oil GDP (the volatility in this indicator stems from revisions to the denominator). Annual growth in the non-oil economy has averaged 12% from 2007-2012.  Revenue out-turn increased from 75% in 2008 to 140% in 2011, 194% in 2012, and 100.6% in 2013, exceeding the target of 90%. Volatility is due to petroleum revenue. Between 2010 and 2012, US$ 300 million in additional petroleum revenues were collected based on tax assessments and audits of company submissions. Non-petroleum revenue out-turns were 101% for 2011, 103% for 2012, and 102% for 2013.  Domestic tax collection as a share of non-oil GDP dropped from 9% in 2007 to 5.4% in 2010, but then increased to 7% in 2011, 7.3% in 2012, and 9.3% in 2013.  Expenditure out-turn increased from 49% in 2006/2007 to 84% in 2011, but then fell to 66% in 2012 and 66% in 2013. The execution rate on current spending was 93% in 2010, 88% in 2011, 92% in 2012, and 87% in 2013. A 2014 budget execution law commits to a reduction in the state budget if execution is not projected to reach 75% by the end of the third quarter. All quarterly budget execution reports in 2013 were published within two months of the end of the quarter. Achievement of the revised objective -- to strengthen capacity in the Ministry of Finance for prudent , effective and accountable planning and management of public finances to promote growth and poverty reduction -- is rated Substantial, based on achievements noted above within the restructured Ministry of Finance. This Review concurs with the ICR's statement that direct project impact on growth and poverty reduction is "difficult to quantify" (ICR, p. 11). 5. Efficiency: Efficiency is rated Modest, due to significant shortcomings in implementation efficiency. The PAD (pp. 26-27, 112-113) did not conduct a traditional economic or financial analysis, but instead qualitatively listed expected program benefits, including improved planning and budgeting, budget execution, revenue collection, accurate and timely procurement, and general governance, leadership, and service delivery. No effort was made to quantify or assess the impact of these gains. The ICR (pp. 38-43) similarly analyzes proxy benefits, demonstrating that budget credibility has improved as budgets have been moderated toward spending capacity, improved efficiency in petroleum tax audits has contributed to the growth of the Petroleum Fund, and an enhanced in-house macro-fiscal framework has led to greater fiscal sustainability and multi-year perspective in planning, policy, and budgeting. However, there is no attempt to assess cost effectiveness and hence no way to judge the project's "value for money." Implementation efficiency was extremely weak in the early years of the project, when its complex structure, coupled with thin MoF management capacity, resulted in significant delays. Language barriers (as well as a low starting skills base at the MoF) limited external advisors' ability to transfer knowledge to counterparts, thereby wasting resources. In many instances, pressure to implement the MoF's reform agenda resulted in advisors taking on direct implementation roles, superseding intended support for capacity development among Timorese staff and delaying development of that capacity. Progress accelerated after the 2010 restructuring, though an MoF restructuring in 2013 (as well as an earlier restructuring in 2009) rotated technical and managerial staff in ways that disrupted day-to-day capacity building work. Turnover of external advisors also interrupted skills transfers. An independent review found that consultant salaries were within international norms (ICR, Annex 4a). The ICR (p. 18) also states that effective donor coordination made technical assistance overall more efficient; it does not provide detail to support this assertion, but cites a 2011 IEG country program evaluation. The project team explained that donor coordination was enhanced through effective government ownership via the creation of a Directorate for Management of External Assistance, through Bank-convened biweekly donor meetings, and through strong informal relationships. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Outcome under the original objectives: Relevance of objectives is Substantial, with the project's objectives tightly linked to country conditions, Bank strategy, and the institutional pillar of the government's 2011-2030 Strategic Development Plan. Relevance of design is rated Modest . While the linkage of the project's planned activities to anticipated outcomes was in principle plausible and logical, initial design was overly complex and required a streamlining restructuring mid-course. Achievement of the original objective -- to support the Recipient's Ministry of Planning and Finance, its agencies, line ministries and district entities involved in financial management to strengthen planning, budgeting, public expenditure management and revenue administration with emphasis on efficiency, effectiveness, accountability, integrity, service culture, and transparency -- is rated Modest, due to lack of evidence related to line ministries and district entities. Efficiency is rated Modest due to delays and inefficiencies in staffing and training. Taken together, these ratings are indicative of significant shortcomings in the project's preparation and implementation, and therefore an Outcome rating of Moderately Unsatisfactory. Outcome under the revised objectives: Relevance of objectives remains Substantial, with the caveat that the inclusion of "growth and poverty reduction" in the statement of objectives was inappropriately ambitious, as the project's planned activities would not reasonably have been expected immediately to impact growth and poverty. Relevance of the revised design was Substantial, reducing the complexity of project design and consolidating activities at the core central level of the restructured MoF. Achievement of the revised objective -- to strengthen capacity in the Ministry of Finance for prudent, effective and accountable planning and management of public finances to promote growth and poverty reduction -- is rated Substantial, based on achievements in public expenditure management, revenue administration, and governance and planning within the restructured Ministry of Finance. Efficiency remains Modest. Taken together, these ratings are indicative of moderate shortcomings in the project's preparation and implementation, and therefore an Outcome rating of Moderately Satisfactory. Following IEG/POCS guidelines, as less than half of Bank-managed funds (35.6%) were disbursed under the original objectives, the final Outcome rating is weighted toward that under the revised objectives, resulting in an overall assessment of Moderately Satisfactory. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Toward the end of the project period, the Minister of Finance requested direct budget support as a follow-on to the project. This support is being provided through US$ 25.5 million (2014-2017) from Australia and the European Union, with Bank advisory services informing policy dialogue. This arrangement is based on donors disbursing a tranche of funds against the MoF's own key performance indicators, with verification of progress by a team of independent experts. Financial risk is therefore low. Institutional risk is also low, as the capacity developed under the project has been institutionalized through an increase in staff skills and consequent reduction in functions performed by consultants. a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: a. Quality at entry: Project preparation was guided by ample analytical work and background documentation (PAD, p. 7). It built on an excellent track record of donor coordination in the country. Lessons taken into account in program design included the need for strong government commitment, alignment of objectives with the country's absorptive capacity, continued effective donor coordination, and close monitoring of potential procurement and implementation challenges (PAD, p. 20). Risk assessment focused on challenges with governance and corruption, delays in decision making on IT system upgrades, weaknesses and delays in program financial management, limited government capacity to manage the petroleum tax regime; mitigation measures consisted primarily of targeted assistance and program-wide focus on integrity, transparency, and accountability. The PAD also outlined a contingency plan for use of IDA funds as a stand-alone Grant, in the event that anticipated financing from other donors did not materialize with a highly detailed account of the project's activities over the first eighteen months, establishing the "platform" on which subsequent efforts would be laid. However, this matrix of inputs, outputs, and indicators covered a large number of specific directorates/agencies with a large number of indicators; few of these included baselines or targets. The ICR (p. 19) refers to an initially "far too ambitious agenda and complex project design." In particular, insufficient attention and resources were devoted to the line ministries and district entities specified in the project's objectives during the initial years of implementation. In addition, the fact that the project was not initially fully endorsed by all donors led to delays in mobilization of funding, contributing to a slow start. Quality-at-Entry Rating: Moderately Unsatisfactory b. Quality of supervision: Early supervision efforts, particularly given the complexity of project design and the fact that the country was emerging from civil strife in 2006-2007 and the installation of a new government in 2007, were insufficient. Turnover of task team leaders prior to the 2010 restructuring was high. The 2010 Mid-Term Review was a "significant milestone" in the project's life cycle (ICR, p. 6), providing concrete action steps for putting activities firmly on track and marking the beginning of regular implementation support missions. These steps included the creation of the Project Implementation Unit and the restructuring of the project to focus exclusively on the Ministry of Finance. Also, after the 2010 restructuring, the task team leader was based in the country office, further improving supervision effectiveness. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Government ownership of the project was strong, particularly after the project recovered from its slow start. It was consistently willing to identify areas requiring improvement and institute remedies. The Ministry of Finance was a strong manager throughout, using the project to define and implement an ambitious reform agenda. However, the demands of the government program continue to stretch the capacity of spending institutions and systems. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The Project Implementation Unit (PIU), housed in the Ministry of Planning and Financing's Administration and IT Directorate, managed the project effectively. It evolved into the "highly recognized" National Directorate for Management of External Support (to the MoF), developing capacity building plans that are currently being implemented across the MoF (ICR, p. 5). Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The Administration and IT Directorate was responsible for coordinating the preparation and submission of quarterly reports and one annual report on progress against annual action programs and results matrices. The original results matrix contained over 60 indicators with few baselines and targets, making it difficult to monitor progress. b. M&E Implementation: At the February 2010 restructuring, the results matrix was revised to one PDO-level indicator and six intermediate outcome indicators. This revised matrix aligned well with the project's components, which in turn matched the MoF's organizational and accountability structure. By the end of the project period, reporting was channeled through the MoF's own results-based management system, governed by the Ministry's Strategic Plan and five-year/annual action plans. Key indicators at the level of Directorates General and Directorates were measured using real-time government data and the MoF's web-based transparency portal. c. M&E Utilization: By project closing, the project's results indicators were mapped completely onto the MoF's monitoring framework. Building on this achievement and experience, the MoF is now developing a planning and M&E framework for the whole of government. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: The project was Environmental Category C and did not trigger any safeguard policies. b. Fiduciary Compliance: At preparation, risk related to financial management was rated High, and to procurement Substantial. Despite challenges, the government preferred to use country systems wherever possible. By the end of the project period, implementation was largely through government procedures, including contract and human resources management, payments, and audit. According to the ICR (p. 9), fiduciary compliance was consistently rated Satisfactory from the 2010 restructuring onward. The ICR (p. 9) mentions fraud in 2011/2012 but does not provide further details. The ICR does not state whether external audits were on time and unqualified, nor does it provide a separate discussion of procurement. The project team later added that only two audits and 7 of 27 Interim Financial Reports were on time, and two audits were qualified (with those qualifications subsequently addressed). The quality and timeliness of audits and financial reports improved significantly as the project progressed and especially after the PIU was established. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Moderately Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome: Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately The challenges of the government's Satisfactory ambitious reform agenda continue to stretch the capacity of spending institutions and systems. Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (p. 23) cites valuable lessons, including: Consistency and stability of personnel are important. In this case, turnover of both consultants/advisors and task team leaders negatively impacted project performance at various stages of implementation. A modest initial project scope, producing quick and demonstrable wins, can be preferable to early ambition. In this case, the complexity of planned activities and results matrix overwhelmed staff with limited capacity, necessitating restructuring to streamline. Deliberate and focused attention to balancing the implementation and capacity building roles of external technical advisors is critical for long-term success. In this case, the advisors frequently assumed direct implementation roles in response to a demanding government agenda, delaying absorption of skills and capacity among local, permanent staff. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provides a clear, concise account of the project's preparation and implementation experience, and of the rationale and mechanisms by which early challenges were addressed. Its analysis acknowledges the difficulties in attributing growth and poverty reduction outcomes to project-financed activities. However, there are numerous shortcomings. The ICR's discussions of efficiency, Bank and Borrower Performance, M&E, and procurement/financial management are somewhat thin. Guidelines for determining the project's outcome rating are not correctly followed; in particular, the ICR rates achievement of components rather than objectives. In addition, the ICR's Data Sheet (p. iii) indicates that a split rating methodology was followed, but there is no indication of this methodology in the main text. The ICR also does not account for US$ 4 million in RETF financing that had been disbursed at the 2010 restructuring (Program Paper, p. 1). The ICR's lessons are insightful and well derived from project experience, but they are not adequately fleshed out. On balance, the ICR quality is satisfactory but marginally so. a.Quality of ICR Rating : Satisfactory