Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) Report Number: ICRR0021837 1. Project Data Project ID Project Name P157035 Technical Assistance Facility Country Practice Area(Lead) Afghanistan Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-A2839 30-Jun-2020 1,252,781.20 Bank Approval Date Closing Date (Actual) 28-Jun-2016 10-Jan-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 5,000,000.00 5,000,000.00 Revised Commitment 1,252,781.20 1,252,781.20 Actual 1,252,781.20 1,252,781.20 Prepared by Reviewed by ICR Review Coordinator Group Srilal Mohan Perera Judyth L. Twigg Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) was to strengthen the fiscal management and the budget planning performance of the Government of Afghanistan (Project Appraisal Document [PAD], p. 15). b. Were the project objectives/key associated outcome targets revised during implementation? No Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) c. Will a split evaluation be undertaken? No d. Components Component 1: Supporting the Continuous Improvement of the Performance of the Macro Fiscal Performance Directorate General (Appraisal: US$3.9m; Actual: US$0.89m). This component was to support the strengthening of the Macro Fiscal Performance Directorate’s (MFPD) institutional capacity to carry out its mandate as specified in the Financial Performance Improvement Plan (FPIP), among other activities, by: (a) carrying out in-depth analytical studies on how to improve fiscal policy and macro-economic management, including preparation of fiscal management reports and forecasts for budgetary support; and (b) providing secretarial services and support to the High Economic Council (HEC) by structuring the Directorate, through directorate-level staffing, and further developing procedures to enhance such support for the HEC. Component 2: Supporting the Continuous Improvement of the Program Implementation and Coordination General Directorate (PICGD) (Appraisal: US$1.1m; Actual: US$ 0.36m). This component was to support strengthening of the institutional capacity of the Policy Department (formerly PICGD) to carry out its mandated activities as specified under the FPIP, including support for the preparation of updated and costed sector strategies and work plans that are fully aligned with the national budget, and providing secretarial support for development councils and national priority programs. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost/Financing: Appraisal: US$5m; Actual US$1.25m. The project was financed entirely through the Afghanistan Reconstruction Trust Fund (ARTF). IDA did not contribute financing but administered the project. The project was closed on 10 January 2019, 18 months after its inception, based on a formal request by the government to integrate all fiscal management technical assistance projects into the Fiscal Performance Improvement Support Project (FSP). The undisbursed balance of US$3.75m was cancelled. Borrower Contribution: None was planned or made. Dates: The project was approved on 28 June 2016 and closed on 10 January 2019, some 18 months before the original closing date of 30 June 2020. 3. Relevance of Objectives Rationale The project’s objectives were aligned well with the government’s major reform program, Realizing Self- Reliance: Commitments to Reforms and Renewed Partnership. The program emphasized a commitment toward good governance and, among other things, economic and fiscal stabilization, more specifically building the country’s public administration capacity to accelerate utilization of aid and ensure transparency and accountability for public expenditures (PAD, para.4). The objectives also aligned with the Ministry of Finance's (MOF) creation of the MFPD, to support the latter’s activities and to provide critical fiscal policy Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) advice. Support from the project would enable MFPD to prepare the annual Fiscal Strategy Paper, including preparation of internal macro-economic forecasts to guide fiscal policy making and revenue and expenditure parameters (ICR, paras 10-11). In addition, the objectives of the project also supported the government’s five-year rolling FPIP and its request for the creation of a technical assistance facility (TAF) to support MOF’s macro-fiscal policy management and budget planning (ICR, paras 12-14). Importantly, the PDO supported Pillar 1 of the World Bank’s Country Partnership Framework 2017-2020 that emphasized the need to strengthen planning as well as fiscal and public financial management through strengthening of the MFPD and the Policy Department, with support from the TAF (ICR, para 29). Relevance to government and Bank strategy was somewhat diminished at project closure, however, due to the government's decision to integrate all fiscal management technical assistance projects into the FSP. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen fiscal management of the Government of Afghanistan Rationale The government needed to focus efforts on building strong and accountable institutions that would contribute to the its long-term goals of state-building and self-reliance, which required improving the institutional and regulatory framework for public service delivery through stronger fiscal planning and improved public financial management. Consequently, the need to strengthen MFPD and the Policy Department by utilizing the TAF to support programs to monitor the efficiency and effectiveness with which public expenditures were made, accountability for utilization of such funds, and accurate future estimates, or forecasting for public expenditures, were all deemed vital for the government's overall reconstruction program and for attracting aid (ICR, paras 28, 29). The project targeted areas to achieve this objective by supporting the strengthening of performance of the MFPD’s fiscal management department to: (a) develop and implement a medium-term fiscal framework, and (b) develop and implement forward-estimates for the fiscal management framework (ICR, Table 2, p. 15) to support budget preparation and provide institutional support for HEC. Outcomes Development and implementation of a Medium-Term Fiscal Framework (MTFF) and Forecasting: The MFPD’s MTFF has shown significant improvement. The forecasts for key macro-economic indicators have been made available for the current budget year and the three years ahead in the annual Fiscal Strategy Paper prepared by MFPD. This Paper includes forecasts for key macro-economic indicators such as growth, inflation, and balance of payments. It also includes additional documents that justify the forecasts. The Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) MFPD prepared the Fiscal Strategy Paper in consultation with other relevant authorities such as the Central Bank, the National Statistics and Information Authority, and key line ministries. This process of consultation has now been institutionalized. Overall, macro-economic and fiscal forecasting has improved significantly, as indicated in the 2018 Public Expenditure and Financial Accountability (PEFA) for Afghanistan. PEFA performance indicator PI-14 was used to rate macro-economic and budget forecasting through three different dimensions: macro-economic forecasting, fiscal forecasting, and macro-fiscal sensitivity analysis (ICR, para 35). The PDO indicators point to successful implementation of this supported activity under Objective 1. The baseline for this activity was D (adopted during the TAF design in 2016) with an end target of B. At actual project closure, the activity was assessed as an A. The result was also consistent with the intermediate results indicators for the activity (ICR, Table 4, TAF Results Framework). However, the ICR (p. 17) states only that the baseline was "estimated to be a D" during project design, with no information provided on the methodology used for this estimate, and therefore the actual degree of improvement experienced over the project's lifetime is uncertain. Forward Estimates Framework: The corresponding activity of developing a forward estimates framework has enabled the MTFF to independently forecast financial and economic indicators. In the past, MTFF was wholly reliant on indicators produced by the International Monetary Fund (IMF), and efforts were made to conform with those indicators. The ICR, however, comments that “there is still substantial room to improve overall depth and quality of the analytical outputs” (ICR, para 34). The ICR also reports that there is substantial inconsistency in the quality of the indicators and forecasts for preparation of the 2019 Budget. In addition, the Fiscal Strategy Paper was not approved by the Cabinet, as it did not include information for budget formulation as in the previous three years (ICR, para 33). This activity (Development and Implementation of Forward Estimates Framework) had a PEFA baseline of D (adopted at TAF design in 2016) with an end target of B. At project closing, the activity was assessed as a B, well above the baseline and corresponding to the end target. The end rating of B was also above the intermediate result indicator rating of C (ICR, Table 2). As noted earlier, the baseline score of D was estimated, and therefore the actual extent of improvement resulting from the project is unclear. Budget Preparation and Reliability: The ICR reports that that there was constant “low budget reliability." As observed, there has been very little reliance on the activities supported by the TAF for budgetary allocations (ICR, para 39). Applying the 2018 PEFA ratings through three indicators, to review overall budget, public expenditures, and public revenue, the ICR observes that “overall, performance indicators I, II and III received scores of D, D+ and D,” with most of the sub-components being rated the same or worse than the previous PEFA of 2013 (ICR, para 35). An activity that showed improvement, through support from TAF, related to improvements by MFPD in planning of and outputs from HEC. With MFPD’s support, HEC now meets more regularly than before and has been organizationally strengthened with the introduction of plans, procedures, and activities all in support of budget preparation. This better enabled HEC to contribute towards more informed budgetary decisions through production of reports and consideration of development projects proposed by line ministries for financing. Also, a practical Operational Manual was developed. Employment of project development and evaluation consultants enabled more efficient review of project proposals for public funding with more rational and acceptable budgetary allocations (ICR, para 36). This specific activity, however, has not been rated. Overall, significant gains were made in forward estimates and forecasts for macro-economic indicators, and in development of the organizational strengthening of the HEC. These gains, however, were diminished by Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) the reported lack of reliance on the forecasts and a lack of information for annual budget formulation. Achievement of this objective is therefore rated modest. Rating Modest OBJECTIVE 2 Objective Strengthen budget planning performance of the Government of Afghanistan Rationale The Policy Department required support for effectively carrying out its mandate as the source that provides work plans, policies, programs, and inputs for preparation of the national budget. The work plans had to be prepared twice yearly to support development councils whose work mirrored the operations of a reorganized National Cabinet that directly corresponded to the individual development councils. Accordingly, the related ministries could then develop strategic frameworks and prioritize programs to increase efficiency in budgetary allocations. Importantly, the Policy Department was tasked with ensuring that development council work plans were implemented. Regular high-level meetings were organized, and proceedings were recorded and archived. In addition, the Policy Department needed support to develop new National Priority Programs (NPPs). The Policy Department was also required to maintain existing NPPs and financial plans to ensure their implementation. In addition, the Policy Department was tasked with developing a handbook outlining rules, procedures, and operation and coordination mechanisms for the development councils. Ultimately, these activities were to support effective national budget planning by the Policy Department (ICR, para 21). Outcomes The TA supported activities to strengthen the performance of budget planning by the Policy Department and the Joint Coordination Monitoring Board (JCMB). These activities focused on supporting the Policy Department and JCMB in developing reform strategies, especially through proposing new NPPs such the Women’s Economic Empowerment (WEE) and the Citizen’s Charter NPP, utilization of human capital, and the role of the private sector in development. In developing these, the TAF provided the Policy Department with key technical inputs towards development of strategies that underpin the government's accountability for budget preparation. Consultants developed and completed NPPs and associated project proposals to implement the NPPs (ICR, paras 37-38). However, despite the development of NPPs, national budget allocations to implement the projects associated with the NPPs have not been forthcoming. The ICR points to an incongruence between budgetary allocations to support NPPs and linking them to align with the National Development Strategy, and “expenditures by sector have been relatively unchanged over the past five fiscal years” (ICR, para 35). There is also no evidence in the ICR on whether the important handbook, referred to above, was actually developed (ICR, para 39). The activities designed to achieve this objective had an estimated PEFA baseline of D, an end target of C+, and a C at project close; this end line rating was below the target but above the estimated baseline. Achievement of this objective is therefore rated modest. Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Efficacy of Objective 1 was modest. MFPD has shown significant improvement in the quality of fiscal analysis and revenue projections. However, there is evidence of low budget reliability on the projections and significant improvements were required to enhance reliability. Objective 2 also receives an assessment of modest. The Policy Department did make contributions through public investment and portfolio reviews, however, “the transition from producing a macroeconomic forecast and NPPs to operationalizing them within the national budget allocations…. and implementation has yet to materialize” (ICR, Para.40, pgs.18-19). Objective 1 and Objective 2 having been assessed as modest, the overall efficacy rating is therefore assessed as modest. Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency The PAD did not provide a cost-benefit analysis for the project, because the nature of the activities contemplated did not lend itself to standard economic or financial analysis. The ICR points to the following factors based on overall value for money from work performed under the TAF, the shortened length of the project term, and overall impact on objectives, to gauge efficiency:  Efficiency in payments based on deliverables: Nearly 62 percent of the consultant contracts supported by the TAF were based on payments due on delivery within a specified time frame, with the other 38 percent of the support based on traditional technical assistance contracts with daily payments tied to a timeline. By using short-term contracts based on deliverables, the MFPD and the Policy Department avoided sustainability issues as they were able to obtain results in a timely manner. The results obtained this way strengthened the respective units in the short term and set reliable precedents that could be used long-term (ICR, para 42).  Utilizing core government units directly for implementation: Without the need for processing funds through a project implementation unit (PIU), TAF funds were directly made available to the operational units (of the MFPD and Policy Department) responsible for achieving the PDOs. Expending of approximately 76 percent of the funding (US$900,000) to support analytical outputs or training of these operational units substantially reduced administrative costs and helped build a knowledge base and strengthened ownership of products of the MOF. The remaining 24 percent was utilized to provide equipment that supported the two operational units to fulfill their technical functions (ICR, para 43).  Benefits gained from output-based consultant contracts: The ICR states that, while there were delays caused initially by the lack of a PIU to implement the project because of uncertainty about structuring output-based consultancy contracts and lack of staff to perform administrative duties, evidence was available to show that there were MOF staff who had experience in structuring output- Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) based contracts. Thus, the negative impact on efficiency that may have been created by these initial delays was diminished by locating staff in the operational units in MOF who had some experience in output-based contracts and who were dedicated to implementing the project. This resulted in reduced administrative costs, avoiding unnecessary expenditures and dependence on a PIU-based approach to implementation (ICR, para 45).  Contribution towards achieving FPIP: The term of the project was significantly shortened, from 5 years to 18 months. This had a significant negative impact in measuring the project’s long-term impact. In terms of cost expenditures of approximately US$1.25 million, a strong foundation or head start for the MFPD and Policy Department in achieving the FPIP had been achieved. (ICR, para 46). However, relatively few benefits were achieved even in an environment of comparatively low costs, rendering overall efficiency modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The PDOs were substantially relevant. They were well aligned with country conditions, but became less relevant to the government's and the World Bank’s strategies for Afghanistan after the government's decision to integrate all fiscal management technical assistance support under a different project. Efficacy is deemed modest, due to shortcomings mentioned above. The project’s long-term objectives could not be fully achieved because the project was terminated prematurely. Efficiency was modest. The ratings are indicative of significant shortcomings in the project's preparation and implementation, producing an Outcome rating of Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) The project's termination, only 18 months into its implementation, exposed the already laid groundwork to the discontinuation or abandonment of the progress made in the short run, particularly in relation to developing a forecasting framework, strengthening the MFPD and Policy Department’s capabilities, and proposing of new NPPs. However, the ICR argues that this risk is minimized because the achievements and their continuance have been subsumed into the FSP that has better financing options and a better long-term horizon for forecasting (ICR, para 85). The TAF successfully identified new NPPs, such as WEE, that were targeted for budgetary allocations for their implementation. With early project closure, there is a high risk that NPPs may not be identified and supported. Moreover, the overall macro development could be adversely affected. The ICR, however, states that the consolidation of TAF activities with the FSP has enhanced the quality of policy and continuance of the technical dialogue on macro-fiscal policy management and budget planning, thereby reducing the risks associated with sustainability of the programs supported by TAF. In addition, the ICR states that any high risk to development operations of the FSP can now be complemented by tailored World Bank- executed technical assistance projects and the new programmatic Development Policy Grant, all of which will tangibly support the government's efforts to develop work plans and establish evidence-based targets (ICR, paras 85-87). 8. Assessment of Bank Performance a. Quality-at-Entry The conceptual framework for the project was at the initiative of the Office of the President. The project sought to provide readily available short-term, on-demand technical assistance for the core areas of policy reform, strategic development, and strengthening specialized capacities and capabilities within the newly created MFPD and Policy Department. This formulation corresponded with the "government’s self-driven agenda to transform economic and social policy and governance reform, and establish citizen- focused service delivery” (ICR, para 76). The World Bank carefully reviewed the proposed conceptual framework and recommended that the project objectives and results frameworks be more narrowly defined and tightly focused. It appears that these recommendations were taken on board (ICR, para 77). Thereafter, the project framework was redesigned to support the Public Financial Management (PFM) Roadmap 2016-2020 (PFMR-II) and its implementation vehicle, the FPIP. The PFMR-II was comprehensive but focused very much on macro-economic policy analysis and on strengthening the alignment between national policy and the government's budgeting process. The project’s objectives narrowly targeted these areas for TAF support, which in turn then supported FPIP implementation by the MFPD and Policy Department units. These reformulations were endorsed by the World Bank’s second review (ICR, para 78). Project design was informed by lessons learned from similar technical assistance projects, e.g. in Rwanda, Afghanistan and Sudan as well IEG reviews of such TA projects. The design took into account methods that would avoid traditional implementation delays by including in the PAD discrete activities that were streamlined, including timelines for implementation for each project component. This was accomplished by agreeing to an activity and output table for implementation (during the first year) that had timelines within which the activities were to be achieved. (ICR, para 79). Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) Finally, the project focused on the government taking on a pilot project that could minimize risks through ownership and accountability by enabling the technical assistance programs to align with national priorities. By enabling the results framework to be used effectively, scaling up or down where necessary depending on implementation needs and requirements of GoIRA, the project was designed to minimize risks. (ICR, para 80). Despite the above advantages, the project design had a few shortcomings in the practical application of measuring indicators for the activities for M&E. The measuring indicators were too broad-based, and lack of intermediate results indicators in particular. This made it difficult to assess systematic progression in the project towards achieving the PDOs (ICR, para 67). On May 29, 2020, the ICR Team provided IEG with supplementary and complementary evidence (ICR Response) about the Bank’s comprehensive dialogue and engagement with GoIRA, to design the TAF as a stand-alone TA project that was flexible enough to respond quickly with resources needed to enhance TA/knowledge. As a result, the design of the TAF sought to improve budget planning, macro fiscal management and development policy in a timely and effective manner. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The ICR reports that the World Bank’s supervision of the project was inconsistent. Overall, the World Bank’s supervision was satisfactory, particularly at the inception of the project. However, there could have been better guidance and better strategic direction by the World Bank through hands-on support to enable better implementation. The World Bank’s assistance in advising on technical and operational aspects was ad-hoc, with the task-team leaders only making themselves available on call (ICR, para 81). In addition, the ICR notes that the World Bank team should have been consistently available to advise the two key units tasked with implementing the project – the MFPD and the Policy Department – on optimizing and timely use of the TAF funds supporting the project activities. For example, the World Bank team did not make themselves available to assist the units to prepare and execute output-based consultant contracts, thereby causing early delays (ICR, para.82). Another serious problem was the difficulty in determining the reliability of the achievements under the results framework that was based mostly on self-assessment under the FPIP. As mentioned above, the Bank’s lack of involvement in the design of the results framework at the beginning to have appropriate measuring indicators included and dependence on self-assessments by FPIP units likely resulted in skewed and often inconsistent assessments and undermined appropriate supervision of the project activities by the Bank (ICR, para 68). The ICR Response, has, however, provided additional evidence of mitigating circumstances that highlighted the constraints experienced by the Bank supervision teams. While it is difficult, prima facie, to perceive why there should be tension between the Bank’s hands-on engagement and use of country systems, overall, the ICR Response is persuasive and convincing about the practical and constraining Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) factors that enable supervision based on ground realities. In the circumstances, the Bank supervision has been overall, moderately satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The methodology for M&E was designed around a reporting structure and processes based on the existing PFM system, that was in turn based on the FPIP and its existing framework for M&E. This design of the reporting framework was conducive to the narrowly defined scope of the project, based on specified TAF activities and indicators, its grant size and short period. It was also in line with IEG’s own reviews of smaller and narrowly defined technical assistance projects in other countries (ICR, para 79). Appropriately, the Director-General of the MFPD was tasked with responsibility for overall M&E for the project, consistent with his role as the Project Manager (ICR, Para.65, p.20). The project’s results framework was designed to evaluate performance on targeted priority issues and actions based on the FPIP. The narrowly defined activities, among others, development and implementation of medium-term fiscal frameworks, support for HEC, development of NPPs, and support for Development Councils, all contributed to an effective M&E design. Given the rolling nature of the project activities over its duration, the number of results framework baseline indicators was kept low so that progress of activities could be monitored efficiently (ICR, para 66). It was, however, difficult to measure activities such as budget allocations determined by NPPs and government/donor coordination on the National Development Strategy based on the FPIP under the broader M&E framework, when the baseline indicators were few and were kept artificially low. The baseline score of D for all PEFA indicators was estimated, and the ICR does not provide information that would permit an assessment of the quality and robustness of that estimate. The ICR indicates that the M&E framework could have been strengthened with additional intermediate indicators. The ICR therefore comments that targeting specific sector or program areas would have been realistic in order to determine whether the PDOs had been actually achieved (ICR, para 67). The non-inclusion of these factors, undermined the future accurate assessment of the activities under the project. b. M&E Implementation Basing the M&E results framework on the FPIP presented limitations. Sometimes PEFA ratings were used as seen above and at other times, for different activities other factors such as timeliness, quality and effectiveness were used to measure achievement. There was incompatibility with the factors that were taken into account in the FPIP M&E framework and the activities targeted for M&E under the TAF, for measurement. An independent assessment of M&E implementation results was also undertaken. In addition, the ICR states that the quality and detail of the government's FPIP reporting based on the FPIP M&E framework with reliance on its own internal assessment and that of the independent assessment Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) were inconsistent and difficult to reconcile. For example, the ICR implies that these inconsistencies resulted in the forward estimates not being used for budget formulation (ICR, para 68). It is clear from the foregoing, and from the limitations in the M&E framework, that there was lack of alignment between the specific activities targeted for assessment under the project and the related generic activities under the PEFA rating methodology utilized by FPIP. Therefore, the measuring factors for M&E of the activities under the project were inadequate and unreliable, yielding inconsistent overall ratings. Taken together, with the PEFA scores, serious confusion was caused in comprehending how performance was assessed by the FPIP units. c. M&E Utilization In the end, M&E utilization relied on PEFA scores (that had a 2016 PEFA baseline score for all three activities and the final ratings at project closing in 2019), rather than on FPIP’s performance reports, since those reports were unreliable as seen above (ICR, paras 68- 69). The shortened life-span of the project also short-circuited the time within which to measure deliverables. The 18 months during which the project was active only enabled the implementing units to hire the first batch of consultants, and it was therefore not possible to make realistic assessments of their work output, since results could only be determined through deliverables and their impact on PDOs (ICR, para 70). M&E Quality Rating Modest 10. Other Issues a. Safeguards The project did not trigger social and environmental social safeguards. The project was an environmental assessment Category “C” technical assistance project. b. Fiduciary Compliance The project amount of US$5 million was committed out of the ARTF. Of this, nearly 26 percent was disbursed by closing, 18 months after effectiveness. The ICR records that this rate of disbursement corresponds with other projects of a similar nature. The project was in compliance with the covenants of the Financing Agreement. The financial management rating for the project was deemed satisfactory. All audits, while initially delayed, were appropriately completed, and no outstanding ineligible expenditures related to the project were reported (ICR, para 73). However, fixed asset management remained weak throughout the project, and no updated fixed-asset register was kept (ICR, para 74). There were some Delays in clearances of operational advances, although clearances have now been received (ICR, para 75). Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) c. Unintended impacts (Positive or Negative) There are no reported unintended impacts or outcomes. d. Other While the project did not directly target gender, the development of an NPP WEE will contribute to women and children’s inclusivity into the mainstream economy. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Modest 12. Lessons The ICR does not contain a Lessons Learned section, but several key lessons can be discerned from the text:  Utility of Direct Funding without a PIU: One of the unique initiatives under the project was that investment of project funds was made directly to the two implementing units – the MFPD and the Policy Department - without the need for a central PIU unit to oversee implementation. This way, the two units were more directly accountable for and took ownership over the project activities and the achievement of the objectives. In a larger context, it reflected the government’s accountability and ownership of the project's implementation and outcomes. Concurrently, this resulted in reduced costs. It could be a good model to follow for technical assistance projects. However, it must be noted that this initiative involved a fairly small investment over a short period of time, ideally conducive to a technical assistance project, and probably is not ideal for projects with larger investments and complex frameworks to achieve objectives.  Need for Effective Capacity Building: Linked to the above, this model assumes a generally high degree of capacity and capability of both units to undertake the designated implementation activities. The ICR at various points states that there were delays caused because of the two units’ unfamiliarity with preparing and executing short-term output-based consultancy contracts (ICR, para 45), as well as a dearth of in-house economists (civil servants) with expertise in forecasting and fiscal planning that had been previously done by external consultants (ICR, para 50). The skills transfer to civil servants caused initial delays Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) in implementation and also points to the need for more capacity building in specialized areas that ought to have been considered as a key component of the project.  Importance of NPPs and Budgetary Allocations in Fiscal Planning: The other important outcome supported by the project was the development and adoption of NPPs that took priority consideration for budgetary planning. A particularly important initiative was the development of NPPs such as WEE and the Citizen’s Charter. Unfortunately, while important NPPs were developed and proposed, the required budgetary allocations to support these NPPs were never undertaken. Therefore, while appropriate NPPs are extremely important to prepare and propose, there must be tangible budgetary allocations to support and sustain the implementation of the NPPs.  Importance of a Practical and Effective M&E Framework in Design of Technical Assistance Projects: It seems that the baseline PEFA ratings of D in the results framework (ICR, para 66, Table 4 TAF Results Framework) for all three activities were artificial ratings and without a proper basis. That should have been avoided. It only provided for a situation where, unless the project totally collapsed, even the most marginal activity would have to be scored above the low base rating. This is a design flaw that should be avoided because it gives a skewed view of actual achievements. This important exercise was further compromised by agreeing to a FPIP self-assessment of activities that bore completely inconsistent and totally confusing results. It is clear from the ICR that dependence on M&E assessments based entirely on inflexible and generic methodologies such as used by the FPIP M&E framework can lead to inconsistent results (ICR, para 66). The World Bank must have a more influential role in designing M&E methodologies that are tailored to assess project characteristics to enable more accurate assessments.  Early Termination: The project was terminated prematurely after only 18 months upon the request of the government, with acknowledgment in the ICR that it had, however, laid a good foundation to achieve the main objective of fiscal management and budget planning. Integration of all TA projects on fiscal performance improvements can have positive effects of consistency, coherence, transparency, and accountability. 13. Assessment Recommended? No 14. Comments on Quality of ICR To the extent the project life was limited to 18 months, the ICR documented and provided a good overview of the project, its implementation, and its trajectory towards achieving the PDOs. The project’s development objectives and the rationale for the project’s activities to achieve the objectives were clearly and succinctly stated. The Executive Summary was particularly well done and useful. For the most part the ICR followed the key guidelines in assessing the activities and achievements. However, it had shortcomings:  At times it lacked evidentiary substantiation for its final rating. For example, the discussion in the ICR of budgetary forecasting is quite confusing. While stating that there was improved forecasting, the ICR concludes that there is still low budget reliability, evidenced by very low PEFA scores that had been Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Technical Assistance Facility (P157035) recorded in the past (ICR, para 35). There are no examples given, however, of why low reliability is still a problem. What were the forecasts or indicators that could not be relied on for rational budget allocations, and why? Yet, as a matter of ratings for this activity, the ICR reported a baseline of D, a formally revised target of B, and a rating of B at closing, and the conclusion that “This indicator was fully achieved” (ICR, Annex 1, pp. 35-36). It is also noteworthy that the ICR does not provide information on the methodology used during project preparation to arrive at the estimated baseline score of D for all PEFA indicators.  At times there is no logical explanation to connect the project's supported activities and the achievement of objectives. The rational alignment between outputs (activities), outcomes, and achievement of objectives lacks clarity in the narrative. For example, there is no logical explanation on the link between the objectives and TAF support for HEC, giving contextual relevance to how critical HEC is in fiscal planning and management in Afghanistan.  There were often unnecessary repetitions that made the narrative more lengthy than necessary. For example, Tables 2 and 4 are identical. The benefits of early termination of the project (paras 6, 26, 27, 46, 85), and circumventing the need for PIUs in TA projects (paras 3, 43, 45, 49, 58) are other examples.  While for the most part the ICR followed the guidelines, an important omission noted was a Lessons Learned section. The lessons learned had to be discerned from various parts of the ICR. The ICR Response, is, however, correct, that during a Webinar held in March, 2020, between the IEG Evaluator and the TTL/ICR Team, the latter informed that the Lessons Learned Section had been mislabeled as “BANK PERFORMANCE, COMPLIANCE ISSUES,AND RISK TO DEVELOPMENT”. (Paras 88 – 91). Even so, the mislabeling is misleading and has affected the quality of the ICR. This section also contained some lessons that are generic to many TA projects and not unique to the TAF. a. Quality of ICR Rating Modest Page 14 of 14