IEG Report Number: ICRR14659 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 05/21/2015 Country: Vietnam Project ID: P075399 Appraisal Actual Project Name: Public Financial Project Costs (US$M): 71.45 87.72 Management Reform Project L/C Number: C3767 Loan/Credit (US$M): 54.33 69.15 Sector Board: Public Sector Cofinancing (US$M): 9.99 10.46 Governance Cofinanciers: UK DFID Board Approval Date : 05/22/2003 Closing Date: 02/28/2009 10/31/2013 Sector(s): Central government administration (75%); Sub-national government administration (25%) Theme(s): Public expenditure; financial management and procurement (50% - P); Debt management and fiscal sustainability (50% - P) Prepared by: Reviewed by: ICR Review Group: Coordinator: Gianni Zanini Iradj A. Alikhani Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Development Credit Agreement (DCA, p.14) the project development objective (PDO) of this Public Financial Management Reform Project (PFMRP) is to strengthen the Borrower’s capacity to plan, execute and report on its budget and to improve the transparency and accountability of its budgetary systems and processes. The Project Appraisal Document (PAD, p. 2 and its Annex 1, pp. 28-29) omits the transparency part of this statement. This Review will use the PDO in the DCA. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project had four components: 1. Strengthening treasury and budget management (US$61.42 million at appraisal; US$77.94 million at closing). This was the largest component, involving the (i) implementation of an integrated Treasury and Budget Management Information System (TABMIS), which was expected over time to support a move from pure cash accounting towards accrual and, together with the new inter-bank transfer systems, a move to a treasury single account. The turn-key procurement of the new TABMIS included hardware, software, and technical advice to implement cash management, a Treasury Single Account (TSA) and a government-wide uniform chart of accounts (COA). Reforms under this component were expected to include also (ii) the consolidation of the Treasury’s new status as the lead agency responsible for maintenance of the treasury and budget management information; (iii) the integration of then disparate accounting systems and their consistency with international standards; (iv) the consolidation of extra-budgetary funds into the budget; and (v) the routine disclosure of detailed, reliable budget plans and final accounts. 2. Strengthening state budget and investment planning (US$4.18 million at appraisal; US$3.13 million at closing). This component was to support: (i) the development of a Medium Term Fiscal Framework (MTFF) by the Ministry of Finance (MOF), in close coordination with the Ministry of Planning and Investment (MPI); (ii) the piloting of Medium Term Expenditure Frameworks (MTEFs) in four sectors and four provinces; and (iii) the consolidation of MOF’s new status as the lead agency responsible for the budget, along with better partnership arrangements among MOF, MPI, line ministries and provinces. 3. Strengthening management of public debt and monitoring of state-owned enterprises (SOEs) fiscal risks (US$2.88 million at appraisal; US$2.43 million at closing). This was to include technical assistance (TA), training, capacity building, and systems implementation to support: (i) the recording of domestic debt in the Debt Monitoring and Financial Analysis System (DFMAS), interfacing or integrated with systems for recording of external debt that were already under development with United Nations Development Program (UNDP) assistance. Other key policy and institutional reforms supported under this component were (ii) the operationalization of MOF’s legal mandate to manage all components of the country’s public debt; (iii) the formalization of public debt management objectives and guidelines; (iv) the clarification of reporting relationships between different agencies involved in debt-related transactions; and (v) the adoption of risk management systems and practices. 4. Project management support (US$2.97 million at appraisal; US$4.22 million at closing), to cover Project Management Unit’s (PMU) staff and consultants’ costs, training, in-country travel, office space, telecommunication costs, consumables, transport, and audit services. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The original project cost at appraisal was US$71.45 million, to be financed by an IDA credit of Special Drawing Rights (SDR) 39.90 million (US$54.33 million equivalent), a UK DFID grant for the equivalent of US$9.99 million, and a Government contribution for the equivalent of US$$7.14 million. In January 2011, IDA provided additional financing of SDR 9.20 million (US$14 million equivalent at that time), bringing total cumulative IDA commitments to SDR 49.10 million (US$68.33 equivalent). As of February 2014, the project disbursed a total of US$87.72 million, with IDA disbursing SDR 45.10 million (equivalent to US$69.15 million), UK DFID US$10.46 million and the Government US$8.1 million. The Bank conducted a Midterm Review (MTR) in December 2006 mainly to address implementation issues and delays, followed by three extensions of closing date and three restructurings beginning in early 2009. The project was restructured three times. Under the first restructuring in February 2011, the scope of several sub-components was adjusted and original funds were reallocated. Nearly 65 percent of the additional financing or about US$ 9.1 million) was allocated to original activities that faced cost overruns during the first seven years. The rest was devoted to scale up the implementation of other selected activities and to add a limited number of new activities. The scaled-up and new activities were also partly related to TABMIS (budget allocation module extended to major supervising central agencies; a new portal; and a TABMIS-based roadmap for a longer-term Government Financial Management Information System or GFMIS) and partly to Component 2 (timely preparation for adoption of MTFF and MTEF following their piloting, and integration of these frameworks into the annual budget process) and to Component 3. Overall, encompassing both original and new activities, about 80 percent of additional financing was allocated to Component 1 (TABMIS). Under the second restructuring in February 2013, selected project activities were dropped, delivery times for other pending activities were adjusted, related output indicators were updated, and allocation of project funds re-aligned to these changes. The third restructuring in September 2013 also removed selected activities and updated sector and output indicators. The project was approved on May 22, 2003 and became effective on September 4, 2003. The original closing date was February 28, 2009. Due to implementation delays and expansion in the scope of key project activities (mainly TABMIS and DMFAS), a first extension (no date provided in the ICR) moved the closing date by two years to February 28, 2011, a second extension (coinciding with the first restructuring) pushed it back by another two years to February 28, 2013, and a third extension (no date provided in the ICR) delayed the final closing date by eight months to October 31, 2013. Actual implementation took ten years. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial. After about a dozen years of transition from a planned socialist economy to a mixed economy, Vietnam in early 2003 had a four-tiered structure of government, with a national-level administration, 61 provinces, 695 districts, and 10,500 communes and wards. The share of sub-national expenditure in total public expenditures had increased steadily from 26 percent in 1992 to close to 50 percent in 2003. During this same period, the existing treasury and budget management regulations and systems had improved in three important areas: (i) clarifying the roles and responsibilities of central and sub-national government; (ii) improving transparency and accountability, including with an enforcement mechanism to make communes post their budgets outside offices; and (iii) promoting administrative rationalization and streamlining of responsibilities, including for the Ministry of Finance (MOF), the State Treasury Department (within the MOF), the Ministry of Planning and Investment (MPI) and the State Bank of Vietnam. Mechanisms were also adopted to prevent over-spending, misappropriation of resources, targeting and equitable distribution of spending on social sectors--especially on education. Public Financial Management (PFM) was one of the four focus areas within the Government’s Public Administration Reform Master Program for the period 2001-2010, endorsed by the Prime Minister in September 2001. Within this framework, a high-level Steering Committee for the Modernization of PFM identified five areas requiring priority attention, among which were the following areas covered by the PFMRP: (i) budget management, including via the establishment of an integrated TABMIS and the introduction of multi-year expenditure programming; (ii) debt management; (iii) SOE fiscal risk management; and (iv) public asset management. Improving governance (encompassing PFM) remains a pillar of the Government’s development strategy, as set out in its 2011-2020 Socio-Economic Development Strategy (SEDS) and its Socio-Economic Development Plan (SEDP) 2011-2015. The completion of the PFM reform agenda, encompassing the priorities listed above as well as organizational restructuring, pay and employment reform, institutional development, revenue management and financial market supervision remain highly relevant to this day to ensure aggregate macro-fiscal discipline, the strategic allocation of resources, and efficient service delivery. PFM has also been a consistent and central priority of Bank assistance throughout the past dozen years, and was an important policy area under its analytical and advisory services and two Poverty Reduction Support Credits (PRSC) series. The project’s development objectives were fully consistent with the Bank's 2002 Country Assistance Strategy, where “improving information, transparency and public financial management” was the first area under Theme 3, and remain highly relevant under the current (2012-2016) Country Partnership Strategy. On the other hand, with a large technical assistance and Financial Management Information System (FMIS) project like the PFMRP complementing other lending (e.g. PRSC) and analytical and advisory services addressing the PFM reform agenda by the Bank and by other donors, the objectives of this project should have gone beyond strengthening only the “capacity” to plan, execute and report on the budget and should have been more outcome-oriented (e.g. “strengthening of the planning, execution and reporting on the budget”). b. Relevance of Design: Substantial The PFMRP was appropriately focused on three clear priority areas, with other important areas related to PFM, public expenditure management, and external debt management supported through other operations (such as the PRSC series) or advisory services provided by the Bank and/or by a dozen other donor agencies (see PAD, pp. 17-18). The project’s components and sub-components were well related to the objectives, and their design addressed all the key institutional and technological challenges, in light of the available lessons from the international experience. The choice of lending instrument (a single, technical assistance project) was also reasonable. The alternative of breaking down the financing into two or more tranches (as in the case of an Adaptable Program Lending, which had been the chosen instrument in some cases (e.g.; Russia) had been considered during the preparatory phase. However, in the case of Vietnam the choice was made of procuring TABMIS under a large, turn-key contract covering all the hardware, the software, and the implementation services. This approach addressed concerns over weak contract management and technical expertise available within the Government to oversee separate contracts for these components. Under this approach, breaking down the financing into two or more phases (Adaptable Program Lending) would have been impractical and inefficient. The results framework for this project was limited to the results matrix in Annex 1 of the PAD, which reflected the standard logframe approach in use at the World Bank at the time of the project’s appraisal. One of the general weaknesses of this approach (no longer utilized by the World Bank) was the lack of explicit discussion about how projects’ outputs would translate into the postulated intermediate and final outcomes and these into the project’s specific objectives. Moreover, two of the key associated outcomes proposed by the PAD for this project, that is, “better planning” and “fiscal sustainability” (see the next section of this review), were not specific with the former being a tautology and the second being a higher-level objective and not substantially attributable to the project. Nonetheless, the logical chain from outputs to intermediate outcomes and to specific objectives or their key associated outcomes that was implicit in the project’s results matrix was broadly plausible. 4. Achievement of Objectives (Efficacy): The project’s development objective was to “strengthen the capacity to plan, execute and report on the budget and to improve the transparency and accountability of the budgetary systems and processes.” Given the large overlap in budget reporting and transparency, however, the efficacy of the project for these two specific objectives will be evaluated together as one. A. Strengthened Capacity for Budget Planning : Modest One of the proposed key outcome indicator in the PAD/ICR to assess the achievement of this specific objective is a tautology, that is “better planning of the State Budget and the Public Investment Program to achieve the growth and poverty reduction goals set out in the Comprehensive Poverty Reduction and Growth Strategy” (referred to in the PAD/ICR as PDO indicator No. 1). Against this level of generality, the ICR points out two achievements (pp. 16-17 and 62-63), in line with targets at appraisal:  The pilot implementation of MTFF and MTEF having improved the quality of socio-economic forecasting and medium-term fiscal policy and public resource projection. Benefits reported by the pilot ministries and provinces included improved analysis and baseline settings of expenditures, better reflection of the cost implications of new policies and decision on the recurrent and capital budgets, and improved coordination between finance and planning departments. However, no independent survey or qualitative analysis or quantitative indicator, however, is cited in the ICR as evidence of strengthened planning in these pilot ministries and provinces. The evidence provided is therefore weak.  The demonstration effect of the pilots helping build consensus for the MTFF/MTEF approach. According to the project’s Task Team, the 2014 Public Investment Law, which went into effect on January 1, 2015, has already incorporated the rolling medium-term 3-year planning framework. The implementation of this law, however, cannot yet be verified and moreover its adoption is not attributable to the project, as it was a prior action under a recent operation. The following two outputs that were delivered by the project (also in line with plans at appraisal, save for the delays) contributed to the outcomes listed above: (i) The project successfully piloted the application of MTFF and MTEF in four line ministries (Education, Health, Transport and Agriculture) and three provinces for the three most recent budget cycles through 2013. The delayed implementation of this component and consequent late outputs was due to the lack of incentives for officials to adopt this new approach, as the current Budget Law did not require it. (ii) In preparation for the nationwide implementation of MTFF and MTEF once the new draft Budget Law is approved by Parliament (expected in mid-2015), the relevant Guidelines have already been finalized. However, on the basis of the weak and vague evidence presented in the ICR regarding the outcome of the pilots (unlike for outputs) and the present lack of nationwide integration of forward-looking planning and budgeting (a key recommendation of multiple studies conducted in the early, mid- and late 2000s), the project’s efficacy with respect to the first specific objective is rated as only modest. The 2013 Public Expenditure and Financial Accountability report (PEFA) scores in areas related to this objective neither demonstrate improvements over time because of a lack of historically comparable indicators or the direct contribution of the project to the current situation. The PEFA overall score of the extent and adequacy of a “multi- year perspective in fiscal planning, expenditure policy and budgeting” (PI-12) is a letter grade of only C (on a scale of A to D). The weaknesses relates to lack of forward estimates of fiscal aggregates in the context of formulating annual national budgets, inadequate linkage between investments and recurrent expenditures (PEFA sub-ratings of D for both). Medium-term sector strategies in a few key sectors (health, education, agriculture and transport) have been prepared, but are not (yet) consistent with the aggregate fiscal projections, a reflected in a PEFA sub-rating of C. B. Strengthened Capacity for Budget Execution : Substantial. The proposed third PDO indicator in the PAD/ICR to assess the achievement of this specific objective is “accuracy, timeliness, relevance, transparency and compliance with international best practices in budget execution and reporting at each level of government.” Note, however, that the reporting and transparency dimensions of this indicator are discussed below under part C of this Section 4. Some of the other dimensions like “accuracy” and “timeliness” lend themselves easily to quantification, as is common in many other Bank projects with a budget execution component (e.g. the Benin PRSC-8). The PAD and the ICR, however, do not provide any such quantitative baseline, target, or actual measures. The following results are reported in the ICR:  Accuracy (and consistency) in budget control, payment processing, and accounting was achieved as planned according to the ICR (p. 15 and p. 61), which, however, does not provide even qualitative information on status at baseline, what the original and revised targets were, and on actual outcomes.  Regarding timeliness, the ICR (p. 14-15 and p. 61) praises the government’s procurement practices (despite the multi-year delays in the project’s implementation, including in procurement) and its improved capacity to allocate and execute the state budget on a real-time basis, thanks to the availability in TABMIS of real-time data on government receipts and expenditures and on cash and fund balances.  Regarding “relevance of budget execution,” the ICR has no definition, mention or discussion about this nebulous concept. The following project’s outputs under Component no. 1 (the most complex, challenging and costly within the entire project) are all in line or exceeding plans and contributed to the actual and expected outcomes mentioned above: i. A TABMIS was developed and is fully operational at 1500 treasury offices and financial agencies in all 63 provinces and 37 spending ministries (and three major departments of Hanoi City) since October 2012, replacing multiple distributed treasury legacy systems located throughout the country. ii. A unified Chart of Accounts (COAs) was developed and applied throughout all four levels of government, replacing multiple COAs. iii. Large state budget commitments are being controlled and monitored through TABMIS, thus helping a move toward accrual accounting. iv. Data on government receipts and expenditures, cash and fund balances available in TABMIS now flow on a real-time basis enable all treasury and finance offices to produce instant reports for management purposes. v. Centralized payment processing. vi. Move to a Treasury Single Account (TSA) was fully completed in mid-2014, with only four bank accounts (with four state-owned banks) replacing 700 prior bank accounts. Despite the weak evidence provided in the ICR on the actual key associated outcomes, IEG rates efficacy towards this specific objective as substantial, on account of the general evidence from international experience of the positive impact on countries’ capacity for budget execution of a fully functioning FMIS and the other institutional reforms implemented under the project. A 2008 unpublished draft report covering PFM, the 2013 PEFA and the 2014 Fiscal Transparency Report (FTR) provide useful checks also in this area. There are still shortcomings in extent (C is the PEFA grade), composition (D+), and variance (D) related to the expenditure out-turn compared to the originally approved budget and also a long list of other weaknesses related to the classification of the budget (D) and omissions, carry-over practices, and overly aggregated capital allocations. However, according to the project’s Task Team, the former shortcoming (that does indeed still undermine the credibility of the budget) stems from a chronic under-forecasting of revenues (outside of the purview of the project) by the government rather than its control over its expenditures (which the project aimed at strengthening). And other dimensions of budget execution appear to fare better, with a letter grade of B+ for the predictability in the availability of funds for commitment of expenditures, B for the recording and management of cash balances, debt and guarantees, B again for the effectiveness of payroll controls, and A for the availability of information on resources received by service delivery units. The PEFA rated with a D letter grade the effectiveness of expenditure commitment controls before budget year 2013, but it also noted that their full implementation was expected once TABMIS would become fully operational throughout the country, which it did during 2013. On balance, therefore, the available external evidence PEFA ratings support IEG’s substantial rating of efficacy in this specific objective. C. Strengthened Budget Reporting and Transparency : Modest Progress towards this third specific objective has been mixed, but on the whole limited. The improved accuracy and consistency in budget accounting, noted above in part B above of this Section 4, helped according to the ICR also to enhance transparency throughout all levels of government. Moreover, the ICR mentions the achievement of transparent and real-time monitoring of the state budget, with instant reports available. All such results, however, are only for internal management purposes. In fact, another key expected result, that is, the publication within 15 days monthly and quarterly on MOF website of budget executions results from TABMIS (Intermediate Indicator No. 16), is yet to be realized, as it needs the legal authorization that only the new Budget Law will offer only after its expected approval in mid-2015. Another key outcome in the PAD/ICR is that of “greater fiscal sustainability through improved and more integrated recording of external and domestic public and publicly guaranteed debt, improved capacity to monitor SOE liabilities, and improved ability to assess associated fiscal risks”. While this outcome has also implications for fiscal planning, it is discussed here because it is mainly linked to the transparency specific objective. The ICR states that analytical capacity building in debt management of relevant MOF officials has been considerably enhanced, but does not provide further qualitative or quantitative evidence (and the PAD did not set any targets) about the improvement in capacity to assess fiscal risks from debt liabilities. No detailed discussion is to be found in the PAD and no information is to be found in the ICR to assess the key outcome of improved fiscal sustainability. It is possible that many of the project’s outputs already noted above in part B above of this Section 4, especially those related to the TABMIS, unified COAs, centralized payment processing and Treasury Single Account, contributed positively to the reporting and transparency specific objectives. Various institutional reforms and improvements took place also in debt management. Among these, the ICR includes the establishment of the Debt Management and External Finance Department in 2009, but such institutional reform was not envisaged at project’s design and is unclear whether the related Government’s decision is attributable to the PFMRP. The Government also passed a Law on Public Debt Management, another reform not envisaged at design. Others achievements listed by the ICR that are clearly attributable to the project are: (a) the approval and publication of debt management objectives within a long-term strategy for public debt and external borrowing; and (b) the installation of and training for DFMAS, which is now being used as the debt management and recording system for both domestic and external debt; (d) the development and operationalization of an interface between DFMAS and TABMIS; and (e) the making of financial information of SOEs available to the public. These represented indeed major improvements in debt management compared to the situation assessed in various government and Bank reports in the early and late 2000s. Notwithstanding such outputs, including stronger capacity for debt management, and the better and timely internal management budget reports, IEG rates the efficacy of the project towards strengthened budget reporting and transparency as only modest due to the failure to publish monthly or quarterly public reports of budget execution (and explicit intermediate outcome indicator), the lack of clarity and supporting evidence in the ICR of actual improved transparency, and the absence of information on fiscal sustainability (or of capacity to assess fiscal risks). External evidence to the ICR is again consistent with this rating. The 2013 PEFA praises the transparency of inter-governmental fiscal relations (B+) and public access to key fiscal information (B). However, the 2014 Fiscal Transparency Review TFR points to a number of practices that still undermine clear reporting and transparency, such as persistent difficulties to interpret and communicate the content of the budget, lack of consistency in classification between budget plans and execution reports, and lack of breakdown of recurrent expenditures in published tables, among others. And the PEFA’s scores for many key relevant areas that were supposed to be addressed under the project are in the D and C range, such as the extent of unreported government operations and the quality and timeliness of in-year budget reports (D+) and of annual financial statements (D+). Debt sustainability analysis, on the other hand, has been undertaken annually with the assistance of the IMF and the World Bank, drawing a PEFA sub-rating of A for its scope and frequency. While overall fiscal management according to the IMF has also improved, thanks in part to the more accurate, integrated and comprehensive domestic and external debt recording system supported by the project, the overall PEFA score for “oversight of aggregate fiscal risk from other public sector entities” (PI-9, encompassing extra-budgetary funds, subnational governments, and SOEs) is only C+. D. Strengthened Accountability : Substantial The ICR failed to identify key relevant outcomes to the achievement of this specific objective, and did not present any other evidence of results. Subsequent information provided by the Bank team and Borrower indicate that accountability was strengthened thanks to: (i) the project’s many outputs listed above; (ii) more timely and more comprehensive reporting on budget execution, external debt and SOE finance to the National Assembly and auditing institutions; (iii) clear roles and responsibilities of each entity involved in budget execution, public debt recording and monitoring and control of budget expenditure. Many of the project’s outputs related to TABMIS, unified COAs, Treasury Single Account and DMFAS are likely to have contributed positively to this objective. Among others, they include: (i) the issuance of regulations on control and monitoring through TABMIS of large expenditures (Intermediate Indicator No. 5 in the ICR), (ii) certain transactions done on accrual basis (No. 8), (iii) implementation of a domestic debt recording system under DMFAS ((PDO Indicator No. 3 and No. 18), and (iv) improved and more comprehensive budget execution and debt reports provided to the National Assembly and auditing institutions. In 2008 a joint Government-Bank report (unpublished) had praised the expanding state auditing at all levels of government and the enhanced and strengthening of budget oversight through the National Assembly and its committees and the Provincial Peoples’ Councils with reviews of budget estimates and budget execution reports. However, at that time the project’s outputs noted above were yet to be delivered and thus attribution to the project of such progress in accountability cannot be established. Moreover, as recently as 2013 the PEFA still assigned a grade of only C+ to the “scope, nature and follow-up of external audits.” On the other hand, the project’s scope was never intended to deal directly and comprehensively with the accountability and auditing institutions in Vietnam, but only to support some of reforms and actions that are necessary (although of course not sufficient) to ensure improved accountability. Thus, in recognition of the project’s partial but tangible contributions through 2014 towards improved accountability, IEG rates the efficacy of the project towards the achievement of the specific objective as substantial. Overall, across all four specific objectives there is only mixed and often weak evidence provided in the ICR, as well as in the exchanges between IEG and the Bank that the project substantially achieved its intended outcomes, as opposed to the ample and convincing documentation related to the achievements of the project’s original and revised outputs. Evidence available from external sources of analysis to the project, such as a 2008 unpublished draft report covering PFM and the July 2013 PEFA (and in particular its PFM Performance Assessment) is consistent with the mixed picture in the quality and extent of budget planning, execution, reporting and transparency, and accountability that has been described above and linked directly to the project’s contributions. 5. Efficiency: Modest. Neither the appraisal team nor the ICR team attempted to estimate internal and economic rates of return (IRR/ERR) and they also failed to establish the project’s overall cost effectiveness. The ICR only mentions benefits accrued through governance reforms, institutional strengthening, capacity building, operational efficiency and effectiveness of public expenditures and makes the hypothesis (p.12) that savings of about US$35 billion may have been accruing since 2012, but fails to provide a rigorous basis for this. No mention is also made in the ICR of a handful of smaller activities that were not completed. Evidence available from two studies (see Section 10 of the 2011 retrospective study by the World Bank about its experience with FMIS projects during the previous 25 years and also a 2014 Handbook on IFMIS for Government) suggest that unit costs as well as the time to implement this project were in line with similar projects in Pakistan, Indonesia, Kazakhstan, the Russian Federation and other countriess. Moreover, most of the project’s procurement was undertaken under international competitive bidding (ICB), and thus there is also a general presumption of cost effectiveness. However, the project suffered from long implementation delays (4.5) years, which cannot be solely explained by the need for additional financing, as this was approved two years after the original closing date. Moreover, such delays occurred notwithstanding the fact that various activities were dropped during the last two restructurings. The project also experienced large cost overruns, only part of which can be explained by the unexpected higher number of users (three times more) and total budgetary expenditure processed by TABMIS (five times more) compared to the levels envisaged at the design stage. Weighing the extent of ICB in the overall project’s cost and the external comparative evidence on the project’s cost effectiveness against the long implementation delays, large cost overruns, and inadequate evidence in the ICR on the project’s cost effectiveness, IEG rates efficiency as modest. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: It seems likely that certain project outcomes will take time to be realized. At the time of this ICR Review, the project achieved important and significant outputs, such as the establishment of the core (TABMIS) of a complex and far-reaching FMIS and of a domestic debt recording system. Information provided by the Bank and the Borrower has provided evidence on additional results associated with the project. However, the evidence on the achievement of the project’s specific objectives and performance indicators identified at appraisal is mixed and at times reflect limited progress. Combining substantial relevance of objectives and design, substantial efficacy for two of the specific objectives (strengthened budget execution and accountability), modest efficacy towards the other two specific objectives (strengthened capacity to plan and improved reporting and transparency) and a modest efficiency rating, IEG rates the project’s overall outcome as moderately satisfactory. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: With both TABMIS and DFMAS being fully operational, have been well received by their users. They are the only such systems used by the relevant agencies at all levels of government. Additional staff have been recruited to the state treasury. The knowledge of how to operate these systems havie been effectively transferred, and the MOF is administering both systems’ databases and security with support from an outside contractor. These systems are this likely to be sustainability appears not in doubt, and he Government is now planning to expand TABMIS to an all-encompassing Government Financial Management Information System (GFMIS). With respect to the medium-term expenditure planning and budget framework, the Government is on track to pass the required legislation in summer 2015 to mandate it nationwide across all levels of government. In debt management, the Government on its own has made welcome institutional changes and has passed a new law. The Bank also plans to continue to support the country with both analytical and advisory services and with operations in complementary areas of reform, notably the modernization of revenue collection, the reform of public expenditure management and oversight, and reforms to various public investment and budget laws. The International Finance Corporation (IFC) is also providing support to priority reforms related to SOEs’ soft budget constraints and weak corporate governance and intends to offer financial support for restructuring of the SOE sector, if so requested by the Government. A minor concern is mentioned in the ICR regarding provision of adequate financial resources at the lower level of government for maintenance expenditures of TABMIS. Another issue is that, as reflected in the PEFA (ratings C or D), significant weaknesses still remain in a handful of areas that unless addressed may affect the overall sustainability of outcomes. These risks are not considered significant. a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: a. Quality at entry: The project built on important reforms already undertaken, under the first PRSC series and extensive diagnostic work by the Bank, including a 2000 Public Expenditure Review, a 2001 Country Financial Accountability Assessment (CFAA) and early consultations on the 2004 Public Expenditure Review and Integrated Fiduciary Assessment, and technical assistance (either completed or ongoing) provided by thirteen other donors in various PFM-related areas. Moreover, as stated in the ICR, during preparation and appraisal, the Bank took into account all major technical, financial, and institutional aspects of the project, including procurement. Major risk factors were addressed and lessons learned from other earlier projects were considered. A number of technical alternatives were explored before arriving at the chosen design, especially the decision to procure an off-the-shelf commercial TABMIS. The services of an experienced international consultant in TABMIS were engaged in advance of appraisal (financed by a Bank-administered PHRD grant or Policy Human Resource Development-grant) to identify the system’s functional and technical specifications. From the start a sound decision was made for the systems implemented under the PFMRP to be open and modular, so as to enable future integration with other systems and expansion to other parts of the government. Project preparation was carried out with an adequate number of specialists. The readiness of the project soon after effectiveness in terms of procuring TABMIS is indeed impressive. Various shortcomings in quality at entry, however, resulted in major and unexpected delays during implementation, cost overruns requiring additional financing, and difficulties in the subsequent monitoring and evaluating (M&E) the project’s achievements: (i) Underestimating the time to implement a complex change as that represented by TABMIS. In particular, as clearly shown in a 2011 retrospective by the Bank of its FMIS-related project experience in various regions of the world, the average implementation period for a TABMIS component is 7.9 years. The appraisal team should have pushed for and Bank management should have accepted a much longer implementation horizon than the five years maximum for Bank investment and technical assistance projects, especially in a country like Vietnam, with its background of rigid planning and a highly decentralized structure of government. (ii) Underestimating the incentives for the main officials in the pilot provinces of the lack of a budget law requiring rolling MTFFs/MTEFs; (iii) Underestimating the number of users of TABMIS three-fold and the total budgetary transactions processed by the system five-fold between its design and full roll-out; (iv) Misidentifying two key associated outcomes (better planning and fiscal sustainability) and failing to identify outcomes related to transparency and accountability that are monitorable and measurable in the results framework at the design stage; and (v) Failing to specify suitable quantitative and qualitative indicators of outcomes, even for those measurable outcomes that could readily be identified (e.g., accuracy, timeliness and reporting of budget execution) and not establishing adequate baselines and targets. Quality-at-Entry Rating: Moderately Unsatisfactory b. Quality of supervision: The Bank devoted large and adequate resources to supervision, with frequent missions and field office-based staff overseeing closely day-to-day operations of the project. There was also good Bank oversight over fiduciary compliance. The restructuring actions and additional financing, including reallocation of funds, and changes to selected activities all appear justified ex-post (e.g. given the three-fold increase in the number of TABMIS users across the country and across all level of government). However, detracting from the quality of Bank supervision, both the additional financing and the three restructurings were bunched in the last three years of implementation (2011-2013), after the project had experienced already considerable delays (four years following the original closing date). Despite consistent ratings of development objective (DO) and implementation progress (IP) as satisfactory through February 2008, the Bank ought to have been concerned earlier at the slow pace of disbursements (US$10.50 million cumulative or 20% of the original project cost only 12 months away from the original closing date). The Bank should have reflected this in its supervision ratings and should have focused on the need for project restructuring (and additional financing) much earlier than 2011. Had the Bank intervened sooner to mitigate the delays, especially with the MTEF pilots and the (main) TABMIS components, it could have helped the project to achieve the same outputs sooner than a decade (or longer) after effectiveness. Issues with the M&E framework and indicators (such as ill-defined international best-practice) should have been apparent early-on and addressed during supervision. To sharpen the project’s PDO and results framework and remedy the lack of quantitative benchmarking and monitoring of its key associated outcomes, the Bank could have sought a deeper restructuring of the project involving revisions of the objectives and/or the establishment mid-way of some outcome indicators and baseline values. From this standpoint and other issues listed above, the Medium-term Restructuring in 2006 constitutes a missed opportunity to undertake a deeper restructuring sooner. Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government exhibited a commitment to the project’s objectives, prepared timely actions plans, procurement plans, and disbursement estimates acceptable to the Bank, and worked closely with the Bank’s project team. The institutional set-up that it put in place to manage the project, and especially a well-functioning steering committee, was effective, according to the ICR. The project never suffered from lack of counterpart funding. On the other hand, the Government should also be held partly responsible for some of the shortcomings noted above under Bank performance, and especially those related to the inadequacy of the M&E framework and for the late response to the initial delays in implementation. The Government, moreover, failed to address in a timely manner during the past dozen years from this project’s effectiveness other key reforms related to the broader PFM agenda but essential to achieve some of the project’s specific objective. This includes for example drafting and submitting much earlier than it actually occurred to the National Assembly a new Budget Law to ensure nationwide integration of budget and planning and implementation of the MTFF/MTEF, as recommended by the 2004 PER and the 2008 (unpublished) CFAA report and subsequently highlighted in the 2013 PEFA and 2014 FTR, and also to ensure monthly and quarterly public reporting of budget execution. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The MOF officials performed well, according to the ICR, on all fronts: leadership, commitment, institutional and implementation arrangements, bid evaluation, staff allocation, counterpart funding, setting up a dedicated telecommunication network for TABMIS, developing interfaces with commercial banks, training of staff, and issuing necessary regulations and guidelines. The ICR actually commends the MOF for its efforts to resolve the technical and contractual issues related to TABMIS and DMFAS that delayed the project in the later years (it is silent on its role during the early years). Project financial management and procurement arrangements were adequate and in compliance with Bank’s procedures. Status reports by the MOF were timely and informative. On the other hand, the MOF, the PMU and other implementing agencies (MPI, pilot sectoral ministries and pilot provinces) ought to share some of the responsibility for the initial delays in implementation and the subsequent late response. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E framework was inadequate, due both to outcome indicators that were not readily measurable and to the lack of any quantitative baseline value and target. According to the project’s task team, there were no Bank-wide guidelines in the early 2000s regarding such outcome indicators for FMIS-type projects. However, budget execution in key ministries could have been easily benchmarked and targeted in quantitative terms, as in fact was the case for other Bank operations in different countries. The time delays in budget publications could have been similarly measured. Also, it was reasonable to expect more of an effort at the outset by the project’s appraisal team to benchmark and then monitor the quality of budget planning, execution and reporting. b. M&E Implementation: During project implementation and at completion, no effort was made on the part of the Bank project team or the project’s Inter-Ministerial Steering Committee (set-up before project approval) to review and identify outcome indicators and gather the data that could have helped establish mid-project benchmarks and targets. There were several opportunities to retrofit given the ten year implementation period that include additional financing and three extensions Their focus was only on the project’s implementation challenges and on the delivering of the promised outputs. M&E implementation might have been enhanced through additional/ad-hoc monitoring of results such as whether civil servant trained were retained and used skills thus acquired in their jobs. c. M&E Utilization: The only indicators and information provided in the ICR refers to the project’s outputs rather than the PDO and its key associated outcomes. Recent indicators of PFM-related performance in the form of scores or summary qualitative assessments found in the 2008 CFAA (unpublished), the 2013 PEFA and the 2014 FTR, and also in the WBI Governance Indicators and the annual Country Performance Institutional Assessment (CPIA) ratings are ignored in the ICR, even though they can shed some useful light on the current state (but not the trend over time) of various PFM dimensions relevant to the project. Similarly, there is no effort to identify and present any indicators relevant to macroeconomic, fiscal and debt sustainability that are normally found in the economic reports of the Bank and the IMF. On the other hand, the most recent PEFA and the FTR confirm that the Bank is continuing to monitor PFM performance. And these recent assessments provide useful guidance for priorities for future reforms and for related Bank support. M&E Quality Rating: Negligible 11. Other Issues a. Safeguards: No safeguard issues are raised in the ICR. b. Fiduciary Compliance: No major compliance issues are raised in the ICR, which mentions close monitoring by the Bank during implementation of fiduciary compliance and describes the project’s financial management generally in compliance with Bank procedures and fiduciary covenants. Procurement was also conducted in full compliance with the Bank’s rules and requirements, according to the ICR. Most procurement documents and contracts were reviewed ex-ante by the Bank and found to be of high quality and observing of all applicable procedures. Only two minor issues are noted to have occurred during the early stage of the project, related to the quality of interim financial reports and some tax withholding on payments to IBM, the main TABMIS contractor. Both issues were later remedied. c. Unintended Impacts (positive or negative): The ICR lists a number of unintended impacts. However, most if not all should have been expected and do not really constitute unintended outcomes. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Moderately Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome: Bank Performance: Moderately Moderately IEG’s lower rating reflects various Satisfactory Unsatisfactory shortcomings in QAE and during supervision, especially the inadequacy of the M&E framework and the failure to address the mounting implementation delays at the time of the project’s medium-term-review. Borrower Performance : Satisfactory Moderately IEG’s lower rating stems largely from Satisfactory the Government’s partially shared responsibility for some of the shortcomings noted under Bank Performance and for the slow pace of adoption of other key PFM reforms. Quality of ICR: Unsatisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR draws many important lessons, with the three most important summarized below and adapted to reflect IEG’s views.  The timeframe for a FMIS project addressing many layers of government needs to be realistic and based on international experience , so that unnecessary extensions of the closing date can be avoided . Alternative lending instruments such as Investment Project Financing (funding most of the cost of the system, with the possibility of later additional financing) could also be considered to deal with long and uncertain implementation timeframes.  Consensus building among various stakeholders and a campaign of information at all levels of government would be advisable before rolling -out a new FMIS. This is due not only to the complex nature of an information system replacing many old legacy ones, but also to the institutional changes it requires in a government’s established budget and asset classification, in administrative responsibilities, in transparency, and in general in its business processes.  The incentives motivating the officials in charge of or otherwise affecting the success of pilot projects that are outside the current legislative and regulatory environment need to be understood and addressed effectively. There were difficulties in implementing the pilots of MTFF and MTEF in four selected ministries and four selected provinces in Vietnam. Despite efforts to integrate the MTEFs into their annual budget cycle, given that it was only a pilot, many officials saw them as an added burden rather than as part of their ongoing duties. In order to earn their commitment, interest and participation for the implementation of pilot projects, the government should align its guidelines, staff work plans, any financial incentives (such as bonuses), personal performance assessments and career promotions to the pilots’ goals.  [Lesson from IEG Evaluator ] Outcome indicators need to be clearly identified and baseline and target values need to be established , to allow for adequate monitoring and evaluation of progress towards the development objectives and not only implementation progress of project activities . At completion, available evidence needs to be drawn from all internal project sources as well as external studies and primary sources to shed light on the key outcomes. Only reporting on the project’s outputs is not sufficient to argue that the desired outcomes have been achieved, even when a results chain is presented. 14. Assessment Recommended? Yes No Why? Given the many gaps noted in the currently available evidence for assessing the project’s outcomes, the next phase of PFM-related reforms, and the confidence expressed in the comments by Bank management and the Government that strong positive outcomes from the project will be clear within three years, it would be useful to verify in more depth the development outcome and impact of the project’s outputs so far and those still pending, such as approval expected by mid-2015 of the new Budget Law. This would be appropriately achieved through a combined assessment of the PFMRP with the PFM-content of PRSC operations, the overall PFM-related AAA services, and the planned project in support of GFMIS. 15. Comments on Quality of ICR: The ICR provides the available evidence (especially qualitatively) at the time of writing on the outputs listed in the PAD. It also explains well the rationale for the delays that required implementation to be stretched to ten years, the need for additional financing towards the end of the project, and the restructuring of some of the projects’ activities. The lessons learnt section is also quite extensive. It suffers, however, from a general failure to explain and justify the basis of many of ratings. It has other important weaknesses discussed below that detract from ICR quality:  The report is written in simple language that is generally easy to read. However, it contains too many typos and awkward turns of phrase, as well as pro forma qualitative statements that are poorly substantiated. Furthermore, important aspects of implementation experience are not mentioned; for instance, the role and effectiveness of the Inter-Ministerial Steering Committee (mentioned in the PAD) are not discussed in the ICR.  It does not explain why disbursements were little and slow until well after six years after Board approval, despite initial speedy effectiveness.  Given the weaknesses of the M&E framework especially for outcomes, the ICR team could have used other sources of data to supplement output level indicators including recent PFM assessments or the 2011 World Bank retrospective study of FMIS projects that contains some information useful for cross-country comparison of unit cost indicators.  The ICR’s section on relevance of design is incomplete and superficial, the section on efficacy lacks evidence of outcomes and does not even mention the transparency objective, the sections on Bank and Borrower performances underplay the underestimation at appraisal of costs and time of implementation as well as the inadequacy of the M&E framework, though the ICR itself describes it elsewhere as flawed because of its almost exclusive concern for outputs.  There is no reference to the second PRSC series, which included substantial policy actions in the PFM, area and how the two Bank teams collaborated. It also does not draw any lessons on how development policy and technical assistance operations such as the PFMRP may complement one-another. a.Quality of ICR Rating : Unsatisfactory