ICRR 14536 Report Number : ICRR14536 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 11/08/2014 Country : South Sudan Project ID : P098639 Appraisal Actual Project Name : Core Fiduciary US$M ): Project Costs (US$M): 6.0 16.3 Systems Support Project (tf056336) L/C Number : Loan/ US$M): Loan /Credit (US$M): 3.0 13.4 Sector Board : Public Sector Cofinancing (US$M): US$M ): Governance Cofinanciers : Board Approval Date : 02/21/2006 Closing Date : 09/30/2008 03/31/2013 Sector (s): General public administration sector (100%) Theme (s): Other accountability/anti-corruption (50% - P); Public expenditure; financial management and procurement (50% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Iradj A. Alikhani Michael R. Lav Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the original Grant Agreement (GA) the project development objective (PDO) was to “assist the recipient in putting in place a robust fiduciary framework for channeling of Multi Donor Trust Fund (MDTF) and Government of South Sudan (GoSS) counterpart funds and provide reasonable assurance regarding the use of these funds.� The Final Project Proposal (FPP, equivalent to a Project Appraisal Document, p .10) uses a differently worded but equivalent definition . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project provided fiduciary support to other projects funded by the MDTF and consisted of two initial components to which a third was added in 2010. The components were the following : 1. Accounting – Cost at appraisal US $3 million; actual cost US $4.5 million : Support the contracting of a consulting firm experienced in project accounting to work with the GoSS (through the Ministry of Finance and Economic Planning, MOFEP ) to establish the Project Disbursement Unit . This Unit will be responsible for preparing and issuing the Project Implementation Guidelines and accounting for all project-related expenditures that are either funded out of the MDTF or pooled with GoSS for a period of two years. It will take responsibility for providing fiduciary assurance to GoSS and MDTF regarding the use of funds . 2. Auditing - Cost at appraisal US $3 million; actual cost US $4.5 million : Support the contracting of a consulting firm experienced in auditing to undertake audits in Southern Sudan . 3. Procurement - Cost at appraisal US $0 million; actual cost US $2.0 million : Provision of technical assistance, goods, services and operating costs to enhance and strengthen the Recipient ’ s public procurement functions including : (i) finalization of public procurement legislation; (ii) strengthening the procurement policy unit in MOFEP; and (iii) retaining the services of a procurement agent to support MOFEP in carrying out day-to-day GoSS procurement and facilitate the decentralization of public procurement functions of Recipient’s ministries, departments and agencies . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Total funding of US$16.3 million for the project was provided through : (a) two Trust Funds (TF) under the MDTF, totaling US$13.4 million (first TF funded US$8.2 million and second TF funded US$5.3 million – US$0.1 million difference in total is due to rounding off ) and Government counterpart funds amounting to US$ 2.9 million. The grants, both of which benefitted from additional financing (as listed below) during project implementation were fully disbursed at closing. The required Government contribution was initially estimated at US$ 3 million. As a consequence of savings on one contract, US$ 2.9 million was disbursed The Table below provides details and key dates related to the project (ICR Tables 2, 4 and 5). It should be noted that MDTF projects are not subject to approval by the World Bank Board . Consistent with the governance framework for such projects all restructurings are approved by the region ’s management. TF056336 TF 056336 – Original TF Purpose Date Features Approval 02/21/06 US$3 million Effectiveness 09/06/06 Amendment 05/18/07 100% financing of expenditure till 09/07 Additional 12/14/07 US$2.6 million Financing Amendment 10/28/07 Extend closing date to 09/30/12 (also actual) and procurement eligibility Amendment 10/28/07 Eligibility of operating cost Amendment 09/15/09 Additional financing of US$2.6 million and 100% financing of expenditure TF097653 TF 097653 – Second TF Purpose Date Features Approval 09/01/10 US$3 million and procurement component added Amendment 06/22/11 Extension of closing date (both TFs), reallocation of funds, and revised /updated (not listed in results framework. ICR) Amendment 03/12/12 Additional financing of US$1.36 million Amendment 08/02/12 Additional financing of US$1.0 million and extend closing date to 03/31/2013 3. Relevance of Objectives & Design: a. Relevance of Objectives: High – The project was originally identified as part of Joint Assessment Mission (JAM, ICR para. 3) which identified putting in place adequate fiduciary capacity as a precondition to funding and implementing all the various projects under the MDTF (up-to 2010 procurement support was financed by a different operation ). It was thus a key part of the joint country and donor strategy at the time and seen as critical element for sound implementation of portfolio of projects . It addressed two concurrent priorities : (a) ensuring adequate fiduciary capacity for implementation of projects; and (b) contributing to the medium- to long-term objective of building the country’s fiduciary systems. World Bank country strategy at the time of project closing was underpinned by the 2013/14 Interim Strategy Note (ISN). Continued support to Public Financial Management (PFM, including procurement) was one of eight focus areas of the ISN (Table 3) as well as for other donors. Activities envisaged under the ISN would be mainly in terms of knowledge products that would deepen initiatives introduced under the project – including support to the audit chamber . Similarly this area is a key priority for Government identified in its present strategy (the South Sudan Development Plan ). b. Relevance of Design: Substantial - The technical assistance lending instrument for the project was appropriate . The simple design was flexible to accommodate additional financing . The project objective was clearly stated and modestly ambitious – capacity building within Government which in practice became part of project implementation might have been specified more explicitly from the outset . The original project components were well -aligned with project objectives, as was the procurement component added in 2010 once earlier funding under another parallel MDTF project was completed. The chain of logic provided adequate conceptual links between project activities and expected results and outcomes, including for the additional component . Attribution was also clear. The link between output and outcomes was initially imperfect, because indicators were well stated theoretically, but many were not readily measurable. This problem was adequately addressed quite late during the 2010 restructuring and subsequent 2011 amendment of indicators (ICR Table 1). The rating for design takes into account that the minor weakness in the chain of logic was addressed and did not ultimately undermine the ability to measure impact . 4. Achievement of Objectives (Efficacy): Substantial - The single objective of the project is framed clearly and simply around putting in place a robust fiduciary framework for channeling MDTF funds; in practice this can be interpreted as both building capacity within government (a medium- to long-term goal) while, more importantly from the standpoint of narrow project objectives, providing adequate fiduciary systems to support the implementation of projects through outsourced experts. Notionally, while not stated explicitly in the PDO, a fiduciary framework has two dimensions: Financial Management and Procurement (which may be fully mapped to the first two and last component respectively even if the related component was added in 2010). Evidence presented in the ICR and associated with indicators The ICR (datasheet and annex 2) reports the progress on retrofitted indicators, which as noted in the earlier section on design are fully attributable and by and large adequately measure project contributions to attainment of project objectives. All PDO targets were thus met: • For each MDTF project: (i) monthly bank reconciliation; (ii) quarterly statement of source and use of funds; (iii) annual project audits; and (iv) annual audits of GoSS accounts . The latter (for 2005-2008) was submitted to the National Legislative Assembly . • Six Ministries, Departments and Agencies comply with procurement regulations . This indicator was met through the adoption of guidelines and establishment of procurement plans . However, it does not quantify outcomes; for example the number of times and /or value of contracts for which exceptional procedures (e.g., sole source) were used might have been a better measure of improved quality of procurement, even though data collection could have been more challenging . In addition, a Procurement Bill was prepared (ICR para. 27) and subsequently approved by council of ministers . It has not yet been discussed by the national assembly due to political turmoil during 2014. On the other hand, the procurement data management system was not installed due to another instance of a failed tendering process (ICR para. 32). Furthermore, all the output targets were achieved (a handful are very close to and /or overlapped with a PDO indicator or were process inputs and are not repeated here ), including the following: • Draft audit legislation and code of conduct adopted (detailed in para. 31). • Training to 31 Ministries, Departments and Agencies, a large number of staff of implementation entities provided – this included hands-on training on procurement (full list in ICR Annex 2). Evidence from other MDTF projects being supported The ICR provides limited evidence of project impact from the standpoint of other MDTF projects . The table below summarizes findings by finalized IEG reviews of seven completed such projects . It indicates some mixed results in terms of efficacy when the support provided by the present project is assessed from the standpoint of its impact on beneficiary projects . In summary, the following points should be highlighted : (i) Most projects had to hire fiduciary specialists to address problems that could apparently not be resolved through the external expertise that had been provided . This raises the question whether some duplication of assistance occurred . (ii) On the positive side, most audits were unqualified and timely for most projects and FM issues appear minor – however, attribution to the present project is partial as most individual projects had their own fiduciary staff. (iii) Some projects also needed to build capacity within their Project Implementation unit (PIU). In the case of two projects, shortcomings with the quality of work of the procurement agent hired by the project are specifically noted, while in another case such support is deemed satisfactory . The above observations are consistent with the conclusion that by and large support provided by the project was generally useful and had a favorable impact, but needed to be complemented with additional technical assistance financed by beneficiary projects . Therefore, the fact that only modest fiduciary problems were experienced by the MDTF project cannot be solely attributed to the contributions of the fiduciary project discussed herein. Project Financial Management (FM) issues Fiduciary Issues Noted in IEG noted in IEG ICRR Reviews ICCR Reviews 1. Multidonor Rehabilitation Problems at first but resolved by None Education project end. Evidence of internal recruitment of expert. 2. Umbrella Program For Challenges at first, resolved later . Initial problems, but resolved. Health System Development Audits unqualified. Evidence of internal recruitment of expert. 3. Support To Agriculture And Challenges at first, resolved later . Notes problems with procurement Forestry Development Audits unqualified. Evidence of agents. Resolved through capacity internal recruitment of expert. building at PIU 4. Livestock And Fisheries Challenges at first, resolved later . Procurement was a major issue that Development Project Evidence of internal recruitment of pervaded all projects managed by expert. the Multi-donor trust fund. Although the Government put in place a centralized procurement agency within the Ministry of Finance and Economic Planning with the support of a consulting firm as procurement agent and an individual consultant as Procurement Advisor, procurement did not work as expected due to inadequacy of hands-on support provided by the Procurement Advisor. 5. Hiv/aids Financial management was Procurement was satisfactory satisfactory throughout project throughout implementation. implementation, largely due to a Independent procurement reviews substantial amount of project were carried out by an independent investment and staffing in this area . team hired by the Bank to review Audits were unqualified. procurement in all MDTF-supported projects. 6. Umbrella Program For An external agent was contracted to Procurement was effectively carried Health System Development provide fiduciary management. No out. major fiduciary problems were reported. Audits were unqualified. 7. Private Sector Development No major issues, but had PIU staffed The private agent contracted to with financial management experts. handle procurement did not perform One qualified audit due to lack of well. reporting by UN agency. Overall, the above evidence of achievement seems consistent with a “Substantial� rating for the project, especially as some activities such as training to various parts of government went beyond narrowly defined project objectives. However, there may have been some shortcomings in achieving the relevant goals, especially during the initial project implementation period, problems were experienced in two out of seven beneficiary projects with respect to procurement support they received, and there were issues with providing training to counterpart staff (ICR para. 20). These suggest moderate shortcomings in the achievement of objectives. 5. Efficiency: Substantial . Rates of returns were not calculated for this technical assistance project . Two factors would affect the efficiency of the project with respect to utilization of resources : (i) The alternative to the project would have been to fund required fiduciary experts for each MDTF project and separately build capacity within Government separately . Such an alternative approach would appear much less efficient. However, evidence from other projects (presented in the previous section ) suggests that such duplication may have taken place nevertheless and if so this constitutes a somewhat inefficient use of project resources – however given mixed information from beneficiary projects this may have been a minor issue . (ii) The project was able to accommodate successive additional financings, obviating the need to prepare new projects. It also helped ensure that fiduciary support to projects was continuously available throughout the period without any discontinuity . The ICR also cites (para. 36 and Annex 4) a value for money analysis conducted by the Monitoring Agent which concluded that support provided was utilized relatively efficiently by beneficiary projects . It should be noted that this was not a totally independent assessment as the Monitoring Agent was funded by the project and that most projects employed fiduciary staff which in the absence of a more detailed analysis makes attribution unclear . A minor source of inefficiency relates to effectiveness delay and initial disbursement lag till 2010 due to procurement delays. Once the decision to hire individual experts instead of firms was implemented, the project was able to disburse four and half times the original grant over a period that was about three times longer that envisaged at time of initial project approval – this suggests relative efficiency in use of project funds . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Strategic relevance was high as the project was positioned to provide fiduciary support needed to implement the rest of the MDTF portfolio. Relevance of design was substantial because of good logical links between objectives and components . Weaknesses in indicators were eventually addressed in a way that did not undermine the ability to measure project achievements throughout the implementation period . Project efficiency was substantial, with certain achievements exceeding stated project objectives, while information from projects that were supported suggesting there may have been some shortfalls . Finally, even though there were initial implementation delays, the project was implemented over a reasonable period and value for money analysis points to substantial efficiency . In summary, the case presented in the ICR is consistent with a Satisfactory rating for project outcome, with some of its aspects perhaps warranting an even higher rating . However, once additional evidence is considered, notably from projects it supported, it is apparent that weaknesses were present . Nevertheless, on balance, overall project outcome is satisfactory . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR rates the risk to development outcome as moderate, largely because support to projects was provided as needed and will no longer be required once these projects were /are completed – these gains would be irreversible. However, the project also provided capacity building to the government which seemed sustainable at the time the ICR was prepared. However, the conflict that has been ongoing during 2014 is likely to undermine some institutional capacity and reduce the sustainability of project activities related to this area . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project as designed originally was well focused and did not overreach – a risk that is often not well managed in similar operations. It included appropriate governance structure (ICR para. 23) and the short initial implementation period was not an issue as the design was robust enough to allow for easy scale -up and extension of activities – as was the case. The design integrated adequate country and sector knowledge and considered lessons learnt in countries in fragile and conflict circumstances in Asia (Timor Leste. Afghanistan and Iraq). It also took into account consultations with stakeholders (ICR para. 66) However, the replicability of the approach entailing recruitment of firms that had worked well in East Timor (and more recently in Iraq in the case of an energy operation ) proved problematic. Delays in procurement related to this design affected initial implementation and may have been partly due to imperfect TOR defined during preparation. Similarly, weaknesses in M&E indicators had to be addressed during implementation . Considering that the project was prepared under time pressure, these were relatively minor weaknesses consistent with Satisfactory quality at entry (the ICR rating of Moderately satisfactory seems conservative and perhaps does not consider additional financing as part of the analysis ). at -Entry Rating : Quality -at- Satisfactory b. Quality of supervision: The Bank supported project implementation under difficult country circumstances . Key procurement tenders were initially problematic. Issues were eventually sorted out by pragmatically opening up the tender to individual experts. The Bank team processed efficiently a number of additional financings and other enhancements to the original project design . Project supervision was undertaken periodically and, given weak capacity at the level of counterparts, the team provided hands -on advice as needed. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government was generally supportive of the project and chaired the oversight committee . However, its weak capacity was an issue throughout project implementation, even though collaboration with team of experts funded by the project alleviated this issue . Notable under-performance on the government side included (a) indecision in recruitment of the procurement advisor (para. 20), and (b) the suspension of auditing staff (ICR para. 26) which resulted in delayed establishment of audit functions . Both these issues constitute moderate shortcomings reflected in the Moderately Satisfactory rating for Government performance. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: Much of the implementation support was provided by outsourced consultants who appear to have performed satisfactorily. The implementation agencies, notably the Ministry of Finance, acted appropriately and fulfilled its expected responsibilities . The other implementation agency (National Audit Chamber) also appears to have been quite effective (ICR para. 77). Delays in procurement were primarily due to poor response by consulting firms and not attributable to the agencies . There were no fiduciary issues affecting project implementation and overall implementation agency performance appears satisfactory . Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Project objectives were clearly specified . The FPP lacked a consolidated presentation of M&E . The three original PDO indicators (FFP p.10) were logically connected to project activities but were largely conceptual as they lacked sufficient specificity and, at times, baselines – although for some indicators one was not needed . For example, two indicators refer to improved transparency and accountability but indicators needed to measure these were undefined. Other output indicators for the consulting firms are also presented (FFP p.12) and could be considered adequate intermediate outcomes for the project – some, somewhat reworded for increased clarity are presented as such in the ICR (pp.iv-v). The 2010 and 2011 revisions to components and indicators largely addressed the initial shortcomings of the project’s M&E by clarifying the original indicators, breaking them down and ensuring that they were quantifiable . Indicators related to the new procurement activity were also added, even though these were subsumed with the indicator on use of IT – which did not have a specific indicator throughout the project . Project logic would have been served better with the introduction of a new PDO indicator related to procurement and the addition of indicators measuring impact at the level of projects that were supported – as done in the review herein. It would have also been preferable to revise the indicators much sooner, for instance in the course of one of the amendments that were processed after the project became effective . Notwithstanding shortcomings, the M&E design and its subsequent detailed specification was simple and indicators were closely related and attributable to project activities . It provided a good basis for measuring expected project impact in line with development objectives – which were output-oriented in nature. b. M&E Implementation: The key aspect of M&E implementation was the improvement brought to indicators, albeit somewhat late in the project implementation process. Furthermore the Monitoring Agent carried out a value for money study, summarized in the ICR (Annex 4) which complements the project M&E. However, taking a more dynamic view where new projects, irrespective of funding sources, enter the portfolio, the ability of the Government to continue monitoring the fiduciary aspects of these projects may be contingent upon continued technical assistance and capacity building in the medium-term. c. M&E Utilization: Project stakeholders notably the Steering Committee met quarterly and utilized the various monitoring reports prepared by the project to review progress . The M&E thus provided a feedback loop which was used to redirect activities as needed. M&E Quality Rating : Substantial 11. Other Issues a. Safeguards: The project did not trigger any safeguards policies . b. Fiduciary Compliance: The project’s main focus was to building fiduciary capacity for implementation of MDTF and was thus a direct beneficiary through strong fiduciary compliance . According to the ICR (para. 38) financial management was robust throughout the project and mitigating measures to address risks and shortcomings proved effective . All reports were submitted and there were not any significant control issues identified . The project also benefitted from effective procurement capacity and there were no procurement issues per -se – i.e., in terms of governance etc. However, it experienced significant issues in implementing key procurement tenders based on the original design (hiring of two firms). According to the ICR and additional information received these issues included : (i) market failure (firms unwilling to take the country risk and not responding to tenders); (ii) the firm with the best offer being excluded following an investigation; (iii) in one case the other qualified respondent demanding renegotiations of terms and conditions; and (iv) with the benefit of hindsight a different approach to both the terms of reference and outreach to consultants may have proven more conclusive – this latter issue is taken up under lessons learnt . The problem was eventually resolved by recruitment of individual consultants (which may have in practice shifted risk mitigation costs from the consultant to the Borrower) even though in the case of the procurement data management system even this more flexible approach did not succeed . Furthermore, the monitoring agent noted that procurement document reviewed were unclear how short lists had been established (ICR Annex 4). c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Risk to Development Moderate Significant Political turmoil since the ICR was Outcome : completed. Bank Performance : Satisfactory Satisfactory NB: Bank performance rated Satisfactory in the IEG review. Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: An important lesson learnt presented in the ICR (section 6) concerns the importance of, and difficulties in balancing the need to ensure outsourced consultants complete the short -run work they have been hired for, while at the same time undertaking parallel capacity building and skills transfer in areas fairly narrowly related to the project logic, while avoiding taking on broader Government functions . Other more generic lessons listed include the importance of timely appointment of counterparts and ensuring stakeholders have realistic expectations about the project from its outset . Another lesson mentioned concerns the observation that individual consultants may provide more efficient and cost effective services, and quicker knowledge than firms . This observation may be correct but cannot be drawn from the ICR write-up. It also ignores the possibility of capture, possible internalization of security and certain other costs by firms, and seems partly contradicted by experience elsewhere cited in project documents. A final lesson from this review is that difficult post conflict situations require an innovative and flexible approach towards procurement, which may include upstream outreach to qualified firms prior to launch of tenders to gauge their interest and, if needed, adjust terms of reference . 14. Assessment Recommended? Yes No Why? The assessment is recommended for the following reasons : • This recommendation has been made for other South Sudan MDTF projects . A consolidated assessment of MDTF projects would allow relevant cross -cutting lessons to be learnt on how to engage with fragile states . • There is a need to better understand the value added of this project to other MDTF projects and better quantify whether its activities were fully complementary or somewhat redundant because of parallel assistance funded within other projects. • Deepen specific lessons learnt, notably with respect of the relevant merits of contracting fiduciary firms versus individuals (who may be more vulnerable to capture by vested interest ), reasons that led to failed procurements based on the original design, and how might such outcomes be avoided through better outreach to potential service providers, improved terms of reference etc . 15. Comments on Quality of ICR: The ICR provides a concise and clear description of key events during project implementation . Its ratings are by and large well explained and adequately justified . Minor deficiencies include: • Insufficient details on failed procurements . The related lesson learnt on merits of firms versus individual consultants is also debatable . • Not using information on fiduciary performance of other projects that were supported . • 2011 amendment of indicators not cited properly . a.Quality of ICR Rating : Satisfactory