ICRR 13072 Report Number : ICRR13072 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/06/2010 PROJ ID : P075911 Appraisal Actual Project Name : Third Social Action US$M ): Project Costs (US$M): 78.08 76.85 Fund (masaf III) Country : Malawi Loan/ US$M ): Loan /Credit (US$M): 60.0 n.a. Sector Board : SP US$M): Cofinancing (US$M ): Sector (s): Other social services (60%) General agriculture fishing and forestry sector (10%) Micro- and SME finance (10%) Roads and highways (10%) General water sanitation and flood protection sector (10%) Theme (s): Participation and civic engagement (29% - P) Other social protection and risk management (29% - P) Gender (14% - S) Social safety nets (14% - S) Social analysis and monitoring (14% - S) L/C Number : C3782; CH045 Board Approval Date : 05/30/2002 Partners involved : Closing Date : 12/31/2006 12/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : Kris Hallberg Robert Mark Lacey IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: According to the PAD, the project ’s development objectives (PDO) were: “to empower individuals, households, communities, and their development partners in the implementation of measures which can assist them better in managing risks associated with health, education sanitation, water, transportation, energy, and food security, and to provide support to the critically vulnerable through a variety of sustainable interventions .� According to the Development Financing Agreement (DFA), the PDO were "to provide a variety of sustainable interventions aimed at: (a) empowering individuals, households, and communities to better manage risks associated with the provision of basic services in health, education, sanitation, water, transportation, energy and food security; and (b) providing support to vulnerable groups ." MASAF III was an Adaptable Program Loan, the first of a three -phase Community Empowerment and Development Program (CEDP). According to the PAD, the overall program objective of the CDEP is "to achieve capacity building for improved service delivery by communities, Local Governments and Sector Ministries within the Malawi PRSP, with decentralization as a key strategy, so that Malawi can achieve its MDGs ." This ICR Review evaluates the project (the first phase of the CEDP) on the basis of the PDO statement in the DFA as this is more monitorable. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): MASAF III had five components: CDPs ) (estimated cost $28.48 million; actual cost $36.83 million): financing Community Development Projects (CDPs) for Community Managed Projects (CMPs) and Local Authority Managed Projects (LAMPs) involving demand-driven and community-based socio-economic infrastructure. SSP ) (estimated cost $24.0 million; actual cost $19.94 million): support targeted to Social Support Program (SSP) vulnerable groups through NGOs/CBOs to stimulate and support traditional coping systems . Community Savings and Investment Promotion (COMSIP ) (estimated cost $11.87 million; actual cost $2.12 million). Business skills training, provision of 80 percent of the cost of infrastructure required for Savings Clubs, and facilitation of the formation of Community Savings and Investment Groups (COMSIGs). Transparency and Accountability Promotion (estimated cost $6.67 million; actual cost $6.65 million): training of implementing agency staff, support for the Technical Support Services Framework and the Secretariat for the National Advocacy Committee for Community Empowerment and Accountability (NACCEA), and the development of an MIS system and a knowledge and information sharing system (KISS). Institutional Development (estimated cost $7.06 million; actual cost $11.31 million): provision of technical staff to the MASAF Management Unit, and support to local authorities for the implementation of sub -projects.MASAF III had five components: d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: In response to the nation-wide food shortages resulting from the 2004-05 drought, the Government implemented an emergency response program (Public Works Program-Conditional Cash Transfer, or PWP-CCT) within the LAMP sub-component. Most of the $12.1 million reallocated to PWP-CCT came from the COMSIP component, leaving COMSIP with few resources. IDA financing consisted of a SDR 24.10 million (US$32.8 million) credit and a SDR 20.0 million (US$27.2 million) grant. The ICR does not report on actual IDA financing, but it is likely to have differed from appraisal estimates in US$ terms because the IDA credit and grant were denominated in SDRs . The Borrower was supposed to contribute $7.68 million, local communities $5.6 million, and local NGOs $4.8 million. No information is provided in the ICR on actual contributions. The closing date was extended by one year; the ICR does not explain why . Furthermore, the Development Financing Agreement mentions an original closing date of June 30, 2006 while the ICR mentions it as December 31, 2006. 3. Relevance of Objectives & Design: Relevance of Objectives : Substantial . At the time of appraisal, Malawi was one of the poorest countries in the world; an estimated 65 percent of the population lived below the poverty line in 1998. MASAF III was intended to be an instrument to achieve sustainable poverty reduction, in line with the objectives of the Malawi Poverty Reduction Strategy of 2002 and the CAS. Relevance of Design : Modest . The project design underestimated the difficulty of handing operations over to local authorities (LAs) -- the complexity of project implementation and reporting procedures strained the already low levels of capacity in some LAs. The assumption that the Government would fully implement its decentralization policy also turned out to be overly optimistic . The objectives of empowerment and improved ability to manage risk were not well-defined and hence inadequately addressed in design . 4. Achievement of Objectives (Efficacy): (i) provide a variety of sustainable interventions aimed at empowering individuals, households, and communities to better manage risks associated with the provision of basic services in health, education, sanitation, water, transportation, energy and food security : Modest . While virtually all output targets were achieved or exceeded, there are questions about how sustainable the interventions were. The targets and actual achievements were as follows : Number of community management projects complete and operational 590. Actual: 571. At least 230 Local Authority Managed Projects (LAMPs) completed. Actual: 377. At least 150,000 persons reached with cash transfers and increased incomes . Actual: 1.56 million, suggesting a significant over-achievement. At least 72,000 individuals receive wages under LAMPs . Actual: 141,975. At least 72,000 vulnerable persons receive assistance under SSP . Actual: 670,000 vulnerable community members received assistance under the SSP and Public Works Program (PWP). 8,000 savings groups (COMSIGs) formed and operational. Actual: 8,392. 800 savings clubs (COMSICs) formed and implementing economic sub -projects. Actual: 32 COMSICs formed. The shortfall was due to the reallocation of funds from this component . At least 26,000 persons participating in Project Management Committees (PMCs), community-based organizations (CBOs), and Community Savings Clubs. The ICR does not contain data on the actual values of this indicator. Evidence on sustainability of physical infrastructure is mixed . The project had made provision for complementary procurements (such as chairs, tables, blackboards ) and for operation and maintenance, through matching grants to the LAs. However, the ICR reports that resource constraints facing the LAs have made these arrangements unworkable. There were operation and maintenance problems associated with sub -projects (ICR paras 25, 55, 57, 62). Health sub-projects faced particular sustainability issues although funding for maintenance was not assured for some other sub-project types as well. The ICR also notes that the compliance with environmental safeguards was an issue that persisted through the end of the project and poses a risk in terms of the sustainability of the project's development outcomes. The subprojects financed by the project were aimed at increasing local empowerment through supporting community-based procurement and accountability systems, participatory rural appraisals of sub -projects, and community scorecards. The project also aimed to develop a cadre of community -based project managers, accountants, procurement specialists in order to enhance local management capacity . Furthermore the project aimed at helping to integrate community empowerment processes into district planning through requiring LAs to develop by-laws to register Community Based Organizations, produce baseline data and targets required to achieve the 12 selected MDGs, etc, which was achieved in 40 LAs (compared to the target of 28 LAs). The extent to which these measures would result in enhanced empowerment would depend on the quality of these interventions and the complementary measures put in place, data on both of which is insufficient (actual improvements in empowerment were not rigorously monitored). ii) provide a variety of sustainable interventions aimed at supporting vulnerable groups : Substantial . The ICR (p. (ii) 14) states that, under the SSP, 75,336 (as against a target of 72,000) vulnerable people such as widows /widowers, the aged, orphans and foster parents, the destitute, disabled and persons affected with HIV /AIDS as well as malnourished children were directly targeted through interventions channeled through NGOs /CBOs. (Note that the ICR's Basic Information Table says that 75,336 households benefitted from SSP and PWP.) According to the ICR, the early Childhood Development Centers have directly improved child development through improved nutrition as a result of the feeding programs, and other benefits include economic empowerment of the orphans and the physically challenged through life skills acquired in vocational training provided through the project ’s interventions. To the extent that the nutrition programs prevented irreversible negative health impacts and the vocational training developed marketable skills, these interventions had sustainable effects . It also states that this project component has also provided a conduit through which other development partners such as UNICEF could channel their support to the vulnerable children, thereby providing a mechanism for continuity of such support . This was the first phase of an APL . While a number of measures under the project could contribute to the program goal of building capacity in communities, local governments, and sector ministries for improved service delivery so that Malawi can achieve its MDGs, the project M&E system was not well -designed to yield specific information on the program goal. 5. Efficiency (not applicable to DPLs): A cost-effectiveness analysis was undertaken by comparing the unit costs of MASAF III infrastructure (cost of schools per classrooms, cost of roads per kilometer, etc .) with those of similar projects implemented by other development stakeholders, including the DFID Education Program and the EU Micro -Project Program. The results indicated that MASAF III sub-projects were delivered at 15 to 40 percent lower unit cost than the comparator sub-projects. The MASAF III sub-project cost figures exclude community contributions (both cash and in-kind). The comparator programs did not require community contributions, so their cost figures may be expected to be higher than the MASAF III costs. The ICR and Regional comments note that differences in community contribution or quality were not so great as to explain the cost differences . Efficiency is rated substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The shortcomings in relevance of design and the mixed results on the sustainability of interventions amount to moderate shortcomings. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Most of the poverty-alleviating measures of MASAF III -- the CCT program, the SSP component, and wages for community infrastructure sub-projects -- were temporary in nature. Their long-run impact on poverty may come through improved access to economic and social services, and through household investments facilitated by temporary wage increases. There were operation and maintenance problems associated with sub -projects (ICR paras 25, 55, 57, 62). The design of MASAF III envisaged that it would be implemented within a decentralized governance framework, where LAs would be comprised of elected officials who would be supported by Government staff . Although this form of devolution occurred in a few sectors, the decentralization policy was not fully implemented . The lack of progress on decentralization creates a risk that the capacity built through the project at the local level will not be sustained . The ICR notes that the compliance with environmental safeguards was an issue that persisted through the end of the project and poses a risk in terms of the sustainability of the project's development outcomes . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Project preparation underestimated the capacity of LAs, and perhaps overestimated the degree to which the Government's decentralization policy would be implemented . More attention should have been paid during preparation to the monitoring and evaluation framework (see Section 10). Some of the objectives were not clearly defined (e.g., empowerment and improved ability to manage risk ), or measured. These are important failings, especially given that this project had the opportunity to build on the experience of MASAF I and II . Despite implementation delays, the Bank supervision team maintained the momentum of the project . The Bank responded with flexibility to the Government's request to reallocate project resources to components that directly addressed critical needs during the 2005 drought. Staff continuity was maintained throughout the project . The low justification of community disbursed resources, which was a cause of early delay, was addressed through the pro-activeness of both Bank and government teams -- the deployment of Justification Assistants and Auditors at various points speeded up justification rates, which improved from 24% in 2006 to 84% by September, 2007. However, the Bank was unable to ensure full compliance with environmental safeguards . The ICR points out that the issue was noted by the Bank during supervision, and appropriate steps were taken to address it through aggressive awareness campaigns throughout the sub -project cycle coupled with close follow -up by relevant sector staff. However, the problem of compliance persisted till the end of this phase of the program due, inter alia, to weak LA capacity. The ICR also points out that the Bank (and the Borrower) should have agreed earlier on how to retain critical capacity in the PMU to avoid the disruptions that occurred as a result of the downsizing (ICR para 17). At least some of the M&E problems could have been fixed through stronger supervision . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The Government supported the project through provision of technical staff constituting the various committees that appraised and recommended sub -projects. Overall, the project had widespread support at the political level, reflected in the continued high profile accorded to the MASAF program . However, more should have been done to avoid the abrupt changes in senior staff in the MASAF Management Unit (MU), about which the Bank was not consulted, and which, together with high lower level staff turnover, had a significant effect on implementation . The Government remains hesitant in implementing its decentralization policy, with which the project had been designed to support. Counterpart funds were delayed . Even with the slow pace of decentralization, the PMU provided enough support to LAs to enable them to implement sub-projects. It also supported a number of innovative management tools, i .e., score cards, Simba Mindset, Transparency and Accountability Promotion (TAP), communication strategy, etc. However, there were some management weaknesses. For example, there were lags in the justification of LA and community funds which could have resulted in distorting fund accountability and might have created room for fraud (ICR para 23). At the LA and community level, there was evidence of inflated material prices and supply of wrong materials (ICR para 25). The PMU shares some responsibility for the persistence of problems relating to compliance with environmental safeguards. a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Efforts seemed to focus mainly on monitoring of outputs. Because outcomes such as "empowerment," "institutional capacity," and "risk management" were not well-defined, M&E could not adequately track them . Changes in poverty resulting from the project could not be measured using the Government's household survey, which is conducted too infrequently. The project's targeting of specific vulnerable groups was not adequately measured . There was no clear methodology to allow attribution of outcomes to the project . The adaptability aspect of an APL is likely to be compromised by weak M&E. Implementation of the LAMIS system was significantly delayed, and data collection became backlogged . In some cases, the LAMIS system was not institutionalized within LAs, and it became merely a tool for managing MASAF sub-projects. An impact evaluation of MASAF III was prepared in 2008. The findings of the beneficiary assessment included in the impact evaluation are summarized in Annex 5 of the ICR, but evidence is also lacking on some outcomes such as empowerment. The ICR notes that due to limited capacity and institutionalization of the M&E system at the LA level, the sustainability of utilizing the system for effective management of projects at the LA level remains a challenge (ICR para 22). a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Compliance with environmental safeguards was an issue . Neither communities nor local authorities gave high priority to the implementation of Environmental Management Plans (EMPs). The ICR states that the issue was recognized by both the Bank and the Government during supervision, and steps were taken to address it through aggressive awareness campaigns throughout the sub -project cycle coupled with close follow -up by relevant sector staff, however, problems persisted through the end of the project . The ICR reports that there were no major problems affecting project compliance with fiduciary requirements . The financial management system worked well . Auditing arrangements also reportedly worked satisfactorily and the ICR makes no mention of any qualifications . While there were no procurement difficulties at the central PMU, some challenges did arise at the level of the LAs and communities where some suppliers and service providers inflated prices and supplied the wrong materials . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately There were mixed results on the extent Satisfactory to which the project supported sustainable interventions (See Sections 3-5). Risk to Development Moderate Significant Funds for operation and maintenance Outcome : are not assured. Slow progress on decentralization creates a risk that the capacity built through the project at the local level will not be sustained. Bank Performance : Satisfactory Moderately There were shortcomings in quality at Satisfactory entry and quality of supervision (See Section 8). Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Mainstreaming of capacity and resources to ensure compliance with environmental and social safeguards is needed at all levels of sub-project implementation. Sector norms and standards also need to be integrated at the community and local government levels . Sustaining community infrastructure sub -projects requires both plans for operation and maintenance at the sub-project selection stage, and a realistic chance of funding them . Careful attention to M&E is needed at several levels, especially for an APL : at the community level, monitoring and feedback from beneficiaries; at the project management level, a functioning MIS system; and at the national/regional level, evaluation methods and data that allow outcomes to be measured and attributed to the project. 14. Assessment Recommended? Yes No Why? To verify the ratings, examine issues of sustainability, cost -effectiveness, and environmental effects, and document lessons learned in supporting Local Authority capacity building . 15. Comments on Quality of ICR: The ICR has several significant shortcomings, resulting in an unsatisfactory rating for ICR Quality. The discrepancies between the statement of objectives in the Development Financing Agreement versus the PAD are not discussed. Data on actual financing (Annex 1) is missing. There is insufficient focus on outcomes . Reporting on the key performance indicators defined in the PAD is haphazard . Actual values for some indicators are not presented, and some of the targets do not agree with the PAD . There is no explanation how the number of beneficiaries were counted or why the number of beneficiaries was so much higher than the target values . There is very little or no information on the number and distribution of sub -projects under each component . There is no report on the Mid-Term Review or actions taken in response to its recommendations, nor an explanation for the one-year extension in the closing date . The ICR mentions a number of innovative management tools supported by the project, i .e., score cards, Simba Mindset, Transparency and Accountability Promotion (TAP), communication strategy, but does not discuss their pros and cons or lessons learned, thereby failing to maximize the learning value of the ICR . The discussion of Bank and Borrower performance is not in compliance with the guidelines which stipulate separate assessments for QAE and supervision in the case of the Bank and of Government and implementing agency in the case of the Borrower . The cost-effectiveness issue (e.g., community contribution calculations, and the cost differentials across agencies with/without community contribution) should have been discussed more fully . a.Quality of ICR Rating : Unsatisfactory