Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) Report Number : ICRR0021500 1. Project Data Project ID Project Name P143367 AFAAS Second MDTF Country Practice Area(Lead) Africa Agriculture L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-15730 31-Dec-2017 5,536,881.00 Bank Approval Date Closing Date (Actual) 23-Aug-2013 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 5,540,000.00 5,540,000.00 Revised Commitment 5,536,881.00 5,536,881.00 Actual 5,536,881.00 5,536,881.00 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives According to the Project Appraisal Document (PAD) (p. 15) and the Grant Agreement of October 31, 2013 (p. 5) the objective of the project was “to reform and strengthen Agricultural Advisory Services (AAS) in accordance with FAAP principles towards increasing agricultural productivity and food security.” On December 18, 2017 the project’s objective was revised to “support and strengthen Agricultural Advisory Services (AAS) in accordance with the Framework for African Agricultural Productivity (FAAP) principles”. Page 1 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 18-Dec-2017 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project included two components: Component 1: Support to Country Agricultural Advisory Services (AAS) for Engagement in Comprehensive Africa Agricultural Advisory Services (CAADP) Pillar IV (appraisal estimate US$10.5 million, actual US$4.64 million): This component was to finance the support of all AAS stakeholders, both public and private, guided by the FAAP principles, in their engagement in the implementation of CAADP Pillar IV in their various countries. This component was to finance the establishment and strengthening of country fora aimed at supporting AAS stakeholders at the country level. This component was to also support the regional and continental collaboration in AAS by supporting the establishment of the infrastructure and mechanisms to enable AAS actors to network for sharing experiences, finding solutions for everyday problems, accessing global knowledge hubs and facilitating collective innovation in AAS delivery. AFAAS’ membership included 40 North African and Sub-Saharan African countries. Existing Country Fora include Benin, Malawi, Tanzania, Kenya, Uganda and Sierra Leone and they were amongst the initial countries to be supported. Country For a are country-specific multi-stakeholder entities comprised of public and private individuals and organizations active in the Agricultural Extension and Advisory Services (AEAS) space within countries. Component 2: AFAAS Governance, Management and Secretariat Activities (appraisal estimate US$6.50 million, actual US$0.90 million): This component was to support the AFASS secretariat (located in Uganda) by supporting AFAAS governance and management systems, developing and implementing monitoring and evaluation system, undertaking syntheses and strategic studies, and developing and maintaining a database on agricultural advisory services in Africa. According to the ICR (p. 10) a third component (MDTF Management and Supervision (Bank-Executed) with an appraisal estimate of US$1.44 million) was included in the PAD erroneously and was included in the total appraised project cost of US$18.4 million. (See following section for explanation of difference between appraised and actual costs). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$18.44 million, actual cost was US$5.54 million. According to the ICR (p. 10) the Multi-Donor Trust Fund Financing for the Bank-approved PAD in the amount of US$18.4 million did not fully materialize since the financing amount was based on AFAAS’ Page 2 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) Strategic and Operational Plan (which included recipient executed components 1 and 2, and Bank executed component 3) rather than the actual grant funds available for the recipient. Financing: The project was to be financed by the European Commission (US$6.8 million) which was to be disbursed through the MDTF, and the International Fund for Agricultural Development (IFAD) (US$1 million). At the time of appraisal additional funding was anticipated from USAID, AusAID, Canadian International Development Agency (CIDA), Swiss Agency for Development and Cooperation (SDC), Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), Agence Française de Développement (AFD) and other donors who had shown an interest in participating in the AFAAS MDTF. The World Bank was to provide financing in the amount of US$5.54 million, of which US$5.53 million was disbursed. Borrower Contribution: It was not planned for the Borrower to make any contributions. Dates: On December 18, 2017 the project was restructured to: i) change the objective of the project to make it more attributable to AFAAS given its mandate to fit the realities of Comprehensive Africa Agriculture Development Program (CAADP) implementation and national policy reform structures in Africa; ii) adapt the two PDO indicators and several intermediate outcome indicators in the Results Framework to show direct attribution to AFAAS; iii) change components to include only the two recipient- executed components (component 1 and 2) and costs; iv) extend the closing date from December 31, 2017 to June 30, 2018 to allow for AFAAS to fully disburse against approved sub-grants and provide bridge financing, as AFAAS was expecting to secure additional bilateral financing during the extension period; v) reallocate between disbursement categories; vi) change legal covenants; and vii) change implementation schedule. 3. Relevance of Objectives Rationale According to the PAD (p. 11) even though Africa’s agricultural output almost doubled over the past forty- year period it was almost entirely due to increases in land and labor inputs. Increases in factor productivity was marginal and played a minor role in the growth of the sector not improving poverty reduction or the elimination of hunger. Reasons for this situation are soil and water constraints, extremely low levels of human capital in the farm population, health of the rural population, lack of infrastructure, poorly functioning markets etc. Also, the flow of information from the farm to inform research, education and extension systems is limited. In order to improve this situation, fostering agricultural research and developing an integrated approach around the research, extension, and educational “triangle,” with the farmer at the center would be critical to addressing the constraints to growth, poverty reduction and elimination of hunger. The PAD (p.11) stated that women farmers need particular attention in this area as they play a leading role in all aspects of crop production and commercialization. The objective of the project was in line with the agenda of the Comprehensive Africa Agriculture Development Program (CAADP) by the African Union which was initiated in 2003 and aimed to help African countries reach and sustain a higher path of economic growth through agricultural-led development that reduces hunger and poverty and enables food and nutrition security and growth in exports. The CAADP identified under pillar four the challenges of agricultural education, research and technology uptake to the farm level. Page 3 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) Furthermore, according to the ICR (p.14) the objective supported IDA’s 18 goals for agri-food development which identified the importance of supporting advisory services in Africa in order to achieve its second goal “better jobs – developing skills and innovation”. The PDO of the project was realistic given the amount of resources available. Having acknowledged the need, there is no real evidence of a strategic linking between support to a peak body and flow-on benefits by way of productivity improvements over and above individual country level research. Given the emphasis on cross-country collaboration and regional benefits, greater articulation of the strategic advantage of this approach would have been helpful to know what the activities were to add up to. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Original objective: To reform and strengthen Agricultural Advisory Services (AAS) in accordance with FAAP principles towards increasing agricultural productivity and food security. Revised Objective: To support and strengthen Agricultural Advisory Services (AAS) in accordance with the Framework for African Agricultural Productivity (FAAP) principles. Even though the objective was revised, no split rating is required given the objectives “to support” and “to strengthen” remained the same. Rationale The project’s theory of change linked the establishment of a Country Fora as a platform for AAS stakeholder involvement in CAADP at country level, the establishment of a networking mechanism for AAS stakeholder at the regional and continental level, the active participation of AFAAS in the implementation of CAADP’s pillar IV at the continental level and the improvement of AFAAS governance and management to increase agricultural productivity and food security. Outputs: • AFAAS established and supported AAS Country Fora (CF) through participatory review and validation of guidelines for the establishment and strengthening of CF (based on implementation experience) and completion of in-country sensitization in 24 countries. The ICR did not include a target for this output. • When the project closed, 22 CF were operational in Benin, Burkina Faso, Cameroon, Ethiopia, Kenya, Ghana, Liberia, Madagascar, Malawi, Mozambique, Nigeria, Rwanda, Sierra Leone, South Africa, South Sudan, Sudan, Tanzania, Togo, Uganda, Zambia and Zimbabwe. The ICR did not include a target for this output. Page 4 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) • These CF supported AEAS capacity and reform through advocacy such as in Malawi and participation in policy making processes and policy reforms such as in Uganda, Rwanda, Nigeria, Liberia, South Sudan, South Africa, Malawi, and Ghana. Also, CF piloted innovation tools, promoted quality and professionalization of AEAS services in Ethiopia, Nigeria, and South Africa. The ICR did not include a target for this output. • CF used sub-grants to pilot a variety of innovative ideas such as youth-led commercial-based extension services in Uganda, and land info mobile app technology to provide soil and climate information in Kenya. The ICR did not include a target for this output. • AFAAS developed a participatory review of guidelines incorporating AEAS concerns in CAADP compacts, built capacity in experts to backstop CF in their engagement in CAADP processes from an AEAS perspective, and sensitized and mentored CF during events at continental/regional levels. • AFAAS institutionalized the delivery of the biennial Africa-wide Agricultural Extension Week to enable a face to face interaction between AEAS stakeholders from within and outside Africa. The aim of these interactions was to foster knowledge exchange, learning and developing AEAS innovations. AFAAS has held three extension weeks during MDTF project implementation: 2013 in Botswana, 2015 in Ethiopia and 2017 in South Africa). The number of participants has been steadily increasing from 300 (2013) to 400 (2015) and then 700 (2017). • AFAAS participated in CAADP implementation at the continental level which resulted in the development of a strategy for AEAS engagement in CAADP and its embodiment in the strategic plan for CADDP Pillar IV. • AFAAS facilitated the availability and accessibility of appropriate and up-to-date knowledge on AAS from a range of sources in Africa and around the world by establishing infrastructure and mechanisms that enabled AEAS actors to network, share experience, solve problems, access global knowledge hubs, and facilitate collective innovation in AEAS delivery. • AFAAS built 107 partnerships at national, regional, continental, and international levels between AAS and other institutions with similar mandates and interests, surpassing the target of 28 partnerships. Outcomes: • The number of AEAS actors who were directly affected or reached by AFAAS interventions increased from 5,000 actors in 2013 to 28,431 AEAS actors in 2018, surpassing the target of 25,000 actors. • The percentage of AEAS actors who were directly reached/affected by AFAAS that are satisfied with the quality of its services increased from 45 percent in 2013 to 68 percent in 2018, surpassing the target of 65 percent. • The number of CFs established increased from 5 CF in 2013 to 22 CF in 2018, achieving the target of 22 CF. • The number of registered CF members increased from 150 members in 2013 to 3,594 members in 2018, surpassing the target of 2,200 members. • 16 CF developed the capacity to articulate demands, advocate for AEAS reforms, address gender equality, mobilize and manage resources, and act as platform for sharing knowledge, surpassing the target of 15 CF. • 15 CF recommended policy options to policy makers, surpassing the target of 13 CF. • 16 AEAS extension approaches were recommended and/or implemented by stakeholders, surpassing the target of four approaches. • 32,686 individuals utilized IT tools and platforms facilitated by AFAAS & CF, surpassing the target of 12,000 individuals. Page 5 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) • The number of partnerships formed at national, regional, and international levels increased from 20 partnerships in 2013 to 107 partnerships in 2018, surpassing the target of 28 partnerships. • 12 partnerships recommended emerging and cross-cutting issues (gender, youth, climate change) related reforms in AAS, surpassing the target of four partnerships. • The satisfaction level of stakeholders with government procedures, management structures, and policies was ”highly satisfactory” (according to the survey), achieving the target of “highly satisfactory”. • The percentage increase in the amount of funds for supporting the AFAAS strategic plan increased from 31 percent in 2013 to 51 percent in 2018, not achieving the target of 96 percent. • The number of advocacy initiatives that recommended emerging/cross cutting issues (gender, youth, climate change) reforms in AEAS to policy makers increased from six initiatives in 2013 to 15 initiatives in 2018, surpassing the target of 11 initiatives. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale There was considerable achievement in getting the various parties more involved and interactive in supporting AAs activities. However, linking the achievement of these activities with higher level outcomes is weak with limited insight as to how the activity and participation will lead to improvements in agricultural productivity and food security. Therefore, the achievement of the objective is rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: The PAD did not conduct a traditional economic analysis since a full and detailed ex-ante economic and financial analysis was not warranted due to the difficulty of estimating in advance the specific benefits of the activities and investments the project was to support. The PAD (p. 27) stated that based on previous work on the rate of return to investments in agricultural research and extension, the returns of the investments made in AFFAS would be well above typical threshold levels. The ICR did not conduct a traditional economic analysis, stating the same reasons as the PAD. Therefore, we are left to speculate as to whether the investment was likely to have the expected financial returns to justify its costs. In the absence of any economic analysis, it is difficult to confidently support this assertion. Operational Efficiency: Page 6 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) According to the ICR (p. 18) the project achieved efficiency through building partnerships which allowed AFAAS to leverage funds from other projects such as AFAAS and CF reaching farmers with extension models and participating in a policy dialogue at different levels for CF. The ICR also stated that the AFAAS secretariat did not increase salaries as approved by the board, resulting in a reduction of administrative costs. However, the project restructuring was delayed due to slow actions by the Bank and the donors, requiring an extension of the closing date by six months, which suggests an inefficient use of financial sources. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objective was rated Substantial, the project efficacy rating was considered Substantial given the breadth of activities, but taking into account the limited data on broader impacts. Efficiency was rated Modest given minor administrative issues and the lack of any attempt at economic justification for the investment. This results in an overall Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The most critical risk to the sustainability of the project’s development outcomes is AFAAS’ ability to raise sufficient financial resources to continue its operations. According to the ICR (p. 24) AFAAS has not been able to diversify its financial sources such as through projects, membership subscriptions, crowd funding or services to members and other development partners. According to the Bank team (January 18, 2019) AFAAS had Page 7 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) addressed this risk in its new strategy and operational plan which also included detailed fundraising efforts. In October 2018, AFAAS confirmed that IFAD support was moving ahead, with the expectation that funds would be available as early as January 2019. 8. Assessment of Bank Performance a. Quality-at-Entry According to the ICR (p. 20) the project’s design was based on AFAAS’ 2011-2015 strategic and operational plan, which focused on five key areas: i) participating in implementation of CAADP pillar IV; ii) information, communication, and knowledge management; iii) establishing and supporting CF; iv) linkages and partnerships; and v) developing governance, management, and funding systems. The ICR (p. 23) stated that the project design was informed by lessons learned from the first Multi-Donor Trust Fund (MDTF) such as that a second MDTF would help to provide stability for AFAAS to carry out its proposed work program, and that AFAAS’s technical and administrative capacity would need to be strengthened to do so. The Bank identified relevant risk factors. The risk related to having only partially secured the funding for AFFAAS’s five-year operational plan was rated as Substantial. Mitigation efforts were not adequate, and the risk materialized, resulting in AFAAS being able to hire fewer staff than originally planned. According to the ICR (p. 23) the project design could have been aligned with available grant financing to be followed by additional financing once other funds materialized. A shortcoming of the M&E design was that it included PDO indicators which measured the long-term impact of project activities rather than project level outcomes, making the attribution of project activities challenging. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision According to the ICR (p. 24) the Bank conducted regular supervision missions and provided clear guidance to AFAAS on project implementation and any issues that had to be addressed. The Mid-Term Review, which was conducted in April 2016, identified the need to restructure the project. However, the restructuring was first delayed by the Bank and then by the donor due to a slow response time of 11 months resulting in the project only being restructured in December 2017. This negatively impacted already delayed processing. Furthermore, M&E was weak throughout project implementation and the project was not able to systematically report on PDO indicators on a regular basis. However, when a Bank and an Independent Procurement Review identified procurement shortcomings, the Bank provided support to address them. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Page 8 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The theory of change illustrating how key activities would lead to intended outcomes was sound and reflected in the Results Framework. The objective of the project was clearly specified. However, according to the ICR (p. 22) the Results Framework was based on AFAA’s Strategic & Operational (S&OP) logical framework and included PDO indicators which measured the long-term impact of project activities rather than project level outcomes, making the attribution of project activities challenging. The intermediate results indicators were adequate to capture the contribution of the project’s activities towards the PDO. The majority of indicators had a baseline and were measurable and were sufficiently specific. M&E activities were to be conducted by the AFAAS Secretariat, Planning, Monitoring and Evaluation Department. An Evaluation Officer was to monitor implementation progress and link monitoring activities with activities that were to be carried out at regional, sub-regional and country levels. b. M&E Implementation According to the ICR (p. 22), the AFAAS Secretariat Planning and M&E Department was to hire a planning manager, an M&E manager and an M&E officer. However, AFAAS was not able to do so due to a lack of funding. Instead, AFAAS had to depend on support by the Forum for Agricultural Research in Africa (FARA) and/or consultants to conduct M&E activities. The ICR stated that this negatively affected the implementation of M&E activities and the project was not able to systematically report on PDO indicators on a regular basis. and AFAAS was not able to collect data from the Country Fora on a timely basis. However, AFAAS was able to conduct the baseline, mid-term, and end-of-project surveys using program staff and consultants. When the project was restructured in December 2017, the Results Framework was modified to adapt the two PDO indicators and several intermediate outcome indicators to show direct attribution to AFAAS. However, the Bank failed to include outcome indicators at the PDO level. c. M&E Utilization According to the ICR (p. 23), given the shortcomings mentioned above, AFAAS could not use the project’s M&E data as effectively as planned to inform decision making. The ICR stated that AFAAS was able to use data gathered through its own M&E system to modify its work plan and develop its new strategy. M&E Quality Rating Modest 10. Other Issues Page 9 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) a. Safeguards The project was classified as category C and did not trigger any of the Bank’s safeguard policies. b. Fiduciary Compliance Financial Management: According to the ICR (p. 23) a financial management manual and grant award guidelines were prepared to provide guidance on project implementation and sub-grants disbursement. Also, the Bank conducted regular supervision missions and reviewed the project’s financial management to ensure compliance with the Bank’s fiduciary requirements. According to the Bank team (Jan 18, 2019) starting in November 2014, the Bank identified several weaknesses related to financial management including inter-project borrowing (co-mingling of funds) ineligible expenditure relating to staff gratuity, salary advances not supported by HR policy or operations manual, weak records management, segregation of duties and reporting delayed, and lack of an internal audit committee. The ICR (p. 23) stated that the AFAAS addressed and solved these issues. Also, according to the ICR the AFAAS’ internal audit conducted regular reviews and unaudited interim financial reports and annual audited financial reports were submitted in a timely and acceptable manner. Furthermore, the Bank generally rated the project’s financial management as satisfactory. According to the Bank team (January 18, 2019) the audited financial statement submitted by Deloitte & Touche on December 10, 2018, which covered the 18-month period from January 1, 2017 to June 30, 2018 was unqualified. It did, however, draw attention to the lack of a tax exemption certificate, as well as the fact that the MDTF was the primary source of financing for AFAAS, that the MDTF had closed and that AFAAS had not replaced this funding, therefore creating a going concern of material uncertainty. Procurement: According to the ICR (p. 23) the Bank and an Independent Procurement Review, which was conducted in October 2016, identified several procurement issues such as slow and delayed procurement processes, poor contract management and supervision of consultancies, and lack of knowledge of roles and responsibilities by stakeholders in procurement and contract management. These issues were addressed by developing an action plan, creating templates for the procurement of goods and services, providing basic procurement training for stakeholders. c. Unintended impacts (Positive or Negative) NA d. Other --- Page 10 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment There were shortcomings both Moderately in the project's Efficacy and in Outcome Satisfactory Satisfactory the measuring of the project Efficiency. Moderately Moderately Bank Performance --- Satisfactory Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR (p. 24-25) provided useful lessons learned: • It is critical for a project’s design to identify and clarify institutional areas to be supported according to the financing available. This project supported the entire Strategic and Operational Plan of AFAAS without having secured the necessary financing. This resulted in the hiring of fewer staff at AFAAS and relying more on consultants, which might have had a negative impact on capacity building within AFAAS. • Ensuring financial sustainability of institutions being supported under a project is critical. In this project, AFAAS has been and continues to rely on donor funding which puts a critical risk on the sustainability of AFAAS. Also, this uncertainty impacts the planning of activities and the effectiveness of activities due to a limited time horizon. It could have been beneficial if the project had included a sub-component which supported the development of a resource mobilization strategy and built capacity within AFAAS for sustainable resource mobilization. • Country Fora (CF) are an effective way to foster reforms and strengthen public sector support for AEAS systems. In this project, the CF model was built on existing institutions and used local expertise resulting in the emergence of champions and volunteers that were critical for establishing the CF. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided a good overview of project preparation and implementation. The ICR is internally concise, generally candid and consistent. However, the ICR did not attempt to conduct a traditional economic analysis (cost-benefit or cost-effectiveness), and is not particularly outcome driven. The ICR did not include information on procurement shortcomings during project preparation. Page 11 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AFAAS Second MDTF (P143367) a. Quality of ICR Rating Substantial Page 12 of 12