IEG Report Number: ICRR14742 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 08/11/2015 Country: Ethiopia Project ID: P108932 Appraisal Actual Project Name: Pastoral Community Project Costs (US$M): 133.25 119.74 Development Project Ii L/C Number: Loan/Credit (US$M): 80.00 74.52 Sector Board: Agriculture and Rural Cofinancing (US$M): 33.54 37.35 Development Cofinanciers: IFAD Board Approval Date : 05/29/2008 Closing Date: 12/31/2013 12/31/2013 Sector(s): General water; sanitation and flood protection sector (20%); Other social services (20%); Animal production (20%); General education sector (20%); General agriculture; fishing and forestry sector (20%) Theme(s): Rural services and infrastructure (33%); Decentralization (17%); Natural disaster management (17%); Water resource management (17%); Rural policies and institutions (16%) Prepared by: Reviewed by: ICR Review Group: Coordinator: J. W. Van Holst Lauren Kelly Christopher David IEGPS1 Pellekaan Nelson 2. Project Objectives and Components: a. Objectives: The project development objective (PDO) stated for Pastoral Community Development Project II (PCDP II) in the Project Appraisal Document (PAD) was "to sustainably improve the livelihoods of pastoralists living in the arid and semi-arid Ethiopian lowlands". The PAD explained that "a 'livelihood' is defined as a people centered concept comprising four key elements related to the well-being of individuals and families: (i) growth and stability of income; (ii) access to social and public services; (iii) the social relations, institutions and natural environment that facilitate or constrain standards of living; and (iv) reduction of vulnerability to disaster" (para 10). The statement of the PDO in the Financing Agreement (FA) was "to assist the Recipient in increasing its pastoralist communities’ resilience to external shocks; and improving the livelihoods of Beneficiary Communities, and thereby to contribute to overall poverty alleviation in the territory of the Recipient" (Schedule 1). The Financing Agreement did not define "livelihoods", but introduced the additional objective of poverty alleviation in the territory of Ethiopia. For the purpose of assessing this project's achievements this Review will use the objective stated in the Financing Agreement. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? No c. Components: The following is a brief summary of the project's components based on their description and estimated costs in the PAD (pages 11-14 and Annex 4) and their actual costs in Annex 1 of the ICR. The project was implemented in 55 pastoral and agro-pastoral woredas (districts) in the regions of Somali, Afar, Oromia and the Southern Nations, Nationalities and Peoples National Regional States of Ethiopia. Component 1: Sustainable Livelihoods Enhancement (appraisal estimate US$90.79 million, actual cost was US$90.94 million). This component would strengthen decentralized and participatory planning at the woreda (district) and community/kebele (sub-district) levels, operating within the regional government structure. Women and men in pastoral and agro-pastoral communities would design and implement Community Action Plans (CAPs) that reflect their development priorities. There were two sub-components. Sub-Component A - Community Investment Fund (CIF): This was designed to finance community sub-projects giving particular attention to poorer sub-groups and to women. This sub-component was expected to absorb 84% of the sub-component's funding and 59% of total project costs. It was therefore a dominant activity in the project. A Woreda Development Committee (WDC), comprising representatives of the woreda administration, customary of institutions and beneficiary communities, would appraise and endorse CIF requests with the support from the project's mobile support teams (MSTs). Beneficiary communities were required to make at least a 15% cash or in-kind contribution to demonstrate commitment, and this contribution increased if communities aimed to benefit from additional rounds of CIF support. Sub-component B - Rural Livelihoods Program (RLP): This sub-component supported the establishment of pastoral rural savings and credit cooperatives (RUSACCOs) in beneficiary communities at woreda and regional levels, and the capacity-building of associated support services. These activities drew on the experience of the Ethiopian Rural Financial Intermediation Program (RUFIP) - funded by IFAD - and other micro-finance projects in rural Ethiopia. Once registered (with project support), RUSACCOs were eligible for project financing of basic office equipment, account books and promotional material, as well as a grant of up to 200% of their pre-registration compulsory and voluntary savings as seed capital for income-generating activities for beneficiary communities. The limit for loans to cooperative members was Birr 50,000 each (equivalent to roughly US$2,564 each). Component 2: Pastoral Risk Management (appraisal estimate US$28.03 million; actual cost US15.46 million). This component financed early warning systems to develop Disaster Preparedness Contingency Plans (DPCDs) and subsequently Disaster Preparedness Investment Plans (DPIPs). The decline of US$12.57 million between the appraisal estimate and the actual cost was not explained in the ICR. However, a perusal of the "End of Project Performance Report" prepared by the Ministry of Federal Affairs showed that investments in bore holes and irrigation development were both close to 50% of the plans in Sub-component B (Tables 28 and 29). This shortfall could explain most of the lower than estimated actual costs. This component's two sub-components were as follows. Sub-component A - Pastoral Early Warning and Response Program (PEWRP): The pastoral Early Warning System (EWS) provided information used to classify woredas participating in the project into one of five “disaster stages”. Early response would be financed through Disaster Early Response (DER) grants administered at the regional level by the Early Warning and Response Department (EWRD) of the Ministry of Agriculture and Rural Development (MoARD), implemented by the regional Early Warning and Response Bureaus (EWRB), and supported by the Regional Project Coordination Units (RPCUs). The project built capacity at woreda level to prepare Disaster Preparedness Contingency Plans (DPCPs) that identified both disaster mitigation and early response activities. Sub-component B: Disaster Preparedness Investment Program : The Regional Pastoral Development Office/Commission (whether they were called "office" or "commission" depended on their geographic location) integrated Disaster Preparedness Contingency Plans (DPCPs) into a long-term strategic disaster preparedness strategy and a prioritized Disaster Preparedness Investment Program (DPIP) for the different regions. To the extent possible, productive safety net programs (PSNP) catchment management proposals would be integrated into the DPIPs. When the disaster preparedness strategy was approved by a regional Steering Committee, regions would receive funding from PCDP II through Disaster Preparedness Strategic Investment Grants (DPSI Grants) which could include construction and rehabilitation of feeder roads, improved water supply and catchment management, fodder banks and rangeland improvement. Component 3: Participatory Learning and Knowledge Management (appraisal estimate US$1.51 million; actual cost US$1.03 million). This component financed various activities aimed at generating information for the improving management of pastoral resources in three sub-components. Sub-Component A - Participatory Action Learning : Participatory Action Learning pilots would be undertaken in selected beneficiary communities to apply and further develop methodologies for demand-driven approaches to participatory knowledge generation and innovation. Starting on a small scale, facilitators would work with pastoral communities to help them identify their research and knowledge priorities for financing by PCDP II. Sub-Component B - Knowledge Management and Networking : This was supported at regional and community levels including through the establishment of small resource units on pastoral research and development as well as pastoral development networks at the regional level. These networks provided a forum for interested actors to share lessons and information on regional pastoral development issues. Sub-component C - Policy Studies: A modest budget was made available to the Ministry of Federal Affairs (MoFA) and Regional Pastoral Development Commissions/ Bureaux to commission studies they regarded as necessary to inform the implementation of social and economic pastoral development policies. Component 4: Project Management (appraisal estimate US$7.04 million; actual cost US$12.31 million). As for the previous PCDP I, the Federal Project Coordination Unit (FPCU) located in the MoFA was responsible for overall PCDP II management, annual planning, and fiduciary management, liaison with stakeholder groups at federal level, communication, M&E and reporting, staff capacity-building, and mobilization of technical assistance. However, PCDP II management was decentralized to the regional Pastoral Commissions and Pastoral Development Bureaux housed the Regional Project Coordination Units (RPCUs), which had responsibility for PCDP II implementation at the regional level. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs: The estimated project cost at appraisal was US$133.25 million but the actual utilized cost at the project's close was at US$119.74 million which was US$13.51 million (10%) below the the appraisal estimate. Financing: At appraisal it was estimated that IDA would provide US$23.40 million as a credit and US$56.60 million as a grant. The actual amounts of the credit and the grant at the project's close were respectively US$21.75 million and US$52.77 million - both close to 93% but would have been 100% were it not for the fluctuations in the exchange rate. IFAD was expected to provide co-financing of US$33.54 million but actually provided 11.4% more (US$37.35 million) which included an unutilized advance of US$6.1 million as explained in the ICR (page 31). Borrower Contribution : The PAD estimated that the Government of Ethiopia would provide $19.70 million toward the cost of the project, but it actually provided only US$13.97 million (70.9% of the appraisal estimate). Dates: According to the ICR (footnote 1, page 3), the Data Sheet recorded an incorrect original closing date implying a six month delay in project completion. December 31, 2013 was the original closing date in the Financing Agreement (Schedule 2, Section IV) and it was not changed. Restructuring: On December 9, 2011 a level 2 restructuring of the project was approved based on discussions at the mid term review (MTR). The MTR, using the definition of the PDO in the Financing Agreement, agreed to the need for a substantial revision of the results framework including changes in the PDO indicators and Intermediate Outcome (IO) Indicators. These changes were incorporated in the approved Restructuring Paper in December 2011 when about half the project's funds had been disbursed. The revised indicators were recorded in the Data Sheet in the ICR. The restructuring process was aimed at discarding indicators that were not relevant or could not be measured, making some of them more precise, and adding six indicators to measure access to social and public services contributing to improved measurement of livelihoods. The main outcome of the restructuring was the decision to move away from qualitative indicators - a perception based approach - towards quantitative indicators that put in place targets and measured the number of persons that achieved access to the project financed assets. While the changes in the indicators were considerable, they did not result in a change in the scope or quality of the outcomes that the project intended to achieve. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High. The project's development objective was "to assist the Recipient in increasing its pastoralist communities’ resilience to external shocks; and improving the livelihoods of beneficiary communities, and thereby to contribute to overall poverty alleviation in the territory of the Recipient". This objective, with its broad definition of "livelihoods " (see Section 2a of this Review) was highly relevant to the pastoral communities in Ethiopia which have been chronically vulnerable to many shocks such as drought, livestock diseases, and locusts causing losses in capital and income, and disruption of social services due to civil conflict. Pastoralism is practiced extensively in the four regions covered by the project, namely the Afar and Somali Regional States (Regions), parts of the Oromiya Region, and in some Zones of the Southern Nations, Nationalities and Peoples National Regional States (SNNPR). According to the PAD these areas were populated by at least twelve million people, some eleven million animals covering half the area of Ethiopia. The population included (i) a few comparatively wealthy pastoralists with many livestock; (ii) a large numbers of poor “agro-pastoralists” with small herds/flocks who depended on cropping or sale of their labor to sustain their livelihood and (iii) a significant number of pastoralists gradually abandoning pastoral livelihoods due to loss of assets and degradation of grazing lands (para 2). The Government's "Plan for Accelerated and Sustained Development to End Poverty" (PASDEP) dated 2006 noted that "poverty remains particularly intense in the pastoral areas, both in terms of low income and food consumption, and high vulnerability defined in terms of the risk of sudden drops in income (page 191)". This project (PCDP II) addressed some of the core strategies suggested in PASDEP for improving the livelihoods of pastoralists. The strategic objective in the Bank's Country Assistance Strategy (CAS) for Ethiopia for FY08-11 was consistent with PASDEP namely fostering economic growth in order to sustain the emerging economic "take-off", improving access to and quality of basic service delivery, and reducing Ethiopia’s vulnerability to help improve prospects for sustainability (Summary, page vi). In August 2012, 15 months before PCDP II closed, the second pillar of the Bank's Country Partnership Strategy (CPS) for Ethiopia for FY13-16 was released with a pillar aimed at "Enhancing resilience and reducing vulnerabilities by improving delivery of social services and developing a comprehensive approach to social protection and risk management" (para 99). Hence the PCDP II development objectives continued to be highly relevant to the Bank's strategic partnership with Ethiopia until it the project was closed in December 2013. b. Relevance of Design: Modest. The design of the project's Results Framework in the PAD - linking the project's inputs and outputs to outcomes - was inadequate. The design lacked a clear causal chain reflecting the project logic - how support for community saving, lending and investments and complementary support for early warning systems would increase resilience, improve livelihoods and ultimately contribute to poverty alleviation. Instead, the results framework at design was merely a listing of the development objectives and indicators without having established needed logical connections. The relevance of design was also weakened by a lack of information on how the proposed community driven development (CDD) model would be effectively designed and implemented within complex pastoral and existing bureaucratic systems at the woreda and kebele level in order to more effectively achieve the intended project results. The project introduced a CDD model involving women and men in pastoral and agro-pastoral communities who prepared community action plans at the kebele level that were intended to pay particular attention to poorer subgroups and to women. When approved, the CAP would be financed by the project-financed Community Investment Fund (CIF). The CAPs were reviewed by woreda development committees comprising representatives of the woreda administration, customary institutions and beneficiary communities with the support from the project's mobile support teams. There is insufficient information to determine how the overall design of the appraisal and approval processes at the community and district level supported adequate participation of community members within the CAP process (choice and allocation of investments etc). For example, the End Evaluation report for the project highlighted the role that the Woreda Development Committee (WDC) played in appraising and deciding on the allocation of investments - a role that the WDC traditionally plays in the without-project scenario. In sum, the design was not specific enough about how a CDD model would be integrated and sustained within the existing structures and how this model would lead to the intended project impacts, A design feature that was introduced in the second phase - the Rural Savings and Credit Cooperative - was a relevantly designed feature, modeled after pilots tested through an IFAD rural finance project. The design of these cooperatives addressed a financing gap and featured access to finance for pastoral women. Another relevant design feature was the importance assigned to the development of the early warning systems - that directly address the project objective of increasing resilience to shocks and to protect livelihoods. The design featured support for Early Warning and Response Bureaux and woreda Early Warning and Response Desks that were were designed to collect and analyze basic household-welfare data to identify the early onset of disasters at the woreda level. This, together with the subsequent preparation of Disaster Preparedness Investment Programs and coordination of pastoral risk management in the central and regional project coordinating units, was a relevant design feature. The final element of the project's design was support for considerable training of government staff at the woreda level and the pastoral communities, with the main emphasis on improving the management of pastoral resources. This activity was also substantially relevant to the project's objectives. Following the project's restructuring in December 2011, the project enhanced its results framework so that the system was more in line with project's causal chain. Meanwhile, the project retained several relevant design features mentioned above. However, information was still insufficient on how community driven development would be utilized to more effectively achieve the project objectives and how community members would adequately participate in the decision making processes (CAPs, CIF) and how this system would be sustained. 4. Achievement of Objectives (Efficacy): The project's PDO was "to assist the Recipient in increasing its pastoralist communities’ resilience to external shocks; and improving the livelihoods of beneficiary communities, and thereby to contribute to overall poverty alleviation in the territory of the Recipient". For the purpose of assessing the project's achievements the objective has been partitioned into three sub-objectives - as presented below. In addition the PAD definition of "livelihood" noted in Section 2a of this Review will be used to assess the first sub-objective. (A) improve the livelihoods of beneficiary communities; (B) increase the pastoralist communities’ resilience to external shocks; and (C) contribute to overall poverty alleviation in the project area. Information used in this assessment is taken from the Restructuring Paper (December 9, 2011), the ICR and the "End Evaluation" prepared by an independent consultant which was based on a random sample of households from woredas and kebeles which were carefully selected by the regional project coordinating units (PCUs). The bias in this sample undermined the reliability of the survey results. Sub-Objective A. Improve the livelihoods of beneficiary communities . The project Substantially contributed to the achievement of this first sub-objective. Outputs  54% of total community members (42% of female members) attended project related meetings compared with original targets of 75% and 50% respectively  99% of approved (budgeted) community sub-projects completed each year compared with a target of 80%  93% of PCDP (social and infrastructure) sub-projects completed that become operational compared with a target of 90%  100% of woredas posting woreda community investment fund (CIF) plans, budgets and service performance at public centers compared with a target of 90% (dropped at restructuring)  100% of woredas with a complaint redress system for CIF plan established compared with a target of 90% (dropped at restructuring)  Students enrolled (grade 1-8) in PCDP II constructed schools per year but no data in ICR other than baselines for four regions  100% of rural savings and credit cooperatives (RUSACCOs) that were established were offering credit service two years after establishment compared with a target of 80%  70% of all RUSACCO members and 72% of female members had active Loan accounts compared with targets of 70% and 90% respectively.  100% of RUSACCO members and female members had active savings accounts compared with targets of 95% and 70% respectively (PDO 6)  562 water points were completed which reduced distance to clean water from an average of 8 km in the control group to 4 km in the project area Outcomes  Outcomes prior to restructuring were measured by a perception survey. It found that 87% of the targeted community members interviewed were satisfied with service delivery provided by the PCDP II financed social infrastructure as compared with a target of 70% (this PDO indicator was dropped at restructuring).  Post restructuring, outcomes were measured per the definition provided in the PAD, namely as a function of improved access to non-consumables (water, health, road connectivity): - 1.1 million people were provided with improved access to potable water but no target was established. - 43,600 households gained improved access to small scale irrigation services compared with a target 9,500 - 364,900 people gained access to improved road connectivity --rural roads -- compared with a target 200,000. - Communities most often opted for schools as part of the Community Investment Fund (CIF) plans. Focus Groups conducted by the End Evaluation reported that increased access to education helped students obtain jobs in towns and cities, and that increased access to education supported better decision-making. - 2.3 million livestock benefitted from increased access to livestock health facilities compared with a target of 450,000. - 757,648 people with gained increased access to health facilities and services provided by the PCDP II compared with a target of 450,000.  The project fell short of measuring its contribution to achieving sustained growth of beneficiary incomes. However the "End Evaluation" calculated that credit beneficiaries earned on average an additional Br 2,477 of income on loans with a range of between Br 1,562 and Br 2,858 which was equivalent (at an exchange rate of Br17.5=US$1) to about US$89 and US$163. This range of average earnings on loans compares with an average gross national income per capita in all Ethiopia of US$420 in 2012.  It is worth noting that a separate study conducted in December 2014 found there have historically been large disparities between per capita urban and rural incomes in Ethiopia. It reported that per capita rural income in 2012 was 25% of the national per capita income (US$105 versus US$420). The overall conclusion therefore is that the additional average income earned on loans was considerable in relation to estimated average annual rural income per capita. However, more information is needed to draw conclusions on the impact of RUSSACO's income generating activities on annual per capita income and hence an important aspect of livelihoods. B. Increase the pastoralist communities’ resilience to external shocks . The project Substantially contributed to the achievement of the second sub-objective. Outputs  100% (four) of the project regions prepared comprehensive strategic disaster preparedness investment plans - meeting the target of four.  89% of approved (budgeted) Development Preparedness Strategic Investment Plan sub-projects completed each year compared with a target of 80%. Outcomes  Outcomes prior to restructuring were measured by a perception survey. It found that 76% of community members were satisfied with the timeliness, quantity and quality of disaster early response compared with a target of 70% (dropped at restructuring but data reported in the End Evaluation report).  Outcomes measured after restructuring were associated with key quantitative indicators set as part of the process. As a result of the implementation of the strategic disaster preparedness investment plans, the project reported that 94% of community based disaster preparedness infrastructure investments were operational and had maintenance plans compared with a target of 85% and 100% of disaster early response grant financed activities (compared to a target of 80%) were implemented within one month after the request was officially submitted. These requests are triggered when the early warning system identifies a change from “normal” conditions.  Early warning information on disaster risks available for 97% of pastoral and agro-pastoral woredas compared with a target of 100%. C. Contribute to overall poverty alleviation in the project area . The project Modestly contributed to the achievement of this third sub-objective. Outputs  RUSACCOs were particularly valuable to the poorest of the poor, peri-urban pastoralists who had dropped out of pastoralism because of livestock loss often because of drought (ICR, para 82). Outcomes  There was no data in the ICR on poverty reduction or contributions to poverty reduction in the four focus regions or in Ethiopia that could be attributed to the project's implementation before or after restructuring.  However the ICR refers to a number of contributions the project made to non-monetary measures of livelihoods (paras 70 to 71) which the literature has shown could result in increased earning capacity (such as through improved education, better nutrition and enhanced health) thereby contributing to poverty alleviation. 5. Efficiency: Annex 10 in the PAD concluded that a calculation of economic benefits generated by the project was of limited value because of the multiple anticipated spillover effects between the project’s activities. In line with this conclusion the main text of the PAD made no estimate of a rate of return but stated that the project was expected to have significant economic and social benefits in the four project areas. They were: (a) increased capacity for risk management and early response to shocks such as drought; (b) effective citizens’ consultation and participatory development and strengthened ownership of improved access to access to basic public facilities and social services; (c) increased access by the pastoral community to sustainable financial services and derived increases in incomes; and (d) greater effectiveness of public administration through capacity building, community participation and accountability (para 69). The ICR also does not provide an estimate of the rate of return but refers to a calculation in Annex 10 of the PAD that shows that the average annual cost of drought is estimated at US$70 million (although it was not clear how this was derived) and that if 15% of this loss could be avoided it would be valued at more than the actual expected cost of pastoral risk management (PRM) which ranged from $6.5 to 7.8 million par annum. However, the ICR stated that data were not available to make a comparable estimate for the project during implementation - even though a drought did occur during the project's implementation and hence estimates of losses avoided such as were made in the PAD could have been made. Instead the ICR stated that "The only relevant income generating data (IGA) for economic efficiency was that additional income from RUSACCO loans averaged Br 2,477" (para 75) - without indicating the rate of return that these additional incomes represented. The ICR concluded its assessment of efficiency by listing four alternative ways of evaluating the project's efficiency, namely (a) percentage of funds used. 100 percent of IDA credits and grants were used - 100% of IDA and IFAD financing; (b) percentage of output targets met - substantial according to the ICR; (c) superior quality of construction of project infrastructure - for example, PCDP II human health posts cost 42 percent less than government/NGO-financed health posts for comparable construction, and for primary schools grade 1-4 the PCDP II costs were 57 percent less and for animal health posts PCDP II was 43 percent less; and (d) timely completion of the project - no extensions of the closing date (ICR paras 76 to 79). None of these were adequate indicators of efficiency. It may, for example, have been more balanced to note that, despite no change in the scope of the project or its closing date, the costs of project management increased to 75% above the original estimate. This Review considers that the analysis of efficiency in the ICR is partial and inadequate and less comprehensive than in the PAD. On this basis the project's efficiency is rated as modest. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of objectives was high, but the relevance of design was modest because of the weak results framework and insufficient information to determine how the overall design of the appraisal and approval processes at the community and district level supported adequate participation of community members in the preparation of community action plans (CAPs) and how this would lead to increased resilience. Based on the project's measurable achievements presented in the ICR, the achievement of improved livelihoods at the project's close was substantial. Increased resilience was likely to be achieved through the implementation of the investments triggered by the Early Warning systems and therefore achievement towards this objective is also rated substantial. More rigorous measures would be warranted going forward in Phase III for measuring this objective however. There was no information in the ICR on the extent to which the project contributed to poverty alleviation and this achievement was therefore rated as modest. The lack of quantitative assessment of the project's efficiency, together with a considerable proportional increase in project management costs, led to a modest rating for efficiency. Overall ,the project is rated as having had moderate shortcomings in the achievement of its objectives. Its outcome is therefore rated as moderately satisfacto ry. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR noted three risks facing the project in the future, namely (a) inadequate support to project investments, such as staff, supplies and maintenance to keep the assets financed by the project functional and sustainable; (b) the government may abandon the CDD design and resume the top down, supply-driven approach for supporting pastoralists and agro-pastoralists; and (c) early response systems may be difficult to sustain without substantial budget support (para 87) The ICR also mentioned that field interviews suggest that these risks are real, but not likely to fully materialize. With respect to the possible demise of the CDD approach and the return of a top down and supply-driven approach the ICR drew attention to an awakening of the power of communities, often to the surprise of woreda staff. The ICR mentioned the example of the Jijiga woreda where the Woreda Development Committee asserted itself amongst other development partners, particularly NGOs and insisted on a community initiative approach. The ICR interpreted this example as indicating that intensive consultations with communities and prioritization through CAPs will become a standard feature of development interventions in the future (para 87). A different interpretation could be that the event in Jijiga was a reaction by the community to undue pressures from the WDC. The ICR also mentioned that "Regional Governments with sizable pastoral populations are strong advocates of pastoral causes. The PCDP development approach based on a CDD model has become part of the development planning process in these regions. There is also evidence in some areas that the PCDP has unleashed a self-sustaining process. With improved access to social and economic services such as water supply education, health (humans and animals) and rural finance through RUSACCOs have come gradually improving and diversifying livelihoods that generate more income" (para 88). On the other hand the ICR noted that there is concern that a CDD approach would not preserve project investments unless there is also continued technical support from woreda facilitators and the mobile support teams financed by the project. In this context there are plans for testing the feasibility of user fees to pay for services (para 87). Finally, the project's pastoral risk management and early warning systems have led to an awareness of the increasing frequency and intensity of droughts and hence a service that signals the advance of disasters and provides advice on coping mechanisms will remain in demand (ICR, paras 89 and 90). While support for these early warning and advisory services will buttress a strong interest in their sustainability it will not necessarily ensure financing. On balance this review agrees with the ICR that risk to development outcome is significant. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: According to the ICR the Bank used its experience from the first PCDP to improve the design of PCDP II. The improved performance of the early warning system in this project compared with its predecessor attests to the benefit of an improved design. Section 3a of this Review has already noted that the relevance of the project's objective before restructuring was high. However, in light of a weak results framework in the PAD (also noted by the ICR) the project's design before restructuring was rated as modest. The results framework had significant shortcomings because most indicators were inappropriate and needed extensive overhaul during the MTR. The ICR also noted the less than optimal participation of females in the activities of communities vis a vis the community action plans (measured as 16% in the End of Term Evaluation. There were also minor shortcomings in the financial management arrangements because of the reliance on statements of expenditures which for a remote project such as PCDP II was cumbersome to administer. Quality at entry is therefore rated as moderately satisfactory. Quality-at-Entry Rating: Moderately Satisfactory b. Quality of supervision: The task manager for this project was a member of Bank staff in the country office in Addis Ababa. This facilitated prompt assistance to the government's project coordinating units which faced an extremely challenging social and economic development challenge in remote and undeveloped regions of Ethiopia. The MTR was comprehensive and there was timely follow up on important issues such as the overhaul of the indicators in the results framework with the assistance of specialists in monitoring and evaluation in Bank headquarters - which led to formal proposals for the project's restructuring as described in Section 2 of this Review. The ICR recognized that the restructuring could have been done earlier. According to the ICR the Bank established good working relations with the government, the federal and regional project coordinating units as well as with the co-financier (IFAD). It is noted that the project team was recognized by the region (a Vice Presidential award) for excellent work in supervising PCDP II (paras 92 and 93). IEG notes that the monitoring of the Redress Mechanism established during design was eliminated at restructuring. It is unclear why the monitoring of this critical project support mechanism was dropped. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR stated that the Ministry of Federal Affairs provided satisfactory support to the project on the basis that it was effective in the handling the project's administration at the central government level. The Ministry also supported the project's management effectively and facilitated the mobilization of counterpart contributions at the regional level. One area where the ICR stated that government performance could have been smoother was in its coordination with the Ministry of Agriculture and Rural Development and the Early Warning Response Department, because there were occasional lapses in the disbursement of central government funds to the regions and resulting delays in implementation (para 95). Government Performance Rating Satisfactory b. Implementing Agency Performance: The ICR stated that the Federal Cooperative Agency CA satisfactorily implemented the Rural Livelihood Program through the Regional Cooperative Promotion Bureaus and relevant Woreda Cooperative Promotion Desks. The mobile support teams provided basic training and support to pastoral RUSACCO’s at community level in areas where woreda development committees required support. High turnover of the MSTs which created problems in maintaining appropriate expertise (para 96). According to the ICR (para 97) the Early Warning and Response Desks of the Ministry of Agriculture and Rural Development managed Component 1 reasonably well including the regional Early Warning and Response Bureaux and woreda early warning and response departments. Early responses were effectively financed and supported by Regional Project Coordination Units (RPCU). However, there were weaknesses in the project's financial management and procurement, especially in the earlier years (see section 11b below). Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E system was the responsibility of the Federal Project Coordinating Unit (PAD, para 59). The FPCU set up a sound reporting structure that allowed detailed tracking of physical progress of the project. Financial transactions were also well documented even if the flow of data was slow because of the large number of SOEs. Using community data gatherers for the early warning system was a successful design element and worked well in practice, because communities reported that they felt empowered to better track the risk of drought and to address it in advance. The main weakness was that the original M&E framework PDO outcome indicators were inappropriate for the measurement of the PDOs and, regardless, most of the PDOs could not be measured. Three out of five original PDO indicators were dropped and replaced during the project's restructuring in December 2011 and at the same time eight more PDO indicators were added. b. M&E Implementation: More than 80% of early warning monthly reports and quarterly early warning bulletins produced and disseminated (national and regional) by the early warning and Response Department and the Ministry of Agriculture and Rural Development compared with a target of 80%. The quality of project reporting was high. The reports were thorough and closely tracked a very large number of Project activities. In addition, in each region, the RPCUs produced films to document and advertise PCDP II activities. The films feature stakeholder interviews, project personnel and project investments. Most prominent are the stakeholders’ testimonies of how the project has positively affected their lives. These testimonials and success stories have greatly contributed to visibility of PCDP II and as a result the Pastoral Standing committee in the Ethiopian Parliament considers PCDP as a flagship program in pastoral areas. Part of the M&E implementation for the project was the "End Evaluation" prepared by an independent consultant. While there was a considerable amount of interesting information in this report, the sample used to obtain information was arguably unreliable because "the sample of woredas and kebeles were carefully selected by the regional PCUs" (page 9). Hence the sample was not a random sample and this seriously undermined the report's quality and reliability. c. M&E Utilization: To supplement the information obtained from the survey, the End Evaluation used a substantial amount of information from the project's monitoring system. It is, however, not clear from the ICR how much of the information collected in the course of the project was used to manage the project M&E Quality Rating: Modest 11. Other Issues a. Safeguards: As indicated in the PAD, three safeguards policies were triggered: OP/BP 4.01 (Environmental Assessment), OP/BP 4.09 (Pest Management), and OP/BP 7.50 (Project on International Water Ways). According to the ICR, the project complied with all three policies. b. Fiduciary Compliance: Financial Management . Financial management (FM) practices were challenging in the remote regions areas and at the sub-district levels. It proved time consuming to collect statements of expenditures (SOEs) for all of the community investment fund sub-projects, aggregate them at the woreda level, and send them up the chain through the regional governments and to the FPCU. Without the SOEs, the Bank could not replenish the project account and the communities would not have the funds to complete the community investment. This process caused some delays, but did not ultimately negatively affect project development outcome. FM rating was moderately unsatisfactory before the MTR, but improved thereafter with timely approval of annual work plans and budgets, and the reduction in all advances to less than four months (ICR, para 52). The final Implementation Supervision Report upgraded FM performance to moderately satisfactory noting improvements in timely submission of SOEs, full utilization of IDA funds and actions taken on most of the internal audit findings related to FM. Procurement. A review of procurement capacity assessment for this project was carried out. In addition training on contract administration and documentation was arranged. Nevertheless, the ICR noted that "capacity inadequacies were evident in the poor quality of the record keeping and procurement documentation" (para 53). The ISR rated procurement moderately unsatisfactory for these reasons, but raised the rating in the final ISR to moderately satisfactory with improved performance at regional and federal levels. The ICR concluded that procurement by communities was a positive step in capacity building as they gained experience in procuring skilled labor and building materials for CIF sub-projects. Field interviews by the ICR team reported that the procurement capacity that was created "generated greater community ownership and contributed to cost effectiveness and transparency in the procurement process" (para 53). c. Unintended Impacts (positive or negative): None d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome: Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately There were weaknesses in the project's Satisfactory financial management and procurement, especially in the earlier years (see section 11b). Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR drew five lessons from the project. In brief they were (a) Communities gain confidence in their ability to solve their own problems when they receive appropriate training to support decision-making. (b) Community empowerment can be achieved by consultation early in the process on their priorities for use of scarce project funds. (c) An SOE-driven approach to CIF replenishments is too unwieldy and creates implementation delays. A more practical needs to be found for all such projects. (d) Rural savings and loan cooperatives work well in promoting income generating activities that communities are very familiar with, but less so on promoting new income generating activities. (e) Small-scale irrigation interventions, crop yield improvements are almost automatic in terms of implementation, but crop diversification is not because it was found that farmers require more time, along with support services, to diversify crops to such things as onions and tomatoes. This Review concluded that there were five broad lessons from the project: (a) Care needs to be taken in ensuring that a CDD model of development is fully participatory and integrated into local development planning processes, since without this, the ownership and sustainability of community driven development planning will be weak and the process and its effects (pro-poor participation in local development planning, increased relevance and resilience of community demanded assets, better targeted distribution of assets etc) will diminish after project close. (b) Projects that seek to reduce vulnerability of project affected persons require a clear method for assessing and measuring vulnerability and the project attributable impacts, especially in an environment with several donor and government funded initiatives seeking to secure the same effect. (c) Implementing CDD within transhumant societies requires a differentiated design for the agro-pastoralists that are largely sedentarized and the migrating population and their relatives residing in associated hamlets. There are many lesson to learn from this project about how to design and implement a CDD program that takes these differences into account, especially in a fast changing social and economic environment that were not presented in the ICR. Facilitation, for example, will require a different skills set and some assets will need to be mobile. Decision-making processes will vary and will need to be designed in a manner that takes into account the needs of pastoralists who may not be present in decision-making forums. (d) Care also needs to be taken to understand the role of the pastoral projects as a function of the wider political-economy, especially with regard to the role of the project as function of programmatic goals designed to sedentarize pastoral communities. (e) Pastoral Development projects are often implemented in environments that are characterized by a certain level of land and asset contestation. While this project had a grievance mechanism, the effectiveness of this mechanism was not reported in the ICR. Donors should apply a conflict filter in such cases to determine whether interventions in contested areas may help or hurt a fragile environment, and adjust design accordingly. In the case of the PCDP, the project indicated it would avoid areas affected by conflict, or the risk of conflict. However a strong justification is required to ensure that the redirection of aid in these areas will not exacerbate grievances or potentially contribute to pockets of increased poverty and insecurity. The approach needs to weighed against multiple factors. 14. Assessment Recommended? Yes No Why? The PPAR is recommended as an input into the Macro-Evaluation of the Rural Non Farm Economy. A PPAR is also recommended to more fully measure and report on the achievements of the project development outcomes, with regard to increased resilience and and the sustainability of the incremental income that was earned through the investments and the savings and loan cooperatives. This is the second phase of a three phase program that have been under implementation for 15 years. IEG has, to date, not conducted a comprehensive review of its support for pastoral livelihoods in Ethiopia (it was not included as a major theme in the 2008 CAE, for example). Some of the project level issues that could be covered by the PPAR include the questions raised by this review about the relevance and effectiveness of the CDD approach within the complex pastoral and food distribution systems in place in Ethiopia; the combined effectiveness of the early warning systems, the CIF and the cooperatives with regard to increasing resilience as a combined approach; and the relative effectiveness of the Bank's approach in the PCDP compared to other donors' efforts and other Bank programs supporting social safety nets in the project areas. 15. Comments on Quality of ICR: The ICR was candid and useful. However it depended significantly (see Data Sheet) on the "End Evaluation" report without acknowledging that the data in the End Evaluation were based on a non-random sample - mentioned in Section 10b in this Review. The ICR provided a "split" assessment of the project's outcome but without any justification or evidence for the ratings before and after restructuring. Finally, the ICR could have explored some options for assessing the project's efficiency (such as had been done in the PAD) instead of concluding that there were no data. a.Quality of ICR Rating : Satisfactory