Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) Report Number : ICRR0020045 1. Project Data Project ID Project Name P127212 TN-Participatory Service Delivery Reint Country Practice Area(Lead) Tunisia Social Protection & Labor L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-11069 31-Mar-2014 5,000,000.00 Bank Approval Date Closing Date (Actual) 14-Jul-2011 31-Mar-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 5,000,000.00 Revised Commitment 0.00 4,747,554.43 Actual 0.00 4,747,554.43 Sector(s) Vocational training(50%):Other social services(30%):Health(20%) Theme(s) Other human development(100%) Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the Project Proposal (page 8), the project objective was as follows: • To pilot participatory approaches to employment generation through a cash-for-service program for vulnerable Tunisian households. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) The project was supported by a grant from the State and Peace-Building Fund, which aimed to mitigate the negative effects associated with the conflict in neighboring Libya and the political-economic instability in the region following the January 2011 revolution. Therefore, the project was part of a broader effort to: (i) foster social cohesion and stabilization among disadvantaged populations following the revolution, and (ii) mitigate the socioeconomic risks faced by Tunisians returnees fleeing Libyan civil strife and byTunisian communities near the border with Libya affected by depressed cross-border trade and commerce. The project objective in the Grant Agreement (page 4) had identical wording, although it further specified that the project would be implemented in the governorates of Tataouine and Medenine. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components (Note: Appraisal figures drawn from Project Proposal (pages 9-12); actual figures drawn from ICR (Annex 1).) A. Pilot Local Service Delivery Sub-Projects (Appraisal: US$ 4.3 million; Actual: US$ 4.3 million): This component aimed to create short- term job opportunities in social services. Local NGOs or community associations were to prepare sub-project proposals to generate employment in the areas of social care, basic education, and basic health services. Examples of employment included household services, childcare services, teaching assistance, adult literacy training, health promotion assistance, and sanitation work. Selection criteria for sub-projects included responsiveness to supply shortages and local needs, high job growth potential, and readiness for implementation. This component was to finance approximately 40 sub-projects, each with a duration of one year, generating employment for 4000 workers. B. Local Service Delivery Sub-Project Training (Appraisal: US$ 0.43 million; Actual: US$ 0.43 million): This component aimed to develop training programs for workers and NGOs. Training would consist of capacity building for local governments and NGOs on planning and implementing sub-projects (i.e. financial management, human resources management) and vocational training for workers. C. Project Management, Monitoring and Evaluation (Appraisal: US$ 0.27 million; Actual: US$ 0.27 million): This component was to support the project implementation unit (PIU) in the Tunisian social protection agency (UTSS) in day-to-day management and monitoring of the project. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost: • The appraised project cost was US$ 5.0 million. According to the ICR Annex 1, the actual project cost was US$ 5.0 million, indicating 100% disbursement. According to the Project Portal, the grant disbursed US$ 4.76 million. The ICR does not provide an explanation for this discrepancy. Financing: • The project was financed entirely by a grant from the State and Peace Building Trust Fund. Borrower contribution: • There was no planned Borrower contribution. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) Dates: • February 2014: The project closing date was extended from March 2014 to March 2015, in order to allow completion of sub-projects and cash transfer disbursements to the workers, and also to prepare the project beneficiary assessment. 3. Relevance of Objectives & Design a. Relevance of Objectives In January 2011, Tunisia experienced a popular uprising that was driven by discontent over political and social disparities and poor employment prospects, as well as allegations of corruption in the government. The interim government is therefore taking short-term measures aimed at greater economic and social inclusion, particularly for unemployed, marginalized households. Hence, the project objective was and continues to be responsive to the prevailing socio-economic conditions in Tunisia, including the negative impact of the Libyan conflict on Tunisian households. There is high unemployment (18%) and related fragility in social and political cohesion. There has also been a reduction in cross-border trade and remittances, which has contributed to slow job generation and limited delivery of social services. The project objective is consistent with the government's Social and Economic Program (launched in 2011), which focuses on two key pillars of (i) improving transparency and accountability, and (ii) taking immediate measures to relieve the plight of the unemployed and most vulnerable families. Similarly, the project objective is relevant to the Bank's Interim Strategy Note (FY 2013-14, extended to 2015), which identifies job creation, social and economic inclusion, and participation as three key areas of engagement. As noted in the ICR (page 6), the intended outcomes of increased income generating opportunities and increased local participation were not explicitly stated in the project objective, although these could be derived from the key indicators and project activities. Rating Substantial b. Relevance of Design The project interventions were intended to be short-term measures, addressing the immediate conditions of socio-political tension and high unemployment. The creation of employment in the social service sector was likely to lead to the intended outcome of income generation for mostly low- and semi-skilled workers. The project design also featured a participatory approach, in which local communities submitted sub- project proposals according to their local needs. This feature was likely to increase participation in planning and delivery of relevant social services (such as basic education, health and social care), which are currently being provided only to a limited extent by the public sector. The project targeted the two regions of Tataouine and Medenine, both of which border Libya and are marked by high unemployment (23%) and poor economic and social outcomes (13% of families receive cash assistance, and maternal mortality and child malnutrition rates are double that of the capital city). Individuals were determined to be eligible for the project according to proxy indicators of vulnerability, such as adult head of household, semi-skilled unemployed status. Due to Tunisia's limited experience with participatory approaches in public service delivery, the project was intended as a pilot project; however, the evaluative steps to be taken to learn lessons and inform any future scale-up or roll-out of the pilot program were not clearly articulated in the project proposal. Rating Substantial 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) Objective 1 Objective To pilot participatory approaches to employment generation through a cash-for-service program for vulnerable Tunisian households. Rationale Outputs The project financed wages for 70 sub-projects that were intended to create job opportunities in basic social services. The sub-projects were characterized as follows: • 24 sub-projects were implemented in Tataouine: 55% were in social protection, 27% in education, and 18% in health. • 46 sub-projects were implemented in Medenine: 74% were in social protection, 19% in health, and 7% in education. • The sub-projects generated approximately 556,380 person-days of work (number of beneficiaries x six months per person x 15 days worked per month). • 78% of sub-projects were service delivery and 22% were minor civil works. The project also financed vocational training for all sub-project workers. The ICR (page 13) reports that "in most cases, small firms and trade associations took an active part in designing and delivering the on-the-job training, often reflecting the skills demanded most by the local labor market." Outcomes Access to employment • The number of beneficiaries that were employed by the sub-projects was 6,180. This surpassed the target of 4,000. Of these, 70% were women. This surpassed the target of 30%. 91% of beneficiaries had either a primary or secondary education level. • The proportion of beneficiaries who perceived an improvement in job skills as a result of the training acquired through sub-projects was 54%. This fell short of the target of 60%. 75% of beneficiaries indicated that "work experience" was the biggest reason for an improvement in their job skills. • The proportion of beneficiaries who perceived that training responded to job skills development needs was 72%. This surpassed the target of 60%. In addition, 13% of beneficiaries (of the 1,360 surveyed within a year of exiting the project) had found employment outside the workfare program within six months. Access to services • The number of community members benefiting from the services provided through sub-projects was 62,669. This surpassed the target of 12,000. This includes indirect beneficiaries, calculated by estimates of catchment areas rather than direct surveying. • The proportion of community members who perceived services to be responsive to local needs was 67%. This achieved the target of 60%. • The proportion of community members who reported that "additional services provided were beyond those in the public sector" was 67%. The target was 60%. Qualitative accounts indicate that communities and authorities "found the Project opened up career opportunities in sectors and services with unmet needs, either because public services were unable to reach certain households, or because the services had never been created" (ICR, page 14). Participation • The proportion of beneficiaries and community members who perceived an increase in local decision-making (regarding service Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) delivery) was 56% and 47%, respectively. These fell short of the target of 60%. • The proportion of community members who perceived services to be responsive to local needs was 67%. This achieved the target of 60%. Other • 100% of bi-annual project management targets were met by central and regional staff. This surpassed the target of 70%. Given the pilot nature of the project, the project design (Component 3) planned for "periodic interim and final evaluations" of sub-project performance. A beneficiary assessment was conducted by an external agency to assess perceptions of process and implementation of the sub-projects and the relevance of the sub-projects; however, a more in-depth evaluation of project effects was not conducted, although it is being considered as part of the next steps. Other lessons learned through project experience are highlighted in Section 6 (pages 19-20) of the ICR. Achievement is rated Substantial due to evidence of the employment generated by the sub-projects, the perceived increase in job skills, and the increased access to services. Rating Substantial 5. Efficiency There was no economic analysis prepared for the project at appraisal. In terms of efficiency resulting from project design, the project was clearly targeted geographically (two regions on the border with Libya with poor social and economic outcomes) and with explicit criteria for participation in the program (proxy indicators for vulnerability such as unemployed head of household, not currently receiving other social assistance, etc.). The ICR (Annex 3) discusses the potential cost-effectiveness of this project compared to the more traditional social safety net program in Tunisia. The existing government program, which is a non-workfare based program, entails a cost of TND 130 per household per month; benefits in the form of cash transfers also equal TND 130. The project was intended to produce additional benefits of increased productivity of project beneficiaries (as measured by additional income generated from follow-on employment opportunities) and improved local community development (not measured in this analysis), aside from income generated. The project paid out TND 220 per household per month, plus an additional 14% due to vocational training and operating costs. If the figures for the project are extrapolated to the same scale of the existing government program, then the total cost of the project would be TND 107 million, while the cost of the government program would be TND 294 million. Given the lower cost of the project plus the additional benefits, it appears that the project was more cost-effective than the comparator program. Additionally, a survey of a sample of beneficiaries showed that 13% had found employment outside the workfare program after the project ended. Assuming the additional earnings were at least minimum wage level, the additional income generated is TND 16 million. Therefore, the rate of return for the project would be 15%. The figures presented above are hypothetical (not based on actual data) - given the pilot nature of the project, a more robust economic analysis is not to be expected. Initial delays due to inadequate implementation agency and NGO capacity were overcome through significant Bank support. Delays in payments to NGOs and beneficiaries were also subsequently reduced through revising financial management arrangements. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 100.00 ICR Estimate  15.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objectives and design are rated Substantial. Achievement of the objective to pilot participatory approaches to employment generation through a cash-for-service program for vulnerable Tunisian households is rated Substantial due to evidence of increased employment generated by the sub-projects, the perceived increase in job skills (albeit minor shortcoming in achieving the target), and the perceived increase in access to services. Efficiency is rated Substantial due to preliminary evidence of cost-effectiveness of the project, compared to the existing non-workfare based social safety net program. a. Outcome Rating Satisfactory 7. Rationale for Risk to Development Outcome Rating The government has expressed strong interest in scaling up the project as part of its efforts to enhance social protection and labor system. It has sent a written request to the Bank for a follow-on project to scale up this project and to introduce reforms to its existing national cash transfer program. However, the country situation (political, economic, and security) remains highly volatile and the slow pace of implementing social programs by the government poses substantial risk to outcomes. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry Country-specific experience in using participatory approaches for social services delivery was limited; therefore, the project design drew from lessons learned from other countries, in which the participatory approach helped create social cohesion while also generating employment. Although the statement of objectives did not clearly articulate the intended outcomes (increased income generating opportunities, improved local participation), the project components and results framework were coherent and relevant. The supporting operational guidelines (targeting criteria, M&E indicators, etc.) were also consistent with the intended outcomes. The risk assessment appropriately considered project risk as high, given the volatile country situation and the lack of prior country experience in employment generation using participatory approaches and in Bank fiduciary procedures. However, readiness for implementation was still delayed due to inadequate fiduciary training provided to the project implementing unit. The M&E indicators were appropriate, although a plan Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) for evaluating the project as a "pilot" was not clearly defined. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision There were initial implementation delays during the first year of effectiveness due to the need to train new PIU staff on Bank fiduciary guidelines and also to comply with effectiveness conditions (due to government staffing changes). The Bank team responded by providing extensive fiduciary training to project implementation staff. Following these delays across the first year, the pace of implementation improved, as reflected by the number of beneficiaries enrolled in the program. Planned interventions were largely completed, after the initial delay. The continuity and in-country presence of the task team leader and fiduciary staff contributed to consistent implementation support, which included addressing some delays in payments to NGOs and beneficiaries. Implementation of M&E was overall satisfactory, and there were no major fiduciary problems reported. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance There was a lengthy period between Bank project approval and project effectiveness, in part due to the ongoing political transitions in the government. However, the government provided an overall supportive environment for the project, including fostering effective coordination between the Ministry of Vocational Training and the Ministry of Social Affairs to verify eligibility of beneficiaries. The ICR (page 9) reports that "there was little documentation provided on the role and inputs of the inter-ministerial Steering Committee to allow for improvements as needed." Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The Union Tunisienne de la Solidarite Sociale (UTSS), within the Ministry of Social Affairs, was the primary implementing agency. Following extensive training and implementation support by the Bank team in the first year of the project period, UTSS increased in capacity and implemented most project activities as planned. According to the ICR (page 9), Implementation Status Report reviews and interviews with the project director confirmed that there were no major implementation problems. However, there were delays in payments, which was an "important negative effect reported by NGOs and beneficiaries" (ICR, page 16); no specific details are provided, although the delays were eventually reduced over the project period. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) 10. M&E Design, Implementation, & Utilization a. M&E Design The M&E framework was overall appropriate, with indicators directly linked to project interventions and reflective of the intended outcomes (as these were not explicitly articulated in the project objective statement). The M&E arrangements ensured regular monitoring and data collection (by the project implementation unit), and also included measures for participatory monitoring of service delivery (i.e. public dissemination of service delivery data to citizens). Given the pilot nature of the project, the M&E design also planned for "periodic interim and final evaluations" of sub-project performance to inform future scale-up. b. M&E Implementation The PIU included a dedicated M&E specialist to maintain the project database, which collected detailed information on beneficiary demographics, sub-project details, NGO partner organizations, beneficiary perceptions of sub-projects, and follow-up employment data for approximately 1,450 beneficiaries. Evaluative activities included a beneficiary assessment (October-November 2014) of a sample of beneficiaries and NGO partners, which helped assess the project impact on employment generation and local participation. It is not clear whether the planned participatory monitoring activities were carried out. c. M&E Utilization The ICR (page 10) reports that the Project Director "confirmed that the PIU used the [database information] to provide implementation and capacity building support to NGOs experiencing delays in implementation of selected sub-projects." A beneficiary assessment was conducted by an external agency to assess relevance and perceptions of the sub-project process. A more in- depth evaluation to analyze impact and inform scale-up of the program was not conducted, although it is being considered as part of the next steps. Other lessons learned through project experience are highlighted in the "Lessons Learned" section (pages 19-20) of the ICR. M&E Quality Rating Substantial 11. Other Issues a. Safeguards The project was classified as a Category "C" project, and therefore no safeguard policies were triggered. b. Fiduciary Compliance The ICR (page 10) reports that there were no major issues with fiduciary compliance during the project period. Financial management: Financial management arrangements were reviewed regularly during the project period, and overall functions were deemed adequate in providing information on financial flows and appropriate use of funds. Annex 9 reports on initial challenges, including delays in submission of financial reports (due to initial lack of familiarity with Bank procedures), quality issues with audits, over-commitments in certain budget categories, and slow replenishment of the designated account by the central bank. According to the ICR, these issues were adequately addressed to ensure full compliance with Bank guidelines. Procurement: Due to lack of specific expertise on Bank procurement procedures, the procurement arrangements included staffing of a Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) procurement officer in the PIU and the detailing of procurement guidelines in the Operations Manual. In particular, guidelines for selection and implementation of sub-projects were effectively followed, and compliance with procurement procedures was overall satisfactory (ICR, pages 10- 11). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Outcome Substantial Substantial --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Lessons drawn from the ICR (pages 19-20): • Public-private partnerships can be a useful modality for local, participatory, demand-driven service delivery. In the case of this project, despite initial implementation delays in establishing institutional arrangements, NGOs were an effective vehicle for the provision of employment and basic social services, as determined by the local communities. • Linking vocational training and short-term employment opportunities to the profile of local job markets is essential, particularly to ensure transferability of vocational skills. In the case of this project, gaps in local service delivery were identified as potential employment opportunities for semi-skilled workers. 14. Assessment Recommended? Yes Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TN-Participatory Service Delivery Reint (P127212) Please explain A more rigorous evaluation of project impact (beyond the beneficiary survey) would be informative to analyze the effectiveness and efficiency of this workfare-based approach, compared to the traditional non-workfare based approach. Also, an evaluation could provide more evidence of the project's impact on local participation and lessons on the use of participatory approaches. 15. Comments on Quality of ICR The quality of the ICR's evidence is satisfactory, due to the strong focus on indicators directly linked to project interventions, supplemented by findings from the beneficiary survey. The analysis is results-oriented (i.e. assessing impact on employment opportunities, access to services, and participation) despite the vague wording of the project objective. More information on the participatory aspects of the project - local selection of sub-projects, participatory monitoring - would have been informative. a. Quality of ICR Rating Substantial