ICRR 13016 Report Number : ICRR13016 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/30/2008 PROJ ID : P074408 Appraisal Actual Project Name : Social Risk Mitigation US$M ): Project Costs (US$M): 630.5 835.9 Project (SRMP) Country : Turkey Loan /Credit (US$M): Loan/ US$M ): 500 498.9 Sector Board : SP Cofinancing (US$M ): US$M): 0 0 Sector (s): Other social services (74%) General education sector (12%) Health (6%) General industry and trade sector (4%) Central government administration (4%) Theme (s): Education for all (20% - P) Small and medium enterprise support (20% - P) Social analysis and monitoring (20% - P) Social safety nets (20% - P) Improving labor markets (20% - P) L/C Number : L4638 Board Approval Date : 09/13/2001 Partners involved : Closing Date : 06/30/2006 03/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Gayle Martin John R. Heath Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: The general objective was to mitigate the impact of the 2001 economic crisis on poor households and to improve their ability to cope with similar risks in the future . The specific objectives were to: (i) provide immediate support to the poorest affected by the crisis; (ii) build the capacity of state institutions providing basic social services and social assistance to the poor; (iii) implement a social assistance system of conditional cash transfers (CCT) targeted to the poorest 6% of the population conditional on improved use of basic health and education services; and (iv) increase the income-generating and employment opportunities of the poor . The objectives in the PAD and loan agreement were generally consistent. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): A. Rapid assistance to the poorest people (appraisal estimate: US$100 million/16% of estimated cost; actual cost : US$100 million/12% of actual cost). The purpose of the adjustment component was to provide quick -disbursing budget support to deliver rapid relief to vulnerable groups through existing government mechanisms . The budget support financed: the delivery of school attendance packages (shoes, uniforms, books, and stationery ), pharmaceuticals for poor beneficiaries of existing insurance programs, and social assistance in -cash and in-kind (including food and heating support ). A macroeconomic condition and a sectoral condition for the tranche release were agreed upon: (i) satisfactory macroeconomic framework using agreed indicators; and (ii) satisfactory progress in implementing the program defined in the Letter of Sector Policy (on social risk mitigation, social assistance and poverty reduction). B. Institutional development (appraisal estimate: US$36.6 million/6% of estimated cost; actual cost : US$41.5 million/5% of actual cost). The purpose of this investment component was to build the capacity of the main institutions that deliver social services to vulnerable groups : the Social Services and Child Protection Organization, the Social Solidarity Fund and its 931 Social Solidarity Foundations. The component financed: staff development and training; management information system (MIS) and information technology (IT) development; technical support to do monitoring, evaluation and policy research; the statistical survey work and related poverty analysis of TurkStat, the statistics institute; and public information campaigns for the mentioned social assistance institutions . This component also financed the Project Coordination Unit (PCU) within the Social Solidarity Fund, which included financial management, documentary and operational reviews, M&E studies, and beneficiary assessments . C. Conditional cash transfer program (appraisal estimate: US$360 million/57% of estimated cost; actual cost : US$559.7 million/67% of actual cost). The purpose was to finance the introduction of an indicator -based administrative targeting mechanism, targeted to the poorest 6% of families and linked to behavioral change with respect to health and education . The CCT program financed education grants and health /nutrition grants to the eligible mothers of children or other eligible recipients . The program provided an extra incentive for secondary school attendance by girls . D. Local Initiatives (appraisal estimate: US$133.9 million/21% of estimated cost; actual cost : US$134.7 million/16% of actual cost). The aim of the Local Initiatives component was to finance activities to promote sustainable employment among the poor that have been proposed by districts and local communities . The component invested in small loans for income-generating activities (IGAs), and grants for employability (skills) training, social service sub-projects, rehabilitation of schools, and temporary employment . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Dates. Dates The project became effective as planned (12/04/2001), but the disbursement of the budget support under Component A was delayed due to failure of the Government of Turkey (GOT) to meet the macroeconomic condition . The condition was met in April 2002 (4 months after the expected date ). Implementation of other components was further delayed due to failure to meet the disbursement condition (requiring the legal status of the Social Solidarity Fund to be defined by the Parliament ). The loan agreement was amended to allow disbursement to proceed . The law was eventually passed in December 2005 and the General Directorate of Social Assistance and Solidarity was formed. The dated covenants in the PAD (p45) required a mid-term review (MTR) by December 2003. According to the ICR datasheet no MTR was conducted . Project closing was delayed by 21 months in total---largely to accommodate the initial delays in implementation . Costs Actual project cost (US$840.9 million) was 133% of cost at appraisal. Component C (the CCT Program) Costs. accounted for the largest share (67%) of actual costs. Financing . At closing the loan was 100% disbursed (actual: US$498.9 million). Borrower contribution (US$32 million) substantially exceeded the appraisal estimate (252% of appraisal estimate). There was also a front-end fee of US$5 million. 3. Relevance of Objectives & Design: Objectives . The project responded to GOT's request for urgent poverty mitigation assistance following the 2001 economic crisis, as well as support for the country's medium to long term social management and social prevention needs. The project was consistent with the country's long -term development agenda (National Development Plan 2001-23) and complemented existing investments in the human development sectors . The project was, furthermore, consistent with the priorities in the Bank's 2001 CAS Progress Report and remains consistent with the priorities of the 2008 CAS. Design . The hybrid loan combined adjustment lending (Component A) with investment lending (Components B-D). The rationale was that a hybrid loan offered a vehicle for rapidly responding to the crisis, combined with the key medium term institution-building and investment requirements needed to strengthen the country's social safety net . The option of two separate operations was rejected because the risk of the two elements losing coherence was considered high. Targeting was comprehensive: the adjustment component and the CCT component targeted the poorest of the poor, while the Local Initiatives targeted the remaining poor or near poor . There were a few areas were design could have been improved : the choice of conditions for tranche release (i.e., the Law) may not have been fully appropriate; the targeting mechanism for the Local Initiatives could have been better thought through; the M&E plan could have been strengthened; and there were weak links between the information campaigns on the one hand and the CCT and Local Initiatives on the other hand . The ICR makes no reference to the Quality Enhancement Review (QER) (September 2001), nor the follow-up actions taken by the Team . The relevance of both objectives and design is rated Substantial . 4. Achievement of Objectives (Efficacy): Objective 1. Provide immediate support to the poorest affected by the crisis (rated substantial ) Outputs. The project delivered school attendance packages, food, fuel for heating and essential drugs to poor households, disbursing YTL 486 million in 2001 (just short of the target of YTL 550). Government initially financed the assistance because of the delays in meeting the conditions for the US$ 100 million tranche release. Education assistance was provided to 1.05 million children in about 400,000 households. Outcomes and Impacts. While a substantial share of the resources benefitted the poor, some of the poor were excluded because of biases in information dissemination and the reliance on headmen for information about the available assistance. Objective 2. Build the capacity of state institutions providing basic social services and social assistance to the poor (rated modest ) Outputs. The following institutions benefitted from capacity building activities : the Social Solidarity Fund, the 931 Social Solidarity Foundations, the Social Services and Child Protection Organization and TurkStat . MIS and IT systems were procured for the Social Services and Child Protection Organization, connecting the Ankara headquarters with 81 provincial directorates and 600 delivery points. The MIS linking the Social Solidarity Fund and its Social Solidarity Foundations was substantially delayed . Software development only started in October 2007 and the MIS was incomplete at project closing . TurkStat conducted the household income and consumption expenditure survey in 2002 and supported the 2003, 2004 and 2005 surveys through investments in computer equipment and training. Four assessment studies and an external impact assessment was financed in support of M&E capacity . Training was offered to the project coordination unit, although there was some difficulty in transferring skills to the General Directorate of Social Assistance and Solidarity . Outcomes and Impacts. Evidence of the impact of these institution building activities was : the success of the Social Solidarity Fund and the Social Solidarity Foundations in implementing the CCT program; and TurkStat's completion of the household and income survey (2002) and on-going poverty monitoring. The delays in development of the MIS hampered integration of the records of the General Directorate of Social Assistance and Solidarity programs, and its use to manage implementation of CCT, small loans for income -generating activities, training, and other activities . The 2002 household survey formed the basis for key poverty reports (e.g., the Joint Poverty Assessment in 2005), poverty mapping, and informed other policy analysis and planning . Information campaigns were only partially successful at publicizing the Social Solidarity Fund Program, and there were some eligible beneficiaries reported not being fully aware of the available support . Objective 3. Implement a CCT system to the poorest 6% of households (rated high ) Outputs. An indicator-based administrative targeting mechanism, the CCT, was introduced in 2003/4. The program was implemented through the Social Solidarity Fund and the Social Solidarity Foundations reaching 2.6 million ultra-poor beneficiaries (exceeding the targeted 1.1 million). Half of the CCT recipients in grade 1 were female. Implementation experienced delays in payments and fluctuations in payment levels . The assessment by the International Food Policy Research Institute (IFPRI) found that health and education beneficiaries, respectively, received only 57% and 64% of the entitled benefits. The regularity of payments reportedly improved over time . Despite implementation of some information campaigns, there were large information gaps and many eligible individuals lacked information on eligibility and access procedures . (It is worth noting that the challenges of getting information to the ultra-poor was stressed in the QER.) Outcomes and Impacts. The CCT was generally successful at reaching the intended beneficiaries with only modest levels of leakage to non-beneficiaries: half of health beneficiaries and a third of education beneficiaries were from the poorest 10% of households, and no beneficiaries were from the richest 30% of households. Successive surveys suggest targeting effectiveness improved over time . The IFPRI impact evaluation found that the program had no impact on girls' primary school enrollment ** (enrollment increased by only 1.3 percentage points) in part due to already high levels of primary enrollment (>99%). There was a substantial impact on secondary school enrollment : the CCT program was associated with a 11% increase in girls' secondary school enrollment; among 14-17 year old children in rural areas enrollment was 17% higher than among non-beneficiaries; and in rural areas beneficiary boys had a 23% higher secondary school enrollment rate than non -beneficiary boys. Drop-out rate data were not available, and the rate of progression from primary to secondary school was analyzed instead . The impact evaluation found no improvement in rates of progression between beneficiaries and non -beneficiaries. However, at the secondary education level completion rates among beneficiary girls were 8 percentage points higher than among non-beneficiary girls. According to the impact evaluation, the CCT was associated with a 14% increase in full immunization rate in preschool children but no difference in health service utilization . According to the IFPRI analysis, the latter was in part due to supply -side quality problems, and because of lack of information among beneficiaries of the 'health facility use' condition . Objective 4. Increase the income generating and employment opportunities of the poor (rated modest ) Outputs. The project financed: small loans for 5,242 IGAs that benefitted 45,549 people; 387 skills training sub-projects that trained 12,453 people (mainly youth); 2,480 social services sub-projects; temporary employment to 5,125 people through 755 labor intensive public works sub -projects; light repairs in 1,300 schools. Outcomes and Impacts. The Local Initiatives were supposed to use a similar targeting approach as the CCT program. The targeting criteria were rarely applied resulting in the initiatives only weakly targeting the poor . Access to information about availability of the Local Initiatives was disseminated by headmen, and information was sometimes biased by political favoritism, diluting the pro -poor benefit. Income generating activities (IGAs): : No information was available on the number of private businesses started . An assessment study rated two thirds of IGAs either fragile or not -sustainable. There was a relatively high rate of failure : half of sewing and embroidery IGAs, a quarter of street vending IGAs and 68% of bee-keeping IGAs failed. The success of IGA loan repayments was mixed . Evaluation studies reported failure to repay among 14% and 23% of beneficiaries, suggesting the majority were repaying their IGA loans . One study found that two thirds of IGA loans were targeted at households with income less than YTL 380 per month, but the same study showed significant under-reporting of income. Skills training: 12% of trainees dropped out and 43% of trainees were placed in jobs . The outcome of the placements was, however, not followed up. No adult literacy training was provided despite the indicator on adult literacy being included in the results framework . The program was furthermore poorly linked to the Chamber of Commerce and Industry to ensure consistency with the economy's training needs . Social services and infrastructure : The overwhelming majority of beneficiaries (96%) reported success of social services sub-projects but it is unclear how success was defined or how the services were used . According to the ICR parents participated in school repairs and maintenance . Social infrastructure investments were reportedly valued by beneficiaries but no information was provided on impact of investment on vulnerability . Based on the ratings of the individual objectives weighted by the expenditure toward each objective, efficacy is rated substantial . 5. Efficiency (not applicable to DPLs): The first component was an adjustment operation and a rate of return calculation is not required for this component. Evidence was presented on the efficiency of CCT investments (component C), although not on the other components financed through investment lending, for example, the income generating activities under component D . The CCT had high level of targeting effectiveness (i.e., high level of coverage of the intended beneficiaries and relatively low level of leakage). While the evaluation suggested some room for improvement in the criteria for the proxy means test (to reduce errors of exclusion ), the use of this targeting approach contributed to improved cost -effectiveness of program delivery. Some of the project achievements (e.g., substantially increased immunization coverage ) are known to be highly cost-effective and yield high returns . Given that the CCT accounted for nearly eighty percent of the investment operation (Components B-D), efficiency is rated substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on the substantial relevance , efficacy and efficiency, the project outcome is rated satisfactory . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The project had a high level of government ownership, judging from GOT's willingness to pre -finance project activities when the country failed to meet macroeconomic conditions for tranche release, and the fact that the government is continuing the finance the CCT program and selected Local Initiatives through the government budget. The CCT imposed a manageable fiscal burden accounting for 0.6% of government expenditures in 2005 (ICR, p8). Some programmatic risks are worth flagging : The impact evaluation found that 90% of education beneficiaries and 87% of health beneficiaries did not know the conditions they needed to meet in order to maintain their eligibility suggesting some risks to the CCT program benefits . These risks are being mitigated through improved information campaigns according to the ICR . The income-generating and skills transfer activities financed through the Local Initiatives component face some risk . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: at -entry. Quality -at- entry The choice of funding instrument (a hybrid loan) was appropriate as it allowed the Bank to respond rapidly while at the same addressing key longer term investment requirements to strengthen Turkey's social safety net. There was a high level of complementarity between the demand -side interventions financed by the project and the existing Bank -financed supply-side operations within the health and education sectors . The team chose to use existing institutions and the choice of implementing agencies was reasonably well thought through. The CCT criteria and targeting mechanism (the scoring formula), verification mechanism and implementation arrangements for the CCT program was generally well planned . There were, however, some trade-offs between the urgency of making available the adjustment loan and the quality of planning and preparation for the investment operations ---especially the Local Initiatives component and M&E . Another shortcoming was the weak synchrony between the information campaigns, and the CCT and Local Initiatives . It is not clear from the ICR to what extent the detailed feedback provided in the Quality Enhancement Review (QER) (September 2001) were followed up by the Team. Quality of supervision . Missions were regular with strong follow -up by the Ankara-based Bank staff especially on fiduciary matters. The Bank team was generally responsive and sought to resolve implementation challenges, e.g., by amending the loan agreement allowing disbursement to proceed . A PCU was created, but the team could have done better at encouraging the integration of PCU functions to the General Directorate of Social Assistance and Solidarity. at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Government performance . There was a high level of gover nment support and ownership for the Project. This contributed to the speedy implementation although there were some delays when administrations changed in 2002. The Government pre-financed some of the project expenditures (in 2001) and over the project's lifespan budgeted adequately for the counterpart funds . It took on an increasing share of responsibility for financing, higher than mandated in the Loan Agreement . The Advisory Committee was only partially functional, although this did not detract much from implementation because of the high level of commitment from the implementing agencies. Implementing agency performance . A PCU was created and staffed with skilled staff . Project implementation especially fiduciary requirements were well executed . Procurement was timely, with the exception of the impact evaluation which was delayed . There were problems with integrating the PCU into the General Directorate of Social Assistance and Solidarity, and responsibility for this has to be shared between the Bank and the borrower . The size of the operation and the country -wide geographic reach of the project made for a potentially a complex operation. The implementing agencies did well to execute the various parts of the project, in part because of the high level of ownership among the key institutions . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: Design . The PAD proposed the development of an integrated MIS to monitor and manage activities implemented by the General Directorate of Social Assistance and Solidarity and its 931 local Social Solidarity Foundations . Evaluation studies were envisaged, including : a household surveys and an impact evaluation . The choice of indicators was generally comprehensive but there were two main shortcomings : (i) the use of the primary school enrollment rate was questionable given already high levels of enrollment (99%); and (ii) drop-out rates were not readily available and no plan was made to collect the relevant information . Implementation and Use . The MIS and IT systems for the Social Services and Child Protection Organization have been delivered and according to the ICR are functioning . These systems connect Ankara headquarters with 81 provincial directorates and 600 delivery points. There was less success in the MIS linking the Social Solidarity Foundations. Software development phase was only started in October 2007 and work is currently completed using the government’s own funds. The delays hampered integration of the records of the General Directorate of Social Assistance and Solidarity programs . TurkStat conducted the household income and consumption expenditure survey in 2002 and supported the 2003, 2004 and 2005 surveys through investments in computer equipment and training. The survey of 2002 formed the basis for the Joint Poverty Assessment in 2005. The external impact assessment as well as the studies on the various components formed the basis of substantial learning and design modifications to improve implementation. The timing of the studies could have been earlier in the project lifespan in order to better inform implementation. The impact evaluation used a quasi -experimental design to strengthen attribution of the results to CCT . It is, however, unclear from the analysis whether the supply -side effects were controlled for because many of the supply-side interventions (e.g., the 2nd Basic Education Project) also targeted the poor. The IFPRI analysis was based on data from 2006, when only 44% of education beneficiaries and 63% of health beneficiaries were enrolled . The results have to be interpreted with these caveats in mind . The evaluation seemed to use enrollment and attendance interchangeably and in the ICR Review it is assumed to refer to enrollment not attendance rates . a. M&E Quality Rating : Substantial 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards . The project had an environmental category C rating, and hence no environmental safeguards were triggered. Fiduciary . According to the ICR overall performance with respect to procurement activities was satisfactory . The project financial statements were audited annually by independent auditors, and all audit reports had unqualified opinions. 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The following lessons complement the lessons offered in the ICR : The implementation of social assistance programs (such as CCT or Local Initiatives ) has to be complemented by a structured communications strategy to ensure dissemination of consistent and factually correct information (about program benefits, eligibility criteria, compliance criteria and appeals mechanism ) and to reduce the risk of capture by powerful elites . An emergency or crisis situation can offer a window of opportunity to address medium goals of institutional strengthening and delivery on social safety nets . Furthermore, the project demonstrated that adjustment and investment funding within the same operation can be successfully combined in support of short and medium term project objectives. Supply-side quality improvements and delivery capacity have to match the demand generated through the incentives created by the CCT program . The health component of the CCT showed some demand - and supply-side mismatches, detracting from the program's impact on health service utilization . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR generally provided a good summary of the project achievements and supporting evidence, but important gaps remain: In the ICR no reference was made to the QER (memorandum dated September 27, 2001) and how the Bank team responded to the QER. ICR guidelines for investment operations require an annex with the outputs by component . Given that components B through D were investment operations, the outputs should have been provided for these components. While the ICR presented evidence against which project efficacy could be rated (including findings of the impact evaluation), the ICR guidelines call for the rating of each project objective, not project components as was the case in Section 3.2. Although the evidence on the efficiency of the CCT program could be found in various parts of the ICR, the ICR guidelines for investment operations require a section on Efficiency, which was absent in the ICR . Insufficient information was provided on implementing agency performance (under Borrower Performance) to support the rating of implementing agency performance (ICR p13). a.Quality of ICR Rating : Unsatisfactory