ICRR 11129 Report Number : ICRR11129 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 08/06/2002 PROJ ID : P055156 Appraisal Actual Project Name : Regional Initiatives Fund Project Costs 6.3 6.4 US$M ) (US$M) Country : Bulgaria Loan/ US$M ) 5 Loan /Credit (US$M) 4.375 Sector (s): Board: SP - Micro- and Cofinancing .6 .45 SME finance (76%), US$M ) (US$M) Central government administration (24%) L/C Number : L4411 Board Approval 99 FY) (FY) Partners involved : UNDP, Japan Closing Date 04/30/2001 04/30/2001 Prepared by : Reviewed by : Group Manager : Group : Ronald S. Parker Soniya Carvalho Alain A. Barbu OEDST 2. Project Objectives and Components a. Objectives This project, designed as a Learning and Innovation Loan (LIL), aimed to test the social fund mechanism as a means of improving the standard of living among the poor and unemployed during economic transition, test the cost and benefits of using micro-projects as a means to transfer income to vulnerable groups, and use temporary employment creation to build infrastructure for development. It was expected that the project experience would justify the creation of a SIF as an autonomous Government agency with a particular social protection mandate. b. Components The project consisted of three components: Micro Projects (75 percent of project costs); Labor Market Analysis (6 percent); and Project Management (19 percent). c. Comments on Project Cost, Financing and Dates The US$5 million loan was made available in Deutsche Marks (DEM 9.09 million). Preparation support was provided by a Japanese PHRD grant (US488,000) and UNDP provided US$400,000. The government contributed $0.3 million as well as BGN 16 million to extend the project. The difference between the appraised and actual loan amounts is due to exchange rate fluctuations during the implementation period. 3. Achievement of Relevant Objectives: Achievement of a key project objective, viz, testing the social fund mechanism, was adversely affected by the lack of a systematic monitoring and evaluation system (see section 5 below). With respect to the microprojects component, the project rehabilitated socially important infrastructure benefiting generally poor and unemployed people, but the temporary and long-term employment created was only 36 percent of the appraisal target. With respect to the labor market analysis component, two rounds of a labor force survey were administered and the data has been made public. Implications of the impact evaluation of active labor market programs are, however, unclear -- policy alternatives and recommendations had not been published at the time of ICR preparation. With respect to the project management component, the project created an administrative unit capable of supervising infrastructure construction but the unit was weak in developing (preparing, appraising, evaluating) innovative (non-infrastructure) projects. It was also weak in evaluating business support for proposed endeavors, and the MTR found that (in nine innovative microprojects studied) it placed too many workers in state-owned enterprises, and failed to justify the provision of direct and indirect subsidies to the private sector. 4. Significant Outcomes/Impacts: The project rehabilitated socially important infrastructure benefiting generally poor and unemployed people. Short-term employment may have boosted the morale of long-term unemployed persons who found work in the labor-intensive microprojects. Adherence to technical designs, schedules, and contract values are reported to have improved capacity in the construction sector. Despite weaknesses in this area, a number of non-infrastructure projects implemented by the RIF managed to build the capacity of local NGOs to implement community-driven projects and design initiatives to assist ethnic-minority communities. The Borrower identified with the RIF to the extent that it mobilized additional funding of its own (budget contribution) and brought in other donors to quadruple the original LIL amount. 5. Significant Shortcomings (including non-compliance with safeguard policies): One of the main project objectives was to test the social fund mechanism as a means of improving the standard of living among the poor and unemployed and also to test the costs and benefits of micro-projects. While a social fund mechanism was put in place and was operational, this mechanism was not tested through a systematic monitoring and evaluation system focussed on measuring how the project was contributing to changes in living standards, targeting, and other relevant outcomes and impacts (The ICR notes weaknesses in the project's social analysis and impact evaluation and suggests that existing efforts would have been more useful with better methodologies, quality control, and repetition). Temporary jobs created by the project did not result in any significant skills upgrading or long-term employment. The goal set for labor content during appraisal was 45 percent of microproject expenditures: the final average (estimated by the PMU) was 24 percent. The project failed to plan for appropriate maintenance -- a finding highlighted by an independent technical evaluation financed by the project. Only a few infrastructure works (for example health centers, which were handed over to an agency with staff and budget; or sanitary infrastructure in areas where sewage fees are charged) are assured of proper quality maintenance and coverage of operational expenses. In instances where fees were collected, however, they were often set so high that the poor could not afford service (connection rates for water and sewerage) because of the cost (including installation) of the indoor facilities. Absolutely necessary equipment was not supplied in areas where its subsequent provision was unlikely: schools were handed over without provision for desks or teaching materials, and cultural centers were opened without heating systems. Sewerage systems were constructed without treatment facilities. The RIF Steering Committee composition and its operation raised questions of bias and political favoritism, and its oversight board was not sufficiently diverse, open, or transparent . Project selection/approval procedures were not disclosed beforehand, and justification for decisions was not given ex post. Project management costs exceeded appraisal estimates by half. 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Highly Satisfactory Moderately To achieve a rating of HS, OED Unsatisfactory methodology requires that a project achieve or exceed all of its objectives efficiently and without major shortcomings. This was clearly not the case with this project despite many positive outcomes. One of the main project objectives was to test the social fund mechanism as a means of improving the standard of living among the poor and unemployed and also to test the costs and benefits of micro-projects. This objective was adversely affected by a weak monitoring and evaluation system which provided insufficient evidence of these impacts. Counter to what one might expect in a project aimed at testing and learning, the problems identified during the MTR remained uncorrected. It appears that the decision to follow -up the LIL with a scaled-up social fund project had been taken at the outset without waiting for the results of the testing . The temporary and long-term employment created was only 36 percent of the appraisal target. The efficiency of the project is also in question. It cost BGN12 to transfer BGN1 through wages. Project management costs exceeded appraisal estimates by half. Institutional Dev .: High Modest To attain a rating of high institutional development impact, a project must make a critical contribution to a country's ability to use human, financial or natural resources effectively. Without a solid monitoring and evaluation system and lacking a learning culture wherein any deficiencies are quickly corrected, the project's institutional development impacts are unlikely to have been high . In addition, the project did not result in any significant skills upgrading of those employed . Over 41 percent of beneficiaries surveyed said that the project had no impact on the development of their community. Institutional objectives were modest in scope and focused on raising the PMU to the point where it could carry out the full range of its duties satisfactorily : The ICR states that by the time the follow -on was approved, the PMU was still unable to design innovative projects . Sustainability : Highly Likely Non-evaluable The information provided in the ICR does not provide evidence for a "Highly Likely" rating. Based on available information it is difficult to judge whether the learning or other benefits that materialized from the project are sustainable. Concerning project infrastructure, O&M was not assured. The SAR stipulated that there should be an O&M plan for each microproject that includes institutional arrangements, equipment, and resource commitment. Furthermore it notes that materials required for proper facility functioning should be fully funded prior to grant approval. These things did not happen. Bank Performance : Highly Satisfactory Satisfactory Only marginally satisfactory. Appraisal estimates proved to be unrealistic, and problems identified during the MTR remained uncorrected. The monitoring and evaluation aspects of the project were too weak for a project that was testing an approach for potential replication . Borrower Perf .: Highly Satisfactory Satisfactory Only marginally satisfactory. The shortcomings of this project were at least partly the result of Borrower actions during preparation and implementation . The SIF oversight board was not sufficiently diverse, open, or transparent, and the RIF Steering Committee composition and its operation raised questions of bias and political favoritism . Quality of ICR : Satisfactory NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. NOTE: 7. Lessons of Broad Applicability: A number of lessons were reported in the ICR but, in the absence of systematic evaluative data, the robustness of these lessons is unclear. Two lessons of broad applicability based on the information in the ICR are: 1. Projects such as LILs where the primary purpose is to pilot test require particularly strong and systematic monitoring and evaluation. 2. When the main objective is large-scale employment generation, the experience of this project supports evidence from other sources that social funds are unlikely to be the best mechanism to achieve this. 8. Assessment Recommended? Yes No Why? To verify ratings and lessons of experience . 9. Comments on Quality of ICR: Marginally satisfactory. The ICR ratings are inconsistent with the analysis and evidence presented in the ICR .