ICRR 13277 Report Number : ICRR13277 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 04/09/2010 PROJ ID : P084596 Appraisal Actual Project Name : Employment Support US$M ): Project Costs (US$M): 14.00 20.03 2 Project Country : Bosnia and Loan /Credit (US$M): Loan/ US$M ): 12.00 12.84 Herzegovina Sector Board : SP US$M ): Cofinancing (US$M): Sector (s): Other social services (85%) Central government administration (10%) Sub-national government administration (5%) Theme (s): Improving labor markets (40% - P) Social risk mitigation (20% - S) Social safety nets (20% - S) Education for the knowledge economy (20% - S) L/C Number : C3927 Board Approval Date : 06/15/2004 Partners involved : Closing Date : 09/30/2008 03/31/2009 Evaluator : Panel Reviewer : Group Manager : Group : Alemayehu A. Ambel Roy Gilbert IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: The PAD states two project development objectives : 1. Provision of employment services for an estimated 10,000 people who are either recently retrenched and above 45 years of age and/or those who are actively seeking jobs for two or more years in Republika Srpska and Federation of Bosnia and Herzegovina; and 2. Further the reform agenda to bring behavioral change and business change in the employment services by focusing the Public Employment Services (PES) on providing effective job brokerage services; employment assistance programs including counselling services; programs to assist the unemployed; and analytical and information services including disseminating labor market information . The wording is slightly different in the Development Credit Agreement (DCA): 1. To support the Borrower's efforts to provide employment related services for approximately ten thousand (10,000) individuals; and 2. To further the reform of the Borrower's employment related services . This ICR Review evaluates project performance against the objectives as stated in the PAD . They are more explicit and specific about the target population and the expected outputs and /or outcomes. As has been common practice in Bosnia and Herzegovina (BiH), project responsibility was split between a separate implementing agency in Republika Srpska and another in the Federation of BiH, the two Entities into which BiH is split. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): ALMP ) and Job Brokerage Services . Expected Cost at Appraisal US$10.50 (i). Active Labor Market Programs (ALMP) million; Actual Cost US$16.99 million. The component was based on the experience of the FY 02 Bosnia and Herzegovina Pilot Emergency Labor Redeployment Project (Cr3385) and included: counseling services; on the job training; self employment in agriculture, agribusiness, small businesses; job fair and public works; business advisory services; and skills enhancement training . For those without work for more than 6 months it provided job matching and income earning opportunities for active job seekers and to the hard -core unemployed. (ii). ii). Institution Building and Technical Assistance to the Public Employment Services PES ). Expected Cost at (PES). Appraisal US$1.65 million; Actual Cost US$0.78 million. This included development of procedures, guidelines and regulations; staff training; piloting analytical labor market information systems; and establishment of labor market research units within the Ministries of Labor (of each Entity). iii). Project Management . Expected Cost at Appraisal US$1.85 million; Actual Cost US$2.26 million. This provided (iii). equipment, operating cost support and technical assistance for the project's implementing agencies . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The 43 percent cost overrun (in US dollars) was the result of local exchange rate appreciation and extending the coverage of the services provided . The additional costs were fully met by the Borrower . The six-month extension to the closing date was necessary because Credit Effectiveness was delayed for a similar period because of protracted procedures for local ratification of the project . The exact appraisal cost estimate was US$ 14.0 million. The US$14.025 reported on page 29 of the PAD was incorrect, the result of incorrect summation . 3. Relevance of Objectives & Design: Project objectives: Substantially relevant . BiH still has high levels of unemployment, low levels of formal employment, and low labor force participation rate . The aftermath of the 1992-95 war and the transition to market-oriented economy affected labor market outcomes, aggravating unemployment in the short -run. Policies to promote employment and improve the working of labor markets have been top priorities for Entity Governments in BiH and for the Bank. For instance, policies to promote private sector led growth through new corporate and personal income tax laws were enacted to improve the demand for labor . Also, labor market-related objectives were supported by the recent (2007) and earlier Country Partnership Strategies . However, the first development objective that focuses on the provision of employment services to 10,000 people is output-oriented, rather than outcome-oriented. The policy reform objective was more relevant to BiH priorities for labor market development and the hoped-for long term development impact. Project design : Modestly relevant . The project had two key components : (i) employment service provision; and (ii) policy reform, each of which, in principle, could be instrumental in achieving the respective project objectives . But the employment services component could have been more clearly articulated so that the choice of the number of beneficiaries, the rationale for the temporary support and the exit strategy (at project completion) could be better understood. Why the number 10,000 was targeted and how they were chosen from among the most vulnerable 99,119 job seekers (ICR p. 6) is not made clear in the ICR, raising further questions about its relevance .The component could also have differentiated more among potential beneficiaries . Although transition-induced retrenchment and war disruption and displacement both have negative employment effects, they have different impacts depending on the age group . 4. Achievement of Objectives (Efficacy): Overall Efficacy is rated Modest . IEG's review gives more weight to the second objective that was intended to bring about systemic changes whilst the first objective temporarily provided employment services that ceased upon completion of the project. 1. Provision of employment services . Substantially achieved . Although the PDO was formulated mostly in terms of output (employment services),10,836 active job seekers were placed in paid jobs and 70 percent of them were still in their jobs one year after they were initially placed. Counseling and other employment services were provided to 10,669 persons (target 10,000) in the target group of 40 years of age and above. The services included on-the-job training (7,263 beneficiaries), self employment (2,756 beneficiaries), public works (181 beneficiaries), job counselling (166 beneficiaries), and skills enhancement (103 beneficiaries). In addition, the project provided employment services to 167 disabled persons. 2. Furthering the reform agenda to bring behavioral change and business change in the employment services . Modestly achieved . The ICR reports the achievement of only one intermediate outcome indicator that focused on the adoption of job brokerage and Active Labor Market Programs (ALMPs) by employment services. On others, the ICR reports (p. iv-v) either they were only partially achieved (2 indicators), not achieved (1 indicator) or no information was available (1 indicator). Nevertheless, the project did launch a publicity campaign providing information on issues related to employment/ unemployment and job search as well as the work and functions of employment offices . It also helped initiate some standardization of services to eventually comply with ISO 9001 and 2000 both at cantonal and federal (Federation of BiH) public employment services. Legislation to encourage active involvement of the private sector in providing employment services has been adopted in Republika Sprska, but not in the Federation . 5. Efficiency (not applicable to DPLs): For the first objective, the ICR compares the cost effectiveness of the project with that of similar services by the PES. For example, the cost per beneficiary placement through the project was about half of the cost of the same job placement service by the PES (KM 2,810 Vs. KM 5,343). For the second objective, there is no discussion in the ICR of the efficiency of achieving reform of the employment services . On this basis IEG rates overall Efficiency as Modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of objectives is substantial but design relevance is rated modest . Because of the limited development impacts beyond the temporary provision of employment services, there were significant shortcomings in this operation's overall achievements . Overall, there is no convincing evidence that the core development aim to bring behavioral and business change was substantially or efficiently achieved . Efficacy is thus rated modest as is efficiency. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The priorities supported by the project are likely to remain government priorities because, according to the ICR, some of the key benefits of the project have already been mainstreamed into legislation, especially in terms of private sector involvement in the provision of employment services (p. 17). a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at entry was moderately unsatisfactory . The PAD incorporated essential elements . These, for example, include the results framework with outcome indicators (p. 18-19), eligibility criteria for employment services (p. 24-25), and financial management and disbursement arrangements including financial risks (p. 35-43). The project also benefited from the experience acquired from the implementation of related projects (for example, the Pilot Emergency Labor Redeployment Project ). However, the ICR noted several problems and the 2005 QAG reviews rated quality at entry as moderately unsatisfactory . Specifically, the ICR reports that the lessons were not adequately analyzed and the project did not include alternative approaches to providing employment services to different groups (p. 9). There are other shortcomings related to the design . For instance, some of the project outputs/outcomes were targeted without baseline information . In addition, only one of the two project objectives was outcome -oriented. Quality of supervision was moderately satisfactory . Bank supervision included regular procurement and financial management reviews. In the meeting between IEG and the TTL, the TTL indicated that the mid-term-review was carried out as scheduled and helped the Bank to adjust the target group . The Bank also continued a policy dialogue on labor market issues with main counterparts . In addition, the ICR notes that there were several other intermediate outcome indicators in the PAD that were not included and /or monitored in the ISRs (p.11). Given the insufficient focus upon development outcomes in the project design, supervision could have been more proactive in redressing these shortcomings . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government performance was moderately satisfactory . The Government's ownership and commitment to the project were reflected in its provision of additional funding to the project and also in its mainstreaming intended project results into the legislation . However, the ICR reports that an intermediate outcome indicator on the transfer of different social insurance benefits to appropriate agencies could not be achieved for political reasons; this kept unemployment higher than it need have been because health benefits were not delinked from unemployment benefits. Implementing agency performance was moderately satisfactory . According to the ICR, the implementing agencies in both Entities were staffed with experienced professionals, and staff turnover was low . These implementing agencies produced progress reports regularly, but they did not include data on all the chosen performance indicators. They managed procurement and financial matters in accordance with agreed procedures. Although the reporting was not always complete, the implementing agencies were able to produce enough data to monitor progress in implementing the project, if not to measure project outcomes . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design was negligible . The project did not have adequate monitoring and evaluation indicators . Many project output/outcome indicators lacked baseline values and clear targets . Implementation was modest . Implementing agencies reported project progress as per the original results framework. However, as noted earlier, the ISRs did not follow the original indicators . For example, a discrepancy was observed by the ICR between the second indicator in the PAD (on effective employment service measures for PES) and the indicator actually monitored during supervision . It was also observed that several intermediate indicators included in the PAD were not monitored or included in the ISRs (ICR p. 11). Utilization was modest . The M&E system could only be used sparingly, given the limited relevance of the performance indicators as originally conceived . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): According to the ICR, no safeguard policy was triggered by this project . There were no major environmental issues encountered during implementation . Procurement and financial management activities were also carried out in accordance with agreed procedures . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The achievement of the policy reform Satisfactory Unsatisfactory objective of the project was modest and most of the intermediate outcomes were not achieved. In addition, efficiency was modest. Overall, there is no convincing evidence that the core development aim to "bring behavioral and business change" was substantially or efficiently achieved . Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Project design only partly focused upon Satisfactory Unsatisfactory outcomes and the development impact . There were too few indicators to measure outcomes and little attention to rectifying this problem during supervision. Borrower Performance : Satisfactory Moderately For political reasons the government Satisfactory did not delink health insurance benefits from unemployment benefits and the numbers of persons registered as unemployed remained high. The implementing agency gave only partial attention to monitoring performance indicators. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Of the 13 provided in the Bank and Borrower ICRs, IEG would highlight the following : Employment projects need to be flexible in their choice of target groups and approaches to employment services (for example, brokerage and job search ) in order to be able to take advantage of the changing labor market conditions. Assistance and advice in brokerage and job matching as well as demand driven skills training are more effective than providing a grant . Public private partnerships for employment services, especially engaging private employment agencies in job-brokerage services, can be cost effective . IEG would add: While attending to immediate needs of social protection, Bank support should always remain focused upon reform and behavioral change that can have medium and long -term development impacts after the provision of temporary assistance has ceased . 14. Assessment Recommended? Yes No Why? In order to find out more about the achievements of this project regarding the extent and sustainability of reforms in employment services. 15. Comments on Quality of ICR: The ICR quality is rated satisfactory . The ICR clearly pointed out design shortcomings and supervision problems (pp. 9, 10, 11. & 18). It has also useful information to justify most of the ratings although they were overstated in most cases. However, the ICR did not assess the project along the results chain and could have been more forthcoming about the data limitations preventing a fuller assessment . It also did not clearly differentiate outputs and outcomes, a task made difficult by the shortcomings in the design of the project . a.Quality of ICR Rating : Satisfactory