ICRR 13917 Report Number : ICRR13917 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/31/2012 Country : Lebanon; Lebanon Project ID : P106489 Appraisal Actual Project Name : Lebanon Emergency US$M ): Project Costs (US$M): 1 0.96 Social Protection Implementation Support Grant; Lebanon Emergency Social Protection Implementation Support Grant L/C Number : Loan /Credit (US$M): Loan/ US$M ): 1 0.96 Sector Board : ; Social Protection US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 08/30/2007 Closing Date : 04/30/2009 04/30/2009 Sector (s): Other social services (90%); Health (10%) Theme (s): Social safety nets (67% - P); Health system performance (33% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Hjalte S. A. Sederlof Soniya Carvalho Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (page 6), the Project Development Objective was “to accelerate and improve the quality of the implementation of the package of social sector reforms presented by the Government of Lebanon (GOL) at the Paris III donor conference in the areas of social insurance, safety nets, and health expenditures.� The Trust Fund Agreement (paragraph 1 of the Annex to the Trust Fund Grant letter ) uses the same Project Development Objective, specifying “the Lebanese Republic� and “the International Conference for Support to Lebanon (Paris III), which took place on January 25, 2007�. The Review rates the project against the Project Development Objective in the Trust Fund Agreement . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The original components of the project were : Component 1: Modernization of the Social Insurance System (US$ 0.7 million; actual US$ 0.7). This component aimed at improving the control of expenditures and revenues, and the quality of services, in the National Social Security Fund; and ensuring financial sustainability of national Social Security Fund . Activities included: 1. Provision of technical assistance for the design and implementation of an electronic claims processing and utilization control system; 2. Provision of technical assistance for the design and implementation of a registry of plan members and collection of contributions system; 3. Provision of technical assistance to identify and implement measures to secure the financial sustainability of the health insurance branch including the refinancing of accounts payable and accounts receivable; and 4. Capacity building, including training, and project implementation and coordination support at the National Social Security Fund. Component 2: Strengthening of Social Safety Nets (US$ 0.2 million; actual US$ 0.2). This component aimed at improving the poverty focus of safety net programs in the Ministry of Social Affairs . Activities included: 1. Development of a “proxy-means testing� targeting formula; 2. Development of procedures for a new targeting mechanism and preparation of an operational manual that will detail the mechanism for the distribution of benefits to the poor and vulnerable; 3. Development of an information management system to support implementation of the targeting mechanism, to include provision of equipment and software; and 4. Provision of technical assistance to Ministry of Social Affairs to carry out testing of the targeting mechanism and method of benefit distribution. Component 3: Health Sector Cost Rationalization (US$ 0.1 million; actual US$ 0.1). This component was aimed at controlling health expenditures by the Ministry of Public Health . Activities included: 1. Provision of technical assistance to introduce an utilization review function at the Ministry of Public Health; 2. Provision of assistance in developing 30 inpatient admission criteria within the Ministry of Public Health; and 3. Provision of technical support to prepare a plan to realign Ministry of Public Health coverage towards more cost-effective health services and hospital facilities . During implementation, it was learned that the National Social Security Fund had already outsourced the development of the registration and contribution collections system (Component 1.2) to a firm outside the Project. This activity was replaced in the project by a study of the implicit liabilities of the National Social Security Fund arising from an end-of-service indemnities awarded to retiring private sector workers in the absence of a pension scheme for private workers. While not selected for the original project, the study still contributed to overall stabilization of National Social Security Fund finances and in line with the objectives of the component . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost. Total project costs were estimated at US$ 1 million. Actual spending totaled US$ 957,923. Financing. Financing was secured as emergency financing from the Trust Fund for Lebanon, which is to be drawn on for assistance to sectors that are key to rehabilitating the economy . Borrower contribution. The project was fully financed by the Trust Fund . Dates. The project was approved on September 7, 2007, and closed on April 30, 2009. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High. A Country Assistance Strategy was prepared in 2005, but some of its assumptions and projections had High become untenable as a result of armed conflict, and an Interim Strategy Note was produced in 2006 to support the social reform plan presented in Paris . The plan (and the Interim Strategy Note) included medium term measures to (i) prepare a policy framework for social insurance and coverage expansion; (ii) improve governance and administration in the National Social Security Fund; (iii) guarantee the medium and long term sustainability of the National Social Security Fund; (iv) strengthen income protection programs (pensions and family allowances ); (vi) put in place a social assistance system that is able to target safety net programs to the most vulnerable population groups; and (vi) control costs in the Ministry of Public Health . The project included key building blocks in the three selected areas (components) – improving the control of expenditures and revenues in the National Social Security Fund; improving the poverty focus of safety net programs; and controlling health expenditures . (See also Section 4.) As part of a medium term program, the measures introduced by the project remain relevant . b. Relevance of Design: Modest . The project was essentially a technical assistance project to develop, and in some instances help implement, instruments that gradually would strengthen performance in key areas of the social sector - pension and health insurance, safety nets and health service delivery . The results framework is a straightforward one, but does not provide a link between investments and outcomes; instead, it measures outputs - the putting into place of systems rather than providing some means of measuring how effective the systems are . 4. Achievement of Objectives (Efficacy): To accelerate and improve the quality of the implementation of the package of social sector reforms in the areas of social insurance, safety nets, and health expenditures : Modest . In the absence of evidence on effectiveness or outcomes, efficacy is rated modest . The implementation of project activities was mixed, as discussed below . Improving the control of expenditures and revenues, quality of services, and financial sustainability of the National Social Security Fund . (i) An automated claims processing and utilization control system had been developed and tested, but had not yet become operational at the time of the ICR; (ii) limited capacity building in health policy modeling has been created, but related actuarial capacity has not been institutionalized; (iii) an end-of-service indemnity valuation has been undertaken, but its value is limited by data constraints; (iv) arrears have been assessed and reduced, but are behind schedule; and (v) the registry and collections element that was dropped from the project during implementation (see 2c above). Improving the poverty focus of existing or new safety net programs . The aim was to provide a better targeting system for subsidies and income support programs for the poor . The project has developed a proxy -means test, including related operating procedures and management information system; and the new system has been piloted successfully, with the exception of the payment process, where concerns about the payment channel caused delays in benefit delivery. The project team noted that the issue of payment channels was quickly resolved, and the cash benefit has now been scaled up to the national level . Controlling health expenditures by the Ministry of Health . Of five results indicators, one – on a cost effectiveness plan - has been met, and two, on a utilization review function and on admissions criteria, were in progress at project closing . Two outputs that depend on applying the utilization review with consequent adjustments to admissions criteria - an IT module to track admissions, and training of physicians in the new criteria – remain in abeyance. 5. Efficiency: No conventional quantitative measures of efficiency were estimated for the project , nor were relevant comparisons drawn to efficiency indicators for other, similar, projects . Some 80 percent of the funds were spent on activities that were not fully implemented. Efficiency is rated as modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The ratings for relevance of objectives and design are high and modest; the rating for the achievement of the objective is modest. Efficiency is rated modest. While the project introduced important instruments in key social areas, only the safety net mechanism was fully implemented . The moderately unsatisfactory Outcome rating reflects that results were only partial for the implementation of activities that were supposed to lead to meeting project objectives, as noted in Section 4. In modernizing the social insurance system, only one of four activities (design and implementation of electronic claims processing and utilization ) has been completed and is being acted on in an apparently sustained way; key measures – establishment of a registry and a functioning collection system – were dropped altogether. Likewise, only one out of three health sector rationalization measures was completed . In contrast, the safety net objective has been completed and is being introduced, initially on a pilot basis . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to the development outcome is rated as high . While the political situation may have stabilized, consensus and decision-making at the political level remains difficult . The project was designed and implemented under relatively volatile political circumstances, and it introduced mechanisms that appear not to have had the full support of all stakeholders, including in particular the increased transparency that implementation of some of the instruments would imply. Security risks subsequently may have declined, but the ICR points to continued absence of commitment, including in the National Social Security Fund, the major recipient of support under the project . Moreover, of the instruments that had been developed by the ICR, only the safety net component has been fully implemented, or used as a management tool . This may partly underline the environment of conflict and the absence of priority for the project among decision -makers. a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: a. Quality at entry: Quality at entry is rated moderately satisfactory . The project was developed and implemented under challenging circumstances – a post-conflict environment and political turmoil . And it appears to have been settled on quickly and given a brief implementation period . Admittedly, the ICR indicates that this was sufficient, but it should be kept in mind that at the time the environment was uncertain and skills levels among stakeholders modest and risks high. Moreover, many of the activities were not completed on time - at ICR Review elements had mostly been designed, but had only partly been integrated into the business practices of the participating agencies. The Bank did assess stakeholder capacity and found it wanting . This led to Bank execution of the project, which also ensured that fiduciary and reputational risks were mitigated . Other design elements also recognized the limited capacity among beneficiary agencies, their lack of exposure to the Bank, and occasional reluctance to move ahead. Direct liaison between the Bank team and National Social Security Fund, the main beneficiary agency, was created, and implementation coordinators were appointed for the agencies, and an implementation support team of national and international advisors was set up to work closely with participating agency managers and staff . The results framework was pragmatic and designed to draw clear links between activities and outputs; however, it included no outcome indicators . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: Quality of supervision is rated moderately satisfactory . As a Bank-executed project, implementation required significant inputs by Bank staff . The intensity of supervision was high : some US$ 380,000 were spent from the BB to supervise the project; and significant progress was made on a complex set of components . It seems unlikely that this would have been achieved with less Bank focus . Still, the outputs under the project remain largely incomplete. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Moderately satisfactory The Government, as represented by the Ministry of Finance, appears to have played a constructive role in moving the project forward, displaying both ownership and commitment to its success . The Ministry of Finance offered a sustained effort to deal with institutional roadblocks, in most instances successfully . That said, it is not obvious that there was across -the-board consensus on project elements, reflected in the difficulties that the cash benefit system is facing in getting the payment system approved; and in delays in following the schedule for drawing down arrears . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: Moderately unsatisfactory . Performance of the Implementing Agencies – the National Social Security Fund, the Ministry of Social Affairs and the Ministry of Public Health – appears to have been mostly supportive at the level of the organization, albeit hampered by capacity constraints and, in the case of the National Social Security Fund, by its Board: a concern appears to have been the realization that introduction of new management tools could significantly raise transparency in transactions . While capacity constraints may have been important in all three organizations, they appear to have been particularly severe in Ministry of Public Health, where progress on all the main activities appears to have been slow . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The results framework was basic, and monitoring indicators consisted of recording completed actions and outputs . No baselines were set (the project consisted of instruments to be developed ). While there were no milestones, time schedules for implementation of project components were developed in the Project Implementation Schedule . These allowed progress on institutional processes to be monitored during implementation . No effectiveness or outcome measures were developed. b. M&E Implementation: Implementation (i.e. progress on the putting into place of the instruments that were being developed under the project) was monitored on a continuous basis by the Bank team and implementation agencies, with the time schedules serving as guides to project progress . No effectiveness or outcome measures were added as part of the Bank's implementation support. c. M&E Utilization: The M&E system for the project was designed to monitor progress in building instruments . It was project and instrument-specific, taking strategic directions as appropriate . It allowed monitoring (and adjusting) speed of implementation; and it monitored spending; it would have allowed for re -allocation of resources between activities, if necessary. This proved not to be the case . All stakeholders were engaged in the process . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The Project was rated as environmental category C and no safeguard policies were triggered . b. Fiduciary Compliance: The designation of the project as Bank -executed helped ensure that appropriate fiduciary practices were followed . Financial management and procurement were conducted by the Bank . c. Unintended Impacts (positive or negative): N/A d. Other: 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately In modernizing the social insurance Satisfactory Unsatisfactory system, only one of four activities (design and implementation of electronic claims processing and utilization) has been completed and is being acted on in an apparently sustained way; key measures – establishment of a registry and a functioning collection system – were dropped altogether. Likewise, only one out of three health sector rationalization measures was completed. In contrast, the safety net objective has been completed and is being introduced, initially on a pilot basis. Risk to Development High High Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following main lessons were drawn from the ICR : Governance issues need to be assessed as potentially significant risk factors . Opposition by the National Social Security Fund board made implementation of the Project difficult, and is now threatening results under a follow-up project. While projects may be providing important, and even desirable, services, outcomes cannot be achieved, or can be achieved only partly and not necessarily satisfactorily, unless potential governance issues are recognized, and the risks to project outcomes determined and appropriately addressed . The complexity of the political environment for decision making should not be underestimated, especially in a post -conflict environment . Simply because a piece of policy reform is in the national interest, does not mean that it will be implemented. In Lebanon, the confessional system of governance (i.e. based on religious affiliation) and the resulting diffusion of political authority make reforms subject to collective action, and such collective action is not easy, especially in an environment that has gone through significant upheaval . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provides a good discussion of the project, its implementation and outputs, albeit treating them as outcomes . The report includes extensive description that leads reasonably well into the analytical parts . a.Quality of ICR Rating : Satisfactory