ICRR 10231 Report Number : ICRR10231 ICR Review Operations Evaluation Department 1. Project Data : OEDID : OEDID: C2593 Project ID : P010456 Project Name : Social Action Program project (SAPP) Country : Pakistan Sector : Other Education L/C Number : C2593 Partners involved : Asian Development Bank, Netherlands, DfiD Prepared by : David Greene, OEDST Reviewed by : Linda Ankrah-Dove, OEDST Group Manager : Roger Slade Date Posted : 11/20/1998 2. Project Objectives, Financing, Costs and Components : The project was a sector investment credit for US$ 200 million equivalent, 5 percent of the US$ 4 billion planned outlay for the first three years of Pakistan's five -year Social Action Program (SAP). SAPP sought to improve social services in all four provinces --basic education, primary health care, rural water supply and sanitation and the national population program. SAPP provided the policy umbrella for building and institutionalizing political commitment to social programs and related Bank -assisted projects already underway but faltering, increasing public expenditure levels on social services, and expanding community and NGO involvement . It was to reimburse the federal government up to 10 percent of approved budgetary expenditures for 27 subprojects and operational plans prepared by each main federal unit and provincial government . ADB provided US$ 100 million, the Netherlands US$ 28 million and the UK US$ 17 million. The project earmarked US$ 10 million to improve community involvement and US$ 6 million for monitoring and evaluation (M&E). SAPP focused on cross-cutting issues: improving implementation through better planning, budgeting and personnel policies; improving program design with a focus on the quality of and access to basic services, especially for women and girls, and increasing non -salary outlays on items such as textbooks and medical supplies; increasing the level of effort by raising government spending on social services (from 1.7 to 2.4 percent of GDP in 1996/97). The program included commitments for reforms for the first year (1993-94). The major elements of the SAP framework included decentralization (provincial level); annual performance-based agreements with the government for operational plans and measurable targets; mechanisms for dialogue, coordination and monitoring to reach front -line ministries and the highest levels of government; and a Participatory Development Program . 3. Achievement of Relevant Objectives : Social indicators are reported to have improved significantly during the project period . Rural primary school enrollments rose significantly and girls' enrollment rose even more rapidly --from 2.8 million in 1991 to 3.6 million in 1996. Child immunizations increased and the gender gap disappeared, though use of government clinics was still low. The population program was strengthened and contraceptive prevalence, though low, increased markedly from 14 percent to 22 per cent Infant mortality declined from 101 to 86 per 1,000 live births and female life expectancy increased. The coverage of rural water supply and sanitation systems expanded and the incidence of diarrhea declined. Improved personnel recruitment policies were introduced . Although budgetary allocations were below target each year, SAP expenditures increased relative to GDP, when, because of budgetary stringency, most other categories declined. 4. Significant Achievements : The project was a major innovation in a difficult context, responding to Pakistan's need to overcome chronic problems in delivering social services to the poor and to establish a robust and durable policy, budgetary and management framework. According to focus group reviews, the project was a good first step, focusing attention on the sector and raising expenditures . 5. Significant Shortcomings : As the ICR attests, SAPP had flaws in design and in execution . The design was exceptionally convoluted, overlaying the already complex sectoral and provincial programs and ongoing Bank operations with agreements between central and provincial governments and donors about policies and expenditures, and procedures for procurement, disbursement, monitoring and accounting . During implementation, individual and joint donor interventions and oversight proved excessive and weakly coordinated, creating the impression that SAP was a donor-driven initiative and compromising local ownership . Monitoring was unsystematic, focusing on budget reviews and not the quality of programs . The Bank's inputs were costly, absorbing 728 staff weeks (US$ 1.5 million) of Bank staff time. Questions remain about how far the the government adequately incorporated the reforms in the operational plans at federal and provincial levels, how far the plans were in fact executed and whether the Bank withheld disbursements appropriately when the plans were of poor quality or were not implemented --even though disbursements were supposed to be conditional on these factors . As the ICR states, the reforms worked against traditional approaches and political interests . For example, the implementation of new personnel policies was flawed, allowing teacher absenteeism, frequent staff transfers and political patronage, while the increase in non -salary expenditures went to items other than textbooks and similar "quality" inputs. According to the borrower's report, the program for monitoring the inputs, processes and outputs of line agencies were ill -defined and a program for monitoring the broader SAP program was not put in place . Evaluation programs proved weak in design and implementation. As a result, as the ICR states, it was unable to establish a link between the project and the improved social indicators reported . The borrower's report also concluded that the SAPP resulted in overcentralization of authority and responsibility in planning and financial administration, leading to bad planning, waste of resources, lack of ownership and political interference . It criticized the absence of effective institutional arrangements to prevent problems implementing personnel policies and the failure to involve communities in any meaningful manner. 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Satisfactory Marginally Satisfactory Expenditures, policy reforms and social service indicators all improved. However, the absence of evidence linking these results to the project mean that the operation cannot be considered fully satisfactory rating. Institutional Dev .: Partial Modest Political will at central level was substantial but the institutional environment, reform efforts, and plan execution were weak at other levels . Sustainability : Likely Uncertain Dependency on external financing and renewed economic and political instability render the durability of reforms and services uncertain. Bank Performance : Satisfactory Satisfactory In a grave situation, the Bank was innovative in responding to systemic problems affecting the poor. However, its performance is assessed as only barely satisfactory because many of the deficiencies in the project's design --in the management of risks and partnerships, in monitoring, supervision of provincial plans and in evaluation--could have been better managed. Borrower Perf .: Satisfactory Satisfactory The federal government's courage and far-sightedness in addressing social sector issues was commendable . However, the borrower's own report criticizes the proviincial authorities for inefficiencies and ineffectiveness in their delivery of social services and detracts from the overall favorable performance . Quality of ICR : Unsatisfactory 7. Lessons of Broad Applicability : The ICR identifed four pertinent lessons . The sector investment approach proved effective in establishing policy dialogue and influencing expenditure levels, but time and perserverance are required to establish innovations in the targeting of funds and the reform of service delivery . Improving governance and financial management are important prerequisites for success, including defining rules of behavior and monitoring compliance . To win and sustain ownership, early involvement of provincial authorities and stability of key staff are essential . Supervision of sector investment credits covering all social sectors is demanding and resource-intensive but pays off in terms of dialogue with government agencies at all levels and responsiveness to difficult issues arising ---as long as, as the ICR emphasises, supervision staff have policy-based lending experience and broad -based sector skills. That the SAPP approach may inadvertently promote over -centralization is an important observation in the borrower's report and deserves further examination . It suggests that, before launching complex and untested initiatives, more modest approaches should be tried . 8. Audit Recommended? Yes No Why? The project was an important programmatic innovation involving a key borrower and several donors and a follow-up operation has recently been approved . Information and conclusions in the ICR should be confirmed since the borrower reports significant differences of view about key elements such as institutional reform, decentralization of administrative and financial powers, siting of facilitiies, community participation and M&E, and better evidence linking the project to sectoral performance is needed . 9. Comments on Quality of ICR : The ICR is not entirely convincing . It contains unexplained inconsistencies, in particular with the borrower's report . It omits the aide-memoire of the ICR mission and the views of the other donors . Key issues are not squarely addressed, such as the quality and implementation of the provincial plans and the link between the SAPP and related investment projects. The absence of evidence linking the SAPP with the improved social indicators is also not satisfactorily explained .