41329 Focus on Results: The IDA14 Results Measurement System and Directions for IDA15 International Development Association Operations Policy and Country Services (OPCS) October 2007 ABBREVIATIONS ACRONYMS AND AAA Analytical and Advisory Activities AAP Africa Action Plan ADP Accelerated Data Program AFR Africa Region BP Bank Procedures (statement) CAS Country Assistance Strategy CASCR Country Assistance Strategy Completion Report CASPR Country Assistance Strategy Progress Report CPIA Country Policy and Institutional Assessment DAC Development Assistance Committee DEC Development Economics DECDG Development Economics Data Group DFID Department for International Development (UK) DPO Development policy operation ECA Europe and Central Asia Region FY Fiscal year GDP Gross domestic product GNI Gross national income HD Human Development Network HIVIAIDS Human immunodeficiency virus1acquired immune deficiency syndrome HIPC Heavily indebted poor country IBRD International Bank for Reconstruction and Development ICR Implementation Completion Report IDA International Development Association IEG Independent Evaluation Group IMF Intemational Monetary Fund ISR Implementation Status and Results Report MAPS Marrakech Action Plan for Statistics M&E Monitoring and evaluation MDG Millennium Development Goal NSDS National Strategy for the Development of Statistics OBD Output-based disbursement OPCS Operations Policy and Country Services OP Operational Policy (statement) OECD Organization for Economic Co-operation and Development PAD Project Appraisal Document PARIS21 Partnership in Statistics for Development in the 21st Century PDO Project development objective PEFA Public Expenditure and Financial Accountability PFM Public financial management PREM Poverty Reduction and Economic Management Network PRSP Poverty Reduction Strategy Paper PRS Poverty reduction strategy QAG Quality Assurance Group QEA Quality at Entry Assessment QSA Quality of Supervision Assessment RBCAS Results-based Country Assistance Strategy RMS Results Measurement System SAR South Asia Region SDV Sustainable Development Network SILs Sector Investment Loans SSA Sub-Saharan Africa STATCAP World Bank Statistical Capacity Financing Program WDI World Development Indicators FOCUSON RESULTS:THEIDA14 RESULTSMEASUREMENTSYSTEM AND DIRECTIONS IDA15 FOR Executive Summary........................................................................................................................ i I Introduction.............................................................................................................................1 I1 Progress in Tier 1 Indicators ................................................................................................... 2 I11 Progress in Tier 2 Indicators ............................................................................................... 10 ... A. Country-Level Indicator ............................................................................................ 1 1 . . B. Project-Level Monitoring............................................................................................ 14 IV.Lessons from Experience with Results Management ........................................................ 21 A. Reinforcing a Results Culture................. ................................................................21 B. Improving National Statistics in IDA Countries.......................................................... 24 V Conclusions and Proposals for the IDA15 RMS.................................................................29 . Annexes Annex A. IDA Recent Progress Exceeds MDG Benchmark Rate of Change..............................33 Annex B. Variation in Progress among IDA Countries ...............................................................34 Annex C. Data Availability for IDA RMS Tier 1 Indicators ...................................................... 35 Annex D. IDA Tier 2 Indicators: Methodology ........................................................................... 36 Boxes Box 1:Retrofitting the Indonesia CASPR.................................................................................... 1 2 Box 2: Good Practice in CASCRs................................................................................................. 14 Box 3: Understanding the Institutions for Monitoring Poverty Reduction Strategies ..................18 Box 4: Bangladesh Third Programmatic Sector Development Support Credit:............................19 Box 5: Examples of retrofitting projects and country portfolios...................................................20 Box 6: Impact of IDA-Supported Nutrition Program in Africa ....................................................24 Box 7: Support to Government M&E in Vietnam.........................................................................25 Box 8: Proposed changes for IDA15 to Tier 2 .............................................................................. 32 Figures Figure 1:Annual growth rates in selected groups of countries .......................................................4 Figure 2: Growth and poverty reduction in IDA countries, 1990-2006..........................................4 Figure 3: Budgetary and financial management in developing countries ...................................... 5 Figure 4: Time required registering a new business........................................................................6 Figure 5: Cost to register a new business ....................................................................................... 6 Figure 6: Access to electricity in IDA countries (in %) .................................................................. 7 Figure 7: Telephone access in developing countries....................................................................... 7 Figure 8: Child mortality rates in developing countries..................................................................8 Figure 9: Maternal care in developing countries............................................................................. 8 Figure 10: Primary completion rates in IDA countries ................................................................... 8 Figure 11: Gender equality in education in developing countries...................................................8 Tables Table 1:Progress in Tier 1 Outcome Indicators (updated in June 2007) ........................................3 Table 2: Implementation of RBCASs............................................................................................10 Table 3: Tier Two: Progress in Monitoring IDA'S Contribution to Country Outcomes.............. 11 Table 4: Quality in IDA Projects...................................................................................................15 Table 5: IEG Indicators in IDA Projects ....................................................................................... 15 Table 6: Outcome Baseline Data in ISRs for IDA Operations......................................................16 Table 7: Ongoing Impact Evaluations in IDA countries ............................................................ 23 Table 8: Progress in IDA Countries in the Statistical Capacity Score ...................................... 26 Table 9: Statistical Capacity-Building Efforts in IDA Projects in Health, Education. .................28 Table 10: Outcome Baseline Data in ISRs for IDA Operations....................................................38 Table 11:IDA14 Results Measurement System (RMS) Mid-Term Review Outputs Table.........40 Table 12: Statistical Capacity-Building Efforts in IDA Projects in Health. Education. Water and Transport........................................................................................................................................ 41 FOCUSON RESULTS:THEIDA14 RESULTSMEASUREMENTSYSTEM AND DIRECTIONS IDA15 FOR i. The IDA Results Measurement System (RMS), initially established during the IDA13 replenishment and considerably enhanced during IDA14, has become an integral part of the World Bank's effort to enhance its results orientation. The RMS focuses on both development effectiveness and agency effectiveness with two tiers of indicators. Tier 1 is a set of indicators selected to track trends in "big picture" outcomes relevant to IDA's overall development strategy-fostering the climate for sustainable growth and investing in and empowering poor people. Tier 2 tracks IDA's contribution to development outcomes using indicators of the quality and effectiveness of IDA programs as well as examples of the outputs that IDA produces in four sectors. Experience with implementation of the RMS is encouraging: it captures heartening trends in country-level outcomes in IDA countries, and it is directly shaping a "results culture" in the management of IDA resources. At the same time, implementation of the RMS raises some issues that need to be addressed in the IDA15 period. A. Progress on the Indicators ii. Tier 1 Progress. Overall, IDA countries are achieving improved outcomes, though progress varies by indicator and is uneven across countries. On average in IDA countries real GDP per capita rose by 5.6 percent annually in the period 2003-2006 and the proportion of people living on less than $l/day dropped by nearly two percentage points, from 31.7 percent in 2002 to 29.5 percent in 2004. Indicators for financial management and investment climate also show positive change during the recent period, with the regulatory obstacles to starting a new business continuously falling between 2004 and 2007. Progress in infrastructure development is also apparent: data show an increase in household electrification rates between 2002 and 2005, as well as a strong expansion of phone subscriptions. In the human development area, however, while the progress toward universal primary education and gender equality in primary and secondary schools is strong, the changes observed in prevalence of HIVIAIDS, maternal care, and child mortality are not as impressive and are insufficient to reach the MDG targets especially in the Africa Region. Meanwhile, even as IDA countries make strong progress, the absolute gaps between IDA countries and other developing countries remain large or even increase in some cases. iii. Gaps in Statistical Capacity Remain. IDA is making an important contribution by focusing attention on country-level data gaps in critical thematic areas. While work still needs to be done to compile estimates for the indicators from reasonably up-to-date censuses and various household surveys, the poorest IDA countries in Sub-Saharan Africa still lack adequate systems for gathering basic social and demographic data. This seriously constrains the transparency and effectiveness of decision making and program implementation at local levels. The IDA RMS has contributed to the awareness of the lack of data needed for informed decision making and the need to strengthen the incentives within country systems to generate this data. There is still an urgent need to close these statistical gaps, either through IDA operations or through other initiatives of the broader donor community. Countries participating in the Third Roundtable on Results in Hanoi in February 2007 made particular note of this issue. Country delegations also showed high levels of demand for advice and support to improve their results management systems. An effort to significantly scale up support for improving statistical capacity and build results management capacities through a better coordinated donor effort was launched at the World BanklIMF 2007 Spring Meetings. The success of this initiative requires strong leadership from IDA. iv. Tier 2 Progress. The implementation of the RMS has contributed to positive shifts in the Bank's internal "results culture." Management took all actions that it committed to at the IDA14 Mid-Term Review, and there has been progress in Tier 2 indicators since then. Outcomes as measured by the Independent Evaluation Group (IEG) continue to remain high, despite the Bank's increasing lending commitments and deeper engagement with the poorest countries. The Quality Assurance Group (QAG) is now reviewing IDA operations at entry for the quality of their results frameworks. Partially to ensure greater responsiveness to the IDA RMS, a Results Steering Group has been formed to foster greater attention to managing for results throughout the Bank. In collaboration with OPCS and the Knowledge and Learning Board, Regions and Networks are implementing training programs in developing results frameworks and using such frameworks systematically during supervision and at completion to report on outcomes. In addition, revised guidelines for Implementation Completion Reports (ICRs) have been issued to help with more specific measurement of project achievements. This step, combined with the effort of the OECD DAC's joint venture on Managing for Results to self assess country capacity to manage for results, should help specify and draw national and donor attention to overcoming gaps in statistical capacity. Results-based Country Assistance Strategies (RBCASs), mainstreamed across all client groups, have generated a major culture change in the institution. More CASs are defining realistic results chains, aligning the Bank's strategy with the country's long-term vision and priorities, coordinating with other donor partners on major elements of strategy, selecting a manageable number of expected CAS outcomes, and better incorporating lessons from the past through a formal assessment of the last strategy. A review of Implementation Status and Results Reports (ISRs) shows an increase in the percentage that have adequate baseline data. Reviews of Implementation Completion Reports for projects exiting after the IDA14 Mid-Term Review are useful in showing examples of outputs that are produced through IDA operations-for instance, IDA transport operations constructed over 7,500 km of roads; IDA health operations trained more than 81,400 health professionals; IDA education operations trained 282,500 teachers; and IDA water and sanitation operations supported 87,000 new water connections. However, many ICRs lack figures on outputs; and when they do include outputs, they tend to report in a variety of forms (e.g., different units, no standard categories), which makes meaningful aggregation challenging. Going forward, Management will work to ensure that ICRs allow a more reliable specification of outputs in sector and thematic areas. A recent review of the quality of results frameworks in a sample of IDA operations found improvements in the quality of project development objectives (PDOs), in the percentage of projects with information on baselines, and in the percentage of investment lending project documents that included institutional arrangements for project monitoring and evaluation (M&E). Development policy operations (DPO) documents generally included baselines and noted that monitoring frameworks would rely on existing systems; however, few provided in-depth assessments of weaknesses in country monitoring systems, or analysis of how those systems are being strengthened and used in decision-making. As was reported at the IDA14 Mid-Term Review, the clarification of expected links between program and project documents and their expected contribution to CAS outcomes continues to challenge IDA staff and borrower teams. B. Conclusions v. The introduction of a Results Measurement System for IDA has generated a sharper focus on results at the country level and a stronger internal results culture. IDA is the first international financial institution to introduce a results measurement system that systematically tracks both key outcome indicators (development effectiveness) and its own contributions to these outcomes. This paper shows that IDA countries as a whole continue to make progress on the RMS outcome indicators: growth is increasing, poverty is declining, infrastructure development is progressing, and obstacles to business environment are falling. However, progress across IDA countries continues to remain uneven, and several key human development indicators remain a cause for concern, notably in Africa. Having in place an explicit measurement system, particularly one that measures factors influencing (Tier 2) agency effectiveness that are in the control of staff and managers, can stimulate changes in internal organizational behavior. Recognizing this, it may be appropriate to further refine the IDA15 RMS indicators and related signaling mechanisms to reinforce the focus on results and to signal other changes in IDA behavior. Proposalsfor Tier 1. The Tier 1 indicators provide an overview of the economic and social progress of IDA countries. To strengthen their usefulness, IDA should continue to advocate for improvements in the quality and frequency of data collection and reporting in IDA countries and, where appropriate, provide resources for investments in national statistical systems. Some revisions to the Tier 1 indicators are proposed to better reflect IDA'S strategic direction and challenges, particularly to develop a PEFA-based aggregate indicator to assess the quality of countries' public financial management and to review Tier 1 indicators to better reflect Bank's increasing emphasis on the private sector. Proposals for Tier 2. In Tier 2 some areas of weakness remain and will be addressed: the quality of outcome indicators, the availability of baselines for all outcome indicators at entry in IDA projects, and the quality of output and outcome reporting in ICRs for IDA operations. Moreover, the list of Tier 2 indicators does not capture the contribution of IDA'S analytic and advisory services at the country level, beyond the coverage these activities receive through CAS Completion Reports (CASCRs). Several modifications are proposed for Tier 2: P In addition to tracking achievement of project results at exit, the RMS will track the quality of the PDO at design, adequate baseline during implementation according to a more stringent definition, and the reporting quality of outputs and outcomes in ICRs based on the original results framework. P The RMS will draw on recent innovations in the CAS process to improve the quality of results frameworks in CASs and rate program performance in CAS Completion Reports (CASCRs) by proposing monitoring of self-ratings in CASCRs and their independent validation by IEG. 9 Management will, based on collaborative efforts between Networks and Regions, develop lists of standard output indicators for 4-5 sectors by the time of the IDA15 Mid-Term Review. P During the IDA15 period Management will be working to strengthen the results orientation of analytic and advisory activities (AAA). In developing this work, the Bank will be looking into issues in promoting joint analytic work, more effective dissemination strategies, disclosure and the adoption of more programmatic approaches. A summary of the assessment of previous analyses, current issues, future directions and a review of implications for Tier 2 metrics will be presented at the IDA15 Mid-Term Review. vi. Goingforward, IDA in collaboration with other donors should strengthen thefocus on statistical capacity, includingfostering demand at the country level and examiningfinancing modalities. Gaps in data at the national and sub-national levels seriously constrain country-level monitoring and local decision makers' ability to use evidence to implement policy and improve the effectiveness of programs. Ensuring the collection and use of small-area data on key socioeconomic issues is central to engaging both citizens and political leaders in poverty reduction and growth promotion. Client countries are increasingly demanding support in building national M&E systems to manage for results. IDA and other donors need to scale up investments in this area, and define financing modalities to support country effort. IDA can continue to promote a culture of results internally through signaling, targeted learning activities, revising budget processes, and integrating statistical capacity and use into discussions on country programs. It can also use its convening role to promote more cross-country learning on establishing and effectively operating systems to ensure that country leaders are able to apply results management tools. Specifically: > The review of weaknesses in statistics and their use in decision making in CASs will be enhanced by requiring more comprehensive discussion of these issues in the CAS. These issues will be followed up, when relevant, in DPOs, SILs, AAA, CASPRs, CASCRs and ICRs. To facilitate this process, IDA, in partnership with PARIS 21, will develop targeted guidance and statistical assessment tools to assist country teams to evaluate the quality and capacity of a country's statistical system. P In line with the country based approach to building statistical capacity, Management intends to intensify its efforts with donors and partners to identify 'lead donors' for work in selected priority countries and accelerate efforts to develop a sector wide approach to building statistical capacity in these priority countries. To ensure that these efforts are well integrated into existing and proposed sector programs, Management will encourage the Poverty Reduction and Economic Growth (PREM), Human Development (HD) and Sustainable Development (SDN) Networks to support this effort and to involve them more actively in encouraging support for statistical and results management. P Management will explore the potential for expanding and redesigning the existing multi-donor Trust Fund for Statistical Capacity Building to provide a mechanism to primarily finance and co-finance as well as to supervise investments in statistical capacity and implementation of national strategies for the development of statistics (NSDSs). P The Bank will develop a data standards system to help IDA countries track their progress toward meeting internationally accepted norms of data coverage, frequency, and timeliness. Subscribers to the system can provide information about their data compilation and dissemination practices for posting on World Bank's Bulletin Board (similar to IMF's Dissemination Standards Bulletin Board), allowing IDA to monitor the quality of development statistics. FOCUSON RESULTS:THEIDA14 RESULTSMEASUREMENTSYSTEM AND DIRECTIONS IDA15 FOR 1. The IDA Results Measurement System (RMS), initially established through the IDA13 replenishment and considerably enhanced in IDA14, is an integral part of the World Bank's effort to enhance its results orientation-an effort that was launched in 2002 in the follow-up to Monterrey, drew on three international Roundtables on Results, and received further impetus through the Paris High-Level Forum on Aid Effectiveness (March 2005). The World Bank's agenda on managing for results ("the results agenda") begins with the premise that the ultimate measure of development effectiveness is sustained improvement in outcomes at the country level, and that improving country-level effectiveness requires borrowing countries and their development partners, including IDA, to give attention to results throughout the planning, implementation, and completion processes. It has become conventional to refer to two forms of effectiveness: developmentor country-level effectiveness refers to the achievement of sustainable improvements in outcomes on the ground, and agency effectiveness means the work that organizations in the donor community do to ensure that their business practices conform to the principles of good aid management, have enabling effects on country capacity and institutions, and thus contribute to country-level effectiveness. 2. IDA14 Results Measurement System. The IDA Results Measurement System is a two- tiered system. Tier 1 monitors aggregate progress on 14 selected country outcome indicators grouped into four categories: growth and poverty reduction, public financial management and investment climate, infrastructure, and human development. Tier 1 also tracks work at the project level to strengthen country capacity to collect and use information on results in four sectors: water, transport, education, and health.' Tier 2 monitors IDA'S results orientation and contribution to country outcomes using indicators for country programs and project^.^ 3. Mid-Term Review. IDA reported on the implementation of the RMS at the IDA14 Mid- Term Review in November 2006.3 The review found improved outcomes in IDA countries. It reported good progress in most Tier 2 indicators, though progress varied by indicator and was uneven across countries; and it acknowledged that, in some areas, the Bank needed to intensify its effort. Therefore, Management agreed to take six actions: (a) improve the quality of results frameworks in IDA operations; (b) review the adequacy of the discussion of data quality and statistical capacity in Country Assistance Strategy (CAS) documents; (c) monitor self-assessment ratings in CAS Completion Reports (CASCRs) and their independent validation by the Independent Evaluation Group (IEG); (d) ensure that projects in water, education, health, and roads address capacity to monitor the relevant IDA indicator; (e) revise guidelines for Implementation Completion Reports (ICRs) to ensure the more specific measurement of project outcomes; and (f) establish the percent of projects with baselines in first Implementation Status ' Indicators in infrastructure and human development are under-five mortality rate, primary education completion rate, proportion of population with improved access to safe water sources, and proportion of rural population with access to an all-season rural road. Report from the Executive Directors of the International Development Association to the Board of Governors/Additions to IDA Resources: Fourteenth ~e~lenishrnent,Working Together to Achieve the Millennium ~ e v e l i ~ m eGoals, March 10, n t 2005. IDA14 Results Measurement System: Mid-Term Review Report (IDAISecM2004-0499), November 2006. and Results Reports (ISRs) as a performance indicator in staff Strategy and Performance Contracts. 4. Purpose and Structure of this Report. This paper provides an update on the status of Tier 1 and Tier 2 indicators and on the actions Management committed to in November 2006, draws on the lessons of implementation to identify issues in implementing results management in IDA, and presents proposals to deepen the RMS for IDA15. Section I1 uses the 14 Tier 1 indicators to review progress in IDA country outcomes. Section I11 uses the Tier 2 indicators and reports on the actions Management agreed to at the IDA14 Mid-Term Review to assess IDA'S contribution to country outcomes. Section IV discusses lessons of experience with the implementation of the RMS and reports on the Third Roundtable on Results held in Hanoi in February 2007. Section V presents proposals for the IDA15 RMS. 5. IDA strategy rests on two interdependent pillars: (a) fostering the climate and conditions for sustainable growth, investment, and job creation that are inclusive of poor people, and (b) investing in poor people and empowering them to participate in development. To monitor the "big picture" progress of this strategy in IDA countries, or the overall development effectiveness, Tier 1 of the RMS contains 14 broad development outcome indicators that are grouped into four categories: growth and poverty reduction, public financial management and investment climate, infrastructure, and human development (see Table 1). This section reviews progress in Tier 1 indicators in IDA countries. Given data limitations, the baseline values and most recently available values refer to different years and slightly different time periods for each of the indicators (see Table 1 and Annex C). A. Progress on Tier 1 Indicators 6. IDA countries as a whole continue to demonstrate progress on the indicators measured by the Results Monitoring System. Real income per capita grew by nearly six percent in 2006, slightly faster than in the previous year and much faster than any other year since 1990. Extreme poverty fell two percentage points between 2002 and 2004, continuing the trend since 1999. Some obstacles in the business environment were eased and, as a result, the cost required to register a new business as share of per capita income fell by eight percent per annum. And several infrastructure and education indicators also showed sizable improvement. However, for many outcome indicators year-to-year changes tend to be small and difficult to m~nitor,~and rates of progress vary substantially by country. Moreover, the current rates of progress in reducing child mortality, improving maternal health, and reversing the spread of HIVIAIDS generally fall short of what is needed to achieve the Millennium Development Goals (MDG) targets on time. 1. Growth and Poverty Reduction 7. IDA countries' real GDP per capita grew by 5.5 percent annually between 2003 and 2006; more than twice the average growth rate of 2.2 percent between 1990 and 2003 (see Figure 1). The performance of IDA countries is on par with that of non-IDA developing countries5and These indicators are based on household surveys, which are not conducted regularly or frequently in most IDA countries. Non-IDA developing countries in this section refer to 69 developing countries that are not eligible for IDA grants or credits in FY08. They are mainly middle-income, IBRD client countries. significantly higher than the average growth rate of 2.3 percent for high-income countries. This improvement in growth performance is broad-based, as 9 out of 10 IDA countries have enjoyed positive growth between 2003 and 2006, with substantial increases in Sub-Saharan Africa (SSA) and Europe and Central Asia (ECA).6Accelerating economic growth is also evident for blend countries, and net oillore exporting IDA countries. Table 1: Progress in Tier 1 Outcome Indicators (updated in June 2007) Aggregate outcomes Country-level outcomes Year of Percent of Year of most Percent Of countries baseline Baseline Most countries recent recent making value value making value value substantial progress progress " Growth and poverty reduction Population living on less than $1 a day 2002 2004 31.7 29.5 81 35 (%) Governance and investment climate Public financial management (number 2001102 2004 6 6.5 41 Cost required for business start-up (% 2003 2006 125 92 89 21 .......................................................................................... of GNI per capita) ".................................................................................................................................................................................................................. .".......................................................................................................................................... Time required for business start-up 2003 2006 80 47 39 21 Infrastructure for development Access to an improved water source 1990 2004 64.7 75.4 75 37 ..................................................................................................................... (% ofpopu~ation) Fixed line and mobile phone Access to an all-season road (% of 2000 61 Household electrification rate (% of 2002 2005 57.4 60.7 households) Human development Under 5 mortality rate (per 1,000 live 2000 2005 122 111 76 21 births) ................................................. ................... ........................... .......................................................................................................... .......................................................................................................................................... " Prevalence of HIVIAIDS (% adult 2003 2005 1.68 1.69 36 36 Ratio of girls to boys in primary and 2002 2005 83.3 89.0 67 38 secondary education (%) Source: WDI 2007 and World Bank staff estimates. a Substantial progress is defined as the 75thpercentile of progress observed in all developing countries; see Annex B for details. (..)Data not available. Seven IDA countries achieved an annual average growth rate of 8 percent or higher of GDP per capita during the 2003-2006 period: Azerbaijan (19 percent), Armenia (12 percent), Angola (11 percent), Chad (9.9 percent), Georgia (8.8 percent), Cambodia (8.7 percent), Ethiopia (8.3 percent). During the previous three-year interval (2000-2003), four IDA countries achieved a growth rate above 8 percent of GDP: Sierra Leone (12 percent), Armenia (12 percent), Azerbaijan (9.3 percent), and Tajikistan (8.2 percent). Figure 1: Annual growth rates in selected groups of countries -2% i All IDA Blend IDA only IDA in IDA in SubNon-oiVore OiVOre Non-IDA High- Europe & Sahara exporters exportersdeveloping income Central Africa countries countries Asia Source: WDI 2007 and staff estimates. 8. Extreme Poverty. Rising GDP per capita has been accompanied by falling poverty rates for IDA countries. Since 1990,the proportion of people in IDA countries living on less than $1 a day has fallen from 40 percent to 32 percent in 2002. The extreme poverty rate dropped farther, by more than two percentage points to 29.5 percent between 2002 and 2004 (see Figure 2). Figure 2: Growth and poverty reduction in IDA countries, 1990-2006 - B GDP per capita --e$l/day headcount 1 I Source: WDI 2007 and PovcalNet. While East Asia and South Asia led the way, there were sizable improvements in IDA countries in ECA and SSA.' A few IDA countries also stand out for progress in poverty reduction: Burkina Faso ($l/day headcount fell from 45 percent to 27 percent between 1998 and 2003), Nepal (from 34 percent to 24 percent between 1996 and 2004), Cameroon (from 32 percent to 17 percent between 1996 and 2001), Azerbaijan (from 11 percent to 4 percent between 1995 and 2001). Although the aggregate pace of poverty reduction appears to be strong, it falls short of the progress needed to halve the proportion of poor people in 1990 by 2015.8 2. Public Financial Management and Investment Climate 9. As at the IDA14 Mid-Term Review, Figure 3: Budgetary and financial management in the progress in public financial management developing countries (PFM) remains uneven. Since 2005, when piGKczT the Public Expenditure and Financial Accountability (PEFA) framework was CFIA 13: Quality of Budgetary and FinancialManagement adopted to replace HIPC expenditure tracking for assessing PFM performance, All IDA there have been 45 PEFA assessments fully Blend countries completed and another 21 substantially completed. A review of preliminary results IDA only countries suggests that performance across the key IDA in Europeand CentralAsia areas of budget preparation and execution in low-income countries needs to be Non-IDA developing countries 1 1 -. impr~ved.~Although repeated PEFA scores --5.0- 1.0 2.0 3.0 4.0 are not available to track progress, the +----- high - World Bank's Country Policy and ' low I Institutional Assessment (CPIA) component for quality of budgetary and financial management indicates that IDA countries, on average, have made progress, particularly those in ECA, where the average score for 9 IDA countries increased from 2.9 to 3.6 between 2000 and 2006, quite remarkable given the difficulty of making changes to public systems (see Figure 3). But of seventy-four IDA countries evaluated, only half improved their scores over the period. 10. Regulatory Obstacles. Regulatory obstacles to starting a new business have fallen consistently between 2004 and 2007. The cost required for registering a start-up business dropped by about one-sixth, and the time required to register a new business fell by two-fifths (see Figure 4 and Figure 5). The gaps between IDA countries and non-IDA developing countries have shrunk. In 2007, on average it only took around 46 days to register a new business in IDA countries, down from 78 days in 2004 and quite close to the time required in non-IDA developing countries (43 days). Nineteen IDA countries made positive reforms to make starting a business easier during 2007, and three IDA countries-Ghana, Georgia, and Kenya-ranked third, fifth, and eighth across 178 economies for improvements that reduce impediments to doing business. The IDA countries that made the largest reduction in the time taken to register a new ' Excluding India, the IDA aggregate poverty headcount (based on 50 countries) still drops 2 percentage points. Poverty reduction is occurring. The Global Monitoring Report 2007 projects that in 2015 Sub-Saharan Africa's poverty rate will remain over 35 percent, far above the 23 percent target. IMF "Fiscal Policy Response to Scaled-Up Aid: Strengthening Public Financial Management", May 2007. business between 2004 and 2007 were Mozambique (153 days down to 29 days), Azerbaijan (122 to 30), India (89 to 33), and Honduras (62 to 21). The top-ranked countries reducing the cost of registering a new business during the same period were Tanzania (187 percent GNI down to 47 percent), Niger (445 percent to 175 percent), Mozambique (93 percent to 22 percent), and Timor-Leste (125 percent to 12percent).1° Figure 4: Time required to register a new business Figure 5: Cost to register a new business In developing countries in developing countries Timrequiredto start a business (days) 1 Cost of business start-up procedures(% of GNIper capita) All IDA Blend IDA only IDA in IDA in Sub Non-IDA All IDA Blend IDA only IDA in IDA in Sub- Non-IDA countries countries South Asia Sahara developing countries countries South Asia Sahara developing Africa countries Africa countries Source: Doing Business Project and staff estimates. Source: Doing Business Project and staff estimates. 3. Infrastructurefor Development 11. Household electrification rates in IDA countries increased more than 3 percent between 2002 and 2005, with 59 percent of households having access to electricity in 2005 (see Figure 6). Of 49 IDA countries with two measurements over time, 44 have shown improvement, and nearly one-third have seen a 5 percent or greater rate of change." The telecommunications sector continued to expand rapidly as well: the rate of fixed-lines and mobile telephone subscriptions in IDA countries nearly tripled between 2002 and 2005, from 47 to 130per 1,000population (see Figure 7). Progress was strong in all regions, and nearly half of IDA countries experienced an annual rate of change greater than 30 percent. But IDA countries are starting from a low base. Although the average pace of increase in IDA countries surpassed that of non-IDA developing countries, the absolute gap in the proportion of phone subscribers between the two groups has become larger. l o Doing Business Report, 2008. IDA countries with significant increases include: Bangladesh (from 32 percent in 1999 to 41 percent in 2004), Comoros (from 29 percent in 1996 to 45 percent in 2002), Madagascar (from 11 percent in 1997 to 20 percent in 2003), Mongolia (from 67 percent in 2000 to 86 percent in 2005), Senegal (from 37 percent in 2001 to 47 percent in 2005), and Vietnam (78 percent in 1997 to 96 percent in 2005). Figure 6: Access to electricity in IDA countries (in O h ) Figure 7: Telephone access in developing countrie --.. r-- ..-.. Household electrification rate (% of households) Annual growth of futed ilne and mbtle phone subscribers 50% - - - - - Ail IDA Blend IDA only IDA in South IDA in S u b All IDA Blend IDA only IDA in IDA in Sub- Non-IDA Asia Sahara SouthAsia Sahara developing Africa Africa countries I I I I I Source: IDA14 RMS. Source: World Telecommunication Development Report database and staff estimates. 12. Other Infrnstructure Indicators. Standardized estimates of access to an improved water source are produced by the Joint Monitoring Project (WHO and UNICEF); estimates through 2006 will be available in early 2008; until then, no update is possible. The indicator of the rural population's access to an all-season road is derived from available household survey responses. Only a few IDA countries have more than one estimate, so it is not yet possible to assess progress. To encourage more extensive and frequent data collection, a standardized module will be developed for inclusion in the International Household Survey Network question bank. 4. Human Development 13. Health. Progress toward several key human development MDGs remains a cause for concern. In 2005, 111 of every 1,000 children in IDA countries died before reaching the age of five. This figure represents an improvement over what was reported at the IDA14 Mid-Term Review, and it represents a faster pace of reduction than during the 1990s; however, it is insufficient to meet the MDG child mortality goal (see Figure 8). The slow pace of improvement is due largely to the stagnation of the mortality rate in fragile states, where public health services are weakest. The progress in increasing the proportion of births attended by skilled health personnel in IDA countries was slower than in the past (with almost no improvement in SSA and a decrease in ECA), and far short of what is required to meet the maternal mortality MDG. The gap between IDA countries and non-IDA developing countries remains large (see Figure 9). Although the UN has not updated the adult HIVIAIDS prevalence rates since the IDA14 Mid- Term report, recent evidence indicates progress in some IDA countrie~.'~Data for at least three African countries-Kenya, Uganda, and Zimbabwe-shows nationwide declines in HIV prevalence, and in another five countries HIVIAIDS prevalence in urban populations has declined, although some of these changes reflect improved measurement rather than absolute change. l 2 World Bank (2007), The Afvica Multi-Country AIDS Program 2000-2006: Results of the World Bank's Response to a Development Crisis. Figure 8: Child mortality rates in developing Figure 9: Maternal care in developing countries countries Wrtality rate,under-5 (per 1,000) 1 Births attendedby sldlledhealthstaff (% of total) I 1995 2000 2005 All IDA Blend IDA only IDA in DA in Sub- Non-IDA All IDA --tBlend SouthAsia Sahara developing - k+ IDA onty a Non-IDA developingcountrie Africa countries Source: UNICEFIWHO and staff estimates. Source: WDI07 and staff estimates. 14. Education. The primary school completion rate increased by six percentage points between 2002 and 2005 (see Figure 10)' and 80 percent of children were able to complete their primary education in 2005. Progress was particularly good in South Asia and Sub-Saharan Africa. The ratio of girls' primary and secondary enrollment to boys' increased from 84 to 89 percent during the three-year period (see Figure 11)' and at this rate of progress it is very likely that the MDG target-which calls for the elimination of gender disparity in all levels of education by 2015-will be achieved. While three-fifths of IDA countries are eliminating- gender disparity in primary and secondary education, the impressive aggregate change is due primarily to a few large outstanding performers. Though IDA countries in Middle East and North Africa, South Asia, and Sub-Saharan Africa made strong progress in achieving gender parity in primary and secondary education, on average they still lag behind their peers in other regions. Figure 10: Primary completion rates in IDA Figure 11: Gender equality in education in countries developing countries R~marycorrpiet~onrate,total (% of relevantage - - --- 100 -- - - - - - 7 --a Ratioof girls to boys in primary and secondary education (YO) I I 201 ! 1990 1995 2000 2005 Ail IDA Blend IDA only IDA in IDA in Sub- Non-IDA I All IDA South Asia Sahara developing --; IDA ~nSub-Sahara~ f r c d -+- Africa countries Blend countries .-..+?+--. IDA only I I 1 Source: WDI 2007 and staff estimates Source: UNESCO and staff estimates B. Measurement 15. The availability of data for most RMS indicators has improved in the past year. Several factors have contributed: stronger demand for good quality and relevant statistics, better statistical capacity in IDA countries, and the efforts of the international statistics community to collect and report internationally comparable statistics. Data on household electrification rates have particularly benefited both from a number of newly available surveys with relevant information and a new compilation effort harvesting information from existing surveys. The RMS has played an important role in stimulating these efforts: the compilation effort, initiated mainly for RMS purposes, not only doubled the number of IDA countries with recent estimates but also made the measurement of progress possible. Efforts to improve the availability and reporting on existing indicators will remain a focus for IDA15. 16. Measurement Gaps. An important contribution of the RMS is the attention it has brought to data gaps across IDA countries in critical thematic areas. Annex C provides an update on data availability for Tier 1 indicators and shows that, even though data are available on all major indicators, substantial gaps remain across IDA countries and have to be addressed systematically in order to improve quantity and quality of available data. 17. Many IDA countries do not have the resources or capacity to implement and sustain a regular survey program. As of August 2007 data were available from only 51 IDA countries to calculate an aggregate poverty headcount rate. The lack of data on rural populations' access to an all-season road is another good example. Not many existing household surveys collect this information, and those that do may not generate comparable estimates. Other survey-based indicators-such as births attended by skilled staff and access to an improved water source-are still not measured regularly. 18. Most IDA countries do not have adequate statistical infrastructure, which casts doubt on the quality of statistics in general. For example, only two countries in Africa (Mauritius and Seychelles)13have sufficiently robust civil registration systems to enable counting population size accurately-a problem that seriously impedes the quality of survey work and means that the denominators for most indicators of poverty reduction are based on estimations and assumptions. Over one-third of Africa's population lives in countries that have not conducted a census in the last 10years.14 19. Some measurement gaps also arise from technical level. The original PFM indicator "the number of HIPC benchmarks met" is no longer tracked and has been replaced by the PEFA scores, although only a fraction of final PEFA reports are released into the public domain. The analysis linking the original 15 HIPC benchmarks with the newer 28 PEFA indicators was only completed recently. The measurement of progress in the HIVIAIDS epidemic also remains problematic. New survey-based data on biomarkers have led to significant revisions in l 3 Setel, Philip W., et al., "A Scandal of Invisibility: Making Everyone Count by Counting Everyone," The Lancet, forthcoming October 2007. l4 Of 39 IDA countries in Africa, 13 (36 percent in terms of population) have not conducted a population census in the last 10 years (1997-2006), and 24 (40 percent in terms of population) have conducted less than two poverty surveys (which would be needed to measure trends in poverty rates) during the same period. Source: World Bank staff estimate; based on the information available from the DDP Microdata Catalog. prevalence estimates for some countries. But surveys collecting blood samples remain too expensive and, therefore, too infrequent to provide a basis for cross-country or time-series analysis. A direct measurement of HIV incidence (new infections) would be preferable to the current, modeled estimate of prevalence, but such data are not generally available. 20. Goingforward. While the ability of the IDA countries to provide more accurate statistics has been enhanced in the past year, the task of improving the collection and use of existing indicators would remain the focus for IDA15. IDA will continue to advocate for improvements in the quality and frequency of data collection and reporting in IDA countries and, where appropriate, provide resources for investments in national statistical systems. Further it is proposed to develop a PEFA-based aggregate indicator (to replace "number of HIPC benchmarks met") to measure the quality of countries' public financial management. Finally, Management will review Tier 1 indicators to better reflect Bank's increasing emphasis on the private sector. 21. Tier 2 indicators assess the quality and outcomes of IDA's operations, from initiation to completion-factors in country-level effectiveness that are more directly under the control of l ~ ~ - s t aandf management. They provide information on the ugency 's effectiveness-the f work that IDA as an organization in the donor community does to ensure that its business practices conform to the principles of good aid management, have enabling effects on country capacity and institutions, and thus contribute to country-level effectiveness. This tier contains six indicators selected to track IDA's contribution to development effectiveness across three levels of the Bank's work-program, project, and aggregate outputs. The first indicator, focused on the country level, tracks the number of results-based CASs (RBCASs) prepared (see Table 2). The next four indicators report on project-level results: (a) the percentage of projects with satisfactory quality-at-entry ratings; (b) the percentage with satisfactory outcome ratings at exit; (c) the percentage of ICRs of satisfactory quality; and (d) the percentage of first ISRs with satisfactory baseline data. The sixth Table 2: Implementation of RBCASs indicator reports on aggregate progress on Indicator FY03 FY04 FY05 FY06 FY07 selected outputs in four sectors-health, Number of IDA RB 1 4 4 loa 11" education, transport, and water-in which CASS IDA is most active across a range of Cumulative IDA 1 5 9 19" 30" countries. This section updates progress on RBCAS~ Tier 2 and reports on the actions Number of RBCASs 1 6 10 15 16 Cumulative RBCASs 1 7 17 32 48 Management has taken to deepen the target - 16 30 45 process of into every aspect Source: OPCS, World Bank.a Includes one blend country of IDA's work (Table 3 summarizes progress on all six indicators). Table 3: Tier Two: Progress in Monitoring IDA'SContribution to Country Outcomes Project level 1.& 2.Percentage of projects with satisfactory outcome rating & percentage of projects with satisfactory ICR quality by exit year (updated numbers for FY05 and FY06) FY03 FYO4 FYOS FY06 FY07' Percentage of vroiects with satisfactow outcome rating 70 77 76 76 68 percentage of projects with satisfactoj ICR quality 85 84 91 86 92 3. Percentage of operations with satisfactorv aualitv at entry 4. Percentage of ISRs with satisfactory outcome baseline data 6. Outputs: Examples Indicator FYOI-03 FYO4-05 FY06-07' Output examples in health Health professionals tra~ned 41,300 75,000~ 81,400' Output examples in education Classrooms built or rehabilitated Teachers trained Teachers recruited Output examples in water supply and New water connections sanitation New sanitation facilities 61,900 111,600 33,000 Output examples in transport Roads built (km) 21,600 5,200 7,500 Jobs created 344,000 26,000 ( . . . ) I Source: IDA (2005), Additions to IDA Resources: Fourteenth Replenishment. Working Together To Achieve the Millennium Development Goals. Also, IEG, QAG, OPCS and World Bank staff calculations. See Annex D for more information on ISR and ICR reviews. "Partial sample as October, 9, 2007 based on the review of only 57 ICRs, As a reference, figures for FY06 and FY05 were based on 122 and 164 ICRs respectively). Figures are subject to change as additional ICRs are completed and evaluated. * 1t includes all ISRs submitted in FY07. Data are not strictly comparable to those for earlier years because of different criteria for identifying ISRs. FY07-submitted ISRs includes a substantial number of ISRs from FY06-approved operations (-64 percent). 'Atthetimeofthereview(July31,2007) only 13ICRswere available fromatotalof 67tobe completed forprojects exiting in FY07. In addition, 8 FY06 ICRs were not available. *Onehealth project also reported in this period on 10 million insecticide-treated bednets. 'Doesnotinclude1.6millionIntegratedChildDevelopment Service(ICDS)workers. 'Does not count educational facilities, e.g., 12 percent of the reviewed ICRs report on classrooms, while 65 percent of them report on general educational facilities that include over 12,000 schools constructed, as well as other 2,000 resource centers, such as libraries or laboratories. Does not include operations that report on increased teacher-pupil ratios and does not include other educational related staff such as Shiksha Mitras (over 100,000). While no projects during the FYO1-03 period reported on the number of new water connections, 548 km of water supply lines were built during this period. ' While no projects during the FY01-03 period reported on the number of jobs created in the transport sector, projects report that 1,228 professionals were trained. A. Country-Level Indicator 22. For a CAS to be considered results-based, it must contain a results matrix,I5a completion report that evaluates the previous strategy period and is reviewed by IEG, and a discussion of l 5 The foundation of an RBCAS is a results framework. It includes logical links between country goals and Bank activities, intended CAS outcomes achievable within the four-year CAS period, measurable indicators to track key links, and arrangements for collection, analysis, and use of data to monitor results. A snapshot summary of this complex underlying framework is presented in the CAS document using a standard template, the results matrix. capacity building in statistics and in monitoring and evaluation (M&E). The number of RBCASs is growing (see Table 3), and there is evidence that they are improving. Country teams across the Bank now use the approach, Regions are gaining expertise in formulating and evaluating CAS results matrices, and good practices are emerging. 23. CAS Progress Reports. Even when a country does not have an RBCAS, the CAS Progress Report (CASPR)-developed about two years into CAS implementation--can improve the results focus of the Bank-funded program. When preparing a CASPR for a CAS that was designed originally without a results framework, some teams have added a results matrix to sharpen program implementation during the remaining years of the CAS. (Box 1 provides an example from Indonesia). In the Africa Region 15 country teams prepared or retrofitted CAS and Interim Strategy Note results frameworks to strengthen outcome orientation and alignment with country priorities.I6 Progress Reports on RBCASs report on the ongoing validity of the results matrix, as well as confirming the overall relevance of the strategy and updating the Bank program. Box 1: Retrofitting the Indonesia CASPR The Bank had to adjust its program significantly and quickly to help respond effectively to the impact of the tsunami that hit Aceh on December 26,2004. The CASPR proved to be an excellent tool to reflect this shift, as the country team revised the results framework to make it a genuine management tool. The 56 original outcome indicators were regrouped into 27 key outcomes for easier monitoring in the updated results framework, which also included a new pillar on disaster risk management. The results framework is now becoming the basis for more frequent but lighter portfolio reviews to track progress toward CAS outcomes. It is also providing the foundation for establishing a systematic and rigorous Country Results Management Reporting exercise to be undertaken jointly with the Government of Indonesia. 24. Lessons from RBCASs. Experience with designing and using RBCASs has brought a range of issues to the fore. Collaboration. CASs produced in collaboration with other donors are central to the harmonization agenda, but they pose challenges of timing and coordination for review by Bank management and the Bank's Board; they require committing jointly to outcomes while reserving clearly defined accountability for each donor partner. A spectrum of collaborative CASs have been developed, ranging from joint donor coordination of timing of strategy processes to fully joint documentation, such as for the Kyrgyz Republic, Ghana, Tanzania, and Uganda during FY06-07. Fragile states. Experience in several fragile states1' has demonstrated that even where capacity is low, information is limited, and risk is high, it is possible to take a results-focused approach. Donor harmonization and clarity of objectives are central in such settings, and a focus on building results management capabilities can bolster early efforts to reestablish core institutional capacities. But approaches to these challenges must be feasible and adaptive to enable responsiveness in uncertain institutional settings. l6 The countries were Burkina Faso, Central African Republic, Democratic Republic of Congo, Gambia, Ethiopia, Kenya, Malawi, Mali, Mauritania, Mozambique, Nigeria, Senegal, Sierra Leone, Uganda and Zambia. I' For example, Afghanistan, Central African Republic, Comoros, CBte d'Ivoire, Haiti, Kosovo, Lao PDR, Liberia, Tajikistan, West Bank and Gaza, and Zimbabwe. Availability of data. When country-level data are limited, it is difficult for country teams to identifl high-quality indicators for all CAS outcomes and to set baselines and targets for them. This situation makes accountability, evaluation, and learning- for improvements in both agency and country-level effectiveness-more difficult. Measuring outcomes. For some areas of Bank work-for example, the effectiveness of analytic and advisory activities (AAA)," for which outcomes rely on complex institutional improvements-it is difficult to specify quantitative indicators of progress. Qualitative assessments of AAA have been conducted periodically by the Quality Assurance Group (QAG) since fiscal year 2000.19 While initially the assessments were focused on individual, country-specific AAA products, recently the emphasis has shifted to include a wider range of AAA activities treating the country as an integrated whole, and framed around country and sector lines. Good design. Although there has been much progress in developing results frameworks linking Bank activities to country-level goals, when establishing intended CAS outcomes and expected causal links country teams still tend to be too ambitious and optimistic, and to overestimate IDA'S contributions to country development. 25. Lessons from CAS Completion Reports. Management agreed at the IDA14 Mid-Term Review that it would monitor self-assessment ratings in CASCRs. Of the 11 IDA CASCRs in FY07, 9 include explicit self-ratings by country teams; but only 6 of the 11 are on RBCASs. While managers and staff find the self-assessment process useful in assessing Bank/IDA contributions to country outcomes, teams find it difficult to rate CASs that were not designed with an explicit results framework. 26. CASCR Strengths and Weaknesses. A review of 16 CASCRs completed in FY06 and early FY07 highlighted strengths and weaknesses. CASCRs are strong in: (a) analyzing sectoral and thematic achievements; (b) focusing on the role of analytic work and its impact on the client; and (c) providing lessons learned to inform the next CAS. They could do better at: (a) distinguishing clearly between CAS achievement and progress in country development goals; (b) analyzing achievements in country capacity to measure and use results information in decision- making; and (c) describing and more fully linking interventions with outcomes-for example, how completion of an environmental assessment is expected to influence environmental policy, or how construction of roads is contributing to outcomes such as distance traveled to health clinics/schools. (Box 2 describes two good practice CASCRs). I s AAA includes six types of activities: (i) Economic and Sector work; (ii) Country AAA; ((iii) Sector AAA; (iv) non-lending Technical Assistance; (v) other Economic and Sector work; and (vi) Global and Regional AAA. l 9 The most recent Phase I1 Country AAA Assessment, currently underway, evaluated seventeen country AAA programs in six regions. The assessment focuses on a number of aspects of AAA including quality, management and oversight, alignment with client development frameworks, CAS objectives and work by other donors, likely impact as well as the quality of dissemination activities. Box 2: Good Practice in CASCRs CASCRs for two IDA countries, Yemen and Bangladesh, received good practice awards from IEG. The Bangladesh CASCR provides a thorough and candid assessment of the effectiveness of IDA'S assistance program. The lessons for the next CAS are well tied to the analysis of the past. Even though the CAS was not results-based, the report effectively constructs a set of outcomes for each CAS objective, using planned measures, milestones, and intermediate performance indicators. The Yemen CASCR excels in deriving lessons for the next CAS, and it does an admirable job of developing benchmarks by which to judge progress, even though the CAS was not results-based. Source: IEG 27. Going Forward. As RBCASs improve, providing clearer objectives and indicators, self- evaluation by country teams will become easier. The experience with results self-assessment is contributing to the development of an internal "results review process" that helps country teams link their results assessment to plans and budget prioritization. In the future, greater rigor in the evaluation of RBCASs will enable more evidence-based and consistent ratings that will be useful for learning. A forthcoming CAS retrospective will discuss the design and content, implementation, and evolution of CASCRs. B. Project-Level Monitoring 28. To monitor project-level performance, IDA draws on a range of data, including data from the Quality Assurance Group (QAG) and project evaluation data from IEG. 29. Quality at Entry. Quality at entry, which is highly correlated with satisfactory project outcomes, serves as a leading indicator of project results. QAG rated 91 percent of IDA operations as satisfactory during its seventh Quality at Entry Assessment (QEA) (see Table 3). At the IDA14 Mid-Term Review, Management committed to strengthening the upstream review of results frameworks; to that end, in the overall rating of quality at entry the eighth QEA is giving special attention to the coherence and consistency of the development rationale and results framework underpinning the project; the consistency of the results focus of the project design with the outcomes listed in the development objective; the clarity, realism, and scope of the project's development objective; the availability and quality of baselines; the appropriateness of arrangements for monitoring implementation; and the appropriateness of arrangements for evaluating impact and measuring outcomes. Findings from QEA VIII will be available in the second quarter of FY08. 30. Quality of Supervision. QAG's seventh Quality of Supervision Assessment (QSA VII), completed in June 2007, shows that the performance of IDA projects improved from 88 percent satisfactory in QSA VI to 97 percent satisfactory in QSA VII, and is now on par with IBRD projects (see Table 4). Key reasons are that the supervision of projects in the Africa Region improved by more than 10percentage points. The quality of supervision in fragile states is close to that in non-fragile states and, for example, the Sierra Leone Institutional Reform and capacity- building project, which is being implemented in a complex post-conflict environment, has been cited as a good practice example of high supervision quality. (For more information on the QAG Assessments see Annex D). Table 4: Quality in IDA Projects QEA IV QEA V QEA VI QEA VII QSA VII Group No. of % No. of % No. of % No.of % No.of % projec satis. projec satis. projec satis. projec satis. projec satis. ts ts ts ts ts IDA 52 90 25 76 51 85 73 91 74 97 IBRD 34 100 21 100 32 85 40 95 41 94 Total 100 94 50 86 85 85 130 92 115 95 Projects reviewed Source: Business Warehouse and QAG. Note: QEA VIII results are not yet available, so QSA VII, completed in June 2007, is reported. 31. Quality at Exit. The IDA RMS tracks: (a) the share of IDA operations that successfully achieve (or are likely to achieve) their development outcomes, and (b) the percentage of projects with satisfactory ICR quality. At the time of the IDA14 Mid-Term Review, IEG had rated 21 projects that exited in FY06. IEG ratings for an additional 69 projects with FY06 exits are now available as well as updated numbers for FY05. Table 5 confirms that IEG outcome ratings for IDA projects are consistently high: in FY06, the percentage of IDA projects rated satisfactory was 76 percent. In addition, the overall quality of reporting of IDA operations at exit in FY06 is 86 percent (see Table 5). According to the preliminary results available for this report for the 57 IDA projects, the percentage of projects with satisfactory ICR quality is 92 percent while the percentage of projects with satisfactory outcome rating is 68 during FY07. These FY07 figures are subject to change as additional ICRs are completed and evaluated. Table 5: IEG Indicators in IDA Projects Category FY03 FY04 FY05 FY06 FY07" Number of IDA projects 146 149 164 122 57 Total IDA commitments ($US million) 6,722 5,144 6,913 6,360 1,884 Percentage of projects with satisfactory outcome rating 70 77 76 76 68 Percentage of projects with satisfactory ICR quality 85 84 91 86 92 Source: Business Warehouse. World Bank. Note: Figures include IDA aid SPFprojects. "Partial sample as October, 9, 2007 based on the review of only 57 ICRs, As a reference, figures for FY06 and FY05 were based on 122and 164 ICRs respectively. Figures are subjectto change as additional ICRs are completed and evaluated. 32. Baseline Data in Supervision Reports. Effective day-to-day management of operations and ex-post evaluation depend on good baseline data. After committing at the IDA14 Mid-Term Review that the percentage of projects with baselines in their first ISRs will be a performance indicator in staff Strategy and Performance Contracts, Management took three important measures to raise task teams' awareness about the need for baselines in the first ISR: (a) preparation of a Guidance Note on Adequate Baseline Data in ISRs; (b) updating of the Bank's core course "Supervision of Bank Financed Projects"; and (c) preparation of an e-learning module on preparing ISRs, including adequate baselines, particularly to reach staff in the field. A recent OPCS review (see Annex D for more details) assessed the availability of baseline data in first ISRs submitted during FY07, judging them adequate (as in previous reviews) if they had at least one outcome indicator with baseline data. The review found that more than 80 percent of the first ISRs reviewed had at least one outcome indicator with baseline data (see Table 6), and 53 percent had outcome baseline data for all indicators. If we take into consideration the quality dimensi~n,~~percent of ISRs included baselines on at least one outcome indicator and were 63 also satisfactory on indicator quality. Table 6: Outcome Baseline Data in ISRs for IDA Operations FY04 FY05 FY06 FY07" No. ISRs % adequate No. ISRs % adequate No. ISRs % adequate No. ISRs % adequate reviewed baseline reviewed baseline reviewed baseline reviewed baseline data data data data 145 59 171 50 163 76 131 80 Source:OPCS. World Bank. aIncludes all FY07 submitted ISRs. Data are not strictly comparable to those for earlier years because of differentcriteria for identifying ISRs, ISRs submitted for FY07 includes a substantial number of ISRs from FY06-approvedoperations (-64 percent). C. Aggregation of Outputs 33. Aggregating outputs continues to present challenges to IDA and its partners: operational practice in defining outputs varies significantly within sectors, even in those sectors that focus on relatively more tangible goods and services; and variations across projects within a sector reflect differences in sectoral priorities and approaches across countries, consistent with a focus on country-level effectiveness. To tackle these challenges, over the past two years Regions and Networks engaged in a considerable effort to develop result frameworks and more standardized output indicators allowing for better aggregation and measurement of outputs. In particular, efforts are now under way to rationalize output indicators through the design of a Results Monitoring System in the Africa region2' as well as through several global initiatives such as Roll Back Malaria, the Water Supply and Sanitation Joint Monitoring Program, Education for A111Fast Track Initiative and the Multi-country HIVIAIDS Program. In addition, the Networks made a substantial progress in their Sector Strategy Papers to develop result frameworks for ten sectors (including water supply and sanitation, rural development, public sector governance, transport, environment, forestry and social protection). 34. In the absence of standardization, it is difficult to monitor year-to-year trends in numbers of outputs within a single sector. Management seeks to ensure that each project defines expected project results (outputs and outcomes) at the project concept stage, and then measures specific achievements during supervision and at completion. To help ensure this, in FY07 Management issued revised guidelines for ICRs, requiring them to include the reporting of project outputs by component, make an overall assessment of development outcomes generated from project activities, and refer to the project development objective (PDO) and associated key indicators that are linked to the original Project Appraisal Document (PAD) results framework.22The revised ICR guidelines were discussed and widely disseminated, and training courses have been prepared or revised to reflect this guidance. (Table 11 in Annex D provides a flavor of common outputs from IDA operations across four key sectors. Figures should not be interpreted as attempting to show trends since many ICRs still lack figures on outputs, or report them in a 20 The quality of outcome indicators is based on whether they cover all important aspects of the PDO and appear appropriate, i.e., not only higher-order (beyond the scope of the project) but also lower-order (e.g., output indicators). AfYica Result Monitoring System was launched in 2006 and provides a list of proposed outcome indicators for nine sectors: governance, PSD, water supply and sanitation, energy, roads, agriculture and irrigation, education, malaria, HIV/AIDS. See: 22 New formats and guidelines for ICRs became effective in FY07. variety of forms-i.e., different units, no standard categories-the aggregate figures shown in Table 11are, at best, approximate.) 35. Going Forward. The Results Steering Group is working to help Regions and Networks support task teams in ensuring that future ISRs and ICRs meet additional requirements that allow a more accurate indicative aggregation of outputs in the four key sectors. In particular, efforts are now under way to provide a good foundation to move toward a more systematic approach. To this end, Management proposes to develop, with active engagement of Networks and Regions, a list of standard output indicators for 4-5 sectors by the IDA15 Mid-Term Review. However, it will be important to manage the risk that the use of standard output indicators dilute project specificity and adaptation to local context. Moreover, it is important to signal that measures of output complement but can not substitute for measures of outcome. IDA projects are expected to monitor both output and outcomes. This work, which will need the active engagement of the Networks, will both facilitate internal reporting and documentation of outputs, and equip operational teams with tools to assist country partners in developing measures of country-level outputs and outcomes. While aggregated results from the standardized list of outputs would only become available in the medium term, the commencement of this work during IDA15 will be an important activity. D. Results Orientation of IDA Operations 36. The paper prepared for the IDA14 Mid-Term Review reported on the quality of results frameworks of IDA operations from a small pilot review of selected FY06 PADS. That exercise reviewed three dimensions of results monitoring: (a) the statement of the PDO and indicators; (b) the link of the PDO with the stated CAS outcomes; and (c) the institutional arrangements for monitoring and eval~ation.~~The report found that in three-quarters of projects the PDO was expressed as a measurable change in the behavior or condition of the target group. In addition, about three-quarters of the projects also contained baselines. However, the PDO's link to the CAS outcome and the overall discussion of institutional arrangements for M&E were uneven. For the present paper, OPCS reviewed another 28 IDA operations approved in FY07 (21 investment projects and 7 DPOs) to assess whether the quality of results frameworks is improving.24This review added a fourth dimension to the quality of DPOs: building capacity in sectorslline ministries to analyze and use information in decision-making. The main findings of this review are as follows (see Annex D for the methodology and a more in-depth analysis of the findings): Development objectives. Overall, PDOs are expressed as clear, concise statements, and they have improved compared to the FY06 sample. In contrast, the development objective statement is often not very clear in DPOs. DPOs do, however, include a - 23 TOthe extent possible, arrangements to collect and use program indicators are to be anchored in the borrower's existing systems, to minimize the burden of the operation's M&E, maximize the prospects for appropriate monitoring and encourage harmonization around a country owned specification of expected results. Where existing systems are inadequate, consideration should be given to ways to strengthen the system. 24 Twenty-six projects used in this review represent an approximate sample size of two-thirds of the projects approved during the March 1 - April 30, 2007 period and include a lending envelope of $1.4 billion or 55.4 percent of the total commitments in that period. Two sectoral DPLs ($180 million) approved in May 2007 were also included. Representation of the regions is in line with the two-thirds sample size as projects from the 6 regions represent approximately the same percentage of sample projects as they do of total projects approved during the period. table showing the objectives and targets of the poverty reduction strategy paper supported by the DPO. Indicators with baselines and targets. Overall, the quality of indicators has improved: projects have fewer and more outcome-focused, indicators. There has also been an improvement in the number of projects with baselines and targets for the outcome indicators: only one of the projects, compared with about one-quarter of the previous sample, lacks baselines. Among the seven IDA countries' DPOs reviewed, indicators in the program matrix generally include baselines and end-of-the-program target values, showing some improvement over the 2004-05 operations reviewed in the 2006 Development Policy Operations Retrospective. Link with CAS Objectives. While 86 percent of PADSreviewed refer to the project's link to the CAS-an improvement over the FY06 exercise-in most cases the discussion merely restates the CAS pillars, themes, or objectives, without specifically mentioning the CAS outcomes to which the project contributes. There does not appear to be improvement in this regard over the IDA14 Mid-Term Review report. DPOs appropriately support implementation of policy and institutional actions to achieve a subset of objectives in the Poverty Reduction Strategy Paper (PRSP), these documents are not as strong in linking outcomes influenced by the DPO to specific outcomes in the CAS results framework. It is expected that the generalization of RBCASs and the improvement of monitoring frameworks and systems at the country level of PRSPs will help strengthen link between PRSPs and CASs, as well as between CASs and DPO results. Institutional Arrangements for M&E. Investment projects in the sample generally include a discussion of the institutional arrangements for project M&E. Three- quarters of the investment projects provide detailed information on the responsible agencies, including the capacity of various agencies and the role the project will play in building capacity. However, for the most part, program documents for the DPOs reviewed do not provide an in-depth assessment of country/sector monitoring systems or steps being taken to strengthen the system. Experiences, and discussions with Bank staff, suggest that focusing dialogue on single and robust sector-level results monitoring arrangements is key both to sustainable project monitoring and to harmonization of donor efforts (see Box 3). However, this is more feasible in the context of sectorwide and programmatic operations than for individual investment operations. Box 3: Understanding the Institutions for Monitoring Poverty Reduction Strategies In 2006, IDA produced a publication in cooperation with Department for International Development (DfID) on the institutional dimension of poverty reduction strategy (PRS) monitoring systems. This publication was innovative, as it went beyond focusing on technical issues related to producing data and setting up monitoring systems. Rather, for the first time, it explored the rules and processes that bring the various actors and monitoring activities together in a coherent framework in order to better understand how information is used in decision-making. The study found that while most countries involved in implementing a PRS have a range of individual monitoring mechanisms in place, drawing them together into a common system is a relatively new undertaking that poses substantial practical challenges. At the root of many of the weaknesses is a dearth of well-functioning institutions. The main messages are that in elaborating and implementing a PRS monitoring system, it is important to recognize that change will be gradual, and to focus on building flexible arrangements that reinforce existing elements and can be adapted over time. Source: Beyond the Numbers: Understanding the Institutions for Monitoring Poverty Reduction Strategies. World Bank, 2006. Sector capacity at the country level to analyze and use information in decision- making, and provisions to strengthen sector capacity. DPOs are normally underpinned by extensive reviews of country fiduciary systems that draw on such analytic tools as the country financial accountability assessment and the country procurement assessment report, which are available for almost all IDA countries where DPOs are being considered. But not all DPOs are underpinned by an analysis of capacities to analyze data and use them in decision-making in sectors targeted by the DPO. The Bangladesh case discussed in Box 4 is a good practice example of what can be accomplished. Box 4: Bangladesh Third Programmatic Sector Development Support Credit: Strengthening Sector Capacity for Results Management To prepare for this loan, the Bangladesh Bureau of Educational and Information Statistics developed a uniform data coding system that allows data on pupil-level examination scores to be linked with data on school characteristics. This alleviates a recurrent problem at the policymakers' level: it had not been possible to integrate several data sources that used different coding systems. In addition, to ensure countrywide availability of data and information on a real-time basis, the Ministry of Education installed an education database and web-based software in 50 district education offices, with the remaining 14 districts to be linked up soon afterward. Simultaneously, the Ministry took several initiatives to enhance its evaluation capacity and use data from evaluations for policymaking: (a) annual student assessments (based on internationally recognized assessments) piloted in Grades 9 and 12; (b) third-party impact evaluation of the girls' secondary stipend program and of the means-tested stipend program, incorporating the lessons fiom these evaluations into program design; and (c) third-party evaluation of several measures and institutional mechanisms, e.g., public expenditure tracking, transparency in functions and processes, and Independent Inspection Bodies. Benchmarking the performance of Bangladeshi students against international assessment standards will serve two functions for policymakers-it will allow them to observe, first, whether the reforms are leading to better test scores over time, and second, whether Bangladesh is closing the performance gap with other countries. It will also help them understand the reasons underlying the gap. Source: Program Document for Proposed Third Programmatic Sector Development Support Credit. 37. Improving result orientation ofprojects andprograms. As a result of the rolling-out of the RMS and increased emphasis on the quality of the Result Frameworks, some country teams engaged with the IDA countries to restructure and/or refine projects and programs to improve their result focus, alignment with the country strategies as well as Bank country strategy. At the project level restructuring usually involves refinement of the indicators and project's objectives, as well as adding the baseline, where req~ired.~'Restructuring of the country or sector portfolio provides an opportunity to assess priorities during the CAS implementation period and restructure operations, as needed, to align them to current CAS priorities. Examples of restructuring of individual projects as well as country or sector portfolio are presented in Box 5. 25 Restructuring of the result frameworks involving changing of the PDO and/or the outcome indicators/targets requires formal Board approval. If changes to outcome indicators do not modify the original project outcome targets, then Board approval is not required. This is called a refinement of the indicators. Box 5: Examples of retrofitting projects and country portfolios Project Retrofitting. Retrofitting of the results framework (RF) for the Macedonia Education Modernization Project, currently under implementation, was identified in the project mid-term review and agreed with the client. While the scope of the PDO was not changed, the retrofitting included refinements to the indicators to enable measuring all dimensions of the project outcomes. It also included revisions to the intermediate outcome indicators to better measure progress towards outcomes for all project components. The revisions to the indicators were developed jointly with the government counterparts. Country Portfolio Restructuring. The Vietnam Country Portfolio Performance Review (CPPR) covered the portfolio of IDA projects, as well as Trust Funds. It reviewed their alignment with the CAS results and assessed the quality of the RF of operations. Through an extensive collaborative effort with sector teams it helped develop sector results frameworks that served as an input for the new FY06-10 Country Partnership Strategy now under implementation. These undertakings were an important one-time investment on the part of the country team, the government and other development partners to raise awareness and understanding among project teams on ways to better manage for results. Sector Portfolio Restructuring. Restructuring of the HIVIAIDS portfolio under the framework of the MAP Program for eight African countries aimed to maximize the effectiveness of Bank support to HIVIAIDS prevention, careltreatment and mitigation interventions at the country level through improved M&E capacity, realigned indicators to national strategies, funds directed to priority and high impact interventions and better coordinated institutional arrangements. The region's progress will be reported through a HIVIAIDS Results Scorecard developed by the Africa Region in collaboration with the Global HIVIAIDS Program and UNAIDSIWHO to facilitate country projects in harmonizing and standardizing HIVIAIDS indicators in compliance with indicators contained in the globally agreed UN General Assembly Session on HIVIAIDS, Millennium Development Goals, IDA indicators, and OECD Paris Declaration on harmonization and minimization of data requirements. 38. IDA'S Role is Key. IDA plays a key role in strengthening country-level effectiveness when it goes beyond the arrangements for project M&E to include dialogue on sector-level results frameworks and M&E tools.26When country M&E systems are sufficiently strengthened, IDA projects can use country indicators and monitoring arrangements. One example is Albania, where IDA supports activities to refine indicators and introduce results-based management and qualitative poverty monitoring through the management of a multidonor Poverty Reduction Strategy Trust Fund. Another is Honduras, where IDA has joined the Swedish government and DFID in cofinancing a technical assistance credit to help the Government improve the coordination and production of sectonvide data and strengthen the monitoring of PRS implernentati~n.~~Best-practice examples of well-functioning monitoring systems are scarce, and donor support to assist governments in improving their systems is not as well-defined and coordinated as, for example, for PFM reforms. IDA supports the strengthening of PRS monitoring systems by analyzing and disseminating lessons and good practice examples. In addition, support for country monitoring systems by IDA and other donors is an essential component of the Africa Action Plan (AAP)." 26 Participants at the 2007 Third International Roundtable for Managing for Development Results in Hanoi expressed interest in developing sector results frameworks at the country level that could harmonize donor efforts to monitor and evaluate specific interventions, replacing donors' multiple and partial monitoring systems. 27 Mackay, Keith (2007). How to Build M&E Systems to Support Better Government, Independent Evaluation Group. The AAF' involves the provision of assistance by IDA and other donors to all 29 PRS countries in Africa to develop and implement M&E plans linked to their national strategies by the end of FY08; see Meeting the Challenge of Africa's Development: A WorldBank Group Action Plan, World Bank, 2005. IV. LESSONSFROM EXPERIENCE WITH RESULTSMANAGEMENT 39. Implementation of the RMS, in conjunction with related Bank-wide efforts to carry out the results agenda, has reinforced the emerging results culture in the Bank and brought increased attention to the importance of helping IDA countries improve the supply of and demand for statistics as well as to work with partners to identify and strengthen country level systems which create demand for their use. A. Reinforcing a Results Culture 40. Knowledge and learning are important drivers of organizational change and culture. To engage staff in the results agenda, the Bank uses a wide variety of knowledge and learning activities. Formal training. In addition to integrating training on managing for results into OPCS core courses, such as "Introduction to Bank Operations" and those on project preparation and ICR preparation, OPCS has tailored activities to specific needs- sponsoring, for example, dialogue and small-group meetings with country directors; seminars on managing for results in RBCASs and investment operations; and events for country office staff. Regions are also very active in developing and delivering formal training to their staff and to government and project counterparts, helping to build staff and partner country skills to improve the quality of operations and ultimately improve project and program outcomes.29 E-learning modules. E-learning modules on ISR preparation and project preparation, with a particular emphasis on measuring, monitoring, and achieving results, have been introduced to provide more flexible training, particularly to staff in the field. Provision of one-to-one andjust-in-time advice to operational staff. Results Team members have helped Regional IDA staff develop project results frameworks, results frameworks in RBCASs and in strategies in fragile states (Liberia is a recent example), and results-oriented program reviews. Informal hands-on guidance from Regional and Network teams makes staff increasingly comfortable with capturing projectlprogram outcomes (rather than just outputs). Development of tools for staff. Tools for staff include the second edition of the Sourcebook on good practices in managing for results.30 SAR Region has been - 29 Examples of such training include a training course on "Focusing on Results in Project Preparation and Supervision" developed by Asia Learning and delivered four times in FY07 in Washington and Bangkok. In the SAR region, 15 targeted clinics were delivered to project teams at the project concept stage to strengthen the results frameworks and help them to incorporate an outcome-based approach to design and implementation. The ECA Region organized seven M&E workshops jointly with WBI in FY07, of which two were held in Washington and the other five in the field (Azerbaijan, Albania, Bosnia, Kazakhstan, and Uzbekistan). More than 220 people attended the workshops, of which nearly 150 were government counterparts. The ECA Region has also held four two-hour learning sessions on ISRs. The LCR region organized 6 Knowledge Exchange Seminars on M&E in client countries (including Bolivia and Honduras), a workshop on practical issues in M&E in Bank projects, and a seminar on current practice on CAS and CASPR results matrices. Finally, in AFR, training was delivered to more than 600 staff and 800 government officials on issues related to better management for results. 30 See htt~:l/www.mfdr.oreiSourcebook/2-2ndEditio1~.htinl particularly active and innovative in developing instruments for staff to better evaluate the impact of policy measures and project inter~entions:~'for example, a scorecard to assess the implementation status of M&E in active projects in the health, nutrition, and population sector; and a Baseline Survey Toolkit to improve the supervision of baseline surveys and more timely impact evaluations in the agriculture and rural development sector. IDA Fact Sheet. The Results Secretariat developed and disseminated the IDA Fact Sheet to inform staff about the Bank's commitments under IDA14 and about the RMS. Guidance Notes. A companion document to the IDA Fact Sheet, the Quick Reference Guide on Results, provides staff with key results guidance for each stage of the project cycle. A guidance note was also prepared on the new Guidelines for Project Restructuring, and notes are being prepared on the need for adequate baseline data in ISRs, on results frameworks in World Bank Operations, and on results in rapid response to crises and emergencies. 41. Changes in Bank Policies. The Bank has introduced changes to its policies, guidelines, and processes to increase the results focus and impact of operations, including in IDA countries. Since the IDA14 Mid-Term Review, Management has issued the following: A revised OP 13.60, Monitoring and Evaluation, which extends the traditional function of M&E to focus explicitly on results; it signals to staff that outcomes and outputs are the starting point in conceptualizing the design of operational activities and M&E. Following up on the new policy, OP/BP 8.00, Rapid Response to Crises and Emergencies, Management is preparing advice to help task teams to develop results framework that suit the uncertainties and flexibility inherent to operations addressing crises and emergencies. Management has issued guidelines on designing investment lending using an "output- based disbursement (OBD) mechanism.32Although interest in the use of OBD is usually associated with middle-income countries, a few IDA countries have begun using it-for example, Albania, in the Health System Modernization Project approved in March 2007. Country representatives at the Third International Roundtable on Managing for Development Results expressed an interest in exploring the use of performance-based management tools, including output- and outcome-based disbursement principles. 3' In recognition of the emphasis that has been placed on results, monitoring and evaluation, IEG recently presented SAR with the Good Practice Monitoring and Evaluation Award for 2007. 32 Under OBD mechanism, the bank establishes credible unit costs for specific expenditures associated with tangible deliverables (i.e., outputs) expected under a project. The Bank disburses loan proceeds against the borrower's delivered outputs, in accordance with the established unit costs. The mechanism is intended to improve project performance by making disbursements subject to tangible deliverables. 42. Extending Results Focus to AAA. These changes in results orientation have highlighted the unfilled gap in IDA'S approach to tracking the contributions of AAA to the achievement of CAS outcomes and country level effectiveness. Through more robust monitoring, it will be possible to strengthen alignment of analytic and advisory services through RBCASs, create space for analyses of the institutional constraints which impede (or facilitate) country commitment to developing and using results management systems. IDA is currently undertaking a self- assessment of AAA services. Initial findings indicate that, in some cases, there may have been insufficient attention to the promotion of joint analytic work, effective dissemination of strategies, disclosure and the adoption of more programmatic approaches to analytic and advisory services. As suggested in a recent analysis by the QAG,33work to improve AAA monitoring should reflect the diverse modalities and purposes of these services, and focus on alignment with the RBCAS, country ownership and the specificity and conduct of dissemination efforts. More work in this area would contribute to the further evolution of Tier 2 indicators. 43. Use of Impact Evaluation. A results culture is facilitated by learning about what works to produce key development outcomes and using that knowledge to inform the design of projects/programs. By comparing the outcomes of a program to a counterfactual that shows what would have happened to beneficiaries without the program, impact evaluation allows attributing observed changes in outcomes to the program being evaluated and can contribute to knowledge of how to produce key development outcomes. The use of impact evaluation is increasing in IDA countries (see Table 7). To increase capacity in IDA countries, IDA uses a learning-by-doing approach and provides cross-country venues for dialogue and networking. An example is the Africa Impact Evaluation Initiative launched by the Bank's Africa Region: its objective is to mainstream rigorous evaluation in Bank-supported operations and develop country capacity for evidence-based policymaking (Box 6 highlights the use of impact information in a program in Africa). Table 7: Ongoing Impact Evaluations in IDA countries Status AFR EAP LAC MENA SAR Total Under discussion 25 1 1 0 2 29 Evaluation Designed 24 0 5 1 1 3 1 Baseline Date Collected 11 4 1 0 6 22 Follow-up Data Collected 0 2 0 0 0 2 Analysis in Progress 5 1 1 0 2 9 Total 65 8 8 1 11 93 Source: World Bank staff estimates. 33 Quality Assurance Group. February 2007. "Country AAA Phase I1 Synthesis Report", Annual Report on Portfolio Performance, FY06. Box 6: Impact of IDA-Supported Nutrition Program in Africa The Uganda Nutrition and Early Child Development Project (NECDP), initiated in 1998 with IDA support, helped poor communities organize nutritional, child health, and early childhood services. At the time of the first round of data collection, children in both treatment and control villages were equally malnourished compared to international standards. An impact evaluation of NECDP services found that, as a result of the program, weight- for-age scores among children under the age of 12 months increased more for the treatment group than in the control group. The additional improvement among the treatment group, which can be attributed to NECDP, is equivalent to narrowing the gap with the international standard by 20 percent, or 0.16 kg. It also found evidence of positive changes in health-seeking behavior: for example, project communities adhered more closely to recommended practices for breastfeeding and the use of de-worming medicines. The Madagascar Community Nutrition I1 Program, a large-scale community-based program started in 1999 with IDA support, promotes behavioral change within the community through growth promotion activities and nutrition and child care education. Results from an impact evaluation of the program found significant effects on nutritional status, including the incidence of underweight for all age cohorts (children aged 0 to 5 years). Children living in participating communities showed improvements in moderate underweight and stunting over and above national statistics, demonstrating that the program had significant effects in terms of longer-term nutritional outcomes as well. Participating communities that started from lower initial conditions-a higher incidence of poverty and malnutrition rates-have managed to catch up over time. The evaluation findings, using monitoring data of the program, helped to increase the government's commitment to setting up a stronger M&E system. In terms of influencing policy, the heterogeneity of results, by age, helped to support policy changes, which were also influenced by a large body of international evidence that shows that focusing interventions on younger children is more cost effective. Source: World Bank. B. Improving National Statistics in IDA Countries 45. This section updates the information provided in the IDA14 Mid-Term Review Report and discusses improvements in overall country statistical capacity and IDA efforts in this area. 1. Enhancing the Demandfor Statistics 46. Improving statistical capacity is not an end in itself; improved statistical capacity will be sustained only where there is demand from country level stakeholders and agencies for the improved results and managerial capabilities that improved information flows enable. Conversely, there will be little demand for poor quality or erratic statistics. The first step in building demand, therefore, is to produce reliable statistics and to disseminate them widely to all potential users: academics, civil society, the press, and public agencies responsible for monitoring government performance and evaluating programs. The Hanoi Roundtable provided compelling evidence that country partners are keenly interested in developing and using statistics: the majority of country delegations developed action plans for strengthening systems to enable the use of information on results in their decision-making. Their action plans noted that the dollar costs of developing these systems are relatively modest in relation to total spending. Country partners see statistics as a crucial element in their ability to manage for results; they emphasize developing local-level statistics, disaggregating data by demographic characteristics and geographical location, and enhancing capacities at all levels to use data. Improved use of data has resulted from improved availability of such statistics; a notable case is the use of population census records and household survey data to spatially disaggregate poverty statistics and produce "poverty maps." In many countries, such as Malawi and Kenya, these maps have provided crucial insights into the location of the poor and have generated policy debate among decision makers. 47. Support for Country M&E Systems. An important aspect of strengthening countries' demand for statistics is developing capacity, strengthening links to public sector budget and civil service systems and strengthening institutions to monitor and evaluate the implementation of programs and projects. The Regions lead the way in working with partner countries to improve national M&E and statistical systems. In Africa, for example, 15 countries have received assistance to strengthen monitoring of their PRSS.~~IEG also works to support evaluation capacity building and has recently summarized key lessons of e~perience.~~(Box 7 provides an example of integrated support to Vietnam's national M&E system, which had positive spill-over effects on local and sectoral statistical capacities). On the international plane, strong country monitoring, evaluation, and results management systems underpin the monitoring for results and mutual accountability processes that are critical to implementing commitments under the Paris Declaration on Aid Effectiveness. Box 7: Support to Government M&E in Vietnam In 2005-6, joint donor support to the renovation of Government of Vietnam's socioeconomic planning process greatly contributed to a more participatory and results-oriented Socioeconomic Development Plan for 2006-10; this in turn served as a framework for a renewed effort on M&E at the national level, in key sectors, and in a dozen pilot provinces. IDA has played a leading role in building local capacities for the implementation of this M&E framework. In particular, it provided support to the General Statistical Office in conducting the Vietnam Household and Living Standards Surveys and 2006 Agricultural Census and analyzing data to allow closer monitoring of development impacts at the national level. At the sector level, IDA (through IDF support) supported the Ministry of Agriculture and Rural Development in developing an innovative sectoral five-year plan and its monitoring framework, which has been recognized as best practice in government-donor collaboration in this area. M&E activities under individual projects funded by IDA and other donors in Vietnam are now increasingly designed to use and thereby strengthen nascent Government M&E systems. Source: World Bank. 2. Enhancing the Quality, Coverage, and Relevance of Statistics 48. The IDA14 Mid-Term Review report discussed improvements in statistical capacity using a score that measures three aspects: statistical methodology, the frequency of data collection, and the availability of statistical outputs.36Over the period 1999to 2007, 53 out of 64 (or 83 percent) IDA countries with scores available in both years showed improvement in capacity, and just over one-quarter saw their scores increase by 20 percentage points or more (see Table 8). While the average improvement in the statistical capacity score for IDA countries is in line with the average in higher-income countries, it is worth noting that there is greater variability of change across IDA countries. Although about two-thirds of Sub-Saharan African IDA countries improved their scores, only 10 percent made substantial progress. Furthermore, 11 IDA countries where the scores declined or remained the same were all in Sub-Saharan Africa. By contrast, 13 out of 28 IDA countries in other regions increased their scores 34 Benin, Burkina Faso, CAR, Congo DR, Ethiopia, Ghana, Kenya, Malawi, Mauritania, Mozambique, Nigeria, Senegal, Sierra Leone, Tanzania, Uganda. 35 Mackay, Keith (2007), op, cit. 36 See Building Statistical Capacity to Monitor Development Progress (SecM2002-0539), October 29,2002. substantially. One reason for the lack of progress in many countries is the failure to conduct a census in the last 10 years (the recommended minimum frequency) and the relative infrequency of household surveys to collect poverty data and other social indicators. Table 8: Progress in IDA Countries in the Statistical Capacity Score IDA countries SSA IDA countries Category Number Percent Number Percent Total countries 81 39 Countries with statistical capacity score for 1999 and 2007 64 100 36 100 Countries showing improvement 53 83 25 69 Countries showing substantial improvement a 17 27 4 11 Countries showing no improvement or decrease 11 17 11 31 Source:World Bank Country Statistical Information Database. aImprovement (decrease) is-measured by positive (negative) changes in the statistical capacity score. Substantial improvements are changes larger than 20 points in the statistical capacity score. 49. Support for Building Country Statistical Capacity. Following the second International Roundtable on Managing for Development Results in Morocco in 2004, the international statistical community adopted the Marrakech Action Plan for Statistics (MAPS). This plan identified six key actions that national authorities and international agencies could undertake to improve national statistics and statistical capacity. A broad partnership is engaged in the implementation of MAPS, which is being supported by IDA through the World Bank's Development Grant Facility. PARIS21 plays a central role in the partnership. Oversight is provided by an international Advisory Board since 2005 which includes representatives from relevant agencies and national governments. Within the World Bank, an internal Statistical Capacity Building Committee, chaired by OPCS, was formed in 2005 to improve internal coordination. Key activities focus on helping developing countries to produce national strategies for the development of statistics (NSDSs) and identify resources for financing their implementation (including through IDA); establishing an International Household Survey Network; implementing an Accelerated Data Program (ADP) to improve national household survey programs and improve the accessibility and use of survey data; and helping countries participate in the 2010 round of population censuses. MAPS has produced results in IDA countries. A major achievement is the engagement of more IDA countries in processes to formulate National Strategies for the Development of Statistics. By the end of FY07, 29 IDA countries had produced a strategy for the development of statistics, and another 28 were preparing one. (Many of the remaining countries are either fragile states or small island economies, where the development of statistical capacity is often especially difficult). IDA has provided resources for implementing NSDSs in Burkina Faso, Kenya, and Tajikistan using the World Bank Statistical Capacity Financing Program (STATCAP)37lending program, and for Nigeria through a larger public sector reform credit. In some cases, such as Kenya and Tajikistan, other donors are also cofinancing the program. Many of the other countries that have produced a strategy 37 STATCAP is a horizontal adaptable program loan applicable to both IBRD and IDA countries, approved in March 2004. T o use STATCAP. IDA countries must obtain resources from their IDA allocation. are working to identify the resources they need for implementation-both from their own budgets and from donor contributions. Currently the pipeline for possible IDA support includes Indonesia, India, Kyrgyz Republic, Sri Lanka and Uganda. 16 IDA countries (of 17 countries in total) are now participating in the Accelerated Data Program to make more effective use of survey data that have already been collected and to support improvements in national survey programs. 50. Mainstreaming Data Quality and Statistical Capacity Discussions. A review of 23 IDA CASs completed in FY06-07 found that only about one-quarter included an adequate discussion of statistics; a further third included discussion of some aspect of the country's statistical programs, but without recommending specific remedial actions. Furthermore, NSDSs have not always been well linked to government and IDA planning processes. Country teams seem to recognize the necessity of including statistical capacity issues in CAS processes; for example, at a meeting in April 2007, country directors and country managers suggested that a review of weaknesses in statistics and their use in decision-making be a topic in CASs. To support this, the World Bank Development Economics Data Group (DECDG) is developing effective and targeted guidance, and statistical capacity assessment tools, for country teams to use in evaluating the capacity of a country's statistical system and for CAS reviewers to use to determine whether the CAS gives adequate attention to statistics. 3. Data Improvements at the Project Level 51. As part of the RMS, IDA committed that, where possible and appropriate, all new IDA operations starting in FY06 in health, education, water, and transport would report on strengthening country capacity to ensure adequate monitoring of key sector outcomes- measured using the under-five mortality rate (health), the primary education completion rate (education), the proportion of population with improved access to safe water sources (water), and the proportion of rural population with access to an all-season rural road (transport). This new requirement was introduced to promote improvements in country capacity to generate regular data for these fundamental development outcomes and to use IDA operations to strengthen this capacity. 52. Review of Operations. To assess progress in this area, 20 IDA Project Appraisal and Program Documents from FY07 in these four sectors were reviewed. As Table 9 shows, almost all operations now link project objectives to PRSP targets or to specific MDGs and refer to the latest data for the specific sector indicator-a clear improvement from the previous two years. Results matrices in the PADs now routinely specify data collection instruments and frequency, and responsibilities for data collection and analysis. Some also mention explicitly how the project will contribute to the IDA14 results measurement agenda and describe opportunities for strengthening national monitoring capacity. Moreover, three-quarters of the PADs elaborate on the use of the national statistical system to monitor key sector outcome indicators. Projects with a sectonvide approach continue to show stronger use of national systems and to be more likely to pay attention to systemic measurement issues-at least in part because of the way they are prepared. Table 9: StatisticalCapacity-Building Efforts in IDA Projects in Health,Education, Water and Transport Number of Percentage with Percentage using national Percentage with a work projects datafor key outcome statistical systemsfor program to support Fiscalyear reviewed indicators project M&E country systems FY05 40 60 63 45 FY06 37 57 97 59 FY07 17 94 71 53 FY05-07 94 65 78 52 Source: World Bank staff estimates. Notes: Operations reviewed were FY05-07 IDA operations in the appropriate sectors. Some subsector operations were not included, such as secondary or higher education, HIVIAIDS, or nonrural roads transport operations. 53. Areas for Improvement. The review also indicated that the diagnosis of data availability and data quality has not always been done systematically. Only about half of IDA countries with FY07 projects in each of the four sectors have data between 2000 and 2005 to monitor the trend of those key outcome indicator^.^' Some PADs have examined data sources and systems in detail (for example, Management Information Systems), while others seem to take the reliability and frequency of data for granted. At the same time, most PADs mention some development actions, but only a few provide work programs that are integrated with the project components and will lead to clear improvement in the availability and quality of key outcome indicators. PADs should include a strengthened assessment of the quality of all (or at least the major) data collection instruments for M&E. 4. Next Steps on Statistical Capacity 54. Participants at the Third International Roundtable on Managing for Development Results recognized that having available nationally developed and owned strategies for statistics for a large number of IDA countries is a significant step forward. They also recognized that further action is urgently needed if these plans are to be translated into increased capacity and better availability and use of statistics. Following the Roundtable, Management has continued discussions with both country and donor partners to identify how investments in better statistics can most effectively be scaled up. There is strong support for increasing levels of investment in statistical capacity and results management using a more coordinated system-wide approach. 55. Role of IDA. Consistent with its platform role, IDA can be an important partner in this effort. The following four explicit actions are recommended. First, review of weaknesses in statistics and their use in decision-making will be enhanced by requiring more comprehensive discussion of these issues in CASs. Options for remedial action, where appropriate, will include use of regular IDA instruments, including investment and Development policy operations, AAA and reviews of country programs. This will enhance the mainstreaming of IDA support for statistics and results management capacities within country programs and should promote closer coordination of National Statistical Development Plans and Poverty Reduction Strategies. To 38 The frequency of estimates is assumed to be one point every three years for under-five mortality rate and access to improved water sources, two points every three years for primary completion rate, and one point every six years for rural access to all- season road. facilitate this process, IDA, in partnership with PARIS 21, will develop targeted guidance and statistical assessment tools to assist country teams to evaluate the quality and capacity of a country's statistical system. 56. Second, as part of the follow up to the Third International Roundtable on Results, the Bank helped to convene representatives of the donor community, PARIS 21 and other interested parties to review options for scaling up assistance to statistical capacity during the World BankIIMF 2007 Spring Meetings. In these discussions, all parties agreed on the primacy of a country based approach to build statistical capacity, identify 'lead donors' for work in selected priority countries and accelerate efforts to develop a sector wide approach to building statistical capacity in these selected priority countries. Management will intensify its efforts on this initiative with donors and partners. To ensure that these efforts are well integrated into operational work programs, Management will encourage the Poverty Reduction and Economic Growth (PREM), Human Development (HD) and Sustainable Development (SDN) Networks to support this effort and to involve them more actively in encouraging support for statistical and results management. 57. Third, IDA will explore the potential for expanding and redesigning the existing multi- donor Trust Fund for Statistical Capacity Building to provide a mechanism to finance and co- finance as well as to supervise investments in statistical capacity and implementation of NSDSs. Fourth, A 'bulletin board' system will be developed to track the pace and results of statistical capacity improvements in IDA countries as well as information on donor support to NSDS. This step will complement efforts being undertaken through the OECD/DACYsJoint Venture on Managing for Results to develop tools to help country partners self-assess country capacity to manage for results. V. CONCLUSIONS PROPOSALS THE IDA15 RMS AND FOR 58. The introduction of a Results Measurement System (RMS)for IDA has generated a sharperfocus on results at the country level and a stronger internal results culture. IDA is the first international financial institution to introduce a results measurement system that systematically tracks both key outcome indicators and its own contributions to these outcomes. Tier 1 indicators confirm that IDA countries as a whole continue to make significant development progress: growth is increasing, poverty is declining, infrastructure development is progressing, and obstacles to business environment are falling, however, progress across IDA countries continues to remain uneven and several key human development indicators remain a cause for concern, notably in Africa. Having in place an explicit measurement system, particularly one that measures factors influencing agency effectiveness that are in the control of staff and managers, can stimulate changes in internal organizational behavior. Recognizing this, it may be appropriate to further refine the IDA15 RMS indicators and related signaling mechanisms to reinforce the focus on results and to signal other changes in IDA behavior. Proposalsfor Tier 1. The Tier 1 indicators provide an overview of the economic and social progress of IDA countries. To strengthen their usefulness, IDA should continue to advocate for improvements in the quality and frequency of data collection and reporting in IDA countries and, where appropriate, provide resources for investments in national statistical systems. Some revisions to the Tier 1 indicators are proposed to better reflect IDA'S strategic direction and challenges, particularly to develop a PEFA-based aggregate indicator to assess the quality of countries' public financial management and to review Tier 1 indicators to better reflect the Bank's increasing emphasis on the private sector. Proposals for Tier 2. In Tier 2 some areas of weakness remain and will be addressed: the quality of outcome indicators, the availability of baselines for all outcome indicators at entry in IDA projects, and the quality of output and outcome reporting in ICRs for IDA operations. Metrics for monitoring the contributions of analytic and advisory activities to country level effectiveness need to be developed. Several important modifications are therefore proposed for Tier 2 (see Box 8): 9 In addition to tracking achievement of project results at exit, the RMS will track the quality of the PDO at design, the adequacy of baselines during implementation according to a more stringent definition, and the reporting quality of outputs and outcomes in ICRs based on the original results framework. 9 The RMS will draw on recent innovations in the CAS process to improve the quality of results frameworks in CASs and rate program performance in CAS Completion Reports (CASCRs) by proposing monitoring of self-ratings in CASCRs and their independent validation by IEG. 9 Management will, based on collaborative efforts between Networks and Regions, develop lists of standard output indicators for 4-5 sectors by the time of the IDA15 Mid-Term Review. 9 During the IDA15 period Management will be working to strengthen the results orientation of analytic and advisory activities (AAA). In developing this work, the Bank will be looking into issues in promoting joint analytic work, more effective dissemination strategies, disclosure and the adoption of more programmatic approaches. A summary of the assessment of previous analyses of the AAA, current issues, future directions and implications for Tier 2 metrics will be presented at the IDA15 Mid-Term Review. 59. Goingforward, IDA in collaboration with other donors should strengthen thefocus on statistical capacity, includingfostering demand at the country level and examining financing modalities. Gaps in data at the national and sub-national levels seriously constrain country-level monitoring and local decision makers' ability to use evidence to implement policy and improve the effectiveness of programs. Ensuring the collection and use of small-area data on key socioeconomic issues is central to engaging both citizens and political leaders in poverty reduction and growth promotion. Client countries are increasingly demanding support in building national M&E systems to manage for results. IDA and other donors need to scale up investments in this area, and define financing modalities to support country effort. IDA can continue to promote a culture of results internally through signaling, targeted learning activities, revising budget processes, and integrating statistical capacity and use into discussions on country programs. It can also use its convening role to promote more cross-country learning on establishing and effectively operating systems to ensure that country leaders are able to apply results management tools. Specifically: > The review of weaknesses in statistics and their use in decision making in CASs will be enhanced by requiring more comprehensive discussion of these issues in the CAS. These issues will be followed up, when relevant, in DPOs, SILs, M A , CASPRs, CASCRs and ICRs. To facilitate this process, IDA, in partnership with PARIS 21, will develop targeted guidance and statistical assessment tools to assist country teams to evaluate the quality and capacity of a country's statistical system. > In line with the country based approach to building statistical capacity, Management intends to intensify its efforts with donors and partners to identify 'lead donors' for work in selected priority countries and accelerate efforts to develop a sector wide approach to building statistical capacity in these priority countries. To ensure that these efforts are well integrated into existing and proposed sector programs, Management will encourage the Poverty Reduction and Economic Growth (PREM), Human Development (HD) and SustainableDevelopment (SDN) Networks to support this effort and to involve them more actively in encouraging support for statistical and results management. %PManagement will explore the potential for expanding and redesigning the existing multi-donor Trust Fund for Statistical Capacity Building to provide a mechanism to primarily finance and co-finance as well as to supervise investments in statistical capacity and implementation of NSDSs. > The Bank will develop a data standards system to help IDA countries track their progress toward meeting internationally accepted norms of data coverage, frequency, and timeliness. Subscribers to the system can provide information about their data compilation and dissemination practices for posting on World Bank's Bulletin Board (similar to IMF's Dissemination Standards Bulletin Board), allowing IDA to monitor the quality of development statistics. Box 8: Proposed changes for IDA15 to Tier 2 Project Level: Quality at Entry: Management will continue to track the percentage of projects with satisfactory quality at entry. Given the important of clearly signaling the need to identify and agree arrangements for the measurement of outputs and outcomes early in the life of an operation, Management will also track the percentage of operations whose outcome indicators in the PAD (for investment lending) and in the PD (for DPO) cover all aspects of the PDO. w Quality of Supervision: Management will continue to track the percentage of ISRs with satisfactory outcome baseline data (defined as baseline availability for either one outcome or one intermediate outcome indicator). Management proposes to also track the percentage of first ISRs submitted during the fiscal year with adequate baselines for all outcome indicators. Quality at Exit: In addition to tracking examples of outputs in ICRs, Management proposes to track the percentage of ICRs that report on all outputs and outcomes from the results framework. Outputs: Management will, based on collaborative efforts between Networks and Regions, develop a list of standard output indicators for 4-5 sectors by the IDA15 Mid-Term Review. Analytical Work: A summary assessment of previous analyses of the AAA, current issues, future directions and implications for Tier 2 metrics will be presented at the IDA15 Mid-Term Review. This will focus on the results orientation of IDA'S analytical work, including joint analytic work, programmatic approaches, mechanisms to ensure that plans for AAA are widely communicated and disclosure and dissemination issues. Country Level: The RMS will draw on recent innovations in the CAS process to improve the quality of results frameworks in CASs and rate program performance in CASCRs by proposing monitoring of self-ratings in CASCRs and their independent validation by IEG. Statistical Capacity Building: The review of weaknesses in statistics and their use in decision making in CASs will be enhanced by requiring more comprehensive discussion of these issues in the CAS. These issues will be followed up, when relevant, in DPOs, SILs, AAA, CASPRs, CASCRs and ICRs. To facilitate this process, IDA, in partnership with PARIS 21, will develop targeted guidance and statistical assessment tools to assist country teams to evaluate the quality and capacity of a country's statistical system. In line with the country based approach to building statistical capacity, Management intends to intensify its efforts with donors and partners to identify 'lead donors' for work in selected priority countries and accelerate efforts to develop a sector wide approach to building statistical capacity in these priority countries. To ensure that these efforts are well integrated into operational work programs, Management will encourage the Poverty Reduction and Economic Growth (PREM), Human Development (HD) and Sustainable Development (SDN) Networks to support this effort and to involve them more actively in encouraging support for statistical and results management. w Management will explore the potential for expanding and redesigning the existing multi-donor Trust Fund for Statistical Capacity Building to provide a mechanism to finance and co-finance as well as to supervise investments in statistical capacity and implementation of NSDSs. w The Bank will develop a data standards system to help IDA countries track their progress toward meeting internationally accepted norms of data coverage, frequency, and timeliness. Subscribers to the system can provide information about their data compilation and dissemination practices for posting on World Bank's Bulletin Board (similar to IMF's Dissemination Standards Bulletin Board), allowing IDA to monitor the quality of development statistics. ANNEXA. IDA RECENTPROGRESSEXCEEDS MDG BENCHMARKRATEOF CHANGE IDA I Aggregate Benchmark rate of change' Measurement Outcome i. oeriod PA) i Countries , End Average annual! IDA part IDA to achieve NorIDA developing Indicator Units I include" 1coverageb End Stan rate of changeo. (1990- MDG targeis countries (IDA I (number) point point value value PA) PA) startpoint) (1990-2015) measurementperiod) IDA recent progress exceeds MDG benchmark rate of change . f official graduation 3. Public financial management umber of benchmar 8. Access to an all-season road i Percent of rural 1 32 i 83 i1995 nla 1 6 1 .. .. nla a Note: (..) means insufficient data, (nta) means not applicable " 80 IDA eligible countries are considered. Serbia and Montenegro is excluded. Percent of relevant population from the total relevant population in the 80 IDA-eligible countries. The average annual rate of change is calculated using exponential growth rate. ANNEXB. VARIATIONPROGRESS IN AMONG IDA COUNTRIES Number of IDA 7 f hpercentile : % of IDA countries with rate of change % of IDA countries Indicator a country level among all i countries making making progress developing : largerprogress progress measurable countries PA) 1 1. GDP per capita I 76 88 5.5 I 22 I 2. Proportion of the population below $US l/day poverty line (%) / 32 8l 3. Public financial management i 22 ............................................................................................................................................................ t............................................................... ' ...................................................... 41 Ii ...,........................................................................ 1......................................................................... 4. Formal cost required for business 1 67 53 -15.7 21 start up i 5. Time required for business start 1 67 31 UP 6. Proportion of population with ' sustainable access to an improved / 79 58 water source per 1,000 inhabitants 9. Household electrification rate I 49 90 13. Primary school completion rate 1 67 63 i 3.6 34 14. Ratio of girls to boys in primary I 69 0.9 38 I and secondary education j Source WDI 2007, IDA RMS database, and staff estimates Note: (..) means insufficient data. " Indicator "Access to an all-season road (%of rural population)" is not shown because country-level progress is not yet measurable. 7jthpercentile rate of change among all developing countries is chosen to represent a good relative progress benchmark. By definition, 25 percent of all developing countries would have performed better than this benchmark rate. ANNEXC. DATAAVAILABILITY IDA RMS TIER1INDICATORS FOR Beginning of IDA14 " IDA14 Mid-Term Review " September 2007 Indicator . GDP per capita (constant 2000 US$) 77 77 1 77 77 78 i j 76 76 76 Population below $1 a day (%) i 15 48 1 25 21 48 1 24 16 51 Public financial management 24 24 1 23 29 22 ; 23 29 22 Cost required for business start-up (% of GNI per capita) i 55 38 1 o d 66 67 61 65 o d 1 Time required for business start-up (days) f 55 39 1 o d 66 65 1 o d 67 61 Access to an improved water source (% of population) ' i 75 34 . 80 79 61 79 79 60 Fixed line and mobile phone subscribers (per 1,000people) 80 79 j 80 79 80 j; 79 74 78 Access to an all-season road (% of rural population) ! 11 10 8 1 10 8 , Household electrification rate (% of households) 18 1 21 2 ; 43 39 47 Under 5 mortality rate (per 1,000)" 80 79 1 80 80 80 ii 80 80 80 Prevalence of HIVJAIDS, (% adult population aged 15-49)' 1 53 " 44" 1 o d 63 63 ; o d 68 66 Births attended by skilled health staff (% of total) 53 52 f 61 40 64 i 63 42 66 Primary completion rate, total (% of relevant age group) ; 62 55 1 62 63 62 : . 62 66 63 Ratio of girls to boys in primary and secondary education (%) 52 52 : 63 64 54 ji 67 69 60 Source: WDI 2005, 2006,2007 and staff estimates. a 80 IDA eligible countries in total, including Serbia and Montenegro but excluding Timor-Leste. SO IDA eligible countries in total, including Timor-Leste but excluding Serbia and Montenegro. " The availability was based on prevalence of HIVIAIDS among women aged 15-24. Previous estimates were removed as either the methodology changed or the methodology only provides estimates for benchmark years. " Availability is based on the official estimates reported by WHO, UNICEF, and UNAIDS, and may under represent the availability of underlying primary data. IDA TIER2 INDICATORS:METHODOLOGY 1. This Annex summarizes the purpose, methodology, and main findings of the four reviews that Management prepared for updating some of the RMS Tier 2 indicators that are referred to in the main document text. Tier 2 indicators are designed to capture the agency effectiveness or the quality of IDA'S interventions at various stages. They are designed to reinforce the focus of IDA staff and managers on: (a) results, by highlighting trends in the quality of operational products, and (b) the use of monitoring and evaluation tools to improve IDA and country partner capacity to use evidence of results in decision-making. Although some of the Tier 2 indicators are systematically recorded in the system (QAG, IEG indicators) and regularly reviewed by Management, others are not. As on previous occasions, OPCS has collaborated with DEC on a number of reviews to assess progress on each indicator. A. Quality at Entry: Results Frameworks in IDA Operations 2. The paper prepared for the IDA14 Mid-Term Review reported on the quality of results frameworks of IDA operations from a small pilot review of selected PADS. That exercise reviewed three dimensions of results monitoring (the statement of the development objective and indicators, the linkage of the development objective with the CAS, and the institutional arrangements for monitoring and evaluation).' The review found that in three-quarters of projects, the project/program development objective (PDO) was expressed as a measurable change in the behavior or condition of the target group, and about three-quarters also contained baselines; but there was no specific link to the CAS outcome, and discussion of institutional arrangements for monitoring and evaluation was uneven. 3. Methodology and Findings. OPCS reviewed the results orientation of 26 IDA operations (21 ILs and 5 DPOs) approved between March 1 and April 30, 2007. In addition, the review also looked at two sectoral DPOs approved in May 2007. During the period, March 1- April 30, a total of 42 projects (35 ILs, 7 DPOs) were approved, with a total lending envelop of $2,555,400,000 (DPOs account for $458,000,000). The 26 projects used in this review represent an approximate sample size of two-thirds of the projects approved during the March 1- April 30 period and include a lending envelope of $1,415,800,000, or 55.4 percent of the total envelope. The two additional DPOs add a further $180 million. Representation of the regions is in line with the two-thirds sample size as projects from the 6 regions represent approximately the same percentage of sample projects as they do of total projects approved during the period. The decision to review two additional DPOYs,both approved after the other sample projects, was based on our desire to look at quality in sectoral DPOs, since the DPOs in the sample were all multi-sectoral DPOs. 4. The review assessed the following areas: (i) Is the PDO or program objective expressed as measurable changes in the behavior or condition of the target group? ' These dimensions were chosen because they are in OP 8.60, which governs DPLs, and in the guidelines for investment loans issued in 2004. (ii) Are the indicators relevant and sufficient to measure progress towards achieving the PDO? (iii) Do the indicators contain baseline values and targets? (iv) Is there a clear link between PDO and CAS outcomes? (v) Does the PADIPD analyze institutional arrangements for data collection (responsible staff / units and time frames), the capacity of the responsible agency and the cost for the results based M&E? If M&E capacity of the responsible agency is assessed to be weak, provides resources to strengthen capacity. (vi) Does the PADIPD say whether monitoring indicators in the results framework are drawn from the borrower M&E system? Both IL and DPO operations were reviewed under the same criteria but are discussed separately because the findings reflect the different nature of the instruments. We have not provided percentages for the DPO review because we feel that the small size of the sample would make percentages misleading. Investment Lending Is the PDO or program objective expressed as measurable changes in the behavior or condition of the 86% target group? Are the indicators relevant to measure PDO or program objective progress? 95% Do the indicators contain baseline values and targets? 90% Are the PDO (or program objective) and indicators linked to CAS objectives? 81% Does the PADIPD analyze institutional arrangements for data collection? If M&E capacity of the 76% responsible agency is assessed to be weak, provides resources to strengthen capacity. Does the PADPD say whether monitoring indicators in the results framework are drawn from the 62% borrower M&E system? B. Quality of Supervision:Adequate Baselines in Supervision Reports 5. Effective day-to-day management of operations and ex-post evaluation depend on good baseline data. After committing at the IDA14 Mid-Term Review that the percentage of projects with baselines in their first ISRs will be a performance indicator in staff Strategy and Performance Contracts, Management took three important measures to raise task teams' awareness about the need for baselines in the first ISR: (a) preparation of a Guidance Note on Adequate Baseline Data in ISRs; (b) updating of the Bank's core course "Supervision of Bank Financed Projects"; and (c) preparation of an e-learning module on preparing ISRs, including adequate baselines, particularly to reach staff in the field. 6. Methodology and Findings. For the IDA14 Mid-Term Review, Management assessed the availability of baseline data in first ISRs submitted during FY07, judging them adequate (as in previous reviews) if they had at least one outcome indicator with baseline data. The review found that more than 80 percent of the first ISRs reviewed had at least one outcome indicator with baseline data (see Table lo), but only 53 percent had outcome baseline data for all indicators. If we take into consideration the quality dimension,' 63 percent of ISRs included baselines on at least one outcome indicator and were also satisfactory on indicator quality, while only 38 percent of ISRs had all PDOs with baselines and satisfactory indicator quality. This The quality of outcome indicators is based on whether they cover all important aspects of the PDO and appear appropriate, i.e., neither higher-order (beyond the scope of the project) nor lower order (e.g., output indicators). means that for around 15 percent of ISRs, though they include baseline data for all PDO indicators either do not capture all aspects of the PDO or seem not to be appropriate. The picture on target values is somewhat better: about 85 percent of the ISRs reviewed have at least one PDO indicator with a target specified, and 73 percent of ISRs reviewed have targets for all PDO outcome indicators. The review also shows that any measurement system needs to guard against creating perverse incentives. While the use of qualitative data is important and teams are encouraged to identify appropriate qualitative indicators, Management strongly discourages teams from using qualitative indicators simply to avoid having to conduct a quantitative measurement of baselines. Table 10: Outcome Baseline Data in ISRs for IDA Operations CRITERIA Number Yo # of ISRs reviewed 131 # of ISRs with no PDO indicators 4 3.1% BASELINE # of ISRs with no PDO indicators baseline # of ISRs with at least one PDO indicator baseline # of ISRs with all the PDO indicators baseline 82 62.6% # of ISRs with at least one PDO indicator baseline and the indicator quality rated "S" 50 38.2% # of ISRs with all the PDO indicators baseline and the indicator quality rated "S" TARGET # of ISRs with no PDO indicators with a target specified # of ISRs with at least one PDO indicator with a target specified # of ISRs with all the PDO indicators with a target specified 89 67.9% # of ISRs with at least one PDO indicators with a target specified and the indicator quality rated "S" 76 58.0% # of ISRs with all the PDO indicators with a target specified and the indicator quality rated "S" BASELINE AND TARGET 62 47.3% # of ISRs with all the PDO indicators with a baseline and a target specified # of ISRs with all the PDO indicators with a baseline and a target specified and the indicator quality 46 35.1% rated "S" # of ISRs with targets specified without a baseline 36 27.5% - . Source: OPCS, World Bank. Note: Data for FY07 are not strictly comparable to that for earlier years because of different criteria used and some overlap of FY07 ISRs with those reviewed for FY06. C. Quality at Exit: Aggregation of Outputs in Completion Report 7. As the IDA14 Mid-Term Review report discussed, aggregating outputs presents several challenges to IDA and its partners: operational practice in defining outputs varies significantly within sectors, even in those sectors which focus on relatively more tangible goods and services; and variations across projects within a sector appropriately reflect differences in sectoral priorities and approaches across countries. Thus, in this review, it is not important to look for year-to-year trends in numbers of outputs within a single sector, but rather to ensure that each project defines expected project achievements at the project concept stage, and then measures specific achievements during supervision and at completion. Management committed at the IDA14 Mid-Term Review that the revised ICR guidelines would include a measure of project achievement^.^ Accordingly, the revised ICR guidelines issued in FY07 require ICRs to report on project outputs by component, make an overall assessment of development outcomes generated from project activities, and refer to the PDO and associated key indicators, which are linked to the original PAD'S results framework. The revised ICR Guidelines were discussed and widely disseminated. A core OPCS course on preparing ICRs, revised to reflect the new guidelines and emphasize results reporting, has been offered three times since January 2007. In addition, an e- learning module on ICR preparation was prepared to facilitate the adoption of the revised ICR Guidelines by task teams and to be responsive to the needs of field-based teams. 8. Methodology and Findings. For the sixth Tier 2 indicator, Management reports on examples of IDA contributions in the health, education, water supply, and transport sectors from projects exiting in FY06 and, when available, FY07. Data for FY06 reported at IDA14 Mid- Term Review were based on the small number of ICRs available at that time; they have been separated out and combined with new data for this report. Table 11provides a flavor of common outputs across the four key sectors from IDA operations. Figures should not be interpreted as attempting to show trends: since many ICRs still lack figures on outputs, or report them in a variety of forms (different units, no standard categories), aggregation remains very challenging, and the aggregated figures shown in Table 11 are, at best, approximate. Going forward, given that there is an increasing demand for this type of information, the Results Steering Group is working to help Regions and Networks support task teams in ensuring that future ISRs and ICRs meet additional requirements that allow a more accurate indicative aggregation of outputs in the four key sectors. New formats and guidelines for ICRs became effective in FY07. Table 11: IDA14 Results Measurement System (RMS) Mid-Term Review Outputs Table Sector Indicator FYO1-03 FY04-05 FY06-07 " Health No. of hos~italsbuilt or rehabilitated 79 97 1.393 laboratories 13,000 Water supply No. of beneficiaries from water supply or 7,545,000 4,760,000 8,000,000 & sanitation sanitation interventions No. of people that gained increase access to (...)" 2,600,000 1,500,000 services (e.g., health facilities, schools, markets) Source: Staff calculations from ICRs of IDA financed operations (does not include analytic work). (...)stands for not available aAtthe time of the review (July 31, 2007) only 13 ICRs were available from a total of 67 to be completed for projects exiting in FY07. In addition, 8 FY06 ICRs were not available. This figure includes hospitals, clinics, and health care facilities; however, it does not include blood banks, VCT centers, nutritional sites, or laboratories. 'Doesnotinclude 1.6millionIntegratedChildDevelopmentService(ICDS)workers. dTheseoutput categories tend to be very project-driven. For example, while there is no information on the number of bednets distributed, over 500 million condoms were distributed during this period. 'These output categories tend to be very project-driven. For example, while there is no information on refrigerator distribution, 435 vehicles/motorcycles/bikes were distributed. '~ o enot count educational facilities, e.g., 12percent of the reviewed ICR report on classrooms, while 65 percent of them report on general s educational facilities that include over 12,000 schools constructed as well as other 2,000 resource centers, such as libraries or laboratories. Does not include operations that report on increased ratios of teacher-pupil ratios and does not include other educational related staff such as Shiksha Mitras (over 100,000). While information on student scholarships was not available, 27,177 grants and fellowships were awarded for teaching and research. ' While no projects during the FY01-03 period reported on the number of new water connections, 548 km of water supply lines were built during this period. 'During this period, 59 dams, reservoirs, and dikes were constructed or rehabilitated. 14,687km of foot and bike paths were constructed or rehabilitated during this period. ' While no projects during the FY01-03 period reported on the number ofjobs created in the transport sector, projects did report that 1,228 professionals were trained. " During thisperiod 3,608bridges and footbridgeswere constructed and rehabilitated. " This figure does not include footbridges. " While there are no data on number of beneficiaries, projects from this period report that road safety was improved for 123,651km. D. Statistical Capacity-Building Efforts at the Project Level 9. As part of the RMS, IDA committed that, where possible and appropriate, all new IDA operations starting in FY06 in health, education, water, and transport would report on strengthening country capacity to ensure adequate monitoring of key sector outcomes- measured using the under-five mortality rate (health), the primary education completion rate (education), the proportion of population with improved access to safe water sources (water), and the proportion of rural population with access to an all-season rural road (transport). This new requirement was introduced to ensure that countries have the capacity to generate regular data for these fundamental development outcomes and to use IDA operations to strengthen this capacity. 10. Methodology and Findings. To assess progress in this area, 20 IDA Project Appraisal and Program Documents from FY07 in these four sectors were reviewed. As Table 12 shows, almost all operations now link project objectives to PRSP targets or to specific MDGs and refer to the latest data for the specific sector indicator-a clear improvement from the previous two years. Results matrices in the PADs now routinely specify data collection instruments and frequency, and responsibilities for data collection and analysis. Some also mention explicitly how the project will contribute to the IDA14 results measurement agenda and describe opportunities for strengthening national monitoring capacity. Moreover, three-quarters of the PADs elaborate on the use of the national statistical system to monitor key sector outcome indicators. Projects with a sectorwide approach continue to show stronger use of national systems and to be more likely to pay attention to systemic measurement issues-at least in part because of the way they are prepared. The review also indicated that the diagnosis of data availability and data quality has not always been done systematically. Only about half of IDA countries with FY07 projects in each of the four sectors have adequate points of estimates between 2000 and 2005 to monitor the trend of those key outcome indicators.' Some PADs have examined data sources and systems in detail (for example, Management Information Systems), while others seem to regard the reliability and frequency of data as certain. At the same time, most PADs mention some development actions, but only a few provide work programs that are integrated with the project components and will lead to clear improvement in the availability and quality of key outcome indicators. PADs should include a strengthened assessment of the quality of all (or at least the major) data collection instruments for M&E. Table 12: Statistical Capacity-Building Efforts in IDA Projects in Health, Education, Water and Transport Number of Percentage with datafor Percentage using national statistical Percentage with a workprogram Fiscalyear projects reviewed key outcome indicators systemsfor project M&E to support country systems FY05 40 60 63 45 FY06 37 57 97 59 FY07 17 94 71 53 FY05-07 94 65 78 52 Source: World Bank staff estimates. Notes: Operations reviewed were FY05-07 IDA operations in the appropriate sectors. Some subsector operations were not included, such as secondary or higher education, HIVIAIDS, or nonrural roads transport operations. The frequency of estimates is assumed to be one point every three years for under-five mortality rate and access to improved water sources, two points every three years for primary completion rate, and one point every six years for rural access to all- season road.