IDA14 IDA13 Results-Measurement System: Spring 2004 Update International Development Association May 10,2004 Table of Contents I. Introduction .......................................................................................................................... 1 I1. Summary Findings ............................................................................................................... 1 I1. IDA Inputs ........................................................................................................................... 2 I11. Country Outcomes ............................................................................................................... 4 IV. Lessons Learned................................................................................................................. 13 Annexes: Annex I: IDA13 Input and Outcome Targets........................................................................ 15 Annex 11: List of Analytical Inputs including Client and Delivery Date ............................... 17 Annex 111: Quality Assurance and Enhancement Mechanisms for ESW ................................ 22 Annex IV: Technical Note on Primary Completion Rate and Measles Immunization Coverage Rate ................................................................................ 24 Appendix A: Data Sheet for Measles Immunization Coverage Rate ...................37 Appendix B: Data Sheet for Primary Completion Rate........................................ 41 Annex V: Technical Note on Time and Cost of Business Start-up ....................................... 44 Appendix A: Data Sheet for Time and Cost of Business Start Up ....................... 50 Appendix B: List of Procedures Required to Start a Business (Country Example)......................................................................... 51 IDA13 Results-Measurement System: Spring 2004 Update I. Introduction 1. The introduction of a results-based framework into the compact between donors and recipient countries and between donors and the International Development Association (IDA) was an innovation of the IDA13 replenishment. IDA Deputies adopted an interim system to monitor results at both the country and the institutional levels during the IDA13 period and recommended that a more robust system be developed to measure results in IDA14 and beyond. 2. The IDA13 results-measurement system, which has been in place since July 2002, tracks results on two levels. On the institutional level, input indicators capture the performance of the Bank in terms of selected pieces of analytic work that underpin IDA'Sdialogue with governments about the proper use of public resources and other key aspects of development effectiveness: Country Financial Accountability Assessments (CFAAs), Country Procurement Assessment Reviews (CPARs), Public Expenditure Reviews (PERs), Investment Climate Assessments (ICAs), and Poverty Analysis (PA). On the country level, outcome indicators capture the performance of all development partners, including IDA and country governments, in achieving desired development goals in areas that are critical for growth and poverty reduction: education (primary completion rate), health (measles immunization rate) and private sector development (time and cost of business start-up). 3. Targets were established for both the input and the outcome indicators, and additional donor contributions have been linked to the achievement of these targets.2 In April 2003, Deputies assessed the first set of input targets and found progress to be satisfactory; they also reviewed preliminary estimates of progress on the outcome indicators. This paper assesses progress towards the second set of IDA13 targets which includes both input and outcome indicators. 11. Summary Findings 4. Both the input and the outcome targets for spring 2004 have been met. This result reflects progress made by the Bank in scaling up the delivery of critical analytical work in IDA countries as well as progress made by IDA countries in creating healthier investment climates, better quality pnmary schools and better health delivery systems. 5. The experience of measuring progress against the IDA13 outcome targets has brought to the fore two major issues. First, as was highlighted in the initial discussions of the results- measurement system during the IDA13 negotiations, data inadequacies such as lack of country coverage, infrequent data reporting, and routine data revision make it difficult to track straight forward indicators. This paper and the accompanying technical annexes include a rich 1 -4separate paper, IDA Results i2.leasiirerr1entSystem: Proposalsfor lDAI4, is being prepared for the July 10-11, 2004 IDA Deputies' meeting. This paper will make recommendations on the design of the IDA14 results- measurement system and the selection of outcome indicators based on further data analysis and the experience of the IDA13 interim system. See Additioiis to IDA Resoirrces: Thirteeiith Replenishriierit: Siipporririg Poverty Reduction Strategies (IDAiSecM2002-0488), September 17,2002. For a complete list of the IDA13 targets. see Annex I. discussion of these issues, and the IDA13 experience clearly demonstrates the need for more and better data in IDA countries and the need for IDA'S inbolvement in statistical capacity building. 6. Second, the experience with the outcome indicators and targets underscores the need to use caution when attributing aggregate progress to IDA. The difficulties associated with attributing changes in country-level outcome indicators, especially in the aggregate, specifically to IDA'Sinterventions were discussed by Deputies at the time the IDA13 results-measurement system was established, and have been confirmed by this experience. As the paper illustrates, part of the change observed in the outcome indicators is due to data revision and improved methodologies for calculating indicators or for estimating values for nonexistent data. Another part of the observed change is due to broader movements in economic growth and country efforts to improve policies. IDA certainly plays an important role in facilitating overall growth and in improving the quality of country policies, but to varying degrees depending on the country, on the activities of other development partners, and on developments in the global economy. While it is important for IDA Management and donors to keep a vigilant eye on the movement of key development outcome indicators across the group of IDA borrowers and to continuously test and improve the links between IDA interventions and country-specific outcomes, progress in these aggregate outcomes cannot be rightly attributed to IDA alone. 111. IDA Inputs 7. As of April 1,2004, analytic work completed for IDA countries. beginning in FYO1, includes 51 CFAAs, 42 CPARs, 42 PERs, and 16ICAS.~As such, the numerical input targets for Spring 2004 have been met. In fact, the number of CFAAs far exceeds the target. This result can be attributed to two factors: first, over the past several years, the Bank has undertaken to strengthen the fiduciary underpinnings of its programmatic and adjustment lending; and second, the financial management area has moved relatively faster in collaborating with other partners in the production of CFAAs, and this has allowed for a more efficient use of resources. In addition, in 2002, Management made a commitment to the Board to fill the gaps in the core diagnostics in all active borrowers by the end of FY04 (see Box I), and this increased management attention has contributed to achieving and surpassing the overall targets for the three core diagnostic products in IDA countries.' Table 1. Status of Analytical Inputs as of April 1,2004 Completed Targetfor Input FYOl to date Spring 2004 CFAAs 51 40 olw Africa 21 20 CPARS' 42 38 olw Africa 19 19 PERs 42 40 oiw Africa 20 20 I C A ~ 16 11 -' For a full list of CFAAs. CPARs, PERs and ICAs produced since the beginning of FYO1, see Annex 11. Core diagnostic reports are: Poverty Assessments (PAS).Country Economic Memoranda (CEMs). Public Expenditure Reviews (PERs). Country Financial Accountability Assessments (CFAAs). and Country ' Procurement Assessment Reports (CPARs) In the case of CPARs, the tally is based on the number of products delivered as of April 30, 2004. 8. In addition to the numerical input targets, the IDA13 results-measurement system includes the following goal: to underpin every CAS with current poverty analy~is,~and where this analysis is incomplete or unavailable, to identify the gaps in the CAS and to lay out a plan for their completion.' Out of the 27 IDA CASs that were presented to the Board between July, 2002 and April, 2004, 21 are underpinned by current poverty analysis, either in the form of a formal Poverty Assessment, analysis conducted by government in connection with PRSP preparation, or a more comprehensive Bank report that includes a diagnosis of the poverty situation in the country. In the remaining six CASs, thorough poverty analysis was impeded due to lack of household survey data. In each of these cases, the gap in data and analysis is identified, the CAS includes a plan for addressing the issue, and data collection or poverty analysis is currently underway. 9. Africa Share of Analytical Inputs. In addition to the overall targets for the analytical products, Management was asked to ensure that half of the targeted figures for CFAAs, CPARs and PERs be produced for African countries. As of April 22,2004,21 CFAAs, 19 CPARs and 20 PERs were delivered in the Africa Region, thereby meeting this target. 10. Enstiring the Quality of Analytical Work. Efforts to expand the coverage of analytical work should not compromise the quality of individual products or the Bank's ability to respond to individual country priorities. Regional guidelines are in place for the production of Economic and Sector Work (ESW) to ensure that the Bank delivers high-quality analytical and advisory work to its clients. The task leader, who is typically a sector specialist located in the Region, is aided by several quality-enhancement processes, including upstream support and peer reviews. Quality standards for the major ESW products (including those that form the IDA13 targets) are set and maintained by the Bank's Sector Boards. Before an ESW piece is delivered to the client. the Sector Board formally certifies that the product complies with the guidelines that have been e~tablished.~Sector Boards also provide support to the task teams as needed, drawing upon the technical expertise in the relevant network. 11. Each year the Bank's Quality Assurance Group (QAG) evaluates a random sample of ESW products along four quality dimensions: strategic relevance and timeliness; internal quality; dialogue and dissemination; and likely impact. The QAG review process strengthens the accountability of staff and managers responsible for ESW, and the associated synthesis report shared with Management and the Board enhances learning and helps to identify best practices that can catalyze changes in ESW policies, programs and procedures." In the case of ICAs, the tally is based on the number of products delivered as of April15,2004. ' Current is defined as having been produced within the last five years. ' This goal applies to IDA CASs prepared beginning in July, 2002. The Sector Board often designates the regional Board representative to assume responsibility for certification 10 See Annex I11 for more details on the Bank's quality assurance processes for ESW. Box 1: The Important Role Played by Analytic Work Country and sector knowledge is critical in reducing uncertainty and risks and in enhancing project design. thereby improving overall operational outcomes and the effectiveness of IDA'S assistance. Economic and Sector Work (ESW) has been found to have a key strategic role in shaping Country Assistance Strategies (CASs), in building the analytical base needed for the successful design and implementation of projects, and in guiding policy dialogue. Evaluation evidence from the Operations Evaluation Department (OED) suggests that high-quality and timely ESW has frequently made a substantial contribution to successful Bank assistance outcomes, and that in cases where ESW did not succeed in having a satisfactory impact on country dialogue, Bank assistance outcomes were found to be on average unsatisfactory". Research indicates that ESW has not only had a significant positive impact on various measures of quality of Bank projects, but that there was also clear evidence of underinvestment in ESW in the late 1990s.'' Over the last several years, the Bank has increased spending on and production of ESW outputs. Both spending and delivery of core diagnostic reports" doubled between FY98 and FY03. At the same time, ESW has become increasingly participatory. and its quality has notably increased across the board. The Quality Assurance Group's (QAG) last review of ESWI4rated overall quality as being satisfactory or better in 98% of the FY02 cohort considered for IDA countries and found that 93% of these ESW products were satisfactory or better in terms of the likelihood of achieving their intended impact.15 In recognition of the important role played by ESW in laying the analytical foundation for increased programmatic lending?Management undertook to strengthen coverage and update the production of analytical work in key areas underpinning the Bank's assistance to client countries. In 2002 Management set a goal for all active countries to have in place by the end of FY04 a current Poverty Assessment. Country Economic Memorandum or Development Policy Review, Public Expenditure Review, Country Financial Accountability Assessment, and Country Procurement Assessment Report, or their functional equivalents.16 In parallel. as part of the IDA13 replenishment arrangement, targets were established for the completion of CFAAs, CPARs, PERs, and poverty analysis in IDA countries. The renewed effort of the last two years has produced a stock of analytical work that is substantially more up-to-date and broader in coverage and is contributing to better designed and more relevant oDerations. IV. Country Outcomes 12. In addition to the analytical inputs, the IDA13 results-measurement system includes a set of country-level outcome indicators and progress targets for Spring 2004. The indicators measure progress in areas that are critical to growth and poverty reduction: education (primary school completion), health (measles immunization coverage), and private sector development (time and cost of business start-up). In selecting these indicators, Deputies took into account their link to development effectiveness and poverty reduction as well as the relative reliability, accessibility, and comparability of the data across countries and over time. I OED (2004). Aniiitnl Review of Dei~elopmentEffectivetiess2003: The Effectiveness of Bank Sipport,fQrPolicy *'Reforms. Deininger, Klaus. Lyn Squire, and Swati Basu. "Does Economic Analysis Improve the Quality of Foreign Assistance'?'' World Bank Economic Review, (Sep 1998). ." Core ESW diagnostic reports are: Poverty Assessments (PAS).Country Economic Memoranda (CEMs),Public Expenditure Reviews (PERs),Country Financial Accountability Assessments (CFAAs), and Country Procurement Assessment Reports (CPARs) I4 QAG (3003). Qrralih of ESW iii FY0-7: A QAG Assessmerit. 15 Defined in terms of dollar cost. l 6 See forthcoming report to the Board on the coverage of core ESW products (end-FY04). 13. Primary Completion Rate (PCR). The Primary Completion Rate was selected as an indicator for the IDA13 results-measurement system because it is the most direct measure of national progress towards universal primary education, which is one of the Millennium Development Goals. Since this indicator measures both education system coverage and student attainment, it is believed to be a more accurate indicator of human capital formation and school system quality and efficiency than either gross or net enrollment ratio~.'~The data source for this indicator is mostly the grade-specific enrollment and repeater information collected and published by the UNESCO Institute for Statistics (UIS), supplemented by completion or enrollment data collected directly from national education systems by World Bank task teams. 14. As Deputies and Management discussed during the initial development of the IDA13 results-measurement system, all outcome indicators are subject to revision as part of the normal process of compilation and validation of statistical data, and there are weaknesses associated with every indicator in terms of reliability, frequency of reporting, country coverage and comparability over time. In the case of the PCR, which is a relatively new indicator, systems for collecting and standardizing the data from IDA countries are not yet in place, and the database has many gaps, particularly for small countries and earlier years." In addition, different methods of computation have been used in the derivation of the indicator. As a result, the comparability of this indicator across countries or over time is compromised, and extensive revisions are needed in order to correct for anomalies and suspect estimate^.'^ 15. Two targets were set to gauge progress in primary school completion over a two-year period, essentially from 2O0O2O (the latest year for which data was available at the time the targets were set) to 2002 (the latest year for which data is now available).21 The targets are: (i) increase the population-weighted average primary completion rate to 69% with a substantial number of countries reaching a higher rate; and (ii) increase the number of countries with positive growth rates in primary completion rates to 38 countries. 17 Net Primary Enrollment is one of the indicators chosen to measure the MDG of universal primary completion. Primary Completion Rate has been proposed as an additional MDG indicator for the reasons mentioned above, '' but it has not yet been adopted. For the 81 IDA countries. the maximum number of countries with an observation in any single year before 2000 is 37 and there are on average 12 observations per year before 2000. ** l 9 For more details on the data issues involved in calculating PCR estimates, see Annex IV. The 2000 baseline for PCR was composed of the latest year of data available as of Spring 2002 which consisted of data points mainly around 1999/2000,but in some cases reached as far back as 1993. The target period for both the Primary Completion Rate and the Measles Immunization Coverage Rate was defined in terms of the points in time at which Deputies would consider the data. e.g. from Spring 1002 to Spring 2004, rather than in terms of the years in which the data is reported. Table 2: Primary Completion Rate (Original Methodology and Updated Data) 1 1 Target Result 1Implicit Change Actual Change from 1 from Baselinezz Revised Baselinez3 Population-Weighted Average 69% 70% 1.5% 1.7% # of Countries with Positive 6 additional countries 2 additional countries PCR Growth (Relative to 1990) I 45 38 16. As shown in Table 2, both of the IDA13 targets have been met. The population-weighted PCR for the 69 IDA countries included in the original baseline set has risen to 70%, and the number of countries showing positive growth rates in their PCRs, relative to 1990,has risen to 45. However, because the PCR data have been updated since the targets were set two years ago, the revised baseline figures for the two indicators have changed s~bstantially.~~ As such, it is useful to compare the actual progress made between 2000 and 2002 to the rate of improvement implied by the original baseline and targets. As the table shows, the rate of improvement in the weighted average PCR over the two-year period is 1.7%, which is slightly greater than the 1.5% rate of change implied by the original baseline and target. For the second indicator, the number of countries showing positive growth rates in PCR relative to 1990,the new data indicate that the change has been in the right direction, albeit somewhat lower. This can be expected given that the revised 2000 baseline (43 countries) is considerably higher than the 2000 baseline that had been estimated on the information available when the target was set (32 countries). 17. A number of factors contribute to malung it difficult to interpret change in the second indicator. One factor is the phenomenon created when countries have PCRs above Because the PCR is calculated by dividing the number of primary school completers (regardless of age) by the population within the relevant age group, the numerator can be higher than the denominator due to children completing the course late or entering early. Usually, a PCR above 100%indicates that there are a number of over-age children who have repeated one or more grades of primary school but are now graduating successfully. As the internal efficiency of the primary education system improves in these countries, their PCRs should not exceed 100%. As such, a negative growth rate in the PCR in these instances could reflect positive developments in the primary school system, and caution is needed when interpreting an aggregate change in this indicator.26 22 This figure reflects the marginal change needed to reach the target from the 2000 baseline as it was estimated when the targets were set. See Additions to IDA Resocirces: Thirteenth Replenishment: Supportirig Poverty Recliicriorz Strategies (IDA/SecM2002-0488). September 17.2002, Action Plan (Annex 111). 33 This figure reflects the actual marginal change between the revised baseline estimate for 2000 (based on updated data) and the latest data available for 2002. 24 *' Annex IV shows the revised data for 2000 - 2002 and the original baseline estimates. Five of the countries in this set have PCRs above loo%,in 1990. 26 For example. Albania's PCR rose from 101% in 1990 to 1079 in 2000 and then dropped to 100.3%in 2002, causing it to move out of the group of countries being counted towards achievement of the "Number of countries with positive growth" indicator even though the country had universal primary compietion at each of these points in time and no substantive negative change had taken place. - 7 - 18. In addition to the issue described above, another- factor makes this indicator difficult to interpret. Because progress is being measured relative to 1990levels. improvement in countries between 2000 and 2002 is not captured unless the 2002 level is higher than the 1990level. This is particularly relevant given that the IDA countries fall into two distinct groups over the 1990s, one which has experienced positive PCR growth between 1990 and 2000 and one which has experience no growth over this period. Analysis shows that the countries belonging to the first category have made impressive progress in the 1990s as well as in the 2000-2002 period, but they started from a low base (average of 42% PCR). Countries belonging to the second category experienced a decline in the 1990sfrom a much higher starting point (average of 65% PCR), but have reversed the decline decisively over the last two years. Because of the way this target was set (measuring growth since 1990),countries in this second group have not been counted as showing positive growth despite the fact that they have improved recently. This experience demonstrates the importance of talung into consideration the difference in growth potential in Primary Completion Rate among the various IDA countries when establishing a target for impr~vement.~' 19. Most-Recent-Trend Methodology. The results described above are based on the methodology that was used to calculate the original baseline in which, for cases where no data were available for a specific year, the most recent observation from a previous period was brought forward to the calculation year with no other adjustment." While this methodology is widely used and is appropriate in certain cases, especially when loolung at change over a long period, it may not be the best way to measure the short-term progress of an indicator with infrequent observations, such as the primary completion rate. A natural alternative is to take the most recent trend of the indicator value into account in imputing the missing figures. The essential assumption of this most-recent-trend method is that the trend of an indicator remains the same unless a new observation indicates otherwise.29 This approach can give a more accurate picture of progress on the ground over the short time period under examination. The results generated by this methodology are presented below. Table 3. Primary CompletionRate (Most-Recent-TrendMethodology and Updated Data) Implicit Change Actual Changefrom from Baseline Revised Baseline ~ Population-Weighted Average 69% 73% 1.5% 4% I I I ## of Countries with Positive PCR Growth Relative to 1990 43 6 additional countries 3 additional countries 20. According to the most-recent-trend methodology, progress on the aggregate PCR has been much greater over the two-year period under consideration. and the population-weighted 27 For more details on this analysis: see Annex IV. 28 The most-recent-year methodology was also used to report on preliminary first-year results in April 2003. See IDA Resrilrs Measureineiit Systeiri: Progress atid Proposals aid Tecliriical Aiirieses (IDA/SecM2003-0159). April 7, 2003. 29 Annex IV includes an in-depth discussion of the differences between the most-recent-year and the most-recent- trend methodologies. average stands considerably above the target. at 7356.'' According to this approach, the number of countries showing positive growth in their PCRs (43) is lower than the number suggested by the original methodology (43, but is still higher than the targeted figure (38). As was the case when the original methodology was applied, the actual change in the "number of countries" indicator is somewhat lower than what had been anticipated based on the assumption of a much lower starting point. Again, it is important to recognize the difficulties presented by this indicator, especially in terms of failing to capture the progress made by the majority of countries which have either surpassed their 1990value at the beginning of the measurement period and stayed above this level during the period, or have not surpassed their 1990 value at the end point but are on the way to breakthr~ugh.~~ 21. A more comprehensive way to assess progress at the individual country level is to examine the change over time in the distribution of countries according to their PCR levels. Figure 1 presents this "density analysis" for the 62 countries for which at least two data points are available beginning in 1990. This graph shows that progress has been made across the board -thenumberofcountriesbelongingtothelowest twocategories(withPCRsbelow 50%) has decreased steadily, from 12 in 1990 to 5 in 2002 for the lowest category, and from 19 to 13for the second lowest category. At the same time, the other three categories include an increasing number of countries during this period. Figure 1: Progress at country level ~ 0<30 I [30,50) ' ~ 0[50,70) 0[70,90) m>=90 , 1990 2000 2002 Primarycompletion rate (%) 22. The density analysis is also useful in examining the role played by country size when countries are grouped together to calculate a population-weighted average. The formulation of the population-weighted average targets for measles and primary completion rates includes the goal for a substantial number of countries to reach a PCR higher than 69%. While 34 countries 30 Pairwise comparison of PCRs for the years 2000, 1001and 2002 shows that these changes are not merely an artifact of the most-recent-trend methodology. For more details, see Annex IV. ? I For a more complete discussion of these issues, see Annex IV. (out of 70) reached a PCR above 69% by 2002, it may be more useful to examine how different i groups of countries have moved up the spectrum towards universal primary completion over the course of the period, as reflected inFigure 1. 23. In sum, according to both the original methodology and the most-recent-trend methodology, the improvement in the population-weighted PCR from 2000 to 2002 meets or exceeds the target, both in absolute terms and relative to the revised baseline figures. With regard to the second indicator, the number of countries with positive growth in PCR relative to 1990,the absolute target has been surpassed, but the degree of change has been less than what was envisioned based on the data available two years ago. This is due to the fact that the actual 2000 baseline is considerably higher than what had been estimated when the targets were set (and thus a smaller degree of improvement can be expected) and to the fact that the target fails to capture the progress achieved over the last two years in a substantial number of countries whose PCR levels dropped between 1990 and 2000 and, while improving, have not yet once again surpassed their 1990levels. In the future, it is recommended that the number of countries with positive PCR growth not be used to assess progress in this area for the reasons described above and elaborated more fully in Annex IV. 24. Measles Immunization Coverage Rate. The coverage rate of Measles immunization was selected as an outcome indicator for the IDA13 results-measurement system because of its relationship to child mortality and poverty and because of the high degree of data availability and the high average frequency of reporting (1 - 2 years). This indicator provides a proxy measure of the coverage and quality of the child health care system in a country, and while measles is only one cause of child mortality, it is clearly related to the Millennium Development Goal of reducing under-five mortality. The indicator is estimated annually by the World Health Organization (WHO) and the United Nations Children's Fund (UNICEF) for almost all countries. WHO/UNICEF revise previous observations when new observations estimated from survey or administrative data become available, and numerous retroactive revisions are found in the most recent update.32 25. In parallel to the targets established for Primary Completion Rate, two targets were also set to gauge progress in Measles Immunization Coverage: (i) an increase in the population- weighted coverage rate of measles immunization to 60%with a substantial number of countries reaching a higher rate; and (ii) an increase in the number of countries with 80% coverage of measles va~cination'~to 29 countries. 26. Since the time the IDA13 targets were agreed upon, WHO intensified its data collection efforts, and the time lag for the measles data has been shortened from three to two years. The availability of more recent revised data on this indicator has two implications. First, because of the methodology adopted by WHO/UNICEF, previous observations are likely to be revised when new observations estimated from surveys or administrative data become available." Second, the 3' For more details. see Annex IV 32 At the time the targets were set, 80V was believed to be the threshold belrond which measles could be contained Currently. UNICEF estimates that an immunization coberage rate of 90% is the target lebel for containment ;.l For more details on this methodoloey, see Annex IV - 10 - most recent available observations relate to the year 2002. expanding the set of relevant data to a three-year peiiod (1999 - 2002) instead of the originally envisioned two-year period (1999 - 2001).3' In order to present results that relate to the intended time frame upon which the IDA13 targets were set. it is important to assess progress over the 1999- 2001 period. Table 4. Measles Immunization CoverageRate (Updated Data) Implicit Change Actual Change Target Result Result from Revised (2001) (2002) from Baseline36 (99 -01) Baseline37 (99-01) population-Weighted Average 1 60% 61% 65% 8% 9% I I I I # of Countries with 80% 3 additional Coverage 29 27 29 2 additional countries countries3' 27. Based on the revised data available today, the population-weighted average target was met by 2001, and the rate of progress over the 1999- 2001 period was greater than that implied by the original baseline and target. For the second indicator, the number of countries with 80% coverage rates, the revised data show that the 1999 baseline had been overestimated by a substantial margin when the targets were set, and that instead of 27 countries, only 24 countries had reached this level of coverage by that point in time. As such, while the absolute target of 29 countries was not yet reached in 2001, the actual improvement that took place from the lower baseline was substantially greater than the progress that had originally been envisioned at the time the targets were established. 28. The additional year of data that is now available (2002) provides a better sense of progress to date. This data show that additional progress has been made since 2001, bringing the population-weighted average up to 65% and increasing the number of countries with 80% coverage rates to 29, the targeted figure for this indicator under IDA13. 29. The results presented above are based on the same set of 71 IDA countries that was used to form the baseline in spring 2002. Since the time that the targets were set, the data coverage of measles immunization has expanded, and data is now available for 79 IDA countries. The impact on the overall coverage rate of including the additional countries is small however -the population-weighted average for 2002 remains at 65%. The impact on the second indicator is 35 The IDA13 targets for measles immunization were set using 1999data as the baseline because that was the latest year for which data was available at that time. 36 This figure reflects the marginal change needed to reach the target from the 1999 baseline as it was estimated when the targets were set. See Arlclitioris to IDA Resoirrces: Thirteenth Replenishmeiir: Supportirig Po\jerty Reduction Strategies (IDA/SecM2002-0488), September 17. 2002. Action Plan (Annex 111). 3: This figure reflects the actual marginal change between the revised baseline estimate for 1999 (based on '' updated data) and the latest data available for 2001. The revised baseline for this indicator is 24 countries in 1999. - 1 1 - larger. increasing the total number of countries with 80% coverage rates to 31in 2002.'" As with all outcome indicators, data can be expected to undergo periodic revision as they become more precise, but the current estimates suggest a very substantial improvement in the coverage rate of measles immunization in a large majority of IDA countries since 1999. This progress is largely due to significant improvement in some countries such as India, Ethiopia and Democratic Republic of Congo, which reflects renewed effort in immunization with significant support from international partners. WHO/UNICEF estimates that through concerted efforts in 2000-2002, 220 million children were vaccinated against measles in 21 priority countries, including nine undergoing emergencies, preventing an estimated 255,000 measles deaths. 30. An additional observation can be made with regard to the "substantial number of countries reaching a higher rate [of measles immunization coverage]" which is part of the formulation of the measles target for IDA13. Of the 71 IDA countries that make up the baseline set, 50 have measles immunization coverage rates equal to or above the population-weighted average target of 60%, and 29 have rates equal to or higher than 80% in the year 2002. While this observation shows that the improvement in the measles coverage indicator has not been generated solely by a small number of densely populated countries, it is also useful to look at the change in the number of countries at different levels of immunization coverage over the three- year period. I I Figure 2: Progress at country level I I 1999 2002 Measles immunization rate (%) 31. Figure 2 shows that between 1999 and 2002 IDA countries at different levels of immunization coverage have been moving steadily up towards a sustainable level of coverage, and that progress has been made in both the low immunization rate countries as well as the high immunization rate countries. 32. In sum, according to the most recent data available, both measles immunization targets have been met, and the rate of improvement experienced by the IDA countries between 1999 and 39 See Annex IV for data covering all 79 countries. - 12- 2001 exceeds the rate of progress implied by the original baseline and targets for the two-year period. In fact, it is worth noting that estimates for future improvement in these indicators should be made with caution, given the considerable upward movement suggested by the revised data for the past few years. 33. Time and Cost of Business Start-up. Under the IDA13 results-measurement framework, two indicators were selected to measure progress in the area of private sector development: the time and formal cost required to register a new business. These indicators were chosen on the basis of their ability to reflect the general health of the investment climate, their close correlation with poor economic outcomes such as corruption and the share of the informal sector in GDP, their measurability, and their sensitivity to policy change within a short time frame. The data are produced as part of the Doing Business Project in the Private Sector Vice Presidency of the Bank, by tracking the process for a standardized hypothetical company to complete all of the necessary regulatory requirements to register a business formally. The indicators are built through a combination of desk research and expert assessment, and data are reconciled through an iterative process involving several rounds of verification and a sample of 5 - 6 respondents (usually business incorporation lawyers), each of whom deals with hundreds of business registrations per year. The data cover only the mandatory official procedures and costs and therefore exclude voluntary procedures and delays as well as informal payments." 34. For each of the indicators, a single target was set: a reduction in the population-weighted average of 7% between end-2001 and end-2003. As shown in Table 5 below, both of the targets have been exceeded by a substantial margin: the number of days required to start a business has fallen by 12% over the two-year period, and the cost to start a business has declined by 19%. Table 5. Time and Cost of Business Start-up (Updated Data) I I I j I Targeted Change End-2001 End-2003 Actual Change Days to Register a Business (Population-Weighted Average) -77c 85 75 -12% Cost to Register a Business as 35. Twenty-seven countries reported a decline of 10% or more in the time for business start- up, and over the two-year period. many countries reformed to the level of IDA "good practice" which is 30 days or less. 36. On the cost indicator, 18 countries reported a decline of 10%or more in the cost of business start-up as a percentagz of GNI per capita. Cost data has two components: cost in local currency units and GNI per capita. Growth in income per capita, especially in 2003, was a major 40 For more details on the methodology, see Annex V This figure reflects data rebisions since Spring 2003 (on both nominal local costs and on GNI per capita estimates) force behind the observed decline in cost. On average, nominal GNI per capita increased by 22.3% between end-2001 and end-200342while the nominal cost of business start-up increased on average by 7%. In fact, nominal costs of business registration (in local currency units) declined in only seven countries. 37. Because the IDA13 targets are population weighted, the larger countries contributed the most to meeting the target. On the time indicator, India, Pakistan, and Indonesia together accounted for almost 60% of the fall in population-weighted days to start a business. Ethiopia alone surpassed the cost target for the entire IDA group, accounting for an 11%decline in the population-weighted average cost, or almost three quarters of the total change in weighted cost for all IDA countries in the set. Nonetheless, the targets would still have been met if a simple arithmetic average had been used. Using a simple average, the time to start a business would have been cut by 22%, and the cost to start a business would have been reduced by 11%. 38. A variety of approaches were used by the reforming countries to reduce the time and cost of business start-up during the 2001 - 2003 period, but several patterns have emerged as examples of successful reforms. They range from simplifying procedures by reducing their number, to malung procedures more efficient through standardization and computerization, to undertaking comprehensive legal reforms. In some cases, such as in Ethiopia, reforms were supported by Bank policy dialogue and programs. Following the advice of Bank staff, in 2003, the Government of Ethiopia eliminated the requirement to publish public notices of incorporation in two newspapers. This reform cut the total cost of business start-up from 447% of GNI per capita to 78% of GNI per capita.43 39. In sum, the targets for both the time and cost of business start-up were clearly met. The inclusion of these indicators in the IDA13results-measurement framework increased Bank Management's and client governments' focus on reducing the administrative barriers to entry, and in some cases led to reforms in this area. Nevertheless, it is important to ensure that these indicators are used to generate dialogue on broader improvement of the investment climate and not to focus solely on the narrow issue of business registration. V. Lessons Learned 40. According to data available today, IDA countries have made considerable progress on all three groups of outcome indicators targeted for IDA13: primary completion rate; measles immunization rate; and the time and cost of business start-up. The targets that were established for IDA13 have been met and in some cases exceeded, but it is worth making several observations about the lessons learned through this process. 41. Measuring progress against the IDA13 outcome targets has demonstrated the difficulty of assessing progress against a given target when the baseline value needs to be periodically revised due to improved data collection. routine data revision, or when an improved methodology is used. For many indicators, unquantified measurement error combined with the need to 42 Note that 2003 GNI estimates are currently unavailable for Benin, Madagascar, Tanzania and Zambia and that tor Zirnbnbme verified GNI data is unavailable tor both 3-003-and 2003 As such. Zimbabwe has been excluded from the cost calculation. and the latest available GNI data have been used for the other countries mentioned 47 For more details on the lessons learned from the reformers, see Annex V - 14- interpolate and extrapolate low frequency data to a common year for aggregate assessment introduces considerable uncertainty in comparing values or measuring trends over short periods of time. Under these circumstances target values are likely to fall within plausible but unknown margins of error. Given the uncertainty around the estimates, a longer period than two years would improve the confidence level around interpreting change in the indicators. Comparisons over time are further confounded by changes in definitions and methods which, although intended to improve the quality of the resulting statistics, have caused breaks in the continuity of time series data. Over the long term these can be reconciled or smoothed. 42. This experience also demonstrates the need to establish clear principles (including time frame, expected rates of progress, rationale behind this expected progress, and links to policy change) behind any results-measurement system going forward. In this way, the observed changes in aggregate monitoring indicators can better contribute to a meaningful discussion about successful (or unsuccessful) government policy actions and contributions made by other development partners in order to guide future action. 43. Finally, it is clearly important to monitor progress across IDA countries along a range of outcome indicators in order to focus IDA programs on contributing to the achievement of key development outcomes. However, establishing targets at this level is not necessarily the most effective instrument for recipient countries or for the Bank, especially if the given time frame is only two years or so. While the global monitoring of key development outcomes may encourage governments to focus on important issues, borrowers are likely to respond more to changes in country-specific indicators, and their performance can be more accurately measured in terms of their policy actions today than the outcomes of policies that were implemented years earlier. With regard to the Bank, depending on the country and on the indicator, it is difficult to define the precise influence of the Bank's contribution on the final outcome given that so many other factors are involved. The forthcoming paper, IDA Results-Measurement System: Proposalsfor ZDA14, gives further consideration to the attribution issue and makes a number of recommendations in this regard for IDA14. 44. Conclusions. Both the input and the outcome targets for spring 2004 have been met. This result reflects progress made by the Bank in scaling up the delivery of critical analytical work in IDA countries as well as progress made by IDA countries in creating healthier investment climates, better quality primary schools, and better health delivery systems. The experience of collecting and reporting on the input and outcome data has been a useful one, albeit not without complications, and will provide an good foundation for informing choices about the results-measurement system for IDA14 and beyond. - 15- Annex I IDA13 Results-Measurement System Progress by Spring 2003 Inputs 0 A total of 30 Country Financial Accountability Assessment (CFAAs) completed, of which at least 50% for African countries. A total of 24 Country Procurement Assessment Review (CPARs) completed, of which at least 50percent for African countries. A total of 29 Public Expenditure Review (PERs) completed, of which at least 50% for African countries. All Country Assistance Strategies (CASs) prepared since July 2002 underpinned by current poverty analysis. A total of 7 Investment Climate Assessments (ICAs) completed. 0 Initiation of performance measurement system, including outline of approach, baseline data, outcome indicators, and progress targets. Progress by Spring 2004 A. Inputs 0 A total of 40 Country Financial Accountability Assessments completed, of which at least 50% for African countries. A total of 38 Country Procurement Assessment Reviews completed, of which at least 50% for African countries. A total of 40 Public Expenditure Reviews completed, of which at least 50% for African countries. All Country Assistance Strategies prepared since July 2002 underpinned by current poverty analysis. 0 A total of 14 Investment Climate Assessments completed. - 16- B. Country Outcomes Education Increase population-weighted average primary completion rate to 69% with a substantial number of countries reaching a higher rate. 0 Increase number of countries with positive growth rates in primary completion rates to 38 countries. Health Increase overall coverage rate (population-weighted) of measles immunization to 60%,with a substantial number of countries reaching a higher rate. 0 Increase number of countries with 80% coverage of measles vaccination to 29 countries. Private Sector Development 0 Reduce time required for business start-up (in number of business days) by 7% from end-2001. 0 Reduce formal cost of business start-up (in percent of GDP per capita) by '7% from end-2001. Annex I1 - 18 - Country Financial .\ccountahility .\sseswients Region I Country I Delivery Date IFYO1 1 FYOZ /FY03IFYOJ1 LCR OECS Countries 12/ 1212002 1 Dominica 06/25/2003 1 Guyana 0 41912002 1 Honduras 06/27/2003 1 - 19- Dominica - 20 - Serbia and Montenegro 07/22/2002 1 LCR Domnica 0 312/2004 1 Guqana 11/01/2002 1 1 Honduras 0416QOO 1 1 -21 - In\ estnient Cliniate .\ssessnients Region 1 Country I Delivery Date I FYOl 1 FY02 IFY03 IFYOI Annex I11 Quality Assurance and Enhancement Mechanismsfor ESW 1. Processing Arrangements. Individual regional managers are responsible for the quality of Country Financial Accountability Assessments (CFAAs), Country Procurement Assessment Reports (CPARs), Public Expenditure Reviews (PERs), Poverty Assessments (PAS),and Investment Climate Assessments (ICAs), and other ESW produced by the Bank. Regional operations staff-Country Directors have final sign-off authority on each report, while Regional Sector ManagersDirectors are accountable for the quality of ESW (for ESW managed and produced by Network Anchor staff, the relevant Unit Managermirector is responsible for the quality of the final output). Quality assurance is also provided by Bank-wide Sector Boards, which include Sector Managers from each of the Bank's six Regions. Sector Boards are responsible for the overall development of ESW tools as diagnostic products, the monitoring of quality, and identifying actions needed to improve product quality. Before a diagnostic ESW product is delivered to the client, the Sector Board, or its designate charged with responsibility for the product, formally certifies that adequate compliance with the guidelines that have been issued for the product. Upstream support is provided to the Sector Boards by Network Anchor units which review Concept Papers, or Initiating Concept Memoranda, provide peer review assistance and, in limited cases, participate directly in the production of specific ESW products. When necessary, Network Anchor staff also provide general advice to task teams on public financial accountability, poverty, and private sector development issues, and on the application of guidelines which govern the processing and production of each task. A description of the complementary roles of Bank units in the quality assurance process is presented in Box 1. 2. Ex-Post Assessment. Like all ESW, CFAAs, CPARs, PERs, PASand ICAs are subject to ex-post review by the Bank's Quality Assurance Group (QAG) which every year evaluates a random sample of ESW tasks along four broad criteria: strategic relevance and timeliness, internal quality, dialogue and dissemination, and likely impact. Core diagnostic ESW such as the PER and PA has historically shown exceptionally strong overall quality performance (93% satisfactory or better for ESW delivered during FYOl and FY02). A first- time QAG assessment of CFAAs and CPARs was also conducted during 2002 at the request of the Procurement and Financial Management Sector Boards, in order to learn how to best design and apply these relatively new analytical products. Following up on the recommendations from QAG, (and, in the case of CFAAs and CPARs, the financial management community's own reviews of fiduciary ESW), Sector Boards have issued guidelines which have led to changes in the contents of the reports including broader and deeper analysis ,an increased emphasis on decisions made at the concept stage and during the draft report review, and a greater level of involvement by regional quality teams. -23 - Key Participants in the Quality Enhancement Process Regioizs. Each of the six Regional Vice Presidencies has in place guidelines which govern the processing arrangements for ESW products, and detail quality assurance procedures to be followed by task teams in the preparation of these tasks. Quality enhancement mechanisms are periodically reviewed and, when necessary. updated pursuant to recommendations by the relevant Office of the Regional Chief Economist, or by Operational Policy and Country Services (OPCS) and/or QAG staff. Currently, regional quality requirements include the following key steps: Concept Review. A Concept Paper or Initiating Concept Memorandum is prepared for each task estimated to cost more than $50,000, and is circulated to internal staff (both inside and outside the Region-and in many cases also to the IFC Regional Economist and the relevant IMF Division Chief. The Concept Paper covers the context, relevance and timeliness of the task; the content, objectives and scope of the task; participatory processes to be followed during preparation of the product; the expected impact of the work; the financial and human resources needed to deliver a high-quality product, and the timetable for delivery. A Concept Paper review meeting is usually chaired by the Country Director or a designated staff from the country team in order to provide guidance to the team concerning the scope, focus, and the analytic framework of the proposed work and to resolve any particular problems affecting the implementation of the work. Decision Draft Review. A decision draft of the intended report is circulated to all recipients of the Concept Paper, and to other interested parties within the Bank for written comment. Based on the nature of the comments received, a meeting may be held to discuss the suitability and readiness of the draft report for discussion with country officials and other stakeholders subject to agreed revisions. Peer review. The key documents prepared which lead to the delivery of the final ESW product are the Concept Paper and the draft report described above. Both documents are subject to a mandatory peer- review process involving experienced staff from within the Bank and external participants selected jointly by the Regions, relevant Network Anchors, and the Development Economics unit to enhance the quality of the end product and provide the valuable input and insight of others who are not members of the assessment team. Peer reviewers are selected from within the Region, from sectors that are related to public financial management and from development partners that have an interest in the product. The peer- review process offers the task team leader a broader range of professional skills on the assessment team. Network Anchors. Network Anchors enhance the quality of ESW by supporting operational staff through the dissemination of best practice analytical work and useful tools for operational analysis. Network Anchors themselves have in place action plans which lay out specific measure for quality support to Regions; and Network Anchor staff work closely together with regional and other staff on a demand-driven basis. One example of support is the Quality Enhancement Review, which usually consists of a one-day workshop with the task team and a panel of 3-4 experts coming from inside and outside the Bank. Sector Boards. The Bank's Sector Boards serve as focal points for debating strategic, policy and technical. procedural, human resource and financial aspects of the Bank's operational work, and provide key inputs into the institutional work program. Sector Boards have prepared toolkits for CFAAs, CPARs, PERs. PAS and ICAs that are designed to help task teams produce high-quality outputs. Whereas the Sector Boards propose the operating standards that would strengthen institution-wide efforts to improve quality, the responsibility for implementing the agreed operating standards rests with the Re,'mions. Annex TV TechnicalNote for IDA13 Results-MeasurementSystem Spring 2004 Update on Primary CompletionRate and Measles Immunization Coverage Rate SummaryFindings 1. The IDA-eligible countries14are malung significant progress on both indicators according to the most recent data as of April 1,2004, both as a whole and individually: . 65% of children under age one in IDA countries have received the measles immunization vaccine, up from 56% in 1999 and greatly surpassing the target of 60%. Among seventy one countries studied,4529 have reached an immunization rate equal to or higher than 80% in 2002, up by five from 1999. = The overall primary completion rate (PCR) for the 69 IDA countries in the original baseline set has increased to 70% as of 2002, up from 69% in 2000 (based on the most-recent-year methodology). Using the most-recent-trend methodology and expanding the country set to the 70 countries with adequate data, the overall primary completion rate has increased to 73%, up from 70% in 2000. 46 Among countries with a 1990 PCR value, 45 out of 63 have seen positive growth from their 1990 level (according to the original methodology). Based on the new methodology, 43 out of 55 IDA countries with sufficient data have seen positive growth from their 1990 level. 2. In short, the spring 2004 targets set for these two indicators have been met. The following two sections discuss various data and methodological issues as well as their impact on the above results in detail for the two indicators. The appendices provide values of the two indicators by country and by year as well as the weighting variables used in the aggregation. Measles Immunization Coverage Rate 3. The proportion of one year-old children immunized against measles is the percentage of children aged 12-23 months who have received at least one dose of measles vaccine before the age of 12months. This indicator provides a measure of the coverage and the quality of the child health care system in the country. Among the vaccine-preventable diseases of childhood, measles is the leading cause of child mortality. Measles immunization is therefore an essential component for reducing under-five mortality. 14 As of April 1,2003, there are 81 countries eligible for IDA. Timor-Leste was excluded from the original set of countries because it only became independent in 3002. Therefore, much of the discussion will refer to 80 IDA countries. 45 The 71 countries studied here are the same as those included in the original baseline set for which the targets were established and estimates calculated. The most-recent-trend methodology is recommended over. the most-recent-year methodology for analyzing trends over a short time period. This analysis is based on 70 countries, representing 97% of the total population in 80 IDA-eligible countries. The countries are not the exact same as those included in the most-recent-year analysis. For details see the section on the primary completion rate. 4. This indicator is estimated annualIy by WHOiUNICEF for almost all countries. The update simply involves obtaining the most recent round of estimates--2002 at the present time.J7 However, WHO/UNICEF revises the previous observations when new observations estimated from survey or administrative data become avaiIable.@ Revisions are found for a dozen c~untries`~in the most recent update. Among the 71 countries included in the previous baseline and progress estimate, which represent 96% of children under age one in 2002, 1999 immunization rates were revised for three countries; 2000 rates for nine countries; and 2001 rates for 11countries. While most changes are minor, some can be significant. For example, the immunization rate of Indonesia was estimated to be 56% and 59% for 2000 and 2001 respectively in the spring 2003, but the most recent estimates are 71% and 76% for those two years. The immunization rates of Mozambique for 2000 and 2001 have been significantly revised downward, from over 90% to under 60%. 5. The aggregate results are given in Table 1. The first section, "Preliminary Estimates and Target", lists the targets that were adopted in the spring of 2002 and the preliminary estimates that were reported in spring 2003. The second section, "Comparable Spring 2004 Results", gives the updated results for 1999-2001and the latest estimates for 2002. These results are strictly comparable to the previous results because only the countries'@included in the "Preliminary Estimates and Target" are included here. The third section, "Full Spring 2004 Results", displays the results when all IDA countries for which data are now available are included. Notes: 1. Preliminary estimates and target are reported in: for (a) Perfi)rniaric~hlnriagrrnerit in IDA (April 2002); for (b) IDA RrAultc ,2lensitrerrierit 5ystevi. Progre.57 arid Propodr, TechnicalAririexer (April 2003). 17 The currently available data for this indicator only lags two years, one year less than when the baseline was estimated in 2002. 45 For more detail on the methodology see WHO iiacciiic-pre~,eiitnblediseases: iiioiiitoritzg sysreiii, 2003 global SI4Ttf1tlf/ I?'. 49 Three observations (Central African Republic, Mozambique Pakistan) are in 1999;nine observations are in 2000 (Angola. Central African Republic. Eritrea, Indonesia. Kenya, Lesotho. Malawi, Mozambique. Pakistan): eleven observations are in 2001 (Central African Republic, Congo Dem. Rep., Eritrea. Indonesia, Kenya, Lesotho, Mozambique. Pakistan. Rwanda, Sao Tome and Principe. Sierra Leone). 50 Macedonia is no longer an IDA country. but is included in the Comparable Spring 2004 Results because it was included in the previous 2002 and 2003 estimates. 2 Coniparable Spring 2004 Result is based on the same set of IDA eligible Countries using updated datx 3 Full Spring 2004 Result is bascd on all qualified ID.4 countries using updated data. 4. During the initial debelopment of the IDA13 restilts measurement system. only IDA countrics in the top three CP14 performance cltiintilei vme included Following the advice of Deputies. a11 1D.A eligible countries with data were included later As such. the I999 baselinc was revised from 58% to SGC/c. 6. The updated results for 1999-2001 in the "Comparable Spring 2004 Results" section demonstrates the impact of data revisions on the overall results. Despite these changes, the targets set for 2002" are met in both cases. The overall immunization rate is 65%,well above the target of 60%;29 countries have reached an immunization rate of 80% or higher by 2002. 7. The extraordinary progress on the measles immunization rate is largely due to significant improvement in some countries such as India, Ethiopia and Congo Democratic Republic, which reflects the donors' renewed effort in immunization campaigns. WHOKJNICEF estimates that through concerted efforts in 2000-2002, 220 million children were vaccinated against measles in 21 priority countries, including nine undergoing emergencies, preventing an estimated 255,000 measles deaths. 8. The increase in the number of countries reaching a measles immunization rate of 80% or higher is certainly informative, but it does not tell us whether or not the poor performing countries are malung progress if not reaching the threshold level yet. Figure 1shows that progress has been made in both the low immunization rate countries as well as the high immunization rate countries. The 71countries are grouped into four categories depending on whether their immunization rates are less than 40%, or between 40% and 60'33,or between 60% and 80%,or 80% or higher. Four countries (Chad, Congo Democratic Republic, Ethiopia and Niger) have moved out of the lowest category since 1999,and five countries (Cape Verde, Ghana, Sao Tome and Principe, Tajilustan, Tanzania) have moved into the higher category during the same period. 51 The targets were originally set for a two-year period The progiess in this three-year period IS clearly strong enough to meet the targets in both 1999-2001 or 2000-2002 periods - 27 - Figure 1: Progress at country level 1999 2002 Measles immunization rate (%) 9. The "Full Spring 2004 Result" which includes 79 IDA countries" confirms the significant progress observed in the 71countries. The additional countries were not available in the previous WHO/UNICEF database, and their measles immunization rates are likely to be revised in the future. It is clear that the impact of the additional countries on the overall coverage rate is negligible. The overall immunization rate is 64.9%,and 31 countries have reached an immunization rate of 80% or higher. Primary Completion Rate 10. Primary completion rate is the ratio of the total number of students successfully completing (or graduating from) the last year of primary school in a given year to the total number of children of official graduation age in the population. This indicator monitors both education coverage and student progression and is intended to measure human capital formation and school system quality and efficiency. It directly addresses one of eight Millennium Development Goals -achieving universal primary education. 11. This indicator is relatively new, so a regular and systematic data collection mechanism is not yet in place. The previous data compilation was undertaken by the education department of the Human Development Network (HDN) in 2002-03. The effort produced three sets of observations, one around 1990, another around 1995 and another for the most recent year (199912000in most cases). The data source is mostly the grade-specific enrollment and repeater information collected and published by the UNESCO Institute for Statistics (UIS), supplemented by completion or enrollment data collected directly from national education systems by World Bank task teams. 5 2 11\11 IDA countries but Liberia are included Liberia has no data in 1999 Macedonia is excluded. \+bile nine additional countries (Afghanistan, Kiribati, Myanmar, Papua New Guinea, Samoa. Solomon Islands, Somalia. Tonga, Vanuatu) are included - 25 - 12. There tire two major data issues. First, the database has many gaps, particularly for small countries and earlier years. For the 80 IDA-eligible countries, the maximum number of countries with an observation in any single year before 2000 is 37, and there are on average 12 observations per year before 2000. Among the 69 countries included in the previous baseline estimates, only 29 countries have an observation in 2000 and only six have an observation in 2001. Secondly, different methods of computation have been used in the derivation of the indicator. As a result, the comparability of this indicator across countries or over time is compromised". 13. The true numerator should be the number of students who actually complete the last grade of primary school, but this information is often not available. In the majority of observations in the current database, a proxy primary completion rate54is calculated as the ratio of the total number of students in the final year of primary school minus the number of students who repeat the last grade in a typical year, to the total number of the children of official graduation age in the p0pu1ation.j~Due to omission of students who drop out during the year, this proxy completion rate may overstate the true primary completion rate. In some cases when the number of repeaters was not available, the enrollment in the last grade of the primary school was used as the numerator to calculate the completion rate. In these cases, the calculated completion rate further overstates the true value. 14. Since the fall of 2003, the Development Economics Data Group (DECDG) and the education department of HDN have collaborated in collecting and standardizing the data of the three most recent years (2000-02) required for calculation of primary completion rates. To improve comparability across countries and over time, it was decided to use only the proxy method even though the number of primary school completers or graduates is available for some countries. Two factors were considered in the decision: measurability and consistency with previous observations. The numerator information required by the proxy method is routinely collected by national education ministries and reported to UIS. The cost involved in producing the proxy primary completion rate is reasonably low. Further, the proxy method was used in the majority of the available observations around 1990. For example, nearly three quarters of the available observations around 1990 for the IDA countries were estimated using the proxy method, with the rest based on enrollment only. Therefore, the upward bias is less of a concern than the actual completion rate in the measurement of progress for 2000 and forward against the 1990 level. As a result of this recent data collection effort, there is a great improvement in the 53 Another two factors complicating the international comparability are different durations of primary education cycles and different systems of graduation (exams, diplomas. automatic promotion) across countries. The primary completion rates in 2000-02 are estimated using the same cycles as for the 1990 estimates in all but a few countries. The exceptions are mostly due to change of national primary education cycle, and the 1990 values cannot be re-estimated. -=90 <30 12 3 7 2 0 0 [30,50) 19 2 5 6 3 3 [50.70) 16 0 1 8 5 2 [70,90) 5 0 0 1 3 1 >=90 10 0 0 I 0 9 29. Another limitation of the target for the number of countries with positive growth relative to 1990 is that it does not tell us whether the "performing" countries (countries with positive growth relative to 1990 value in 2000) have kept on malung progress after 2000, and it does not tell us whether the "non-performing" countries (countries with no positive growth relative to 1990 value in 2000) have made progress after 2000. It would be interesting to see whether the two groups have been following a similar growth path, both before and after 2000. 62 The seven countries whose 1990values exceed 100percent are included in Figure 2 and Table 4 - 35 - Average completion rate (not weighted) 1990 1995 1999 2000 2001 2002 If countries grouped by: 2000value>1990value 42 51 57 60 61 63 2000value<= 1990value 65 55 53 53 55 56 i If countries grouped by: 1990vahe<70% 42 47 52 53 55 199Ovalue>=70% 85 81 82 85 85 57 84 30. Table 5 strongly suggests two different growth paths for the IDA countries. The "performing" countries, as defined above, have made impressive progress in the 90s as well as in the 2000-02 period, but from a low starting point. The "non-performing" countries, on the contrary, have experienced a decline in the 90s from a much higher starting point, but reversed the decline decisively in the recent years. There are 15 "non-performing" countries64,with a primary completion rate in 1990 ranging from 34% (Rwanda) to 92% (Zambia). Three "non- performing" countries (Cambodia, Cameroon and Guyana) have surpassed their 1990 values in the 2001-02 period. Even if the two groups are defined based on whether the 1990 value is greater than 70% or not, the difference in their growth paths remains. The countries with 1990 values less than 70% have made progress continuously in the 1990s, and similar improvement in the 2000-02 period. But most countries whose 1990 values are equal to or greater than 70%65 have seen their primary completion rates fluctuating around their 1990 values since 1990. 31. This experience demonstrates that the difference in growth potential in primary completion rates among IDA countries is an important consideration in evaluating progress in this indicator. Countries with different initial levels of completion rates may face very different policy options, and the impact of similar policy inputs is likely to vary from country to country. Conclusion 32. IDA countries have made significant progress on both measles immunization coverage rate and primary completion rate by 2002. The targets for both indicators have been met. 33. One lesson that can be drawn from this experience is that the presence of clear assumptions and principles underpinning a results-measurement system can facilitate the assessment of progress, especially when data values are revised or when a different set of countries have to be included due to data availability constraints. Also, when clear links cannot be made between policy interventions and progress. we cannot say what has worked and what has not worked at the country level. 63 The seven countries whose 1990values exceed 100percent are excluded here 64 Armenia, Burundi, Cambodia. C`imeroon. Central African Republic. Congo Dem Rep Congo Rep, Ghana, , Guyana. Kenya, Nigeria. Rwanda, Sudan. Vanuatu, Zambia 63 Armenia. Georgia, Guyana, Indonesia. Nigeria, Vanuatu, Serbia and Montenegro, Zambia 36 - 34. Another lesson is that a results-measurement framework needs to be comprehensive, allowing a straightforward interpretation of progress, and intended to measure progress during a relevant period. The PCR indicator of the number of countries with positive growth relative to 1990 should be reassessed in any future results-measurement system. The change in the number of countries with positive PCR growth relative to 1990fails to capture the progress made by the majority of countries which either have surpassed their 1990 value at the beginning of the measurement period and stayed above their 1990 value during the period, or have not surpassed their 1990 value at the ending point but are on the way to breakthrough. Therefore, the benchmark of 1990 value used should be replaced with one that is comparable across countries and more relevant for the MDG target-universal primary education. References: United Nations (2003). Indicatorsfor Monitoring the Millennium Development Goals: Definition, Rationale, Concepts and Sources. Tim Holt (2003).Aggregation of National Data to Regional and Global Estimates. Committee for the Coordination of Statistical Activities, Geneva, September 1003 World Bank (2002a). Measuring Outputs and Outcomes in IDA countries World Bank (2002b). Pe$omance Management in IDA World Bank (2003a). IDA Results Measurement System: Progress and Proposals and Technical Annexes (IDNSecM2003-0159) World Bank (2003b). IDA Results Measurement System: Update Note Bmns, Mingat, Rakotomalala (2003). Achieving Universal Primary Education by 2015: A Chancefor Every Child - 37 - Appendix A ~. Measles Immunization Coverage Rates Total crude births, 2002 Country Name 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002(thousands) Include? Afghanistan 20 19 22 25 40 41 42 48 40 40 35 46 44 1274 No Albania 88 80 87 76 90 91 92 95 89 85 95 95 96 53 Yes Angola 38 39 39 47 44 46 62 78 65 46 41 72 74 619 Yes Armenia 93 95 95 96 89 92 94 92 92 93 91 31 Yes Azerbaijan 66 28 91 97 99 97 98 98 99 99 97 121 Yes Bangladesh 65 68 69 74 78 79 69 72 72 76 76 76 77 3669 Yes Benin 79 60 70 67 78 65 60 66 66 75 68 65 78 243 Yes Bhutan 93 89 86 84 81 85 85 84 71 76 76 78 78 30 Yes Bolivia 53 54 57 57 64 58 61 51 50 79 79 79 79 261 Yes Bosnia and Herzegovina 52 48 57 53 70 86 84 83 80 92 89 48 Yes Burkina Faso 79 69 60 50 45 43 40 41 46 46 46 46 46 496 Yes Burundi 74 78 70 62 43 80 79 77 76 75 75 75 75 272 Yes Cambodia 34 38 33 37 50 62 56 50 52 55 65 59 52 361 Yes Cameroon 56 48 41 40 43 46 49 52 57 62 62 62 62 559 Yes Cape Verde 79 76 82 88 83 66 66 82 66 61 80 72 85 15 Yes Central African Republic 83 62 31 41 51 46 46 46 39 37 36 35 35 134 Yes Chad 32 28 25 19 24 26 22 30 30 30 42 36 55 354 Yes Comoros 87 40 51 56 59 69 43 49 67 69 70 70 71 18 Yes Congo, Dem. Rep. 38 17 25 33 39 27 21 20 20 15 46 37 45 2234 Yes Congo, Rep. 75 64 60 55 47 38 42 18 21 23 34 35 37 152 Yes Cote d'lvoire 56 57 54 52 55 57 65 68 66 62 73 61 56 586 Yes Djibouti 85 53 41 41 42 41 41 31 21 23 50 49 62 25 Yes Dominica 91 98 99 99 92 96 99 99 98 99 99 99 98 1 No -38 - Total crude births, 2002 Country Name - 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002(thousands) Include? Eritrea 18 34 51 58 66 73 81 88 86 84 84 160 Yes Ethiopia 38 17 12 22 54 38 54 49 46 27 52 52 52 2765 Yes Gambia, The 86 87 83 87 89 91 94 92 92 88 85 90 90 51 Yes Georgia 16 61 63 61 65 69 73 73 73 73 73 47 Yes Ghana 61 63 64 66 68 70 71 73 73 73 84 81 81 588 Yes Grenada 85 99 73 99 93 88 85 92 97 94 92 96 94 2 Yes Guinea 35 42 52 55 58 61 61 56 52 52 52 52 54 289 Yes Guinea-BissaiJ 53 52 60 68 68 45 49 51 61 70 59 48 47 67 Yes Guyana 77 81 73 80 83 84 91 82 93 a7 86 92 95 17 Yes Haiti 31 35 39 44 48 49 50 52 53 54 54 53 53 263 Yes Honduras 90 86 89 94 93 89 91 99 98 98 98 95 97 200 Yes India 56 43 51 59 67 72 66 55 51 50 56 56 67 26414 Yes Indonesia 58 59 61 62 62 63 71 71 71 71 73 76 76 4538 Yes Kenya 78 81 84 84 84 83 81 79 78 76 77 78 78 1053 Yes Kiribati 75 62 77 89 70 47 64 82 77 62 80 76 88 3 No Kyrgyz Republic 94 93 88 97 98 98 98 99 98 99 98 98 Yes Lao PDR 32 47 46 46 73 68 73 67 71 71 42 50 55 195 Yes Lesotho 80 80 ao 81 81 83 a2 80 78 77 74 70 70 59 Yes Liberia 52 78 57 138 No Madagascar 47 54 54 54 63 55 46 46 46 55 55 55 61 621 Yes Malawi 81 85 91 87 83 90 90 87 90 83 73 82 69 474 Yes Maldives 96 97 98 86 97 96 95 96 98 97 99 99 99 8 Yes Mali 43 42 40 51 51 54 55 57 54 52 49 37 33 509 Yes Mauritania 38 32 43 49 53 67 66 64 62 56 62 58 81 98 Yes Moldova 92 92 95 99 98 99 99 99 87 81 94 39 Yes Mongolia 92 82 84 84 80 85 88 91 93 93 94 95 98 53 Yes - 39 - Total crude births, 2002 Country Name ___ 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 (thousands) Include? Mozambique 59 55 56 62 65 71 58 58 58 58 58 58 58 708 Yes Myanmar 90 85 84 88 77 82 86 88 85 85 84 73 75 1189 No Nepal 57 57 58 58 58 56 65 73 72 72 71 71 71 760 Yes Nicaragua 82 54 73 83 73 81 90 94 99 99 99 99 98 152 Yes Niger 25 28 21 19 19 40 38 35 35 36 34 51 48 537 Yes Nigeria 54 57 43 40 41 44 38 69 40 40 40 40 40 5076 Yes Pakistan 50 51 52 52 53 47 50 52 55 56 56 57 57 4695 Yes Papua New Guinea 67 69 70 72 73 75 46 41 59 57 68 58 71 169 No Rwanda 83 89 82 74 25 84 76 66 78 78 74 69 69 339 Yes Samoa 89 77 90 92 94 96 96 99 99 91 93 92 99 5 No Sao Tome and Principe 71 77 52 57 65 74 57 60 59 64 69 75 85 5 Yes SenegaI 51 54 57 58 59 80 70 65 62 60 48 48 54 353 Yes Sierra Leone 62 37 53 60 226 Yes Solomon Islands 70 76 70 64 60 68 90 68 64 96 87 78 78 16 No Somalia 30 30 32 33 35 34 33 25 47 38 38 36 45 445 No Sri Lanka 80 79 82 86 84 87 89 94 94 95 99 99 99 332 Yes St. Lucia 83 87 72 94 94 94 95 95 90 95 95 89 97 3 Yes St. Vincent and the Grenadines 96 99 99 99 99 99 99 99 99 87 96 98 99 2 Yes Sudan 57 57 52 49 48 51 59 58 49 53 47 67 49 1069 Yes Tajikistan 84 92 90 88 86 83 81 79 87 86 84 118 Yes Tanzania 80 79 81 77 79 78 78 73 78 72 78 83 89 1314 Yes Togo 73 69 64 60 57 53 48 43 50 57 58 58 58 173 Yes Tonga 86 90 87 87 86 94 95 97 96 97 95 93 90 3 No Uganda 52 54 56 57 59 57 55 54 53 57 56 61 77 1070 Yes Uzbekistan 84 82 71 91 95 88 96 96 99 99 97 542 Yes Vanuatu 66 63 66 74 53 60 61 65 94 94 94 94 44 6 No - 40 - _ _ _______I_ Total crude births, 2002 Country Name 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002(thousands) Include? Vietnam 85 88 90 93 96 96 96 96 96 93 97 97 96 1492 Yes Yemen, Rep 69 53 46 51 31 46 47 46 66 74 71 79 65 700 Yes Serbia and Montenegro 83 76 82 85 81 86 90 92 88 84 89 90 92 128 Yes Zambia 90 80 85 91 96 86 86 86 85 85 85 85 85 395 Yes Zimbabwe 87 87 86 86 87 87 88 84 79 79 70 68 58 380 Yes Macedonia, FYR 98 86 97 91 98 96 98 97 92 98 27 Yes Sourcc: WHO/UNICEF (http://www-nt.who.int/vaccines/globalsummai-y/ljmeseries/TScoverageMCV.htm) .-a- x 5a a 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 z o g o o 0 0 0 0 0 C 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 c oN c o ~ r 0 ~-W 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C ~ m r n ~ ~ w m r ~ ~ r n - cu n ) N - 0 - 2 m r nU- m b o- b- ~m a r nb em -m - c o nr n N Y; Y ; m a a a a a Y ; Y ; Y ; t t t t t t r c? m 6 b Lo . . . .. . . . . . . . .. . . r w - y lfj w c d w 9'4 9 k b r n m m b N N r.0 N m U J U ) N L o U J * m Rm m @ ! U J L o UJm x 2 2 b b N .. . . . . . . . . . . .. .. .. . . N L o o c d Z r n . . . . . . . . ., .. .. . . . . . . . . . . . . . . . . 7 r; T .. . . . . . . . . . . z. z. . . . . . . 7 .. . . . . .. . . . . .. . . . . . . 2 m m 7 b N . . .. :.. . . . . . . 2 r 4 N UJ Lo .. .. . . . . . . .. . . .. .. .. . . .. .. .. ,-- cj N . . . . . . . . .. .. .. .. . . . . . . . . . . . . . . . . . . . . . . . . k N m . . .. .. .. .. . . . . . . . . . . . . . . .. . . . . . . .. . . . . . . 2 N . . . . .. . . .. .. .. .. . . . . . . .. . . . . . . N Lo m .. . . . . . . .. . . . . ., . . .. . . . . . . . . . . 9 - 0 , ? " k c ? In 6Lo K $ In 7 b 2 0 - L o b G Z % % .. .. .. .. .. .. .. m : (D 0 r Lo In N . . . z. .. .. .. . .. . . . . . . . . . . . . .. .. .. .. .. .. .. . . . . . . . . . . . . . . . . . . . . .. .. m 3 Q U a, Q a S U m - 2 0 0 0 0 0 O r . 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 O f 0 0 0 0 0 0 ~ 0 0 0 0 0 0 0 ~ 0 0 0 0 0 0 r . d r b .t n o co ~ N V b c o r . 0 ~ o o m z y) z~ z~ ~-ln m c o c oo o) c o r . r . ~ ~ N~ )C L O N N m b N N ( ~ c o ~ ~ c o r . - m o m y c~No b g " T - N bm mm .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . .. . . . . . . .. . . . . . . .. .. r- 7 Lo co w r. . . . . . . . . . . . . . . . . .. . . . . .. .. .. . . . .. . . . . . .. . . . . . . . . .. .. .. 2 (D .. .. .. .. .. .. . . . . . . . . . . . . . . .. . . .. . . .. . . 2m !2co 2 T- cd N LD .. . . . . . . .. .. .. .. .. .. .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. . . . . . . .. .. .. .. .. .. .. . . . . . . . . .. . . .. . . . . .. .. .. .. .. .. .. .. .. ., .. . . . . . . . . . . . . . . . . . . . . . . -or-bmm .. . . . . . . . . . . . . .. .. .. .. .. .. .. .. ..r-.. .. .. .. .. .. .. .. .. .. .. . . . . T- 7 .. . . . . . . . . . . . . . . . . . . T- . . . . . . . . . . . . . . . . . . .. . . .. . . .. . . . . .. . . . . .. .. 2 m 9 7- 7 t. 7 lo . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. . . . . . . . . . . . . . . . . .. 7 . . . . . . 2 6 (D (D . . :.. .. .. .. .. .. . . . . . . . . . . . . . . . . . . . . . . . . .. . z . . . . . . . . . . . . . . . '9: 2:.. 2..; ? ? T - 2co $m 2 "- 2 T- (D N % N % % $ m m N -r. t o m .. .. .. .. .. .. .. .. N r- d m d d .. z . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . .. . . . . . . . . . . . . '99 coo 7- d d . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. . . . . . . . . 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 * d o 0 0 0 0 0 0 0 0 0 0 0 ~ 0 0 0 w e ~ m t . r . e ~ r - ~c ww w (D N m a o N N in i n t . N m r . - a - w w " 0 - c " - 2 r- c? --"! t. r- a w K m oi d Lo (D In ;$ L o w .. . z x . . . . . .. . . . . . . . . . . . . . . . . . . .. .. .. .. .. m (0 .. .. .. .. .. .. .. .. .. .. x . . . . . . . . .. . . . . . . .. a 2 :d in . . . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. ,-- 0 r- . . . . . . . . .. . . . . .. .. .. .. .. .. .. . . . . . . . . . . . . . . . . . . . . . . .. .. .. .. .. R 0) . . . . . . . . . . .. .. .. .. .. .. .. .. .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. $ 2 " 2 2 : 7 c9 N 10 g = in d d t. . . .. .. : .. . . : . . .. .. .. .. LD a d d m . . . . .. .. .. .. .. .. .. . . . . . . .. . . .. .. .. Ra . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. v) a, ._ c -0 m a, "- 2 2 0, .- c 17 a, a, aL rn 5 a, +- U U -0 C 0 m mS