IDA14 Measuring Results: Improving National Statistics inIDA Countries International Development Association November 2004 Table of Contents I. Background......................................................................................................................................... 1 I1. The Role of Statistics inManaging for Results .................................................................................. 2 I11. The Current Statistical Capacity in IDA Countries ............................................................................ 4 Iv. The Use of Statistics inPRSPs ........................................................................................................... 9 V. Statistical Capacity Issues inthe CAS Process................................................................................. 10 VI. A Harmonized IntemationalApproach: the Marrakech Action Planfor Statistics........................... 12 VI1. A Programto Accelerate Progress inIDA Countries ....................................................................... 13 VI11. Conclusions and Recommendations ................................................................................................. 15 Boxes: Box 1: Leadingand Misleading Statistics ................................................................................................. 3 Box 2: Indicators Selected for the IDA14 Results Measurement System .................................................. 5 Box 3: GoodPractice inCASs .................................................................................................................. 11 Box 4: A Pilot PrograminAcceleratedDataCollection......................................................................... 15 Figures: Figure 1: The Three Dimensions of the Statistical Capacity Indicator for IBRD and IDA Countries ......7 Figure 2: The Relationship of Statistical Capacity Score with GNIPer Capita ........................................ 8 Table 1: Indicators inPRSPs. Compared to Areas Measuredby Key MDGIndicators........................... Tables: 9 Table 2: Statistical Issues in Country Assistance Strategies................................................................... 11 Annexes: Annex 1: The Marrakech Action Planfor Statistics ................................................................................ 17 Annex 2: Methodology for Calculating Statistical Capacity Assessment Indicators .............................. 20 Annex 3: Methodology for Reviewing JSAs ........................................................................................... 21 Annex 4: Example for Statistical Capacity InformationSheet................................................................ 24 Annex 5: Key Statistical Processesand Issues inIDA Countries ........................................................... 25 Annex 6a: Household Surveys in IDA Countries 1999-2005................................................................... 27 Annex 6b: Description of Main Internationally Sponsored Household Surveys...................................... 29 Annex 6c: MDGIndicators Collected Through Household Surveys ....................................................... 31 Acronymsand Abbreviations CAS Country Assistance Strategy DAC Development Assistance Committee DECDG Development Economics Data Group DHS Demographic and HealthSurvey DQM Data Quality Assessment Framework GDDS General Data Dissemination System GDP Gross Domestic Product GMIS Global Monitoring InformationSystem GNI Gross National Income IBRD Intemational Bank for Reconstructionand Development IDA International Development Association IES Income and Expenditure Survey IMF International Monetary Fund JSA Joint Staff Assessment LSMS Living Standards Measurement Study MAPS Marrakech Action Plan for Statistics M&E Monitoring and Evaluation MDG MillenniumDevelopment Goals MECOVI Programfor the Improvement of Surveys and the Measurement of Living Conditions NSS National Statistical System PARIS21 Partnerships for Statistics in the 2lStCentury PRS (P) Poverty Reduction Strategy (Paper) RMS Results Measurement System SDDS Special Data Dissemination Standard STATCAP Multi-Country Statistical Capacity BuildingProgram UN United Nations UNDP United Nations Development Program UNESCO United Nations Educational, Scientific and Cultural Organization UNICEF United Nations Children's Fund USAID United States Agency for Intemational Development WHO World HealthOrganization MeasuringResults: ImprovingNationalStatisticsinIDA Countries "Whether we wish to adjust the size of ourfiscal deficit, increase social spending, pursue macroeconomic convergence in the region, or assess progress in achieving the Millennium Development Goals, accurate, timely, usefil data lie at the heart of all these efforts...,,IAchieving the right policies requires the management of trade-offs informed by good statistics. I.Background 1. Rigorous monitoring and evaluation o f outcomes, evidence-based policy making, and greater transparency and accountability are part of a broad strategy for improving development effectiveness that i s often referred to as "managing for results." Timely and reliable statistics are a key input to the results management process. The IDA Results Measurement System (RMS), endorsed by the IDA Deputies at their July 2004 meeting in Hanoi, monitors the aggregate results obtained by 81 IDA countries on a set o f 14 indicators. Monitoring and reporting on publicly agreed indicators is also a core part o f the Poverty Reduction Strategy (PRS) process and lies at the heart o f the international agreement on the Millennium Development Goals (MDGs). 2. The report on the IDA RMS2noted the need to intensify support within Country Assistance Strategies (CASs) and IDA projects for improving the statistical capacity of member countries and to work in partnership with other agencies to strengthen the intemational statistical system. "Country capacity to measure progress toward core development outcomes i s critical to country-led implementation o f PRS, and i s the foundation for global monitoring of progress toward MDGs and complementary monitoring efforts such as the IDA Results Measurement System. I t i s recommended that, in addition to continuing support for the PARIS21 consorti~m,~ IDA intensify support within its Country Assistance Strategies and projects for statistical capacity building, and work inpartnership to implement the global action plan to strengthen statistical system~."~The global action plan for improving statistics was proposed at the Second Roundtable on Measuring for Results inMarrakech, Morocco, in February 2004. Now commonly known as the Marrakech Action Plan for Statistics or MAPS, it has been widely endorsed by intemational agencies (Annex 1). 3. Because the successful implementation of the IDA R M S -or, indeed, the PRSP process and the MDGs -is limited by the capacity of countries to produce reliable indicators, an expert panel which reviewed the IDA R M S strongly supported "...the Bank's commitment to statistical capacity development and the emphasis on managing for results underpinned by empirical evidence ...." To identify shortcomings in statistical capacity, the panel recommended the development of an indicator of statistical capacity without imposing additional reporting burdens Trevor Manuel, "Data for poverty measurement," Opening address at the Joint NationalTreasury/WorldBank Workshop, Pretoria, 28 June 2004. IDA ResultsMeasurementSystem. Recommendationsfor IDA14.June 2004 http://siteresources.worldbank.org/IDA/Resources/IDAl4resultsrecommendations.pdf PARIS21(PARtnership InStatistics for development in the 21" Century) i s an internationalconsortium of users and producersof statistics. See www.paris21.org. The italics in this quotation are taken from the original report. - 2 - on countries. The panel also reviewedthe Marrakech Action Plan for Statistics and endorsed its emphasis "... on improving both the national and international components of the global statistical system." Intheir recommendations they noted ". ..the strong need for even greater coordination between international agencies and, in the context of MDG and IDA monitoring, a particular needfor a regular and coordinated approach to the various household surveys that different agencies support." 4. The rest of the paper is organized as follows. Section I1looks at the role that good statistics can play inmanaging for development results and some examples of problems caused by bad statistics. It points out the important feedback loop between the production of reliable statistics and their use for monitoring and evaluation. Section 111introduces a set of three indicators used to measure the statistical capacity of IDA countries based on readily available and comparable information. The indicators reveal differences in the strengths and weaknesses of country systems, confirming that country-based programs for capacity buildingwill be needed to produce sustainable results. The Joint Staff Assessments (JSAs), prepared by staff of the World Bank and IMF, are reviewed in section IV for further evidence of countries' statistical capacity and their ability to carry out the monitoring programs built into PovertyReduction Strategy Papers (PRSPs). Recent CASs are examinedin section V to see to what extent statistical capacity issues are being addressedinthe World Bank's business plans. The results suggest that shortcomings inthe statistical capacity of IDA countries are widely recognized, but they have not been systematically addressedin CASs. Where CASs propose statistical projects, they are often aimed at short-term data collection activities rather than long-term capacity building. 5. Section VI reviews the Marrakech Action Plan for Statistics. Because capacity building i s inherently a long-term process, MAPS calls for continued support for the development of national statistical development strategies and increased financing capacity building. However, it also recognizes that there are opportunities for makingnear-term improvementsinthe quality and timeliness of national and internationaldevelopment statistics needed to monitor the MDGs, the IDA indicators, and national development goals. Section VI1proposes a way forward through better alignment, harmonization, and coordination of major international survey programs. 11. The Role of StatisticsinManagingfor Results 6. The paradigm of development effectiveness holds that good policies based on empirical evidence and a clear understanding of the development process will lead to improved outcomes, by directing scarce resources to their most effective use and ensuringthat benefits flow to those in need. A corollary is that rigorous monitoring of programs and evaluation of results will lead to a better understanding of the development process, better policies, and further improvements inoutcomes. This feedback loop, from results to policies to programs, lies at the core of the results agenda. Statistics are the informationcarriers that make the process work. ' Methods for Monitoring the Achievements Made Towards IDA Results Indicators, Committee on Development Effectiveness, October 2004 (CODE2004-0077) - 3n - 7. Good statistics are accurate and timely. They must be collected, compiled and disseminated using reliable and impartial methods. They must be relevant to the purpose for which they are used. Moreover, they should be readily available to all who seek to use them. In Tanzania, for example, good, readily available and relevant statistics have led to better targeting of health programs, with an impact on human development (Box 1). However, good statistics are not costless. Tanzania, like most poor countries, cannot buildan adequate statistical system with its own resources. 8. Good statistics can improve the quality of decision-making and yield significant rewards. Bad statistics may result in misleading assessments, bad decisions, or poorly targeted programs, which are likely to disproportionately affect the poor. InMalawi, poor monitoring of agricultural production disguised an impending food shortage, delaying an appropriate response to the 2002 food crisis. InNicaragua, underestimates o f gross domestic product distorted other indicators, raising the potential for inappropriate policy advice (Box 1). Box 1:Leadingand MisleadingStatistics Making Plans for Success - The Tanzania Essential Health Interventions Project, International Development ' Research Centre, October 2003 Malawi - The Food Crises, the Strategic Grain Reserve, and the IMF.A Fact sheet, International Monetary Fund, July 2002 State o f Disaster. Causes, Consequences and Policy Lessons from Malawi. Devereux, S., ActionAid Malawi, June 2002 - 4 - Impact ofthe underestimationof GrossDomestic nvestment levels, sector becomes th 9. Examples such as these may be found throughout the world. Nor are poor countries the only places where the quality and availability o f statistics limit decision-making. Uncertainties over the measurement o f productivity have led to conflicting recommendations for fiscal and monetary policies in OECD countries. The realization that poor quality statistics and uncertainties over their provenance inhibit the smooth functioning of financial markets ledto the development o f the IMF's Special Data Dissemination Standard (SDDS), to which 26 high- income countries and 31middle- and low-income countries now subscribe, including 4 IDA members. However, for most poor countries, meeting the requirements of the SDDS i s far in the future. For now, they require the basic statistics to provide accountability to their citizens on the use o f resources, allow good policy formulation and monitoring, particularly within poverty reduction strategies, report on progress toward the Millennium Development Goals, and engage inthe global economy.lo 111.The CurrentStatisticalCapacityof IDA Countries 10. Statistical capacity refers to the ability to produce and disseminate reliable, relevant, and timely statistics and to analyze and use them for policy making. Many institutions and agencies are involved in national statistical systems, including the central statistical agency, monetary authorities, and line ministries, such as the ministries of finance, planning, health, and education, which are normally both producers and users o f statistics. Other important contributors to the production, analysis, and use o f statistics are businesses, academic institutions, and civil society organizations. 11. A full evaluation o f a country's statistical capacity considers all of the characteristics of good statistics and the performance of all participants in the statistical system. Such an evaluation, which must be conducted on the ground with the full cooperation of the system's managers and users, provides a detailed assessment o f the system's strengths and weaknesses. Here we adopt a more limited approach, which draws on publicly available information on statistical activities in most countries o f the world, to produce a small set of statistical capacity indicators. The resulting indicators help to identify countries with weak statistical systems and suggest where improvements are most needed. A prototype of this method was employed in the paper on statistical capacity building presentedto the Bank's Board o f Executive Directors in Improving the Poverty Focus of Public Spending. Nicaragua Public Expenditure Review, World Bank, December 200I lo Many IDA members participate in the General Data Dissemination System (GDDS) of the IMF. The GDDS encourages countries to report on existing statistical processes and adopt good practice, and covers basic economic, financial, and social statistics. - 5 - October 2002.'' Further work has been done to produce a more robust and well-founded set of indicators based on widely accepted statistical capacity frameworks, includingthe General Data Dissemination Framework o f the IMFand the Statistical Capacity BuildingIndicators o f PARIS21. Although the small set o f indicators cannot capture all dimensions o f statistical capacity, a major benefit i s that compiling them imposes no additional reporting burden on countries themselves. 12. The framework used for the indicators presented here has three dimensions: statistical practice, data collection, and indicator availability. This approach captures various aspects of data quality, which i s broadly defined as producing statistics that are fit for their purpose. In particular, it presents proxy measures for methodology, data access, timeliness, periodicity, and comparability. There are some limitations. For example, the approach does not assess the efficiency o f statistical systems, or the capacity or willingness o f countries to use the statistics they produce to make improvements in policy and management. These aspects have not been included simply because comparable and complete informationi s not readily available, but further work will be undertaken to refine the system, and to investigate what information can be obtained through existing sources (such as the IMFData Quality Reference Site). However, to provide insightsinto the capacity of countries to analyze and use statistical data, 38 Joint Staff Assessments o f PRSPs are studied in a later section o f this paper to find out what they say about PRS monitoring plans. 13. For each dimension, a country i s scored against specific criteria, usinginformation available from the World Bank, IMF,UN, UNESCO, and WHO. The scores are aggregated to give a result on a scale o f 0-100. Scores on each dimension were calculated for 1999and 2004, and for different country groupings (IBRD, IDA, Sub-Saharan African countries in IDA, and all other IDA countries). 14. The statistical practice dimension reflects a country's ability to adhere to intemationally- accepted statistical standards and methods. This i s captured by an assessment o f guidelines and procedures used to compile macroeconomic statistics, some social data reporting practices, and whether a country subscribes to the SDDS o f the IMF. A score o f 100percent means that a country meets current international standards in all areas assessed. The data collection dimension reflects whether countries conduct data collection activities with specific periodicity standards, and whether data from administrative systems are available, and used to calculate statistical data. Specific criteria include conducting population and agricultural censuses every ten years and surveys to monitor poverty and health every three or five years, and maintaining a complete vital registration system. A score o f 100percent means that countries conduct these basic censuses and surveys at regular intervals and support a complete vital registration system. The availability dimension i s evaluated against the availability and frequency o f seven IDA14 and MDGindicators, plus per capita GDP growth, as recorded in the World Development Indicators database (Box 2). This dimension attempts to measure the extent to which data are made accessible to users through the translation of source data into timely statistical outputs. A score o f 100percent means that these indicators are available with acceptable frequency on a regular basis. Annex 2 gives further details o f the scoring methodology. Building StatisticalCapacity to Monitor DevelopmentProgress, October 23,2004 - 6 - Box 2: Indicators selected for the IDA 14 Results Measurement System12 15. Average scores for groups of IDA and Il3RDcountries are shown in Figure 1. Although there i s significant variability between countries, scores for IDA countries are lower on average than IBRD countries, and no IDA country scored perfectly. This suggests that there i s room for improvement across the board. With a few exceptions, the lowest scoring countries are from Sub-SaharanAfrica. 16. Inthe areaof practice, many countries didnot reachthe mid-point score and improvements in statistical standards and methods are critically needed. Although some progress has been made since 1999,the pace of change has been slow, especially for Sub- SaharanAfrican countries who score lowest in aggregate; countries in other regions appear to have been able to adopt new methods more quickly, particularly ineconomic statistics. The practice dimension reflects the institutional and professional context inwhich statistical activities take place, and comprehensive statistical capacity buildingprograms are needed to address these deficiencies. l2Indicators initalicsarethose includedinthe availability dimensionofthe statistical capacityscore (see section 111). - 7 - Figure 1:The Three Dimensions of the Statistical Capacity Indicator for IBRD and IDA Countries: Average Score by Country Grouping 100% ~~-~ 90% 80% 70% 60% 50% 40% 30% 20% 10% 1 1 other 1 0% IDA - IBRD IDA IDA- IDA- IBRD IDA IDA- IDA- Other AFR AFR Other ~ ~ ~ ~ Practice Collection Availabiliky 1999 HChange between 1999-2004 17. Scores against the data collection dimension also demonstrate weaknesses. Countries that score lowest are those without established data collection systems, and those countries that do not benefit from externally financed surveys and international data collection initiatives. Eastern Europe and Central Asia and from South Asia have higher average scores than other regions because many countries inEurope and Central Asia have established data collection systems, including vital registration, and countries in South Asia have been able to implement regular census activities. It should be noted that some elements of data collection systems are not reflected inthe score. InEastern Europe and Central Asia, for instance, the quality o f data from vital registration systems in some countries is thought to have decreased inthe last few years. 18. International initiatives have increasedthe number of health and poverty surveys in IDA countries in recent years (for example, the Multiple Indicator Cluster Surveys sponsoredby UNICEF, the Demographic and Health Surveys sponsored by USAID, and the MECOVIlivings standards measurement project sponsored by the World Bank and the Inter-American Development Bank). Even so, the gap between IDA and IBRDcountries on data collection remained significant between 1999 and 2004, and progress in Sub-Saharan Africa was slow. In that region, the frequency o f sample survey activities has improved, but the number of countries conducting large scale surveys, such as population censuses and agricultural surveys, declined slightly. - 8 - Figure 2: The Relationship of Statistical Capacity Score with GNI Per Capita 1 100% 90% * * -. 80% ** * **** < * 70% *=. 0 a- v * * vA * **A* A 60% ** * - ** -* * 50% 0 v A A 40% 30% 20% I-- 10% - 0% -b 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 5,000 GNI per capita (US$) 19. Average scores for the availability dimension for IDA countries are comparable to IBRD countries, and have improved considerably inrecent years. This is an encouraging trend. It suggeststhat the recent emphasis on MDGand PRSPmonitoring has stimulated the estimation and dissemination ofkey indicators. To a degree, it also reflects recent efforts of international organizations to locate and utilize data from existing data sources. Insome cases, modeling techniques have beenusedto improve data quality and fill indata gaps. For example, child mortality rates calculated from survey data normally refer to a five-year period, and the calculation of estimates for a single year can only be produced from a model. However, these techniques may mask the lack of recent data and lead to the use of estimates that are very uncertain.l3This underscores the importance o f taking a long-term approach to building capacity indata collectionat the country level. 20. Figure 2 shows the relationshipbetweenthe average score ofthe three dimensions for each country, and per capita gross national income (GNI), for IDA and IBRDcountries with less that $5,000 per capita annual GNI. The chart provides the important insight that higher income does not guarantee better statistical capacity, and that a relatively high level o f statistical capacity can be attained ina low-income environment. l3For example, estimates of under-5 mortalityratesfor the Republicof Congo have beenmade by UNICEFiWHO for the current decade, yet the most recent reliable data on mortalitycomes from the 1974 census. - 9 - IV. The Use of StatisticsinPRSPs 21. Statistics are only of benefit when they are used. Although it i s difficult to assess the extent to which statistics are used in malung policy in IDA countries without a detailed evaluation o f country-level monitoring and evaluation mechanisms, it is possible to review the demand arising from the results-focused approach taken in their poverty reduction strategy papers. 22. Through their PRSPs, countries establish goals for their development programs, propose strategies for achieving those goals, and identify a set o f statistical indicators to monitor progress toward time-bound, quantified targets. Most countries select targets and indicators consistent with the Millennium Development Goals, but many also choose to collect andmonitor indicators related to growth, macro-economic stability, governance and infrastructure.l4Of the indicators included in the IDA Results Measurement System, all countries have included some measure o f income poverty, school attendance, maternal health, and economic growth, with more than 75 percent of countries additionally selecting indicators o f gender balance, child mortality and immunization, access to water, macro-economic stability and road transport (Table 1). Table 1:IndicatorsinPRSPs,Comparedto Areas Measuredby Key MDGIndicators Countries including at least one indicator: Areas covered AII Income poverty, school attendance, maternalhealth, economic growth More than 75 percent Gender balance in education, child mortality, child immunization, access to water, macro-economic stability, road transport More than 50 percent Child malnutrition, school completion, literacy, access to sanitation Less than 50 percent Gender balance in employment 23. Just as PRSPs are intendedto be country-owned development strategies, their monitoring plans should be based on country-owned statistics. However, PRSPs are often ambitious and imply highlevels of statistical capacity. Inmany cases, data requirements exceed the capacity of statistical systems to deliver timely and good quality data. Incountries with weak statistical capacity, this casts doubt on their ability to implement their poverty reduction strategies. However, in some countries, such as Vietnam, Ethiopia, and Tanzania, the data demands o f PRSPs have led to efforts to improve the capacity o f statistical services. 24. Joint Staff Assessments o f PRSPs, conducted by Bank and Fund staff, review the soundness of the PRSP as a basis for concessional assistance. They also provide constructive feedback to countries on the deficiencies o f the PRSP and its monitoring plan, and make suggestions for improvement. Most JSAs provide some commentary on statistical capacity. To 14 Are Poverty ReductionStrategies Undercutting the Millennium DevelopmentGoals?An Emuirical Review (preliminary draft). Makiko Harrison,Jeni Klugman, Eric Swanson, World Bank, April 2004 - 10- assess the adequacy o f statistical systems for monitoring each PRS, JSAs for 38 PRSPs" have been analyzed. The analysis provides additional insightsinto the strengths and weaknesses o f statistical systems in PRSP countries, and in particular adds information about the capacity to use data. 25. The method used for the analysis i s explained in detail in Annex 3. The conclusions are broadly consistent with the analysis o f statistical capacity in the previous section. Almost half o f all PRSPs are constrained by poorly organized statistical systems and more than half are constrained by weak data collection programs. All JSAs express some concerns about the choice o f indicators in PRSPs, and about a third consider the choice to be a significant constraint. In general, the capacity o f the country to conduct in-depth policy analysis and evaluation i s also judged a serious limitation. These findings are reinforced by the recent Operations Evaluation Department evaluation o f the Poverty Reduction Strategy initiative, which concludes "The development o f country-specific indicators and monitoring systems to track them i s at a preliminary stage in most countries."16 V. StatisticalCapacity Issues inthe CAS Process 26. CASs summarize the World Bank's business plans for client countries. Buildingon the country's development plan, the CAS identifies opportunities for and obstacles to Bank assistance and investment programs. For IDA countries, the CAS i s normally based on the Poverty Reduction Strategy. Recognizing the need to strengthen the results focus of CASs, the World Bank initiated a pilot program for results-based CASs, which start with the long-term objectives o f the government to determine the best project portfolio and place greater emphasis on outcomes. The measurement o f development results and monitoring o f progress are an essential component o f a results-based CAS. 27. A review of 50 CASs17for IDA countries approved since 2000 shows that about two- thirds recognize that countries lack adequate statistics or that existing statistics are not reliable or comparable over time (Table 2). These deficiencies directly affect both the ability of the Bank to offer meaningful policy advice and the ability o f countries to monitor and evaluate their own development programs. There has been little change inthe rate o f diagnosis over time. As of July 31,2004. Most JSAs are of full PRSPs, while some are of PRSPAnnual ProgressReports.For one country (Bangladesh), the JSA is based on an interimPRSP. l6 The Poverty Reduction Strategy Initiative. An IndependentEvaluationof the World Bank's Support Through 2003, World Bank OperationsEvaluationDepartment, July 2004 Out of fifty reviewed documents, three are InterimCAS (Comoros, Cote d'Ivoire,Nigeria). - 11- Table 2: Statistical Issues in Country Assistance Strategies Approved between 28. One half of the CASs propose new data collection activities. While data collection i s an important function of statistical agencies, there i s a risk that ad-hoc efforts based on specialized surveys will not contribute to sustainable improvements in the statistical system and m a y even reduce the capacity o f the system by diverting resources from other core activities. Less than a half of all CASs propose broad-based statistical capacity buildingefforts, although this has been increasing over time. 29. The five pilot results-based CASs for IDA countries provide a more thoughtful consideration of system-wide needs across statistical sectors and long-term nature capacity building. These CASs put less emphasis on data collection activities and greater emphasis on capacity building. In general, they pay more attention to data and statistical capacity constraints than do past CASs. For example, a results-based CAS provides a more careful review of the availability and quality of statistics in the country. Insome cases, however, this interest is directed principally at data issues related to the monitoringand evaluation requirements of the Bank's assistance program, rather than a more broad-based concern for the country's own monitoring, evaluation, and management activities. Box 3: Good Practice in CASs 30. With the increasing attention paidto results and the need for better statistics, it i s likely that future CASs will increasingly need to discuss statistical capacity issues and may include specific proposals to address defi~iencies.'~The statistical capacity indicators previously l8 The five pilot CASs included in this review are: Armenia, Cameroon, Mozambique, Sri Lanka and Zambia. l 9 "Specifically, the CAS should highlight the weaknesses o f the M&E capacity in government (which can be based on the JSA of the PRSP or from more in-depth analytical work such as a readiness assessment), and how the Bank or other partners are supporting capacity building. This capacity building may include strengthening - 12- discussed and the underlying information system on which they are based are a useful tool to assist with this diagnosis and planning work. They could also be usedas inputsto development planningand policy making at national level (for example in a country's PRSP) and by other international agencies (for example, as part of the U N ' s UNDAF/CCA process). To simplify statistical capacity assessments and to deepen the informationbase, we plan to work with the UN Statistics Division, PARIS21, and other partners to pool informationon country statistical systems and to make them available on-line for development planners, policy makers, and analysts. As a starting point, there i s already a preliminarytool available to produce the information sheet shown in Annex 4. VI. A Harmonized International Approach: the MarrakechAction Planfor Statistics 31. The FirstRoundTable on Better Measurement, Monitoring and Managingfor Results (Washington DC, June 2002) called for a better approach to statistical capacity buildingby development agencies. The Development Committee subsequently requested a time-bound, costed action plan for improving statistics for measuring development outcomes. As a result, the MarrakechAction Plan for Statistics was presented and agreed at the Second Round Table on Managingfor Development Results (Marrakech, February 2004, Annex 1). This plan aims to develop national statistical capacity, to improve the international statistical system, andto make improvements inthe key areas of population censuses, householdsurveys, and indicators for the MDGs. Itbuilds on other initiatives such as PARIS21andSTATCAP, a new Bank lending instrument to support statistical capacity building inmember countries. 32. Country-driven strategic statistical planning, based on principles of comprehensiveness and user-focus, has proved an effective tool to develop sustainable and prioritized capacity buildingprograms. It is the recommended approach for IDA countries. Inall countries, and particularly the poorest, a sustainable level of statistical capacity will be reached through the implementationof comprehensive plans, prioritizedthrough a careful and realistic assessment of user needs and the demand for data, and existing capacity in statistics and the public sector as a whole. Likely scenarios for the level of resources, and political realities, need to be taken into account. MAPS recommends that statistical capacity buildingprograms shouldbe based on national statistical development strategies, and that the international community should assist all countries that wish to adopt this approach. MAPS sets a specific target of assisting all low- income countries to develop these plans by 2006. 33. Progress i s beingmade in implementingMAPS. Many countries are developing national strategies for statistical development, assistedby the development community through PARIS21 and through the Trust Fundfor Statistical Capacity Building managed by the Bank (which has so far provided support for 22 strategic plans). STATCAP was approved by the Board inMarch 2004. Three countries are currently talung advantage of the new STATCAP program (Ukraine: $32 million, BurlunaFaso: $10 million, and Kenya: $20 million), and discussions with other countries are ongoing. Seven other STATCAP projects, with a total loan or credit amount of more than $100 million, are in various stages of preparation, representing a major increase in the the country's statistical base thus informing a statistical master plan." Country Assistance Strategies: Retrosuective and Future Directions, Committee on DevelopmentEffectiveness, Operational Policy and Country Services, World Bank, March 2003. - 13 - fundingfor statistical capacity. Furthermore, some ten percent of all active World Bank projects includea minor component aimed at improvement of statistics, although their impact on the wider statistical system i s limited. 34. Progress i s also being made on other elements o f MAPS. Activities to prepare for the 2010 global census round have been started by the UnitedNations Statistics Division, and the Bank i s helping formulate proposals for funding. The Bank and other international partners have established a new International Household Survey Network (IHSN), which aims to better coordinate internationally-sponsored surveys and provide better access to survey results and information. The Bank i s performing the secretariat function for the IHSN. A key focus o f the IHSN's work is the expansion of a survey data archive, developed at the World Bank, which now contains records from over a thousand household surveys that were not previously easily accessible. A new tool to assist countries in archiving and disseminating their survey results i s also under development and will be made available free of charge to all IDA countries. The UnitedNations Statistics Division i s also developing a new set o f principles and practices to improve coordination and accountability in the international statistical system. 35. Although much has been achieved, more needs to be done by the international community and national governments. The expert panel convened by the Bank to review the IDA Results Measurement Systemwelcomed the strong emphasis placedby the Bank on strengthening national and international statistical capacity and the objectives o f the Marrakech Action Plan for Statistics. However, the panel also recommendedthat urgent progress shouldbe made to strengthen the database o f key indicators needed for the IDA R M S and for monitoring the MDGs. VII. A Program to Accelerate Progress inIDA countries 36. The IDA 14Results Measurement System will produce annual reports on the aggregate outcomes for the 14 selected indicators, usingthe methods o f aggregation, gap-filling and calculating rates of change recommended by the expert panel. Estimates of indicators will draw on the information reported through the international statistical system or, in the case o f the private sector and fiscal management indicators, compiled through World Bank sponsored programs. However, the reliability of most of these indicators depends on the performance o f national statistical systems. IDA countries confront many constraints in carrying out even basic statistical functions. Annex 5 discusses the foundations o f a basic statistical system and the problems statisticians in developing countries face. Accelerating progress and increasing the resources available for statistics will require increased collaboration between donors and developing countries and between users and producers o f statistics. While the Marrakech Action Plan provides a roadmap for global efforts to improve national and international statistical, the implementation o f the IDA 14Results Measurement System is an opportunity for donors and IDA countries to undertake certain steps neededto make immediate improvements in the quality and availability of development statistics. 37. Increasingawareness of statistical capacity issues in IDA countries. Improved assessments of statistical capacity are required to inform planningprocesses, particularly during PRSP and CAS preparation, and to guide investment and technical assistance programs. As part - 14- of its Global Monitoring Information System (GMIS),20the DevelopmentData Group will work with other international agencies to increase the information available on the statistical programs of IDA countries. Drawing on this informationbase, the GMIS will provide a country informatioGsheet on statistical activities. A prototype i s shown in Annex 4. The information sheet, including the statistical capacity indicators previously discussed, can be used by World Bank country teams duringCAS preparation and can also be shared with other bilateral and multilateral agencies to help inthe development of their assistanceplans through an appropriate public discussion forum (for example, a web site) where information on statistical practices can be exchanged. 38. PromotingNationalStatistical DevelopmentStrategies. Based on a rigorous analysis of capacity constraints, countries should develop and implementnational statistical development strategies to plan for sustainable improvements in their statistical systems. Plans shouldbe prioritized and realistic, taking into account budget and human resource constraints, but also ambitious and consistent with recognized statistical standards and codes of conduct. Many developing countries will require donor assistance to preparetheir statistical development strategies. To meet the MAPS target, all low-income IDA countries, where feasible, should have strategic plans inplace by the end of 2006. Resources from the Trust Fundfor Statistical Capacity Building and the work of PARIS 21 will be used to support this effort. 39. Implementinga pilot program to harmonize and align data collection activities. Better alignment of data collection instruments and internationally sponsored survey programs, along with improved statistical practices incountries, could substantially increase the availability, timeliness, andreliability of key indicators. Annex 6 shows details of recent survey programs in IDA countries, including their relationshipto MDG indicators. As these tables show, surveys are often ad-hoc and isolated exercises, conducted where and when donor funding i s available. The sequencing of surveys i s not always optimal, resultingin data gaps in some countries, and duplication of effort in others. Moreover, developing a sound statistical system i s widely recognized to be a long-term process,2' and capacity building initiatives may overlook the opportunities for increasing the efficiency of internationally sponsored survey programs. Box 4 describes a proposal for a pilot programin 12countries that would show the value of a more coordinated approach and would make rapid improvements in the frequency and quality of statistics available to national and international policy makers. 2o DECDG i s working with the Global Monitoring Secretariat to construct a web-based system to make available indicators on the performance of developing and developed countries and international financial institutions. See, for example, "Some guiding erinciples for good practices in technical cooperation for statistics," UN Statistical Commission (Economic and Social Council), March 1999 - 1 5 - Box 4: A Pilot Program inAccelerated Data Collection VIII. Conclusions and Recommendations 40. The evidence presented in this paper shows that there i s a pressingneed for improvements in the production and use of statistics in most IDA countries. The Marrakech Action Plan for Statistics has been widely accepted as a framework for national and international efforts to improve statistics for development decision making and monitoring. To support IDA 14 activities and accelerate progress toward the objectives of MAPS, a three-step programis proposed by the Bank under IDA14. a. Implementation of the IDA14 Results Measurement System: 0 Working with international partners, compile information on the statistical systems of IDA countries, including indicators of their statistical capacity. Based on this 22 HouseholdSurveys and the MillenniumDevelopmentGoals, Juan Munoz and Kinnon Scott, PARIS21,2004 - 1 6 - information, the Bank will prepare profiles of countries' statistical capacity (Annex 4). These country profiles will be made publicly available on the Bank's external website and can feed into the preparation of Country Assistance Strategies and PRSPs. 0 Monitor statistical capacity building activities undertaken by all development partners in IDA countries. The Bank will prepare anote each year, which will track changes in and describe efforts being made to improve statistical capacity. This note will be shared with the Board o f Executive Directors and made publicly available. 0 Track aggregate IDA14 outcome indicators annually, usingmethods recommended by the expert panel. b. Assist countries inpreparingnational strategies for the development of statistics. To meet the MAPS agenda all low-income IDA countries (where feasible) should have national strategies inplace by 2006. The World Bank's Trust Fundfor Statistical Capacity Building and PARIS21 will continue to support countries adopting a fast-track approach to preparing strategic plans. c. Initiate a pilot program to align and harmonize internationally sponsored surveys and to make improvements in statistical practices consistent with national statistical development plans. Inselecting countries for this work, priority would be given to IDA countries preparing CASs during the IDA14 period. This work should be coordinated through the new International Household Survey Network, formed as part o f MAPS. - 17 - Annex 1: The Marrakech Action Plan for Statistics At the second internationalRoundtableon Managingfor DevelopmentResults inMarrakech, February 200423,key international agencies agreed a plan of action that will leadto improved statistics. The plan has six components: 1.MainstreamStrategic Planning of Statistical Systems. Inparticular, the goal is to support the implementationof national statistical development strategies in every low-income country by 2006. To achieve this goal, the following steps should be taken: - Incorporate national statistical development strategies in result-based strategic planning processes such as the PRSP and include them inthe policy dialogue between developing countries and donors. - Ensurethat all donor-specific statistical programs support and complement national statistical plans. - Continue advocating and providing training and financial support from PARIS21 and the TFSCB. Based on the new repositioning of PARIS21 and World Bank's Trust Fund, earmark a significant part of the TFSCB to exclusively support countries' planning work. 2. Prepare for the global 2010 population census round. Based on the findings of the PARIS21 Census Task Team, prepare for the 2010 round of censuses by developing an overall strategy for funding and conducting censuses in low-income countries. The first priority is to build consensus on the importance of the 2010 CensusRound, recognizing the role census data will play in measuring the MDGs in 2015. Because such an effort should bringtogether donors andnational statistical agencies, PARIS21could act as the convener with leadership on the coordination and substantive work coming from the UNFPA and UNSDwith support from other key organizations. As a first step, the task force shouldreview the recent proposal by UNSDto set up a global trust fund for UN's support of census work and consider options for scaling up this proposal to meet the expected need of the least developed countries. The expected outcome could be a trust fundto support the preparations by the neediest countries for their 2010 census. The resources needed for a Census Trust Fundare estimated to be about $5 million a year for the next three years to support about 15-20 countries per year. The trust fundwould beusedto: - Conduct researchinto census costs and operational methods to determine what practical measures can be taken to reduce costs, as well as how to maximize the timely dissemination and use of census results. - Conduct research on improved methods for preparingregular population estimates at the national and sub-national level during intercensal years. 23 Managing for Development Results, Second International Roundtable, Marrakech February 4-5 2004. See http://www.mfdr.org - 18 - - Assist national statistical offices to advocate for conducting regular censuses and securing the necessary funding within countries and from the donor community. - Buildnational capacity at the technical level and develop the management skills neededto prepare an overall strategy and costed plan and to coordinate and negotiate with donors and users, pooling potential contributors in a cost-effective strategy. 3. Increase financing; for statistical capacity building;. Integrate financing needs from different agencies and different initiatives usingthe model are for PARIS21 and the World Bank's Trust Fundto make it easier for the donors to see the full picture o f needs and make reliable commitments. Bringdonors together inan annualjoint event, perhaps through DAC senior level meetings, and try to engage new donors; the first meeting should take place in 2004. A number of IFIs andbilaterals are already major funders of statistical capacity building. Butmost of this work has been done as part of investment projects inother sectors. Inthe future statistical capacity buildinginvestment projects shouldbe better identified and linked with general budget support andPoverty Reduction Support Credits, using a strategic planning process based on a sector-wide and multi-donor approach. Support long-term statistical investment projects with STATCAP-type financing programs. An issue to be resolved is findingthe right balance between grants, loans, and country resources. 4. Set up an international household survey network. Organize a Household Survey Network for the purposes o f sharing information and mobilizing international support for more efficient approaches to conducting household surveys in developing countries. Develop a set o f recommendations for household-based economic and social data, taking into account current and planned multinational survey programs and the needs o f developing countries to monitor their own development progress. Work with experienced data archivists and data users to establish a global information center containing household survey and metadata; establish good dissemination practices that promote analysis and research while protecting the confidentiality o f survey respondents. 5. Undertake urgent improvements for MDG monitoring in 2005. Consistent with the report of the MDG Indicators Expert Group, the following actions are proposed: - A review of the principal MDGindicators for poverty, education, health, the environment, and global partnership should be undertaken b y working groups composed o f experts from participating agencies, which would report back to the MDG Indicators Expert Group in fall 2004 with recommendations for improvements or changes to be made in the MDG indicators after 2005. - Establishment of a small, interagency editorial board to work with the Office o f the Secretary General on the production o f a five-year review o f the MDGs in 2005. - 19- - Provide training and tools to increase understanding of the MDGs at the national level and to improve country capacity to monitor and report on MDGs and other national goals. This would include UNDP's planned dissemination of the DevInfo data system and associated training modules. 6. Increase accountability of the international statistical system. - Adopt a statement of principles describing the responsibilities of international organizations for carrying out their official statistical activities. Encourage all international agencies to adopt the statement of principals as core values guiding their activities. - T o further improve the coordination of interagency activities, establish a mechanism through which international agencies would report on their core work program and exchange views on improvements needed. The UNCommittee for Coordination o f Statistical Activities, in which membership is open to all official statistical agencies, could provide a suitable forum for organizing this activity. - Systematically collect information on current and planned levels of international spending on statistical activities by agency, by functional area and by intended results. This information would be usedto assess the effectiveness of current spending and to identify areas where additional resources are required. - 20 - Annex 2: Methodology for Calculating Statistical Capacity Assessment Indicators I.StatisticalPractice years 9. Vaccine reporting to WHO Nationally reported data on measlesvaccine coverage consistent with WHO estimates Otherwise 1 10.IMF'sSpecial Data Dissemination Subscribed Otherwise 1 Standard 11.Data Collection 1.Periodicity of population census 110years Otherwise 2 2. Periodicity of agricultural census 110years Otherwise 2 3. Periodicity of poverty related surveys (IES, LSMS, etc.) 13years 55years Otherwise 2 4. Periodicity of health related surveys (DHS, MICS, Priority survey, etc) 5 3 years 5 5 years Otherwise 2 5. Completenessof vital registration system Complete Otherwise 2 - 21 - Annex 3: Methodologyfor ReviewingJSAs JSAs were reviewed for their discussion of statistical capacity, and to facilitate analysis comments were grouped into four areas: the institutional arrangements for statistical activities (institution); the adequacy of data sources (data); the selection of indicators for monitoring PRS progress (indicators); and the capacity to evaluate and formulate policy (analysis and use). JSAs were placed into one o f three categories by assessing comments against detailed criteria in each area, as follows: No constraint: No deficiencies were foundagainst any criteria in that area. Significant constraint: A severe deficiency noted on at least one criterion inthat area, and deficiencies noted on multiple criteria. This rating implies that only some criteria are met, and significant efforts are needed to make improvements. Some constraint: A single minor deficiency inthat area. This rating implies that the relevant criteria are mostly met, even though some improvement i s still needed. The results of this analysis for the 38 JSAs reviewed are shown in the table below. Most JSAs provide some commentary on statistical capacity in all four areas, although it should be noted that about a third do not provide an assessment o f the institutional arrangements for statistical activities -probably the hardest area to evaluate without in-depth review. It should also be noted that the quality of data, diagnosis, and analysis in.PRSPsvaries widely among countries, and in preparing the JSA, staff tend to focus on the issues that are most relevant in the country context. JSAs may not therefore comment on all the detailed criteria. No constraint 3% 5% 0% 5% Some constraint 16% 42% 63% 37% Significant constraint I 47% I 53% I 34% I 45% ~ Unknowdno discussion 34% 0% 3% 13% The following detailed criteria were used in the assessment, based on issues listed in the JSA guidelines (these are listed in italics). Institution Are current and proposed monitoring and evaluation systems adequate and sustainable? - Adequacy of efsorts to improve data collection and analysis. - Transparency of arrangementsfor, and results OLmonitoring the PRSP, including service delively to the poor. Use of participatory methods of monitoring. - 22 - Accessibility of datafor policy analysis, especially outside govemment. * Involvement of key government agencies and civil society inthe monitoring and evaluation process, and dissemination of results to all stakeholders. Development of a coherent statistical activity plan including an appropriate m i x of survey and administrative data, quantitative and qualitative data, all undertaken at regular periodicity. How adequate are existing poverty data? - Extent of disaggregation of poverty data by regions and by demographic groups, including by gender. - Degree to whichquantitative data were complemented by qualitative information. How well have the nature and deteminants of poverty outcomes (income and non-monetary dimensions) been identified? Have trends in key poverty determinants and outcomes been presented? - Extent of incomekonsumptionand other dimensions of poverty (health, including environmental diseases, education, natural resource degradation, vulnerability, disempowement) and their evolution over time. - Distribution of assets of various types-natural, physical,financial, and human. I s the allocation of expenditures consistent with the strategic priorities? - Comprehensiveness of budget data-fiscal data is adequate to track poverty reduction spending. N o major quality concern on key statistics. National accounts, trade, and balance of payment data are produced timely in accordance with internationally accepted standards. Data i s also available at some sub-national level. Quality administrative data i s available annually for health, education, and agriculture sectors. Indicators Does the PRSP define medium- and long-term goalsfor poverty reduction outcomes (monetary and non-monetary), establish indicators of progress, and set annual and medium- t e m targets? Are these indicators and targets appropriate given the assessment of poverty and the institutional capacity to monitor?And are they consistent with thepolicy choices in the strategy? - Selectivity in the choice of monitorable indicators and targets, in line withpriority public actions and capacity. - Inclusion of indicators related to the MDGs, recognizing that the appropriate indicators, as well as specific targets, will vary among countries. - Indicators and targets that appropriately capture disparities by social group, gender, and region. Consist of a good mix of indicators measuring every phase on the result chain-inputs, outputs, intermediate outcomes, longer-term outcomes, and impacts. - 23 - Analysis & use Adequate use of results monitoring and evaluation in policy formulation. To what extent have the growth and distributional impacts of past policies and programs been assessed? - Macroeconomicpolicies, including the ability to respond to exogenous shocks. - Structural and sectoral policies, including the distributional impacts of past reforms and policies aflecting private sector development, the operation of product andfactor markets, environmental management. - Equity, effectiveness, and efficiency of existing pattem of public expenditure, service delivery, and systemsfor budget management,financial management, and procurement. - Policies with regard to gender inclusion, and social inclusion. Make good use of available data to input in policy formulation. - 24 - Annex 4: Exampleof a Statistical Capacity Country Profile Burkina Faso Statistical capacity at a glance Basic Country Information Practice I AFR 100% -Burkina Faso 11.8 million 80% ! - IDAAverage pita $US 300 60% Millennium Development Goals Seriously off-track Seriously off-track Seriously off-track Seriously off-track Maternal mortality Seriously offdram AvailabiIity Collection Environment ILodata Statistical Information Law 040/96/ADP of November 8, 1996 Cahier de charaes statistiques et proaramme national statistiaue (2000-20041 Schema Dtrecteurde la Statistiaue du Burkina Faso 2004-2009 1990 (base year) BPM4 (manual in use) Actual Not available 1996 (base year) Not available GDDS subscriber Not available 1996 (latest year) Coverage not complete 1993 (latest year) Priority Survey (2003, 1998, 1994); DHS (2003, 1998, 1992); Enqu6te 1-2-3(2003); Enquete permanente agricole (Annual); Enquetesur les depenses des menages de Ouagadougou (1996); MlCS (1996) Development of the National Statistical System (IBRDADA-STATCAP,2004) Statistical Master Plan (TFSCB, 2003) None lnstitut national de la statistique et de la demographie (INSD)' http://www.insd.bf Bamorv Ouattara, Director General, National Institute of Statistics and Demoaraphv ltnsdQcenatrin.bf) Lists of publications are available from IMF'sGDDS webstte' 2000, Prowess Report in 2002 2000. Prowess Report in 2003 (even/ 3 vears) 2003 In preparation As of 2003, INSD has a total staff of 78, of which 49 are technical staff. Notes 1 Other relevant institutions: Afrtstat (http.//wwwafristat ora/) lnvesttr en Zone Franc (httD,//www tzf net/izf/index htm) 2 Major publications by CSO Annual L'Annuaire Stattstique du Burkina Faso, Le Bulletin d'lnformattons statisttques et economiques; Les Comptes Nationaux; La note Rapide de Conjoncture, La Note Raptdesur les Prtx; Le Tableau de bord social; Le Tableau de bord de I'Economie; La Note Raprde sur la Production tndustrtelle, L'Annuatredu commerce exterieur Monthly Harmonized Consumer Price Index for WAEMU Member Countries - 25 - Annex 5: Key Statistical Processes and Issues in IDA Countries Key processes Coordination and leadership. Most countries have established statistical offices, but there are often weaknesses in their ability to coordinate producers, maintain integrity, standards, and comparability, and interact with and assess user priorities. The result i s data that are incomparable or inconsistent, o f poor quality, and, in extreme cases, irrelevant. Improving this situation requires improvements in coordination and management arrangements o f statistical systems, to provide incentives for producers to work more closely with clients and with other producers. Solutions may include changes to legislation, improved arrangements for user consultation and liaison, and the development and implementation o f national statistical development strategies. Censuses. As with household surveys, most countries have experience o f taking censuses, but the cost o f census taking is relatively high, particularly for the poorest countries, and it i s often difficult to maintain the skills and expertise needed from one census to the next. These problems have been recognized in the Marrakech Action Plan for Statistics, which calls for the international community to develop a strategy for funding and conducting censuses for the next global census round in 2010. Household surveys. Household survey capacity i s an important area for the IDA Results Measurement System; eight o f the proposed outcome indicators are normally obtained from household surveys. One success o f international action over the past few years has been the development o f capacity for fieldwork operations in most countries. However, surveys are often infrequent; they do not form part o f regular budgeted activities and they are often relatively expensive, particularly if complex variables, such as income, consumption, or mortality are required. Although many countries need to establish genuine national household survey capacity, the Marrakech Action Plan for Statistics recognizes the role o f the international community in supporting household survey programs and recommends the establishment o f an international household survey network. The objective is to find more efficient methods to support household surveys and to make best use o f data produced from those surveys. Because network.2 Additional investment in household surveys i s required in the short term to fill o f its comBarative advantage in this area, DECDGis taking the lead in establishing this important data gaps and increase the frequency o f key statistics. Economic statistics. The production o f basic economic statistics, including national accounts, price statistics, production statistics, and trade statistics, i s a well-established function o f most statistical offices. But there are often problems, for example with the collection of production data from businesses, with maintaining accurate business registers, and with technical capacity for implementingintemational standards. Countries are often in need of technical support and advice, help with the development o f appropriate standards, and investment in more efficient and effective data collection methods, particularly in the household and agricultural sectors. '' See http://www.internationalsurveynetwork.org - 26 - Sectoral statistics. Most countries have systems for delivering health and education systems, from which timely statistical data can be collected at low cost and used for decision-making at local community level. But there are quality issues, because many administrative systems are relatively weak, and there are data reporting incentives that may bias results. Investment i s needed to develop improvedreporting systems. Depending on country circumstance, there are also needs for specific surveys or data collection instruments on specific topics. Inmany IDA countries, for example, household-based farming activities are important, yet agricultural production statistics and environmental data are relatively weak. Dissemination. A common complaint of data users i s the inaccessibility of data, either in aggregated form or inthe form o f anonymized records from surveys. Many IDA countries operate outdated and costly publication methods, and have not been able to take advantage o f new electronic methods for data dissemination. With notable exceptions, many IDA countries do not utilize the Internet for publishingdata, and delays in publishingdata are still attributed to problems with printingprocesses. Solutions include investment in slulls and equipment to take advantage o f new techniques for disseminating statistical data. Relationship o f key processes to IDA Results Measurement System Process I Relationship to IDA14 indicators ~~ ~~ ~ Coordination and Important for all nationally produced indicators, particularly to ensure and assess quality, leadership and provide comparable estimates over time Censuses Required to estimate denominator incalculations of primary school completion rate and GDPper capita. Also important in many IDA countries in taking samples for household surveys. Household surveys Proportion of population living on less than $1 a day Child mortality Proportion of births attended by skilled health personnel Ratio of girls to boys at school (although normally estimated from school records, where available) Access to improved water source Access to all-weather road Household electrification rate Economic statistics Directly: GDPper capita Good price statistics also required for estimating proportion of population living on less than $1a day Sectoral statistics HIVprevalence rates, from health information systems Primary school completion,from school information systems Ratio of girls to boys at school, from school information systems Proportion of births attended by skilled health personnel, from health information systems (although often difficult to assess from health records and household surveys are a common source) Telephone access Many sectoral statistics also contribute to measurement o f GDP (e.g. agricultural and emdovment data) - 27 - Annex 6a: Household Surveys inIDA Countries 1999-2005 Europe and Central Asia Albania 321 77 86 I I,MI C L L L I Armenia 3 1 1 82 76 I I D 1 c,1 1I - 28 - Statistical Country Popuiation capacity score (%) an estimatein Azerbaijan 8.2 73 Bosnia& Herzegovina 4.1 37 67 M L Georgia 5.1 72 79 I 1,M I c,1 IKvrgvz Rem I 5.1 I 72 71 c.I I I I Mo1dova 4.21 55 Serbia & Montenegro I 8.1 I 47 Tajikistan 6.31 60 Uzbekistan 25.61 . 57 ILatinAmerica and the Caribbean Cape Verde 05 n/a 56 I C I Comoros 0 6 n/a 62 M C Sao Tome and Pnncipe 0 2 n/a 48 I C 1 East Asia and the Pacific I MiddleEast and NorthAfrica Djibouti 0.7 I d a 56 Bhutan 0.91 n/a 52 I M I I Maldives 0.31 n/a 59 I C 1 Key: C: PopulationCensus; D: DHS (Demographic and Health Survey - USAID/Macro International)); I:Income and expenditure survey: L:LSMS(LivingStandards MeasurementStudy Survey -World Bank); M:MICS (Multiple Indicator Cluster Survey - UNICEF): Q: CWIQ (Core Welfare indicators Questionnaire- World Bank). See Annex 6b for a descriptionof each of these surveys. Source: World Bank Development Data Platform (DDP), and staff estimatesbasedon informationsuppliedby survey sponsors - 29 - Annex 6b: Description of Main Internationally Sponsored Household Surveys Population Census Population censuses are household surveys that attempt to enumerate all households in a country. They are capable o f providing results for all subgroups o f the population, including small areas. They collect data on a limited number of demographic indicators, using short questionnaires. In many developing countries, they also provide the basis for taking samples for household sample surveys. Field operations are normally conducted during a very short period using large numbers of enumerators. Censuses are seen by most governments as essential operations, and normally conducted every ten years. Besides their role as household surveys with such special features, censuses are an essential element o f a country's household survey system, because they are needed to develop sample frames for all other household surveys. Demographic and Health Survey (DHS) The DHS survey model follows a long tradition o f demographic research dating back to the 1970-84 World Fertility Survey (WFS) program. Its original intention was to produce internationally comparable measures o f fertility, mortality, contraceptive use, maternal and child health, and other demographic indicators, but its goals have broadened to include urgent topics such as HIV/AIDS, STDs, anthropometrics, and child malnutrition; and to meet frequent requests o f participating countries for data on access to health services and intra-family violence. A typical DHS (of 5,000 households) uses different questionnaires for women and men, requires about eight months o f preparation, including training, and is fielded for three months using a staff o f fifty field workers. The DHS has a reputation for technical excellence in questionnaire and sampling design-including the routine publication o f sampling errors for all relevant indicators. Data management i s very standardized, and includes a well-specified set o f consistency controls to be applied in the data entry and cleansing phases. D H S datasets are available to researchers through a well-documented and accessible database. Since 1985, technical supervision o f the D H S program has been managed by Macro International, a private firminthe U.S. Income and Expenditure Survey (IES) Income and expenditure surveys have several objectives. The first is to find the shares o f different commodities in the budget o f households, to define the composition o f the baskets used incalculating Consumer Price Indexes. The secondis to provide direct measures of household consumption for the system of national accounts. The third i s to calculate poverty lines and poverty incidence. In many IESs, household expenditures are collected usingdiaries, which record each individual purchase made by the household during a specific period, and questionnaires based on expenditures made by the household in the past. IESs in various countries differ in the extent to which they resort to the use o f each kindo f instrument. This type of survey i s generally fielded for 12 months, to take into account the seasonal variations o f consumption patterns. Sample sizes range from 2,000 to 20,000 households, depending on the required level o f geographical disaggregation. Living Standards Measurement Study Survey (LSMS) The LSMS survey instrument was developed by the World Bank in the 1980s as a means to help policymakers understand the determinants o f observed social and economic outcomes and, thus help their ability to design effective programs and policies. The LSMS i s designedto provide a - 30 - comprehensive picture of household welfare and the factors that affect it. The LSMS provide analysts with data that allows them to assess: welfare levels and distribution; the links between welfare and the characteristics of the poor; the levels of access to and use of social services; the impact of government programs; and the causes of observed social outcomes. The LSMS typically incorporates data collection at the individual, household, and community level. The survey instrumentsare designedinclose consultation with policymakers to ensure that data are relevant. Questionnaires, especially the one at the householdindividual level, tend to be quite complex. To maintain highquality data, samples are kept small (2,000-5,000 households) to minimize non-sampling error. Substantial efforts and resources are devoted to data quality (such as month-long interviewer training, use of direct informants, concurrent data entry with in-field corrections taking place at the households, low supervisor-interviewer ratios). MultipleIndicator Cluster Survey (MICS) The MICS instrument has been developed by UNICEFto be low-cost household survey that quickly generates data on key welfare indicators that are inadequately monitoredin other data collection systems, and has been implementedin more than 100 countries since 1995. UNICEF, with partner agencies, defined a set of indicatorsto guide the assessment process (UNICEF 1995). Indicators are meant to monitor progress in health, education and welfare, and include data on the welfare of children in developingcountries (e.g., child labor, birthregistration, disability, orphandalternativefamily care, and early child development). Incollaborationwith a number of other agencies, UNICEFhas harmonizedthe MICS with other major survey programs (for instance, the DHS)to improve the generation of comparable and complementary data across countries. MICS does not allow measurement of poverty based on income, but they can be used to generate population profiles regarding health, education and child labor indicators, which can be usedas proxy variables for poverty status. Core Welfare Indicator Questionnaire (CWIO) survey The CWIQ i s a household survey designed to measure changes in key social indicators for differentpopulation groups: specifically, the focus is on indicators of access to, utilization of, and satisfaction with, core social and economic services. Typically, CWIQ surveys uses methods designed to get reliable national estimates of these indicators in a timely fashion. These include a large sample, a simple questionnaire with multiple-choice questions, high-quality fieldwork, the use of optical mark and character recognition during data entry, pre-programmed validation procedures, and standardized outputs. The CWIQ survey was developedby the World Bank, inclose collaboration with UNDP, UNICEF, and the ILO. - 31 - Goal Indicator LSMS DHS MICS CWIQ 1.Proportion ofpopulationbelow$1(PPP)per day 0 0 0 0 2. Povertygap ratio [incidence x depthof poverty] 0 0 0 0 1 3. Share of poorest quintile innational consumption 0 0 0 0 4. Prevalenceof underweight children under-five years of age 0 0 0 0 15. Proportion of population below minimum level of dietary energy consumption I 0 I 0 I 0 I 0 I 6. Net enrolment ratio inprimary education 0 0 0 0 2 7. Proportion of pupils starting grade 1who reachgrade 5 0 0 0 0 8. Literacy rate of 15-24year-olds 0 0 0 0 9. Ratios of girls to boysinprimary, secondary, and tertiary education 0 0 0 0 3 I10.Ratio of literate females to males of 15-24year-olds 1 ~ 11. Share of women in wage employment inthe non-agricultural sector 0 1o ~ 1 ~ 1 . 0 0 13.Under-five mortality rate 0 0 0 0 4 14. Infant mortality rate 0 0 0 0 15.Proportion of 1year-old childrenimmunized against measles 0 0 0 0 116.Maternalmortality ratio l @ l . l . l @ l 17. Proportion of births attended by skilled health personnel 0 0 0 0 18. HIV prevalenceamong 15-24 year old pregnant women 0 0 0 0 19.Condom use rate of the contraceptiveprevalencerate 0 0 0 0 20. Number of childrenorphanedby HIV/AIDS 0 0 0 0 22. Proportion of population inmalaria risk areas using effective malaria prevention and treatmentmeasures 129. Proportion of population using solidfuels I o I o I o 1.1 30. Proportion of population with sustainable access to an improved water source,urbanand rural 31. Proportion of urbanpopulation with access to improvedsanitation 0 0 0 0 32. Proportion of households with access to secure tenure (owned or rented). 0 0 0 0 45. Unemployment rate of 15-24year-olds, each sex and total 0 0 0 0 8 47. Telephone lines and cellular subscribers per 100 population 0 0 0 0 48. Personalcomputers in use and Internetusers per 100population 0 0 0 o I IHousehold electrificationrate Key: 0 Indicator canbemeasuredwiththis survey 0 Indicator canbemeasuredwiththis survey,butsomechangesto methodologymayberequired 0 Indicator wouldnotnormallybeestimatedwiththis survey Note: IDA 14 RMS indicators -or most closely relatedMDG indicator - are shown in bold. Only indicators that can be measuredwith household surveys are shown: for some indicators shown, data may also be obtainedfrom administrative sources. Source: HouseholdSurveys and the Millennium DeveloumentGoals, Juan Mufioz and Kinnon Scott, 2004. - 32 - wb10494 N:ksloan\IDA14\Mailings\Athen~-GreeceDec. 2004\final papersweasuringResults.doc November29,2004 2:Ol PM