96525 Independent Evaluation of the Marrakech Action Plan for Statistics Evaluation Report Brian T. Ngo March 2015 1 Independent Evaluation of the Marrakech Action Plan for Statistics Contents I. Executive Summary................................................................................................................ 4 II. Introduction ......................................................................................................................... 9 1.1 Background ..................................................................................................................... 9 1.2 Purpose of the evaluation .............................................................................................. 10 III. Scope and Methodology of the Evaluation ....................................................................... 11 2.1 Scope ............................................................................................................................. 11 2.2 Methodology ................................................................................................................. 11 2.3 Limitations to the evaluation ......................................................................................... 12 IV. The International Household Survey Network (IHSN) and Accelerated Data Program (ADP) .......................................................................................................................................... 13 V. Preparation of National Strategies for the Development of Statistics (NSDSs) ............... 21 VI. Improvement in Gender Statistics..................................................................................... 28 VII. Help countries participate in the 2010 Population Census Round.................................... 36 VII. Lessons and Recommendations ........................................................................................ 42 References ............................................................................................................................... 44 Annex A: DGF/MAPS funding 2006 - 2014 (US$ thousand)................................................. 50 Annex B: PARIS21 Logical Framework (2010-14) ................................................................ 51 Annex C: External Evaluation of the IHSN/ADP Programs: Major Findings, Recommendations (Executive Summary) ............................................................................... 58 Annex D: IHSN Management Group - Response to the Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP) .... 66 Annex E: Persons Met or Consulted ....................................................................................... 70 Annex F: Terms of Reference Independent Evaluation of the Marrakech Action Plan for Statistics ................................................................................................................................... 71 2 Abbreviations and Acronyms ADP Accelerated Data Program AfDB African Development Bank ACSS ASEAN Community Statistical System DECDG Development Data Group DGF Development Grant Facility ECOSOC Economic and Social Council (United Nations) EU European Union IDA International Development Association IHSN International Household Survey Network LIC Low-income Country Logframe Logical Framework MAPS Marrakech Action Plan for Statistics MDG Millennium Development Goals MOWs Microdata Outreach Workshops NADA National Data Archive Application NSDS National Strategy for the Development of Statistics NSO National Statistical Office NSS National Statistical System OECD Organization for Economic Co-operation and Development PARIS21 Partnership in Statistics for Development in the 21 st Century PRESS Partner Report on Support to Statistics PRSP Poverty Reduction Strategy Paper RSDS Regional Strategy for the Development of Statistics SCB Statistical capacity building SRF Statistics for Results Facility TFSCB Trust Fund for Statistical Capacity Building (of the World Bank) UIS UNESCO’s Institute of Statistics UNECE United Nations Economic Commission for Europe UNECLAC United Nations Economic Commission for Latin America and the Caribbean UNSC United Nations Statistical Commission UNSD United Nations Statistics Division` WB World Bank 3 I. Executive Summary Introduction i. This report presents the results of a light evaluation of the partnership effort between the World Bank, through the Development Grant Facility (DGF), and a number of key agencies involved in the building of statistical capacity in developing countries under the Marrakesh Action Plan for Statistics (MAPS). This evaluation covers the five largest partnership programs which include the International Household Survey Network (IHSN), the Accelerated Data Program (ADP), the preparation of National Strategies for the Development of Statistics (NSDSs), the urgent improvement in key statistics, including Gender Statistics and the support given to countries to participate in the 2010 census round. The grants for the first three programs are managed by the Partnership in Statistics for Development in the 21 st Century (PARIS21); the United Nations Statistics Division (UNSD) managed the other two programs. The DGF is being phased out this year. ii. The objectives of the evaluation are i) to assess the effectiveness, efficiency and sustainability of the five partnership programs; ii) to assess the extent to which MAPS actions remains relevant to the statistical capacity; and iii) to provide guidance on how support for statistical capacity could be strengthened and made more effective under the Busan Action Plan for Statistics. iii. The evaluation is based on the desk review of the five partnership programs, the review of the outcomes of the independent IHSN/ADP evaluation completed in August 2013, meetings and interviews of program task managers and MAPS Unit staff at the World Bank and, if supported, where relevant, by additional sources of information including interviews with former PARIS21 staff and PARIS21 consultants. Quantitative methods were used to assess program effectiveness where relevant. In view of the limited scope of this evaluation, there were no efforts to seek feedback from the beneficiaries of the programs. Moreover, important considerations such as governance and management arrangements of the programs and financial sustainability particularly given the phasing out of the DGF will not be assessed in this report. Main findings iv. On IHSN/ADP1:  Relevance: Interviews by the external independent IHSN/ADP evaluation team confirmed the general consensus that archiving of and promoting access to household survey data was important for improving the evidence base for policy and for monitoring progress towards the MDGs; 1 The results of the assessment of the IHSN/ADP programs rely to a great extent on the findings of the external evaluation report of 2013, on management responses to the evaluation together with follow-up activities supplemented by other information where relevant. 4  Effectiveness: The majority of statistical producers contacted by the external evaluation team have used the IHSN microdata management toolkit and rated it as very useful. The inventory, documentation and archiving of national survey data have been achieved in a significant number of countries. The external evaluation confirmed that IHSN/ADP has created greater awareness of and improved the availability of and access to microdata as shown by the large number of visits and downloads from the National Data Archive (NADA) catalogs. Efforts to remove the obstacles to microdata access are the main driver for the work on developing the anonymization tools under IHSN/ADP. But the impact of the programs on effective data use is harder to measure. The external evaluation also reported that progress on the quality of data and surveys has been slow but did not make reference to the indirect benefits derived from the training and expertise building in data documentation and dissemination. The external evaluation found limited impact of the two programs on promoting the collaboration between data producers and data users. Cognizant of this gap, ADP has since 2013 supported a series of Microdata Outreach Workshops aimed at fostering the interactions between microdata producers and users. The external evaluation team found the limited effectiveness of IHSN on survey coordination at country level. This has led to a rethinking of the role and mandate of IHSN and to the decision to engage the UN Statistical Commission (UNSC) in this process;  Sustainability: The external evaluation team revealed that most countries which have received training in the use of the toolkit are continuing activities on their own and that new countries are joining the ADP. Most NADAs are still operating and an automated monitoring system to check the working of NADAs has been set up making it possible to provide remote assistance and support when needed; and  Efficiency: The external evaluation finds that the use of financial and human resources by both IHSN and ADP has been quite economical. Overheads have been low, and close to 90% of grant funding has gone directly to implementation. The number of full-time equivalent staff in both programmes has been kept small thanks to the reliance on international consultants many of whom from countries in the South. v. On NSDSs:  Relevance: The IEG Global Program Review of the MAPS, PARIS21 and Trust Fund for Statistical Capacity Building undertaken by the World Bank in 2011 highlighted the strong international consensus for the support to NSDSs as a key goal of the MAPS. The review of relevant documents, interviews with DECDG staff held in the context of this evaluation and various discussions at PARIS21’s annual Board meetings confirmed the strong relevance of the partnership program’s objectives and activities;  Effectiveness: Overall progress has been notable in relation to the design and implementation of NSDSs, a process which has taken hold in all regions. As of March 2014, 88% of all IDA countries are implementing, designing or awaiting adoption of their NSDS. This ratio is even higher (95%) for sub-Saharan Africa where statistical capacity is, on average, weaker. While the relationships between the NSDS process and national statistical capacity are challenging to establish, it is worth noting that the aggregate scores of the World Bank’s Statistical Capacity Indicator (SCI) for 82 IDA-eligible countries 5 have, on average, increased by 5% over the 10-year period since the NSDS process started. NSDSs have also helped to strengthen the status of statistics in the development agenda, fostered a growing attention to the data demand, through a dialogue with users, particularly with sector ministries;  Sustainability: While other programs and activities have undeniably contributed to overall statistical capacity building, the review of the relevant literature and consultations undertaken for this evaluation confirmed the critical role played by PARIS21 in assisting developing countries with advice and technical support relating to the development of national statistical development strategies. Some 50 developing countries, of which about half are in sub-Saharan Africa, have benefited from PARIS21 support; and  Efficiency: Funding for PARIS21’s core program related to NSDS processes including country and regional programs, advocacy, and donor coordination have registered very moderate changes in the last few years. Management costs have been held steady over the years at around 15–21 percent of total expenditures. vi. Improvement in Gender Statistics  Relevance: The program on gender statistics carried out by UNSD has been shaped by demands from four international conferences on women and the relevance of gender statistics gets renewed at regular intervals as new issues such as violence against women and time use issues receive more attention as a focus of policy dialogue at the highest level;  Effectiveness: The evaluation confirmed that the partnership program has helped to advance the global agenda on gender statistics. First, the program has made it possible to broaden the coverage of gender statistics from a fragmented approach to the adoption of a minimum set of gender indicators composed of 52 quantitative indicators and 11 qualitative indicators which needed to address key policy concerns as identified in the Beijing Platform for Action. There has been an increase in the availability of gender statistics. The majority of countries are now able to produce sex- disaggregated statistics on population, enrolment, employment and parliamentary representation and increasingly, data in some newer areas such as child labour, time use and violence against women are becoming available. The other major contribution of the partnership program was the efforts to mainstream gender statistics into national statistical systems. But based on a recent global review of national gender statistics programs, much more remains to be done in this area;  Sustainability: The proposal of a stand-alone goal on “achieving gender equality and empower all women and girls” in the on-going deliberations on the Sustainable Development Goals which are expected to replace the MDGs after 2015 will likely reinforce the demand for gender statistics; and  Efficiency: Lacking the information on the overall budget of UN Social and Housing Statistics Section, efficiency in this evaluation is assessed by delivery of the agreed outputs. Consultations with UNSD and DECDG staff confirmed that in the eight years of the partnership programs (2006-14) covered by six annual arrangements, UNSD has 6 consistently delivered outputs agreed at the yearly DGF arrangements without cost overruns. vii. Help countries participate in the 2010 Population Census Round  Relevance: Helping countries participate in the 2010 population census round comes directly from a call by the United Nations that all member countries undertake a population census in the decennial period;  Effectiveness: The impact of the partnership program can be assessed first through the rate of participation: out of 235 countries, areas or territories, 214 have conducted at least one census or reported detailed population statistics in the 2010 census round spanning the period 2005 to 2014 covering an estimated 6.39 million persons or 93% of the estimated world population. Compared to the previous round, the number of countries that have missed the 2010 Round declined from 26 to 21 countries. For Africa and Oceania where demographic data gaps are the largest, there has been significant catching up. Countries with advanced statistical capacity have reported a shift toward the increased use of alternative census methodologies, the expanded use of multiple sources of collect data but developing countries have also experienced greater collaboration in alternative data collection methodologies and in data dissemination and analysis;  Sustainability: While issues of rising costs and other considerations will continue to pose challenges to future census rounds, the review of reports and presentations by the various regions showed a quickening of the pace of adoption of technological advances which have played a role in sustaining some of the outcomes and impacts described above and in helping to limit cost increases of population censuses; and  Efficiency: As was the case for the gender statistics partnership program, the lack of information precludes a more detailed analysis of the cost efficiency of the program. The consultations for the evaluation again confirmed that UNSD has consistently delivered its activities and outputs within the allotted budgets under the various yearly arrangements of the DGF. Recommendations viii. The five partnership programs covered by this evaluation have in many different ways contributed greatly to the production and availability of key statistics. The review of partnership programs’ effectiveness highlights four lessons three of which are related to the design and implementation of the programs and indirectly to overall statistical capacity support. The fourth lesson concerns the operationalization of the DGF arrangements.  Effectiveness requires clear objectives in terms of outputs and outcomes. The review of the five partnership programs shows that clearly defined objectives in terms of outputs and outcomes have helped to enhance the chance for effectiveness of the programs. This is the case for the two partnership programs with UNSD on improving gender statistics and promoting the participation of countries in the 2010 population census round. A monitoring and evaluation system could further add to achieving the specified objectives. 7  Setting specific timelines could help to generate the expected outcomes. Although they may be perceived as constraining, timelines combined with clear objectives and benchmarks could enhance the chance of success. The achievements under the IHSN, Gender and Population Census Round may partly be attributed to the strict timelines for delivery.  Some selectivity may be needed to achieve progress in statistical capacity building. The IHSN/ADP external evaluation made the observation that the rush to respond to a rapidly rising demand from countries has meant that interventions at country level were poorly coordinated. An alternative option would be to work with a smaller number of countries but in greater depth and in full coordination with the NSDSs which could have provided more substantive experiences to share across countries.  For capacity building activities, multiannual funding may be more desirable. The DGF annual grant agreements have imposed various costs on the partnership programs such as the repeated need to request extensions for disbursement as well as challenges in terms of medium term planning particularly for staffing purposes. 8 II. Introduction 1.1 Background 1. The Marrakesh Action Plan for Statistics was created in 2004 as an important component of an informal partnership whose main goal is to help develop and strengthen national statistical systems through six specific actions with target dates aiming to prepare the way for sustainable improvement in statistical capacity. These actions aimed at: i) Mainstreaming strategic planning of statistical systems and prepare national strategies for the development of statistics for all low-income countries by 2006. ii) Ensuring the full participation of developing countries in the 2010 census round; iii) Increasing financing for statistical capacity building and international coordination; iv) Setting up an International Household Survey Network; v) Undertaking urgent improvements needed to monitor the Millennium Development Goals (MDGs) and other development goals; and vi) Increasing accountability of the international statistical system. 2. Under MAPS, the World Bank’s Development Grant Facility has provided $41.4 million of financial support over the period 2006-14. The DGF is being phased out with calendar year 2015 being the last year of new commitments. Financial support under the DGF is established on the basis of a partnership effort between the World Bank and a number of key agencies involved in the building of statistical capacity in developing countries. This evaluation covers the five major MAPS Partnership programs2 managed by PARIS21 and UNSD3. The five major MAPS Partnership programs cover: i) The International Household Survey Network: The priority areas of work of the IHSN aimed at: a) coordinating internationally sponsored survey programs with emphasis on better timing, sequencing and frequency; b) fostering adoption of international standards and best practices by harmonizing data collection instruments; c) developing and maintaining a central survey data repository; and d) developing tools and guidelines for improving survey data documentation, preservation, anonymization, cataloguing, dissemination and archiving of survey data. The IHSN management group is headed by the World Bank which also serves as the IHSN secretariat coordinator. Under the DGF grants, PARIS21 has managed this activity as a ‘satellite’4 program. 2 The MAPS Partnership is taken here to mean the recipients of grants provided by the Development Grant Facility. 3 Smaller partnership programs involved the United Nations Economic Commission for Europe (UNECE, gender statistics program in South and Eastern Europe); the UNESCO’s Institute of Statistics (UIS, education statistics); UN - Habitat (Urban indicators, the International Comparison Programme for CIS countries (ICP-CIS which is designed to produce internationally comparable price and volume measures for gross domestic product and its components ), and United Nations Economic Commission for Latin America and the Caribbean (UNECLAC) which, together, account for less than 5 percent of total DGF funding for the MAPS program. 4 Satellite programs are managed independently of PARIS21’s core work programs, allowing for more flexibility in their management. 9 ii) Accelerated Data Program: The ADP’s objectives were to: a) assist countries to improve the planning and financing of data collection activities; b) foster better inter-agency cooperation in country-level activities under the umbrella of the IHSN, c) build national capacity in micro-data preservation, analysis, and dissemination, and d) generate updated estimates of key indicators by collecting new data including censuses and administrative data5. As the main agency6 for implementing the ADP, PARIS21 has run the ADP as a ‘satellite’ program. iii) Preparation of National Strategies for the Development of Statistics: The DGF grants have made it possible for PARIS21 to provide direct assistance missions to least developed countries in developing their NSDSs and in accessing financial instruments for that purpose; to produce and disseminate guidelines on strategic statistical planning; and to help national statisticians to develop advocacy materials on the importance of statistics. iv) Urgent improvement in key statistics, including Gender Statistics : Women's empowerment and the promotion of gender equality are key to achieving sustainable development. DGF grants have contributed significantly to the work of the UN SD’s Social and Housing Statistics Section in the area of gender statistics including: a) the development of technical materials; b) the compilation and dissemination of online data and meta-data for the minimum set of gender indicators; c) the organization of the Interagency and Expert Group meetings; and d) the building of country capacity through regional workshops. v) Help countries participate in the 2010 census round: Population censuses are essential tools for policy and planning purposes. DGF grants have contributed to the work of the UNSD’s Demographic Statistics Section in: a) developing technical materials; b) improving the capacity of countries to plan and conduct population and housing censuses; c) providing assistance to countries in census data documentation, analysis, archiving and dissemination; and d) monitoring the implementation of the 2010 World Programme On Population and Housing Censuses and providing support to countries that may be facing problems. 1.2 Purpose of the evaluation 3. The purpose of the evaluation is to: i) Assess the performance (effectiveness, efficiency, and relevance) of the major partnership programs in implementing actions under the aegis of MAPS, and provide guidance on how to improve this performance where relevant; and 5 This objective was mentioned in the very early days of the ADP but was subsequently abandoned as other agencies had begun to estimating key indicators. Besides, the ADP program never had a budget to collect data. 6 The implementation of the ADP program has benefitted from the contributions of other partners including regional agencies such as the African Development Bank, the Economic and Social Commission for Asia and the Pacific (UN- ESCAP), L’Observatoire Économique et Statistique d'Afrique Subsaharienne (AFRISTAT) and the Secretariat of the Pacific Community (SPC) and specialized UN agencies on sector-specific datasets including UNICEF on the Multiple Indicators Cluster Surveys (MICS) datasets, the World Food Program (WFP) on food security surveys datasets, and the Food and Agriculture Organization of the United Nations (FAO) on agriculture census datasets. The World Bank is also a key player in the ADP implementation, with financial support from DFID. 10 ii) Assess the extent to which MAPS actions remains relevant to the statistical capacity improvement agenda, assess sustainability, and advise on/make recommendations as to what a successor program such as the Busan Action Plan for Statistics (BAPS) could do differently to be most effective7. III. Scope and Methodology of the Evaluation 2.1 Scope 4. The evaluation shall address the questions noted below. Where appropriate, illustrations of good (or bad) practice/outcomes will be provided. i) Efficacy: Assess the performance of the MAPS partnership programs in achieving its desired results and more specifically in terms of outcomes, impacts and their sustainability:  To what extent have the MAPS partnership programs/DGF funded activities supported actions at the national and the international level?  To what extent have the MAPS partnership programs/DGF funded activities contributed to increased statistical capacity in countries?  To what extent have the MAPS partnership programs/DGF funded activities contributed to an improvement in the availability of key statistical data?  Are these outcomes and impacts likely to be sustainable?  Did they trigger more local awareness and coordinated efforts? ii) Efficiency or cost effectiveness of the programs  Have the programs delivered as promised and cost more or less as planned? 2.2 Methodology 5. The evaluation is based on: - A desk review of DGF-funded activities of the largest five partnership programs which include the ADP, IHSN and NSDS (PARIS21), and the 2010 Population Census Round and Gender Statistics (UNSD); - A review of the outcomes of the independent IHSN/ADP evaluation; - Meetings and interviews of task managers; - Interviews with MAPS Unit staff at the World Bank; and 7 PARIS21’s work programme was revised following the approval of the BAPS and the PARIS21 Board’s decision to assign to PARIS21 Secretariat the responsibility of assuring the Secretariat of the BAPS, and to incorporate new work streams emerging since Busan. 11 - Additional sources of information including interviews with former PARIS21 staff, consultants undertaking work for PARIS21 and other procedures including quantitative methods where relevant. a) Approach to the assessment of efficacy 6. Not having access to stakeholder analysis and more particularly to the direct feedback from the beneficiaries of the MAPS Partnership programs given the relatively limited scope of this evaluation, the evidence on efficacy -- the extent to which activities and outputs generated by the five programs have helped to advance their broad objectives -- is drawn from past evaluations of these programs where available and from a variety of sources as noted above. For the IHSN and ADP programs, a comprehensive external evaluation was completed in 20138. For the NSDS program, the detailed evaluation of 2009 which looked over the first 10 years of PARIS21’s existence provided some useful insights. Key conclusions of the 2009 evaluation were subsequently corroborated by a further evaluation conducted in 2011 by the World Bank’s Independent Evaluation Group of three related programs – PARIS21, the Trust Fund for Statistical Capacity Building (TFSCB) and the MAPS. The PARIS21 Secretariat is also undertaking an internal review of key documents issued during the period 1999-2014. UNSD’s programs on gender statistics and on the promotion of country participation in the 2010 Population and Household censuses have not been evaluated in recent years. The absence of recent external evaluations on the NSDS partnership program is partly compensated by a growing volume of information on the implementation and review of NSDSs at country level and new initiatives on sub-regional initiatives9. 2.3 Limitations to the evaluation 7. In view of the limited scope of this evaluation, important considerations such as governance and management arrangements of the programs and financial sustainability particularly given the phasing out of the DGF which contributed the bulk of funding for the three partnership programs managed by PARIS21 will not be assessed in this report. Suffice to say, however, that based on a cursory review on these issues in the external evaluation reports, there do not appear to be noteworthy inconsistencies or anomalies in the governance or management arrangements for the five partnership programs. The phasing out of DGF funding will, however, require substantially different approaches in the three partnership programs managed by PARIS21. PARIS21 Secretariat is in the process of revising its strategy. 8 Thomson Anne, G. Eele and F. Schmieding (2013), Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP), Final Report . 9 Such as for example, SADC in Africa, the Small Island Development States (SIDS) of the South Pacific region and ASEAN in Southeast Asia. 12 IV. The International Household Survey Network (IHSN) and Accelerated Data Program (ADP) 3.1 Goals and objectives 8. The IHSN is an informal network and the voluntary membership is currently limited to organisations and agencies providing financial and/or technical support to the implementation of survey programs in developing countries. IHSN does not provide technical or financial support to producers or users of official statistics in developing countries. 9. The IHSN objectives were to improve the availability, accessibility and quality of survey data, to avoid duplication of data collection activities, to improve cost-effectiveness of surveys, and to reduce the burden of international survey programs on national statistical systems. Priority areas of work were: ‐ Improved coordination of internationally sponsored survey programs, by fostering better timing, sequencing and frequency; ‐ Fostering the adoption of international standards and best practices by harmonizing data collection instruments; ‐ Fostering better use of existing survey data by establishing and maintaining a central survey data repository; and ‐ Developing tools and guidelines for harmonization of international survey methods and instruments. 10. The ADP is a technical assistance and training programme to help developing countries adopt and use the standards, tools and guidelines generated by IHSN. Its main goal is to improve survey programs and increase the use and value of survey data in developing countries which is to be achieved by: ‐ Building national capacity in microdata preservation, analysis, anonymization10, and dissemination; ‐ Working with national data producers and secondary users on the production of updated estimates of key indicators, by further exploiting existing datasets and collecting new data; ‐ Foster better inter-agency cooperation in country activities, under the umbrella of the IHSN, and ‐ Assisting countries that do not have a coherent long-term survey programme in strategizing their data collection activities. 10 Statistical disclosure control management of microdata. 13 3.2 Progress of activities and outputs A. The International Household Survey Network 11. Based on the yearly Progress Reports submitted by PARIS21 to the World Bank for the years 2006 to 2012, PARIS21 reported a wide range of activities performed under the DGF grants. To ensure consistency with the broader list of projects presented in the IHSN website, these activities are presented using the headings of the IHSN website: Survey planning, coordination and integration Maintenance of the IHSN Information System on Planned Surveys and Censuses: ‐ this activity which was only mentioned once in the first Progress Report for 2006; there has been no available update since11; and ‐ Contributions to the Development of the Survey Management Toolkit: including the development of survey budgeting tool and a survey planning checklist. Survey methods and assessments ‐ Contributions to the development and maintenance of the IHSN Microdata Management Toolkit (MMT): Regular update and improvement of the Toolkit. A light web-based Survey Management Tool (SMT) was developed to help document surveys on a real time basis. The SMT was not developed as a separate tool, but its functionalities were included in the key components of the Toolkit (NADA in particular); ‐ Development of the IHSN National Data Archive: The National Data Archive (NADA) is an open-source application which can be customized and integrated in national data producer’s websites of ADP participating countries. NADA 1.1 was launched in 2008 and is on its fourth version; and ‐ Contribution to the development of a ‘Question Bank’: the application has been developed to harmonize data collection instruments at country level and is a central feature for implementing Task 2 of the ADP. Data analysis and use ‐ No activity reported. Data curation ‐ Development of Microdata Anonymization Tools: The tools have been developed and are being tested in regional workshops12. Survey cataloguing ‐ Maintenance of the IHSN Central Survey Catalog: 4191 surveys available on the IHSN website with multiple partnerships established with data depositors. Microdata dissemination 11 The system was developed, but never fully implemented as it proved to be impossible to get timely and reliable information from statistical agencies and sponsors. 12 The World Bank is also doing a large-scale desk-exercise testing of the tools. 14 ‐ Production of guidelines on data documentation, dissemination and archiving and on survey design, budgeting, implementation and integration. B. The Accelerated Data Program Country Level Activities 12. The DGF grant agreement stated that the ADP objectives should be achieved in participating countries by: (i) making existing survey microdata more widely and easily accessible to secondary users, with a view to fostering better and more diverse analytical work and use of data; (ii) assessing the quality of survey data with a view to improve the relevance and quality of future surveys; (iii) assisting countries in designing improved mid- and long-term survey programs; and (iv) co-financing the implementation of data collection activities. These activities were organized under three ADP Tasks as shown below: Task 1: Establishment of national survey data repositories in participating countries, to make existing survey microdata more widely and easily accessible. - Support including training was provided to about 70 participating countries on survey inventory, documentation, archiving, and making survey metadata freely accessible through National Statistical Organisations (NSOs)’ websites and NADA portals (50 countries), and on developing national microdata dissemination policies (7 countries). Task 2: Assessment and improvement of the quality of existing surveys. Development of a national “Question Bank” to harmonize and improve data collection practices at the national level. - Support to Cameroon on education indicators and to Nigeria on improving the monitoring of access to water and sanitation (as part of the joint UNICEF/WHO monitoring); to Ethiopia and Cameroon on the adoption of national “Question Banks”; and to the Philippines on the assessment of household surveys. Task 3: Design of improved national survey programs and implementation of new data collection activities. This task is not considered as a priority. - Support household survey data entry in Niger and survey data analysis in Congo DRC (2007); support the analysis and dissemination of household survey in Cameroon and water access survey in the Marshall Islands (2008); Vanuatu on the hybrid survey and the Philippines on survey planning (2011). 13. Originally, the plan was to perform all three tasks for 12 countries jointly chosen by the World Bank and PARIS21. Subsequently it was decided to focus on providing developing countries with support to improve the management of survey data as a means to disseminate key socio-economic data (see Task 1) 13. 13 With the DGF budget allocated annually, it would have been impossible to finance data collection activities except in very few cases. So the decision was made in the early days to focus on Task 1 where the demand was highest, then to move to Task 2. Task 3 was put on a stand-by basis. 15 Global level activities 14. These activities involve the partnership with other agencies on ADP implementation and the establishment of regional communities of practice of data documentation and dissemination specialists in Africa and Latin America. 3.3 Relevance of the program 15. As called for by the terms of reference of this evaluation, the review of the IHSN/ADP program should build on the results of the external evaluation of IHSN/ADP completed in August 2013. This section of the report therefore relies to a great extent on the findings of the external evaluation report of 2013, management responses to the evaluation together with follow-up activities supplemented by other information where relevant. For this reason, the assessment of relevance, efficacy and efficiency of the IHSN and ADP programs is done jointly. The executive summary of the external evaluation is attached as an annex to this report. The evaluation of the IHSN/ADP program is therefore richer than that of the other three partnership programs covered by this evaluation. 16. Interviews by the external evaluation team in country visits as well as with a wide variety of people active in statistics confirmed that the establishment of IHSN and ADP was the relevant response to the recommendations of MAPS and that their missions have continued to remain relevant. The two programs have filled a clear gap in statistics since no other statistical program has focused on archiving surveys in increasing access to microdata. In-country visits by the external evaluation team confirmed that the work of IHSN/ADP programs was seen as relevant to improving the capacity of national statistical systems. 3.4 Efficacy: Outcomes, impacts and their sustainability 1) Outcomes: To what extent have the MAPS partnership programs/DGF funded activities supported actions at the national and the international level? 17. The external evaluation concluded that there was a general consensus that archiving of and access to household survey data were important for improving the evidence base for policy and for monitoring progress towards the MDGs. However, one of the original objectives in setting up IHSN -- to improve coordination of internationally sponsored survey programmes at country level -- was not achieved in spite of the wide acknowledgement that better survey coordination at country level is a much desired objective. The external evaluation team advanced a number of reasons on the lack of success on survey coordination including the dependence of most countries on external resources for undertaking major surveys. More significantly, it also suggested that the practice of IHSN of engaging partners on a one-to-one basis rather than working collaboratively which may have reduced the IHSN’s potential impact on promoting survey coordination. 18. On this specific point, field experiences tend to show that, unless national statistical offices are willing to play a greater role on survey coordination, funding agencies, who are also facing their own constraints particularly in terms of funding, have passively resisted coordination. As one of the key actions taken to address concerns expressed by the OPM evaluation, it was decided that the mandate for survey coordination should not reside with the IHSN and to engage 16 the UN Statistical Commission (UNSC) in this process. A paper was prepared for the 2015 UNSC meeting, endorsed by the IHSN Management Group, which makes suggestions including the adoption of a Code of Practice for surveys, and giving a coordination role to UNSD in a support role. 19. In the other areas of activities and outputs, IHSN has performed well. For instance, the eSurvey of statistics producers14 by the external evaluation team revealed that 76% of the respondents had used the microdata management toolkit and the majority rated the toolkit as very useful. A total of 64% of the respondents15 had used the NADA software and 89% of them rated it as very useful. For ADP, the inventory, documentation and archiving of national survey data have been achieved in a significant number of countries as ADP responded to demand. According to the external evaluation, some progress has been made on improving access to microdata but this remains rather limited. 20. While the impact on data use is hard to measure, information on the usage statistics of selected NADAs show that there is a very large number of visits and downloads from NADA catalogs16. According to the external evaluation, the rush in addressing rising demand has prevented the ADP from developing detailed work plans with specific countries with clear milestones and time frames. A start has been made to improving data and survey quality but progress has been slow. The slow progress could be partly attributed to the fact that ADP is not directly involved in survey design and implementation which therefore would minimize its impact on data quality. It should get and the number is your thinking on his which be further noted that the training and expertise building in data documentation and dissemination would likely have a significant albeit only indirect on data quality. 2) Impacts: a) To what extent have the MAPS partnership programs/DGF funded activities contributed to increased statistical capacity in countries? 21. At country level, the results from the eSurvey showed that the IHSN catalogue has been a useful reference tool but that NSOs are not doing as much as they could to improve access. The eSurvey findings also showed that NSOs appeared to have given more importance to microdata management tools than to facilitate and improve access to microdata. Investments in the anonymization tools under IHSN/ADP are in part driven by the need to address the obstacles to microdata access. b) To what extent have the MAPS partnership programs/DGF funded activities contributed to an improvement in the availability of key statistical data? 14 Based on a sample of 58 statistics producers from 49 countries. 15 The NADA catalogs usage statistics show that NADA are extensively used. The World Bank/IHSN NADAs for example have more than a million hits per year. 16 Further discussions with IHSN staff confirmed that a growing number of countries are actively disseminating data compared to 2 years ago and that the recent effort to compile a database of citations to monitor the use of microdata for research and analysis show more than 18,000 as per the latest information. 17 22. In terms of impact, the external evaluation confirmed that IHSN/ADP has improved data availability, both through the IHSN catalogue and through assisting countries to establish their own NADAs. The program has also contributed to more open data. But, due to the lack of information, the external evaluation team was inconclusive as to whether there was an increased use of existing survey data in spite of the fact that access to microdata appeared to have increased in the number of countries. And although IHSN/ADP has contributed to improved relevance, usability, accuracy and timeliness of survey data, the external evaluation team found little evidence as far as improved quality of survey data in participating countries. 23. The external evaluation also found limited impact on the collaboration between data producers and data users. In an effort to foster this collaboration, ADP has since 2013 supported a series of Microdata Outreach Workshops engaging microdata producers and users. The workshops served the dual purpose of raising users’ awareness of existing microd ata and dissemination policies, while helping data producers learn about users’ needs and feedback. PARIS21 has also begun to provide regional trainings to launch the NSDS Data Annex – a module to help in planning data collection operations and estimate field operation costs. The output provides an indispensable information and support documentation on the estimated budget needed to implement statistical operations that would generate indicators identified in the national development plan. c) Are these outcomes and impacts likely to be sustainable? 24. The eSurvey by the external evaluation team revealed that most countries are still either directly receiving support or are continuing activities on their own and that new countries are joining the ADP. A substantial majority of countries that have received training in the use of the toolkit are continuing to document surveys without ADP support. According to the external evaluation, the picture is a little more mixed with respect to the NADAs. The external evaluation reported that one-sixth of the NADAs installed were not running at the time of the evaluation but it was not clear whether this is related to technical capacity and technical problems or to a drop of interest in the country about microdata management including dissemination. Further discussions with IHSN staff confirmed that the actual number of non-operating NADAs is much smaller (4 sites out of 77) and that the IHSN/ADP has established an automated monitoring system to check the working of NADAs on an almost continuous basis, making it possible to provide remote assistance and support when needed. 25. In its assessment of the sustainability of the outputs by IHSN/ADP, the external evaluation team came to the conclusion that IHSN guidelines are not a concern. For the program’s three established outputs -- the ongoing maintenance and upgrading of the software developed by IHSN, the use of the microdata toolkit to archive data and the maintenance of country NADAs --, efforts must be pursued to maintain and upgrade the software. Based on country observations, the external evaluation team did not see any major issues in countries where the NSOs have good technical capacity, a strong commitment for the objectives regarding microdata and a good degree of financial autonomy. For other countries, some degree of institutionalization of data archiving and the maintenance of a national data archive may be needed to avoid undoing the achievements of IHSN/ADP. 18 d) Did they trigger more local awareness and coordinated efforts? 26. According to the external evaluation team, the IHSN catalogue has created greater awareness of what microdata are available. Among respondents to the eSurvey of data producers, 52% were aware of the IHSN guidelines and felt they were of high quality. However, the external evaluation team observed that awareness of IHSN and its work is very limited, even among organisations that are part of the IHSN network. Based on country visits, it was observed that the IHSN catalogue is recognized as a useful reference tool but that NSOs are not doing as much as they could to improve access. The external evaluation team made the suggestion that if ADP had, instead of quickly moving into a large number of countries, focused on a few countries to work through the different stages from activities through to outcomes, particularly in terms of achieving greater use of existing surveys, this might have resulted in a better understanding of what is required, in terms of advocacy, political support and even legal and policy structures to enable a much greater increase in use of surveys and their data. 27. A counterargument to the recommendation by the external evaluation was that the ADP “light touch” approach was in part a deliberate choice to respond to the demand by countries so as not to miss the opportunity to promote the adoption of the DDI standard, which has become a de facto global standard for microdata thanks in great part to the training and support provided by ADP. 3.5 Efficiency 28. The external evaluation finds that the use of financial and human resources by both IHSN and ADP has been quite economical. Both IHSN and ADP have used their financial resources very economically. Overheads have been low, and close to 90% of grant funding has gone directly to implementation. The number of full-time equivalent staff in both programs has been kept very small thanks to the reliance on international consultants many of whom from South. 3.6 Main findings 29. The support to the IHSN/ADP program has helped to raise awareness of the importance and availability of microdata in the large number of developing countries over a relatively short time. It has also helped to improve data availability and contributed to more open data. At the same time, while the findings of the external evaluation were inconclusive on the use of microdata and on their quality in participating countries to the ADP, discussions with IHSN staff confirmed that much progress has been achieved in the recent past. The external evaluation team also found limited interaction between data producers and data users. To foster this collaboration, ADP has begun to support the promotion of user-producer dialogues at country level through Microdata Outreach Workshops (MOWs) and to strengthen the links between ADP and NSDSs by initiating new working arrangements at country level combining the ADP-type ‘data team’ with the NSDS ‘policy team’ and the assistance to provide countries with a Data Annex in their NSDSs. These measures are, however, too recent to be usefully assessed. 30. The external evaluation team found the limited effectiveness and impact of IHSN on survey 19 coordination at country level and recommended that more efforts will be needed in terms of approach – more specifically, working arrangements -- and institutional setup of the IHSN network to begin making progress in this very important area. While survey coordination was one of the main goals for establishing the IHSN, a rethinking has taken place in this area and it was decided that the mandate for survey coordination should not reside with the IHSN and to engage the UN Statistical Commission (UNSC) in this process. 20 V. Preparation of National Strategies for the Development of Statistics (NSDSs) 4.1 Goals and objectives 31. According to the terms of the arrangement, the contribution by the DGF was intended to finance PARIS21’s support to NSDSs with an evolving focus but no specific activities and outputs as was the case for the other three partnership programs covered in this evaluation. In the early years of the partnership program, the main objective was to assist all International Development Association (IDA) countries to design an NSDS in order to have by end-2006 nationally owned and produced data for all Millennium Development Goals indicators by 2010, which was one of the 6 main objectives of the MAPS. The NSDS work program main elements include advocacy, the development of NSDS methodology, most importantly the NSDS roll-out at country level and through regional programs. In addition, the DGF grants also covered costs related to the functioning of the PARIS21 Secretariat and the organization of the yearly Executive Committee meetings and other miscellaneous items. 4.2 Progress of activities and outputs 32. The NSDS partnership program helped to provide funding to PARIS21 for advocacy to promote support to statistics, the NSDS process and funding and coordination which are detailed below: 1) Advocacy: The work program in the earlier years focused on the development of the advocacy toolkit17, the delivery of advocacy messages at several regional statistical meetings to promote increase support to statistics and assistance in finalizing advocacy booklets in several countries with a strong focus on sub-Saharan Africa; 2) NSDS Processes: a) NSDS Methodology: The first set of NSDS Guidelines18 was issued in 2007. Subsequent grants have financed the compilation of good NSDS practices and a guide on the financing and costing of NSDS. The 2013 DGF grant also helped to cover part of the costs relating to the production and finalization of the new NSDS guidelines which take into account lessons learned from the design and implementation of NSDSs in nearly a hundred countries since its adoption and integrate lessons from global statistical development that complements the NSDS approach. They also feature specific country situations including fragile and small-island developing states (SIDS), sectorial strategies, intra-national 17 The Statistics Advocacy Resources Toolkit (START). 18 A Guide to Using the System-Wide Approach to Implement National Strategies for the Development of Statistics (NSDS). - 21 - strategies, and regional strategies among others. The new guidelines will be regularly updated to document new lessons and good practices on NSDS design and implementation. b) NSDS processes at national level: PARIS21’s support to NSDS processes at country level includes direct technical support, the promotion of partnership, the facilitation of peer reviews and sharing of experiences across countries and the preparation of donor roundtables for NSDS financing. In more recent years, PARIS21 has been involved in a series of new activities in the context of NSDSs including mid-term reviews and evaluations of NSDSs, the provision of specific support to fragile states and Small-Island Developing States (SIDS) and encouraging the establishment of dialogue mechanisms at the country level. These cover a wide range of activities including the coordination of all national statistical system actors, the promotion of local donors groups on aid coordination for statistical support, the facilitation of interactions between the producers and users of statistics, and the integration of sectoral strategies in NSDSs. As mentioned above, as part of the new strategy which is been finalized, the PARIS21 Secretariat will promote more active interactions between the NSDSs and the data work program under the IHSN/ADP with the creation of two teams, a data team and a policy team, who will be working collaboratively at country level. It is proposed that a data annex be developed for the NSDS yielding two concrete objectives: i) to integrate the IHSN/ADP work in the NSDS and ii) make data production and dissemination a more specific objective of the NSDS. With these new initiatives, the NSDS work program, which has until now been conceived as a process, will begin to have specific targets as outputs. As part of its overall NSDS work program, PARIS21 initiated, in 2006, the preparation of six- monthly progress reports on the status of preparation and implementation of NSDS. c) NSDS activities at the regional level: These have also evolved over time, starting in the earlier years with regional workshops aimed at promoting advocacy and training in the preparation of NSDSs to helping various sub-regions (Central America, Southeast Asia and Southern Africa) develop specific priorities for regional strategies for the development statistics (RSDSs) to improve data quality and dissemination, to raise the level of appreciation of statistics by senior policymakers in the region and to promote aligning the NSDSs with the RSDSs. 3) Funding and co-ordination of support: Another activity under the NSDS work program revolves around the collection and reporting of donor support to statistical capacity building with the publication of the yearly “Partner Report on Support to Statistics” (PRESS) starting in 2008 and covering the activities of more than 40 bilateral and multilateral partners providing technical and financial support covering all areas of statistics in developing countries. Based on the PRESS experience conducted at international level, PARIS21 helped the launching of an initiative to gather all data relating to the funding of the national statistical system from both domestic resources and external aid known as the “Country Reports on Support to Statistics” (CRESS) in six African countries (Benin, Ghana, - 22 - Ethiopia, Senegal, Cameroon an Malawi) in an effort to promote advocacy and coordination of support for statistics. 4.3 Relevance 33. The IEG global program review of the MAPS, PARIS21 and Trust Fund for Statistical Capacity Building undertaken by the World Bank in 2011 highlighted the strong international consensus for the support to NSDSs as a key goal of the MAPS -- the preparation of NSDSs to improve the availability of key indicators in order to monitor progress towards the MDGs. At the same time, support to NSDs is completely aligned to those of developing countries as articulated in their poverty reduction support papers (PRSPs) on the importance of statistical development particularly in relation to the need for monitoring and evaluation. 4.4 Efficacy: Outcomes, impacts and their sustainability 1) Outcomes: To what extent have the MAPS partnership programs/DGF funded activities supported actions at the national and the international level? 34. Overall progress has been notable in relation to the design and implementation of NSDSs, a process which has taken hold in all regions. As of March 2014, all but 10 of the 82 IDA-eligible countries are either designing or implementing a NSDS. Forty four countries are currently implementing their NSDSs while 28 others are currently designing or awaiting adoption of their NSDS (representing 88 percent of all IDA countries). This ratio is even higher (95%) for sub-Saharan Africa, the region where PARIS21 has been most active. Table 1: Summary of NSDS Status – IDA-Eligible Countries Currently Currently designing Strategy expired or Strategy expired Total implementing a a strategy or absent but or absent and not strategy awaiting adoption currently planning planning one an NSDS # % # % # % # % Africa 27 65.9 12 29.3 2 4.9 0 0.0 41 Asia-Pacific 12 44.4 11 40.7 3 11.1 1 3.7 27 Eastern Europe 4 80.0 0 0.0 1 20.0 0 0.0 5 Latin America and 1 11.1 5 55.6 3 33.3 0 0.0 9 Caribbean TOTAL 44 53.7 28 34.1 9 11.0 1 1.2 82 Source: PARIS21, NSDS Progress Report – NSDS Summary Table for IDA and Lower-Middle Income Countries, March 2014. - 23 - 2) Impacts: a) To what extent have the MAPS partnership programs/DGF funded activities contributed to increased statistical capacity in countries? 35. While important challenges remain with the implementation of the NSDSs (to be discussed later in this report), the aggregate scores of the World Bank’s Statistical Capacity Indicator for 82 IDA-eligible countries showed that on average, the scores increased by 5% over the 10-year period when the NSDS process started indicating that real improvements in the capacity of national statistical systems have occurred in the last ten years. It should, however, be noted that the improvements in data availability from databases from international agencies using estimates from models or other sources have also contributed to the improvement in the capacity of national statistical systems. Most importantly, Table 1 shows a significant catch-up by sub-Saharan Africa IDA countries in the most recent years, rising by 5.7% during the 2008-10 to 2011-13 period compared to an increase of 2.3% in the previous 5 years. In contrast, the SCI score for non-Africa IDA countries which reached a high of 73.3 in 2008-10 – a level comparable to the average of mid- and high-middle income IBRD countries – has experienced a decline in the past few years. Earlier analyses such as the multi-donor assessment of PARIS21 after 10 years19 have highlighted the concern that while Africa’s statistical capacity was improving, its rate of improvement was lower when compared to non-Africa IDA countries. Table 1: World Bank’s Statistical Capacity Indicator20 (scale 0-100), 3-year Averages a/ (2004-06 to 2011-13) Percentage Percentage Percentage 2004-06 2008-10 2011-13 Overall SCI change change change average average average (2005-12) (2005-09) (2009-12) All IDA countries 60.0 62.0 63.0 5.0% 3.1% 1.6% Sub-Saharan IDA countries 55.0 56.3 59.5 8.2% 2.3% 5.7% Non-Africa IDA countries 71.7 73.3 70.7 -1.4% 2.2% -3.6% IBRD countries 73.0 75.0 73.7 1.0% 2.7% -1.7% a/ Using tree-year averages to reduce the impacts of year-to-year fluctuations in some of the components of the SCI. Most noteworthy, the periodicity and timeliness sub-component can vary a great deal depending on whether the country has undertaken a population census or an agricultural census within the 10 year recommended periodicity. 19 PARIS21 Consortium, PARIS 21 at Ten - Improvements in Statistical Capacity since 1999. 20 The Statistical Capacity Indicator (SCI) provides an overview of the statistical capacity of a country and is available for over 140 developing countries. It is based on a diagnostic framework developed with a view to assessing the capacity of national statistical systems using metadata information generally available for most countries. The framework has three dimensions: statistical methodology; source data; and periodicity and timeliness. A score of 100 indicates that the country meets all the criteria. - 24 - b) To what extent have the MAPS partnership programs/DGF funded activities led to improvement in the availability of key statistical data? 36. The Millennium Development Goals have stimulated demand for better statistics which have been partly met by new statistical capacity projects and programs in support of developing countries. The result has been a marked improvement in the quality and availability of statistics on core development outcomes: poverty and income distribution, school enrollments, mortality and morbidity rates, and environmental conditions. Progress in data production in countries, improvements in the reporting from countries to the international statistical system and increased access and understanding by agencies of existing national sources have resulted in a greater availability of data in the database of Millennium Development Goal indicators. Table 2 shows the availability of data for the assessment of trends, measured as the number of countries that have data before and after 2000 (at least one data point) in the database. Availability of trend data is shown as of 2003, 2006 and 2013. In order to compare the current status of data availability with the status in 2003, only a subset of indicator series is considered, which includes those that have remained unchanged over the years in terms of definitions and methods of reporting. The number of countries for which data are ava ilable for a large number of indicator series has continued to increase over the years. In particular, the number of countries with trend data for between 16 and 22 series increased from zero in 2003 to 84 in 2006 and 115 in 2013. Meanwhile, the number of countries with trend data for only zero to 5 series has fallen from 47 to only 7 between 2003 and 2013. 37. Although this is not solely due to actual progress taking place in countries, it does reflect, at least in part, an increased availability of data in national sources and a stronger capacity of national statistical systems in addressing the monitoring requirements. Other factors also played a role, including the improvement in the reporting mechanisms from countries to international agencies and increased access and understanding by agencies of existing national sources. - 25 - Table 2: Number and percentage of countries and territories of developing regions w ith Millennium Development Goal indicators for which trend analysis is possiblea/. As at July 2003 As at July 2006 As at July 2013 Number of indicator Number Percentage Number Percentage Number Percentage seriesb/ 0-5 47 29 15 9 7 4 6-10 66 40 22 13 11 7 11-15 50 31 42 26 30 18 16-22 – – 84 52 115 71 Total 163 100 163 100 163 100 a/ The criteria used to determine whether trend analysis is possible for a given indicator series is that the country has data for the given series for at least one point before 2000 and one after 2000. b/ Refers to indicator series for which reporting is done on countries and territories of developing regions; the calculation for the indicator series on donor countries is presented separately. Source: UN Statistical Commission (2013), Indicators for Monitoring the Millennium Development Goals, Report of the Secretary General (Report E/CN.3/2014/29). c) Are these outcomes and impacts likely to be sustainable? 38. While other programs -- and most particularly the World Bank’s Trust Fund for Statistical Capacity Building (TFSCB)21 and the African Development Bank’s support for statistical capacity building particularly in the context of the Reference Regional Strategic Framework for Statistical Capacity Building in Africa have indeed contributed to statistical capacity building, the review of the relevant literature and consultations undertaken for this evaluation confirmed the critical role played by PARIS21 in assisting developing countries with advice and technical support relating to the development of national statistical development strategies. Some 50 developing countries, of which about half are in sub-Saharan Africa, have benefited from PARIS21 support. Through its ‘more neutral’ approach -- the provision of advice and technical support without having to require recipient countries to deliver specific outputs such as in the case of project managers – PARIS21 is widely perceived by recipient countries as a friendly advocate for statistical development in developing countries with a positive reputation as a valuable part to the architecture of data and statistic development and cooperation. d) Did they trigger more local awareness and coordinated efforts? 39. The review of relevant documents, interviews with DECDG staff held in the context of this evaluation and various discussions at PARIS21’s annual Board meetings confirmed the strong relevance of the partnership program’s objectives and activities. NSDSs have strengthened the status of statistics in the development agenda, fostered a growing attention to the data demand, through a dialogue with users, particularly with sector ministries. The 21 TFSCB provide small grants of up to $500,000 over a period of two or three years mostly to low-income countries to support the preparation of NSDSs, working with PARIS21. - 26 - shift in PARIS21 work program focusing on the new guidelines and the promotion of user- producer interactions which have proven to be critical in promoting the sustainability and quality insurance of statistics have been well-received by the global statistical community. 4.5 Efficiency of the program 40. Funding for PARIS21’s core program related to NSDS processes including country and regional programs, advocacy, and donor coordination have registered very moderate changes in the last few years. Estimated expenditures for the core program in 2014 were Euros 3.6 million. Management costs have been held steady over the years at around 15–21 percent of total expenditures. 4.6 Main findings 41. The evaluation confirms important progress in the NSDS process which has helped to elevate the issue of statistical capacity building throughout the developing world. NSDSs have allowed for fuller visibility of the national statistical offices in the discussion within the government and with development partners. Mid-term and final reviews of NSDS implementation have generated useful lessons which could help in furthering progress in the future. PARIS21 has broadened its areas of intervention on NSDSs with more involvement in the review of NSDS implementation, the coordination of assistance to statistical capacity building through the advocacy for local donors group for statistics and for more sustainable funding arrangements for statistics, the integration of sectors into the NSDSs and a more active involvement in regional strategies for the development of statistics (RSDSs). 42. In spite of the steady progress on the NSDS process, consultations for this evaluation confirm that important challenges remained particularly those concerning the implementation of NSDSs which was also highlighted in the IEG review’s main findings. Causes for the lack of progress on implementation are numerous and only the most significant ones are summarized below: ‐ In spite of much effort in their design and preparation, most NSDSs continued to focus on the outputs of the NSOs and failed to integrate sectoral statistical strategies. This has made it difficult to mainstream NSDSs into the PRSPs or similar national development processes and, as a consequence, have prevented statistics to claim adequate resources from the national and sectoral budgets as a cross- cutting development priority. Past experiences have shown that the process to achieve more inclusive involvement of sectoral priorities in NSDSs usually take longer and require skills and approaches that are inadequate and will need to be developed and mainstreamed. Without a clear strategy for integrating sectors in NSDSs22, ad-hoc 22 Such as the FAO-led Global Strategy for improving agricultural and rural statistics. For more details, see FAO, the World Bank and the UN Statistical Commission (2012), Action Plan of the Global Strategy to Improve Agricultural and Rural Statistics for Food Security, Sustainable Agriculture and Rural Development . - 27 - approaches such as in the case of Sierra Leone where Statistics Units were created in eight ministries with the assignment of a qualified statistician in each of the ministries have not led to meaningful results. In Tanzania, efforts to integrate agriculture in the NSDS have met with little success until recently with the support of the FAO-led, AfDB- operated Global Strategy for improving agricultural and rural statistics; - Most NSDSs continued to be focusing on the production of statistics with weak or inexistent linkages with the users of data. Broadening and deepening the user/producer dialogue on statistics, which is beginning at PARIS21 with the organization of Microdata Outreach Workshops mentioned under the discussion of ADP represents a step in the right direction; - While some countries have been successful in negotiating a relatively efficient and funding arrangement (e.g. basket funds) for external resources such as in the case of Rwanda, Tanzania and Malawi, funding for statistics generally remains uncoordinated in most countries. Facilitating the sharing of country experiences together with advocacy for more coordination in funding is a role that PARIS21 may be well-placed to perform giving its ‘neutral’ attitude vis-à-vis other donors; - The reviews of NSDS implementation have highlighted in some cases the lack of specific target setting in terms of activities and output and the absence of a monitoring and evaluation system as significant impediments. Having concrete results set using a pragmatic, country-based approach could go a long way in making NSDSs more realistic and strategic. Again this is an area where PARIS21 could leverage given its close relationships to the NSOs across the developing world; and - The experiences of regional events such as the High Level Seminar and Workshop on ASEAN Community Statistical System (ACSS) Strategic Plan 2016-2020 where clearly defined priorities in terms of data quality and dissemination have helped to raise the awareness among policy makers of member countries for ASEAN statistics. VI. Improvement in Gender Statistics 5.1 Goals and objectives 43. Women's empowerment and the promotion of gender equality are key to achieving sustainable development. DGF grants have contributed significantly to the work of the UNSD’s Social and Housing Statistics Section in the area of gender statistics including: a) the development of technical materials; b) the compilation and dissemination of online data and meta-data for the minimum set of gender indicators; c) the organization of the Interagency and Expert Group meetings; and d) the building of country capacity through regional workshops. - 28 - 5.2 Progress of activities and outputs 44. Unlike for the NSDS where DGF provides funding for a continued process without specific targets, funding for the improvement in gender statistics are done through annual grant agreements with specific activities and outputs defined for three objectives. The first objective, which is to improve the capacity of countries to produce statistics on informal employment, was a focus area in the earlier years of DGF funding (2006-08) but has dropped out of the yearly work arrangements since. The three objectives and the accompanied activities and outputs are summarized below: Objective 1: Improve the capacity of the participating countries to collect, analyse and disseminate statistics on employment, particularly on women’s and men’s participation in the informal sector and in informal employment, following internationally agreed standards covering: ‐ Development and implementation of national plans for the production of sex- disaggregated data on employment in the informal sector and informal employment; ‐ Gender analysis and dissemination of the data collected on employment in the informal sector and informal employment; ‐ Training on informal sector employment (Lesotho, 2008). Objective 2: The development and application of international statistical standards for gender statistics including: - Development of technical material on gender statistics (2010) - Development of technical material on gender statistics: Guidelines for Developing and Producing Gender Statistics; - Preparation of a gender statistics Manual – Integrating Gender Perspective into Statistics; - Development of IT applications based on the Gender Statistics Manual; - Compilation and online dissemination of data and meta-data for the Tier-I Minimum Set of Gender Indicators (2013); - Finalization of the work on the International Classification of Activities for Time-Use Statistics (ICATUS) to ensure that the classification is consistent with the ILO new definition of work (2012-13); - Participation in major meetings on methodology and data availability of gender statistics (2013); and - Preparation of the World’s Women 2015. Objective 3: Improve the capacity of countries to develop gender statistics consisting of: - Support the organization of Inter-agency and Expert Group (IAEG-GS) meetings (2011, 2012, and 2013); - Support the participation of member states to the Global Forum on Gender Statistics (2010 and 2012); - 29 - - Organization of regional statistical capacity building workshops, based on UN Gender Statistics training manuals (2013); - Organization of a regional capacity building workshop for sub-Saharan Africa, based on the Gender Statistics training manual (2012); and - Support study tours for national statisticians participating in training activities on gender statistics, including on time use and emerging issues (2013). 5.3 Relevance 45. The Statistics Division’s work on gender statistics was initiated in the early 1980s, mid - way into the United Nations Decade for Women: Equality, Development and Peace (1976- 1985) and in response to the call for more statistics on the status of women. The project the requested some of my holding in one of the hotel’s number 89 when he 01648 are and that they would 010 1648 on gender statistics carried out by UNSD has been shaped by demands from four international conferences on women and other conferences, such as the World Summit for Social Development (1995) and the International Conference on Population and Development (1994). The relevance of gender statistics gets renewed at regular intervals as new issues such as violence against women and time use issues receive more attention as a focus of policy dialogue at the highest level. 5.4 Efficacy: Outcomes, impacts and their sustainability 1) Outcomes: To what extent have the MAPS partnership programs/DGF funded activities supported actions at the national and the international level? 46. While it may too early to talk about the outcomes of a series of activities led and coordinated by UNSD, the contributions through the GDF have made it possible to help to advance the global agenda on gender statistics. When the Inter-agency and Expert Group on Gender Statistics (IAEG-GS) was first convened in December 2006, activities which were implemented in response to the growing demand for gender statistics were fragmented and had limited impact. In terms of coverage, the production of gender statistics still focused predominantly on traditional areas – namely, mortality, education, labour force and unemployment – with little attention to emerging areas. Using the primary criterion that indicators should address key policy concerns as identified in the Beijing Platform for Action as a guide, the IAEG identified a minimum set of gende r indicators composed of 52 quantitative indicators and 11 qualitative indicators covering norms and laws on gender equality to guide the national production and international compilation of gender statistics. The selected indicators were further classifie d into three tiers, which make it possible to gradually extent the coverage of gender statistics as new policy issues arose such as asset ownership and entrepreneurship from a gender perspective. Tier 1 indicators are conceptually clear, with an agreed international definition and regularly produced by countries; Tier 2 indicators conceptually clear, with an agreed international definition, but not yet regularly produced by countries; and Tier 3 indicators, - 30 - which generally represent emerging issues for which international standards need still to be developed and are not regularly produced by countries. 47. The other major contribution of the partnership program was the integration of a gender perspective into official statistics which covered the process of mainstreaming gender in national statistics, of identifying gaps in gender statistics and developing coherent and comprehensive plans for the production of gender statistics. These efforts have helped to institutionalized gender statistics as part of the national and international statistical systems. Since its establishment, the IAEG-GS has helped to foster the development of gender statistics in countries and globally, by promoting training activities, data compilation and the dissemination and exchange of experiences and best practices and by improving international coordination and a dialogue with countries in all activities related to gender statistics. The Global Gender Statistics Program is mandated by the United Nations Statistical Commission, implemented by the UNSD and coordinated by the IAEG-GS. The Program encompassed:  Improving the coherence among existing initiatives on gender statistics through international coordination;  Given that a key concern in developing gender statistics has been the need for reference materials, training manuals and handbooks to be used by national statisticians, developing and promoting methodological guidelines in existing domains as well as in emerging areas of gender concern played an important role both in the work of the IAEG-GS and in the partnership program activities; and  Strengthening national statistical and technical capacity for the production, dissemination and use of gender relevant data. 48. A data portal is under development to facilitate access to gender relevant data and metadata. 2) Impacts: a) To what extent have the MAPS partnership programs/DGF funded activities contributed to increased statistical capacity in countries? 49. Thirty five African countries, 34 Asian countries and 10 Latin American countries have participated in workshops, Global Forums, IAEG-WG during the period 2009-2014 which have provided training to over 500 gender-affiliated persons, with a majority of women. Ex-post follow-ups with participants that these training workshops undertaken by the UN Finance and Administration Division confirmed their efficacy in terms of relevance and acquisition of new skills and experiences six months after the workshops. b) To what extent have the MAPS partnership programs/DGF funded activities contributed to an improvement in the availability of key statistical data? - 31 - 50. As discussed in The World’s Women 2010, there has been an increase in the availability of gender statistics. The majority of countries are now able to produce sex-disaggregated statistics on population, enrolment, employment and parliamentary representation. In addition, gender statistics in some newer areas are becoming available. For example, statistics on child labour are now collected by a larger number of countries. Similarly, surveys on time use and on violence against women were conducted in both developed and developing countries although international standards in these two statistical fields have not yet been fully developed. Table 3 provides a synopsis of the current situation in terms of data availability in various domains involving gender. But many problems remain: a) statistics in certain domains are not available for many countries; b) even where they are available, there are often not comparable because concepts, definitions and methods vary from country to country; and c) in other areas, the absence of internationally agreed measurement standards and methods has resulted in a lack of gender statistics. - 32 - Table 3: Production of gender statistics in different areas, 2013 Percentage of countries producing statistics Areas Regularly Irregularly Not producing statistics Labor force 83.3 8.7 7.9 Informal employment 36.5 17.5 46.0 Unemployment 88.1 7.2 4.8 Poverty 70.6 14.3 15.1 Unpaid work 42.1 24.6 33.3 Satellite accounts 7.2 18.3 74.6 Entrepreneurship 27.8 19.1 53.2 Agriculture 44.4 19.1 36.5 Education and training 81.0 9.5 9.5 Power and decision-making 52.4 18.3 29.4 Media 15.1 19.8 65.1 Information and communications technology 38.9 20.6 40.5 Mortality 84.9 7.1 7.9 Morbidity 73.0 11.9 15.1 Disability 53.2 24.6 22.2 Access to health services 65.1 12.7 22.2 Sexual and reproductive health 65.9 14.3 19.8 Child marriage 39.7 11.1 49.2 Adolescent fertility 65.9 11.1 23.0 Violence against women 40.5 31.0 28.6 Access to clean water 37.3 17.5 45.2 Access to sanitation 38.9 16.7 44.4 51. Using the World Bank data dashboards “Gender Equality Data and Statistics”, which provides information on 432 gender-related indicators series, a comparison of data availability over time shows an improvement in the availability of gender statistics most of which from official sources (national governments and specialized international organisations) but some from other sources particularly related to financial information. Table 4 provides information on progress in terms of availability of a gender-related statistics in more recent years (2011- 13) compared to the period prior to 2005. - 33 - Table 4: Progress in gender-related statistics by topic and sub region (Developing countries only), 2005 to latest year with available data Middle Latin America Sub- East East & Pacific South Coverage and the Saharan Asia North Islands Asia Caribbean Africa Africa Causes of death by age groups X X X X X Literacy rates X X X X Mortality rate (infant and under-5) X X X X Contraceptive prevalence X X X X Smoking prevalence X Progression to secondary school X X Employment to population ratio X Labour force with primary, secondary X and tertiary education (% of total) Vulnerable employment (% of female X X X employment) a/ Financial access b/ X X X Savings c/ X X X X X Personally pay for health insurance X X X Purchase agriculture insurance X Notes: a/ Vulnerable employment is unpaid family workers and own-account workers as a percentage of total employment (ILO). b/Financial access information covers a broad range of indicators related to the percentage of women age 15+ holding of an account at a formal financial by a woman, a credit or debit card and the use of electronic payment methods, information on access to loans from various sources, on the use of mobile phone to pay bills and to receive and send money and on the use of the loan proceeds. c/ Savings information refers to the percentage of women who saved any money in the past year, where the savings ware made and purposes for using the savings. c) Are these outcomes and impacts likely to be sustainable? 52. The inclusion of a stand-alone goal on “achieving gender equality and empower all women and girls” in the Outcome document of the Open Working Group (OWG) for the Sustainable Development Goals (SDGs), adopted in July 2014, reinforces the importance of gender and will likely increase the demand for gender statistics. The World Bank committed to developing a renewed gender strategy for the World Bank Group over the coming year —with more ambitious targets, new ways of measuring progress, and an agenda for pushing ahead in new frontiers with the potential for transformational impact. In more practical terms, considerations of gender equality are now fully incorporated in the analysis, content, and - 34 - monitoring of all programs and Country Assistance Strategies; all regions of the World Bank are implementing gender equality action plans. d) Did they trigger more local awareness and coordinated efforts? 53. At the country level, one of the major challenges in producing gender statistics is that they are often seen as addressing a ‘women’s issue’ and are generally not yet mainstreamed. As a result, in most developing countries, there are no clear institutional arrangements in place to coordinate their production and limited resources dedicated to data collection and dissemination. A global review of national gender statistics programs and the extent to which gender statistics are mainstreamed23 into national statistical systems, undertaken by UNSD in collaboration with four UN regional commissions (Africa, Asia Pacific, Europe and West Asia) in 2012 revealed that much more remains to be done. Altogether, less than 20% of countries have fully achieved the mainstreaming of gender issues and about half of the other countries have partially achieved that objective. West Asia has the lowest “fully achieved” ratio (less than 10%) but 86% of the UN member countries in that region reported to have a fully or partially achieved the mainstreaming of gender statistics. In Africa, about 10% of the countries have fully mainstreamed gender statistics and another 45% have partially achieved that objective. In terms of budgets for gender statistics, out of the 126 responding countries only 13% had a dedicated gender statistics budget, 47% relied on ad-hoc/project funds and the remaining 39% had no funding at all. 5.5 Efficiency of the program 54. Lacking the information on the overall budget of UN Social and Housing Statistics Section, efficiency in this evaluation is assessed by delivery of the agreed outputs where consultations with UNSD and DECDG staff confirm that in the eight years of the partnership programs (2006-14) covered by six annual arrangements, UNSD has consistently delivered outputs agreed at the yearly DGF arrangements without cost overruns. 5.6 Main Findings 55. The work done under the partnership program has led to the establishment of a concrete program to improve gender statistics under the able coordinating role provided by UNSD of a network of international agencies including the World Bank. UNSD does not itself collect information which is forthcoming from specialized agencies such as the World Health Organization for health, UNESCO for education, the International Labor office for employment etc. The guidelines produced by the gender statistics work program and training on using the guidelines have helped to fill the numerous gaps in the international statistical framework on gender statistics, thus allowing the program to steadily tackle increasingly more complex gender indicators. The consultations for this evaluation also 23 Meaning that gender issues and gender- based biases are taking into account the systematically in the production of all official statistics and at all stages of production. - 35 - confirmed significant knowledge sharing at the various training events organized under the partnership program. 56. Although DGF funding for the gender statistics partnership program is relatively small compared to the other four policy programmes covered by this evaluation – US$ 1.45 million over a seven-year period -- discontinuation of the DGF therefore poses very important challenges to this partnership program particularly given the fact that the DGF has been the major provider for funding. VII. Help countries participate in the 2010 Population Census Round 6.1 Goals and objectives 57. The 2010 World Programme is primarily aimed at ensuring that each Member State conducts a population and housing census at least once in the period from 2005 to 2014 and disseminates the results and is as one of the key areas of action for improvements in national and international statistical capacity under the MAPS. DGF grants have contributed to the work of the UNSD’s Demographic Statistics Section in: a) developing census methodological guidelines; b) assisting countries in improving their statistical capacity to plan and conduct population and housing censuses; c) providing assistance to countries in census data documentation, analysis, archiving and dissemination; and d) monitoring the implementation of the 2010 World Programme on Population and Housing Censuses and providing support to countries that may be facing problems. 6.2 Progress of activities and outputs 58. The work programme covering the 2010 Population Census Round is time-specific covering the decade 2005-14 with the objectives that all countries, areas and territories would have undertaken a population census. Most of the work covered during the earlier part of the decadal round involves the finalization of guidelines to allow for their adoption by countries. In the latter years, the focus is on assisting countries in the implementation of the 2010 Round. The activities covered by the partnership programs are summarized below: Activity 1: Support activities relating to the implementation of the 2010 World Programme on Population and Housing Censuses in developing countries (2006-09). 2006 - Finalize the draft United Nations Principles and Recommendations for Population and Housing Censuses; - Promote the understanding by member states of the revised Principles and Recommendations for Population and Housing Censuses (P&R) and the 2010 Round of Population and Housing Censuses; - 36 - - Establish regional initiatives for developing joint projects in planning, conducting, data processing and disseminating population and housing censuses (2006); and - Develop technical guidelines on census cartography and the use of new technologies such as the Geographical Information Systems (GIS) and Global Positioning System (GPS). 2007-08 - Organize regional workshops on international standards which also helped to assess countries’ needs and plan technical assistance accordingly; - Monitor country progress in conducting population and housing censuses and their financing needs; - Assist countries to implement the United Nations Principles and Recommendations for Population and Housing Censuses; and - Build technical systems to help countries conduct their population and housing censuses (preparation of handbooks and guidelines and of training and reference materials). Activity 2: Implementing the 2010 World Programme on Population and Housing Censuses in developing countries (2009-12) 2009-10: - Enhance progress by countries in conducting population and housing censuses and in disseminating information on census activities and on national practices in census methodology; and - Develop guidelines and technical materials pertaining to the different stages of a census (post enumeration, data capture). 2011-12: - Improve capacity of countries to analyse and disseminate census data; - Improve capacity of countries to document and archive census data; - Provide technical assistance to help countries to plan and conduct population and housing censuses; and - Monitor and disseminate information on implementation of the 2010 world programme on population and housing censuses. 6.3 Relevance 59. Helping countries participate in the 2010 population census round comes directly from a call by the United Nations that all member countries undertake a population census in the decennial period. Population censuses are the primary source of basic national population data required for administrative purposes and for many aspects of economic and social research and planning. The first Millennium Development Goal (MDG1) which calls for of the proportion of people whose income is less than $1.25 per day is based on the size of the population. Increasingly more advanced economies are generating updated population figures on a regular basis through registers as part of the need for economic policy decisions. - 37 - Furthermore, population centres information is increasingly being used by the private sector to make important decisions about their businesses, including investment strategies, hiring plans, and location of facilities. 6.4 Efficacy: Outcomes, impacts and their sustainability 1) Outcomes: To what extent have the MAPS partnership programs/DGF funded activities supported actions at the national and the international level? 60. Out of 235 countries, areas or territories which are being monitored for carrying out a population census in the 2010 census round spanning the period 2005 to 2014, 214 have either conducted at least one census24 or reported detailed population statistics using population registers, administrative records, sample surveys or other sources or a combination of those sources. Together these 214 and areas account for an estimated 6.39 million persons or 93% of the estimated world population. As of December 1 st, 2014, 21 countries, which together have an estimated population of 504.4 million or 7% of the world population, have not conducted a census for the 2010 round. In terms of regional distribution, 9 of the 21 countries in Africa, 5 in West Asia, 3 in Europe, and 2 each in Asia and Latin America, 61. Figure 1 shows the overall temporal distribution of censuses over the period of the 2010 round (first censuses only). The peak of the census round in terms of the number of censuses taken is the year 2011 when 60 countries or areas conducted a census. Figure 2 shows the cumulative distribution of the censuses. At the end of 2011 (the 6th year of the 10- year census round), 87% of the world population was enumerated. 24 Some countries undertake more than one population centres during the 10-year period. - 38 - Figure 1: Distribution of censuses by year over the 2005-14 period Figure 2: Cumulative coverage by population censuses - 39 - 2) Impacts: a) To what extent have the MAPS partnership programs/DGF funded activities contributed to increased statistical capacity in countries? 62. Given the diversity in statistical capacity among the 235 countries, areas or territories covered by the 2010 Population Census Round, it is challenging to come to a detailed assessment of the impact of the partnership program on countries’ statistical capacity. A review of by the US Census Bureau25 undertaken in 2013 at the request of the UN Statistics Commission to review the experience of the 2010 World Program on Population and Housing Censuses based on censuses have been completed by end-2011 and to provide insight into the potential direction for the 2020 World Census Program, noted that all countries reported efforts to face mounting demands to improve data quality and to disseminate data faster while facing escalating costs of population censuses. For countries with more advanced statistical capacity, the survey results showed a shift toward the increased use of alternative census methodologies and technological advances throughout all the census phases and the expanded use of multiple sources of collect data -- registers, administrative records, and sample surveys are either replacing field enumeration or are being used to supplement data collected directly by enumerators -- in an effort to reduce costs while taking advantage of new technological developments. Regions were also working together in new ways, consulting with each other, sharing resources, technology (hardware and software), and forming partnerships. 63. While Africa and Oceania continued to rely fully on full field enumerations as the main data collection methodology, it was reported that significant improvement occurred in census cartography methodology and in data capture where OCR/OMR/ICR26 technology has become more prevalent. Four countries have experimented with the use of handheld devices in data collection. Moreover, over 25% of African countries and more than half of the Pacific Islands countries have collaborated in most technologies involved in the various faces of the census including alternative data collection methodologies and data dissemination and analysis. The US Census Bureau survey anticipated increased use of new technologies and emerging trends in census taking27. b) To what extent have the MAPS partnership programs/DGF funded activities contributed to an improvement in the availability of key statistical data? 25 US Census Bureau, “Mid-Decade Assessment of the United Nations 2010 World Population and Housing Census Program”, 2013. 26 Optical mark recognition (OMR) is the process of capturing human-marked data from document forms such as surveys and tests using optical character recognition (OCR) software. Intelligent character recognition (ICR) is a more specific OCR that helps to recognize handwriting. 27 Among the key recommendations by the US Census Bureau survey figure the revision 3 of the UN Principles and Recommendations for Population and Housing Census. DGF funding has contributed to the finalization of the revision 2 of this document. In addition, the survey recommended that UNSD plays a significant role in planning for the 2020 round by providing opportunities for countries to exchange information on their 2010 experiences and promote the full range of possible census methods, which indirectly contribute to statistical capacity building for all countries. - 40 - 64. Compared to the 2000 Population Census Round, participation rates have increased with the 2010 Round. The number of countries having missed the Round declined from 26 to 21 countries. For the two sub-regions where the availability of statistical data remains the weakest, this has meant the improvement in data availability. For Africa, while the number of countries which have carried a population census in the 2010 Round only rose marginally – from 39 countries in the 2000 Round to 40 countries in this round --, 4 countries which have missed both the 1990 and 2000 Census Rounds have made up their lack of demographic data by having participated in the 2010 Round. As a result, Angola now has a population census after 44 years; Togo after 29 years, Djibouti after 26 years and Liberia after 24 years. For the Pacific Islands, all countries did participate in the 2010 Round. Finally for Europe, 50 of the 52 countries participated in this Round, up from 45 countries in the 2000 Round. c) Are these outcomes and impacts likely to be sustainable? 65. While issues of rising costs and other considerations will continue to mount challenges to the implementation of future census rounds, the review of reports and presentations by the various regions showed rising exposure and learning from advanced census methodologies and technologies that helped to reduce the cost of population censuses. Cross learning within regions and across regions has helped to quicken the pace of adoption of technological advances thus playing a role in sustaining some of the outcomes and impacts described above. As the new technologies will allow for more regular updates of key demographic data, the training component developed for population censuses and its impact will likely become more permanent instead of dissipating once the census was completed as was generally the case in past population censuses. d) Did they trigger more local awareness and coordinated efforts? 66. The experience in terms of impact of the population censuses on local awareness varies greatly across countries and depends to a significant extent on the country’s level of institutional capacity. Presentations made at the November 2013 United Nations Expert Group Meeting on Revising the Principles and Recommendations for Population and Housing Censuses show a wide range of experiences in terms of raising local awareness. In the case of South Africa, the government saw the census not only in terms of helping to generate demographic and socio-economic information but also a means to provide insight into the performance of government policies and programs and to promote economic literacy through increased statistical literacy. A well implemented marketing strategy was developed and all derived products in printed and electronic formats are available on their dedicated website (www.statssa.gov.za). But for the majority of African countries where in-house capacity on technical aspects continues to be a major challenge, dissemination of census results remained restrained with limited analysis of data and no or few thematic reports. However, local awareness can take various forms. The government of Samoa has, for instance, initiated discussions with parliamentarians and the representatives of key sectors on the results of the census. - 41 - 6.5 Efficiency of the program 67. As was the case for the gender statistics partnership program, the lack of information precludes a more detailed analysis of the cost efficiency of the program. The consultations for the evaluation again confirmed that UNSD has consistently delivered its activities and outputs within the allotted budgets under the various yearly arrangements of the DGF. 6.6 Main findings 68. The partnership program aimed at helping developing countries to participate in the 2010 census round has proven to be critical for the success of the round particularly in the preparation and dissemination of technical guidelines and assistance provided to countries in various phases of the census. The changes in census technologies which are becoming more widespread in the 2010 round have added a new dimension to statistical capacity building which make population censuses a very useful tool for the building of countries’ capacity. Also , new methodologies (administrative registers among others) which allowed for more regular updates on demographic information while helping to reduce the cost of population censuses have also helped in sustaining and maintaining the capacity of the statistical apparatus developed to undertake population censuses. 69. Contributions by the World Bank to this partnership program should be continued and preferably using a medium term (2-3 year basis) to allow for a more comprehensive approach to helping countries in the 2020 Census round. Preparatory works on the 2020 Round have already begun with the revisions of the guidelines and are expected to accelerate in 2015 which is the first year of the 2015-24 decennial round. VII. Lessons and Recommendations 70. The five partnership programs covered by this evaluation have in many different ways contributed greatly to the production and availability of key statistics. The main findings were presented under each of the five partnership programs separately and need not be repeated here. The review of partnership programs’ effectiveness highlights four lessons three of which are related to the design and implementation of the programs and indirectly to overall statistical capacity support. The fourth lesson concerns the operationalization of the DGF arrangements.  Effectiveness requires clear objectives in terms of outputs and outcomes. The review of the five partnership programs shows that clearly defined objectives in terms of outputs and outcomes have helped to enhance the chance for effectiveness of the programs. This is the case for the two partnership programs with UNSD on improving gender statistics and promoting the participation of countries in the 2010 population census round. The absence of specific objectives in the case of NSDSs and for the management of microdata surveys has, at least in part contributed to the lack of clear results in terms of implementation for NSDSs. Similarly, according to the external evaluation on the IHSN and ADP programmes, the lack of clear objectives may have led to the inconclusive result in terms of the use of - 42 - microdata under the ADP program. Tracking the use of microdata remains a problem for the ADP making it difficult to attribute the programme’s effectiveness in responding to the demand by data users. A monitoring & evaluation system could further add to achieving the specified objectives.  Setting specific timelines could help to generate the expected outcomes. Although they may be perceived as constraining, timelines combined with clear objectives and benchmarks could enhance the chance of success. The achievements under the IHSN, Gender and Population Census Round may partly be attributed to the strict timelines for delivery.  Some selectivity may be needed to achieve progress in statistical capacity building. The IHSN/ADP external evaluation made the observation that the rush to respond to a rapidly rising demand from countries has meant that interventions at country level were poorly coordinated. An alternative option would be to work with a smaller number of countries but in greater depth and in full coordination with the NSDSs which could have provided more substantive experiences to share across countries.  For capacity building activities, multiannual funding may be more desirable. The DGF annual grant agreements have imposed various costs on the partnership programs such as the repeated need to request extensions for disbursement as well as challenges in terms of medium term planning particularly for staffing purposes. - 43 - References General PARIS21 Steering Committee Meeting, Record of the Proceedings, various years. PARIS21 (2013): Report from the Task Team on Defining Implementation Arrangements for the Busan Action Plan for Statistics. World Bank Independent Evaluation Group (2011): Global Program Review (Marrakech Action Plan for Statistics, Partnership in Statistics for Development in the 21st Century, and Trust Fund for Statistical Capacity Building). The Marrakech Action Plan for Statistics: Better Data for Better Results, an Action Plan for Improving Development Statistics. Presented to the Second International Roundtable on Managing for Development Results, Marrakech, Morocco, February 4-5, 2004. Statistics for Transparency, Accountability, and Results: A Busan Action Plan for Statistics. Presented at the Fourth High-Level Forum on Aid Effectiveness (HLF-4), Busan, Korea, November 29-December 1st, 2011. ADP PARIS21 Secretariat, Transition 2016: Post ADP Data Group at PARIS21 (draft). __________, ADP assists countries with NADA upgrades (November - December 2013). __________, Accelerated Data Program (ADP), DGF 401006-04, Final Report, April 2006 - June 2008. __________, Accelerated Data Program (ADP), DGF 401007-04, Final Report, January 2007 - June 2008. __________, Accelerated Data Program (ADP), DGF 401008-04, Final Report, January 2008 - June 2009. __________, Accelerated Data Program (ADP), DGF 401009-04, Final Report. __________, Accelerated Data Program (ADP), DGF 401010-04, Final Report. __________, Accelerated Data Program (ADP), DGF 401011-04, Final Report. __________, Accelerated Data Program (ADP), DGF 401012-04, Final Report. Nigeria Education Data Outreach and Harmonization Workshop, May 26-28 2014, Hawthorn Suites, Abuja, Nigeria. Organised by Accelerated Data Program (ADP), National Bureau of Statistics (NBS), and EDOREN. Uganda Microdata Outreach Workshop, 19-20 April, 2014, Uganda Bureau of Statistics. With support of the Accelerated Data Program (ADP), PARIS21. Gender UN Statistical Commission. Gender Statistics; Report of the Secretary-General, various issues. - 44 - UN Statistics Division (UNSD), Improving Gender Statistics on Employment Consolidated Report for 2009-2010 and 2011. ___________, Workshop on Time Use Statistics in Arabic Speaking Countries - Conclusions And Recommendations. Amman, Jordan, 25-28 April 2011. ___________, World Programme on Gender Statistics: Progress Report from January 2012 to December 2012. ___________, World Programme on Gender Statistics: Progress Report from January 2013 to December 2013. UN Departments of Economic and Social Affairs, Guidelines for Producing Statistics on Violence Against Women. 2013. __________, The World’s Women 2010 - Trends and Statistics, 2010. __________, Report of the Interregional Workshop on the Production of Gender Statistics. New Delhi, India, 6-10 August 2007. UNSD, Handbook for producing National Statistical Reports on Women and Men. IHSN IHSN Management Group (2013). Response to the Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP). October 2013. PARIS21 Secretariat, International Household Survey Network (IHSN), DGF 401006-06, Final Progress Report, April 2006 - June 2007. __________, International Household Survey Network (IHSN), DGF 401007-06, Final Report. __________, International Household Survey Network (IHSN), DGF 401008-06, Final Report. __________, International Household Survey Network (IHSN), DGF 401009-06, Final Report. __________, International Household Survey Network (IHSN), DGF 401010-06, Final Report. __________, International Household Survey Network (IHSN), DGF 401011-06, Final Report. __________, International Household Survey Network (IHSN), DGF 401012-06, Final Report. __________, Progress Reports, various years. Thomson Anne, G. Eele and F. Schmieding (2013), Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP), Final Report . NSDS African Development Bank, Ecole Nationale Supérieure de Statistique et d’Economie Appliquée (ENSEA), PARIS21 and UN Economic Commission for Africa, Assessment of the NSDS 2008-2012 in Sierra Leone. Report of the Evaluation Team (January 2013). - 45 - African Development Bank, PARIS21 and UN Economic Commission for Africa, Nigeria National Strategy for the Development of Statistics (2010-2014) - a Review Report by a Multi- Agency Mission (20-31 January 2014). AFRISTAT-PARIS21, Rapport de la mission conjointe d’évaluation à mi-parcours de la SNDS 2010-2014 de l’Union des Comores, Moroni du 1er au 5 juillet 2013. AFRISTAT and PARIS21, Workshop for Trainers in Statistics and Public Relations, Bamako, Mali, 3-7 November 2014. Institut National de la Statistique et de l’Analyse Economique (République du Benin) and Consortium PARIS21, Atelier de haut niveau sur le financement de la statistique publique au Bénin, Compte-rendu des travaux, Cotonou, 22 juillet 2014. Instituto Nacional de Estatística (Mozambique), PARIS21 and Southern African Development Community (SADC), “Recommendations from the PARIS21/SADC Regional Workshop on “Better Dialogue in Statistics for Better Development Results”, n.d. FAO, the World Bank and the UN Statistical Commission (2012), Action Plan of the Global Strategy to Improve Agricultural and Rural Statistics for Food Security, Sustainable Agriculture and Rural Development . Rome. Madaya, Norah, Evaluation of the Implementation of the Liberia National Strategy for the Development of Statistics (2008-2012), April 2013. PARIS21 Bureau, Special Committee on Logical Framework, Governance and Funding Issues, PARIS21’s strategic planning for 2010-2014: logical framework, governance and funding issues, June 2010. PARIS21 Consortium, PARIS 21 at Ten - Improvements in Statistical Capacity since 1999. __________, Dakar Declaration on the Development of Statistics, November 2009. __________, The Partnership in Statistics for Development in the 21st Century, October 2009. PARIS21 Document Series, National Strategies for the Development of Statistics (NSDS): Some Issues in Design and Implementation Planning, March 2007. PARIS21 Secretariat, Review of Key PARIS21 Documents 1999-2014 - Summary and Observations, draft. __________, Forum on Advancing Statistical Development in SIDS (Small Island Development States) in the Post-2015 Era, 7 August 2014. __________, Proceedings, Main Discussion Points and Decisions. High-Level Seminar and Workshop on ASEAN Community Statistical System (ACSS) Strategic Plan 2016-2020, Bogor, West Java, Indonesia 17-16 June 2014. __________, PARIS21 Recommendations on Updating Cambodia’s Statistical Law and its Secondary Statistical Legislation, n.d. __________, Advocacy at a Glance, n.d. - 46 - __________, PARIS21 Final Expenditure and Narrative Report for Grant Number DI130062 (DGF file 401013-05), 14 February 2014. __________, PARIS21 Final Expenditure and Narrative Report for Grant Number DI120030 (DGF file 401012-05), 13 February 2013. __________, PARIS21 Final Expenditure and Narrative Report for Grant Number DI110041 (DGF file 401011-05), 19 March 2012. __________, PARIS21 Final Expenditure and Narrative Report for Grant Number DI290305 (DGF file 401010-05), 05 May 2011. __________, PARIS21 Final Report for Grant Numbers DI270715 and DI280755 (DGF files 401008-0-NSDS and 01009-05), 28 June 2010. __________, PARIS21 Final Report for Grant Number DI270715 (DGF file 401008-05-NSDS), 4 March 2009. __________, PARIS21 Final Narrative Report for Grant Number DI270299 (DGF file 401007-05), 9 December 2008. __________, PARIS21 Progress Report for Grant Number DI26215 (DGF file 401006-05), 23 February 2007. __________, Partner Report on Support to Statistics – PRESS, 2013 Round, Final Report, December 2013. Previous editions are available yearly for 2008-12. __________, National Strategies for the Development of Statistics - Progress Report: NSDS Summary Table for IDA and Lower Middle Income Countries, various editions (March 2014 is the latest). __________, Advocating for the Development of National Strategies for the Development of Statistics, Country Level Tool Kit, May 2010. __________, New NSDS Guidelines, (http://nsdsguidelines.paris21.org/node/305), 2014. __________, NSDS2 in a Nutshell, 2014. __________, A Guide to Use a System Wide Approach to Implement National Strategies for the Development of Statistics, October 2007. __________, PARIS21 at Ten: Improvement in Statistical Capacity since 1999. __________, The NSDS Approach in a Nutshell - National Strategy for the Development Of Statistics, n.d. __________, Progress Report for the Partnership in Statistics for Development in the 21st Century (PARIS21), Reporting Period: various years. Philippines Statistical Authority and PARIS21, Regional User-Producer Dialogue on Inclusive Growth and Disaster Risk Management: Role of Statistics, Legazpi City, Albay, 13 February 2014. - 47 - Rajan, Silwal, Millicent Gay Tejada, and Romulo Virola, Roadmap for the National Strategy for the Development of Statistics of Nepal (NSDS, Nepal), November 2013. Roberts, Richard and Claudine Voyadzis. Mark Baldock, Evaluation of PARIS21 (Final Report), __________, 2010 World Programme on Population and Housing Censuses - Progress Report, Spitz, Pierre, PARIS21 Light Evaluation, Final Draft Report, June 2006. Statistical Service Ghana and PARIS21, Country Report on Support to Statistics (CRESS), Ghana 2010-2013, August 2013. UN Statistical Commission, Statistical Capacity Building: Report of the Secretary-General (various issues). _________, Indicators for Monitoring the Millennium Development Goals: Report of the Secretary-General, 9 December 2013 (Report E/CN.3/2014/29). _________, Report of the World Bank and The Partnership in Statistics for Development In the 21st-Century on Statistical Development: Note by the Secretary-General, 24 December 2013. Willoughby, Chris, Anne Thomson, Xavier Charoy and Juan Munoz, An Evaluation of PARIS21, An Initiative to Improve the Contribution of Statistical Services to Development, Oxford Policy Management, August 2003 2010 Population Census Round UN Departments of Economic and Social Affairs, Principles and Recommendations for Population and Housing Censuses, Revision 2, 2008. __________, World Population Policies 2013. UN Statistics Division, 2010 World Programme on Population and Housing Censuses - Progress Report, January to December 2006 . __________, 2010 World Programme on Population and Housing Censuses - Progress Report, March 2007 to February 2008. __________, 2010 World Programme on Population and Housing Censuses - Progress Report, March 2008 to February 2009. __________, 2010 World Programme on Population and Housing Censuses - Progress Report, March 2009 to February 2010. __________, 2010 World Programme on Population and Housing Censuses - Progress Report, July 2010 to May 2011. __________, 2010 World Programme on Population and Housing Censuses - Progress Report, 1 June 2011 to 31st May 2012. __________, Progression of the 2010 Census Round. - 48 - UN Statistical Commission, Population and Housing Censuses: Report of the Secretary- General, various issues. United Nations, Demographic Yearbook 2012 . US Census Bureau, Mid-Decade Assessment of the United Nations 2010 World Population and Housing Census Program, 2013. - 49 - Annex A: DGF/MAPS funding 2006 - 2014 (US$ thousand) MAPS P21 P21 P21 UNESCO UN UNSD UNSD ICP - UN- Secretariat NSDS ADP IHSN UIS Habitat Census UNECE Gender CIS ECLAC Total 2006 120 1,500 2,000 900 180 200 1,000 5,900 2007 100 1,300 2,000 750 200 1,000 350 5,700 2008 150 1,300 2,000 750 200 1,000 200 100 5,700 2009 150 1,000 2,200 1,000 1,000 150 200 5,700 2010 150 700 2,200 1,000 1,000 250 150 250 5,700 2011 50 500 2,000 1,000 1,000 150 4,700 2012 300 2,000 1,000 200 3,500 2013 300 1,100 700 400 2,500 2014 300 750 500 400 1,950 Total 720 7,200 16,250 7,600 180 600 6,000 950 1,450 250 150 41,350 - 50 - Annex B: PARIS21 Logical Framework (2010-14) Goals, Purpose, and Outputs Objectively Verifiable Indicators Narrative Summary Assumptions/ Risks Indicators Sources Partnership Super-Goal: Achieve national and international MDG indicators UNSD Development Goals, especially the Millennium Development Goals (MDGs). Partnership Goal: Develop a culture of Management for OECD Progress Report on the Better policies and budgeting processes based and Indicator G1: % of countries that have Development Results (MfDR). Target Implementation of the Paris managed through better use of better statistics developed sound frameworks for Group: Countries eligible for IDA Declaration on Aid Effectiveness contribute to achieving development objectives monitoring development results resources plus all LMICs and all and the Accra Agenda for Action including the MDGs. The involvement of policymakers African countries (118 countries as of Baseline 2008: 9% (AAA) and progress towards development objectives will Dec 2009) Milestone 2012: 24% sustain the MfDR capacities in the long-run. Target 2014: 40% Indicator G2: Average score on the use of PARIS21 Scoring System on Use of statistics in policy making process Statistics Baseline 2010: 52.9 Milestone 2012: 60 Target 2014: 70 Partnership Purpose: - 51 - Objectively Verifiable Indicators Narrative Summary Assumptions/ Risks Indicators Sources The availability of improved statistics tailored to the needs of Build and sustain the capacity to Indicator GP1: % of IDA World Bank policymaking generate the statistical data needed to borrowers (with a population and monitoring and the full implementation of the support development through an of 1 million or more) whose commitments countries of and donors in the Paris Declaration will lead to partners effective use of statistics to inform policy, World Bank Statistical Capacity Indicator the use of to manage for development results. Declaration monitor its implementation, promote scores have increased. accountability and measure development results. Main target: Baseline 2009: 62% Milestone 2012: 70% Target 2014: 80% Sub target 1: Percentage of IDA-eligible countries having conducted at least one population census in the current round Baseline 2009: 33% Milestone 2012: 86% Target 2014: 96% Sub target 2: Increase in the percentage of IDA eligible countries with a periodicity of 5 years or less for poverty related household surveys Baseline 2009: N/A Milestone 2012: 5 Target 2014: 10 - 52 - Objectively Verifiable Indicators Narrative Summary Assumptions/ Risks Indicators Sources Indicator GP1bis: % of 117 target UNSD countries having at least three non- modelled data points for at least 20 of 28 selected MDG Indicators Baseline 2010: 7.6% Milestone 2012: 12% Target 2014: 15% Partnership Outputs: 1. Partner countries design and Indicator GPO1: % of target countries that PARIS21 NSDS Progress (i) Well-designed NSDSs are integrated into national implement comprehensive and have adopted an NSDS that covers the report development policy processes meeting their needs, operational NSDSs that fulfil needs logframe reporting period funded through national budget and donor support, especially of Development policies and are implemented. (ii) High level policymakers are and Performance Budgeting. committed to the successful implementation of NSDSs. Baseline 2009: 52% (iii) Donors are involved in the design of NSDSs and Milestone 2012: 80% fully align their assistance frameworks on the NSDSs. Target 2014: 93% (iv) Systemic approach will lead to more efficient NSSs. Indicator GPO1bis: % of IDA- eligible (i) NSDS Quality Assessment countries with an NSDS covering the Framework and possibly (ii) Peer logframe period that scores more than 75% reviews of the total score defined by the NSDS QAF Baseline 2010: Milestone 2012: Target 2014: - 53 - Objectively Verifiable Indicators Narrative Summary Assumptions/ Risks Indicators Sources 2. Support to Statistics is adequate, Indicator GPO2: Share of aid to statistics Partner Report on the Support to (i) ODA to statistics is increased by donors consistently well aligned with NSDSs and well (as reported in PRESS) in the total ODA Statistics (PRESS) & DAC ODA with the MfDR commitments. (ii) Partners and Donors coordinated. over a 3-year period starting from 2007- figures fully implement the Paris Declaration and AAA to 09 enhance the effectiveness of assistance to the development of statistics. (iii) Better dialogue between Partners and Donors will lead to more and better Baseline 2007-2009: 0.2% Milestone 2010- investment in the statistics. 2012: 0.3% Target 2012-2014: 0.5% Indicator GPO2bis: Share of PRESS aid to statistics (as reported in PRESS) aligned with NSDS Baseline 2010: 31% Milestone 2012: 45% Target 2014: 55% Indicator GPO2ter : % of countries who PRESS have a Country-Donor coordination mechanism in the field of statistical development and where this mechanism is perceived as satisfactory or better Baseline 2010: 27% Milestone 2012: 53% Target 2014: 70% - 54 - Objectively Verifiable Indicators Narrative Summary Assumptions/ Risks Indicators Sources 3. International and Regional Indicator GPO3: a perception indicator PARIS21-administered Implementation agencies are committed to ensure Statistical Programs are on a measuring developing country satisfaction Perception Survey (S1 and S2) their alignment with the NSDSs and build statistical sustainable financial and technical with 9 international programmes. (Scale capacity in partner countries while reinforcing the footing. 1-5. 1=ineffective; 5=excellent) regional and global data comparability. Baseline 2010: 3.7 Milestone 2012: 4.0 Target 2014: 4.2 4. Countries increase PARIS21-administered Perception (i) A strategic approach is adopted for these programs Indicator GPO4: % of countries with a capacities to document, archive, Survey (S1) at the global and regional levels to ensure their national data archiving system disseminate and harmonise survey effectiveness and their sustainability. (ii) Dissemination micro-data and metadata in line Baseline 2010: 25% of innovation products through the use of ICTs is with IHSN Principles. Milestone 2012: 40% systematic. (iii) Donors are committed to a stable Target 2014: 60% funding of IHSN/ADP and similar programs. Indicator GPO4bis: % of countries giving PARIS21-administered Perception access to micro-data or metadata through Survey (S1) website Baseline 2010: 15% Milestone 2012: 25% Target 2014: 50% 5. Governance of the global Indicator GPO5: Perception indicator on PARIS21-administered (i) Governments, international organisations and statistical system is improved the quality of governance of global Perception Survey (S1 and S2) development agencies work closely and effectively to statistical system. (Scale 1-5. 1=ineffective; enhance the global statistical system and the availability 5=excellent) of improved data to monitor international commitments. (ii) The status of statistics in the major international initiatives is fostered. - 55 - Objectively Verifiable Indicators Narrative Summary Assumptions/ Risks Indicators Sources Baseline 2010: 3.3 Milestone 2012: 3.4 Target 2014: 4.0 Indicator GPO5bis: % of the 118 logframe focus UNSD countries attending annual UNSC meetings Baseline 2010: 59% Milestone 2012: 70% Target 2014: 75% Indicator GPO5ter: % of the 118 logframe focus UNECA, UNECLAC, countries attending statistical committees of UN UNESCWA, UNESCAP, regional commissions UNECE Baseline 2010: 73.7% Milestone 2012: 75% Target 2014: 80% Secretariat Outputs: 1. Facilitate better design and successful Indicator SO1: Perception indicator on the quality of PARIS21-administeredPerceptionSurvey (i) PARIS21 Secretariat has access to Partner countries’ implementation of NSDSs in developing PARIS21 Secretariat advisory services. (Scale 1-5. (S1andS2) policymakers that are committed to the successful countries 1=ineffective; 5=excellent) implementation of NSDSs. (ii) The Secretariat receives adequate funding. Baseline 2010: 3.5 Milestone 2012: 4.0 Target 2014: 4.2 - 56 - Objectively Verifiable Indicators Narrative Summary Assumptions/ Risks Indicators Sources 2. Promote an increase of PARIS21-administered (i) Partner countries’ policymakers and Donors’ Indicator SO2: Perception indicator on resources devoted to the Perception Survey (S1 and S2) representatives are committed to enhance aid stakeholders satisfaction with the Development of Statistics, in efficiency. (ii) All donors respond to the PRESS advocacy work. (Scale 1-5. 1=ineffective; conformity with the Paris Declaration Survey. 5=excellent.) and the AAA Baseline 2010: 3.6 Milestone 2012: 4.0 Target 2014: 4.2 3. Provide support to Annual PARIS21 Progress Report Programs Implementation Agencies cooperate with Indicator SO3: % of countries that have a Developing Countries to the Partner countries and the Secretariat to ensure national data archiving system with ADP document, archive, disseminate their mainstreaming in NSDS processes. support and harmonise survey micro-data and metadata in line with IHSN Baseline 2009: 15% Principles. Milestone 2012: 40% Target 2014: 70% Indicator SO3bis: % of countries giving Annual PARIS21 Progress Report access to micro-data or metadata through website with ADP support Baseline 2009: 8% Milestone 2012: 30% Target 2014: 70% 4. Contribute to a better Indicator SO4: % of Board members who Annual Survey of PARIS21 Board All the Partnership members are determined to coordination between members find Board meetings "good" or "very members enhance the coordination and the PARIS21 of the Partnership good" across 9 questions Baseline 2010: Governance. 80% Milestone 2012: 85% Target 2014: 90% - 57 - Annex C: External Evaluation of the IHSN/ADP Programs: Major Findings, Recommendations (Executive Summary) Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP) Final report Anne Thomson, Graham Eele, Felix Schmieding 31 August 2013 - 58 - Executive Summary Introduction This report presents the results of an independent evaluation of the International Household Survey Network (IHSN) and the associated Accelerated Data Programme (ADP). Oxford Policy Management, a UK based consulting company has been contracted by OECD to carry out the evaluation. The objectives of the evaluation are: (i) to assess the impact, effectiveness, efficiency and sustainability of both the IHSN and ADP and of their institutional arrangements; (ii) to assess the relevance of both programmes in the context of the evolving statistical capacity development agenda; and (iii) to provide guidance on how the programmes could be strengthened and improved in the future. The evaluation was carried out over a six month period and the main evaluation instruments were: a review of all relevant documents, stakeholder interviews, five country case studies, and two eSurveys covering data producers and users. The report falls into two parts: the evaluation of IHSN/ADP activities over the period since inception, and an assessment of the options open to IHSN and ADP. The International Household Survey Network (IHSN) was established in 2004 following the adoption of the Marrakech Action Plan for Statistics at the Second Roundtable on Managing for Development Results in February 2004. The Accelerated Data Programme (ADP) was launched in 2006, to assist developing countries to implement improved practices in survey design and implementation and in microdata management. The IHSN/ADP has been implemented jointly by PARIS21 and the World Bank. The objectives of the Network are: (i) to improve coordination of internationally sponsored survey programmes; (ii) to increase the availability of coordinated and practical technical and methodological guidelines for all stages of the survey life cycle; (iii) to establish and promote access to a central survey data catalogue; (iv) to develop and promote the adoption of standards, tools, and guidelines that allow data producers to document, disseminate, and preserve microdata; and (v) to improve collaboration between data producers and users. The ADP is a technical assistance and training programme to help developing countries adopt and use the standards, tools and guidelines generated by IHSN. Its objective is to increase the use and value of survey data, in particular to help data producers and users in developing countries make better use of existing data and to align survey programmes and statistical outputs to priority data needs. When the IHSN and the ADP were established no log-frame or intervention logic was defined and there has been no formal monitoring and evaluation system as such. A logical framework for IHSN and ADP was developed in 2011, but this has only been in place for one year. In order to assess the OECD DAC criteria of relevance, efficiency, effectiveness, impact and sustainability, the evaluation team developed two theories of change to determine whether the activities and outputs of the IHSN and the ADP have been appropriate and have led to the desired outcomes and impacts. - 59 - Findings The evaluation concluded that the establishment of IHSN and ADP was a relevant response to the recommendations of MAPS and its mission has continued to remain relevant. The two programmes have fulfilled a good part of their remit, and have filled a clear niche. No other statistical programme has focused on archiving surveys and increasing access to microdata. However there are parts of the original mission that have not been addressed, especially improved coordination of internationally sponsored survey programmes and improved collaboration between data producers and data users. In terms of efficiency, for IHSN most outputs have been achieved. For ADP, Task 1 outputs have been achieved in a significant number of countries. Task 2 activities have only started at a fairly late stage and have yet to result in significant outputs. The evaluation finds that the use of financial and human resources by both IHSN and ADP has been quite economical. There has been good use of South consultants and efforts to promote South- South sharing of expertise and experience. The number of full-time equivalent staff in both programmes has been kept remarkably small for the results achieved; by situating ADP at OECD the management fees for the Secretariat have been kept at a lower level than other comparable programmes. However, more effort could have been made in recent years to shift some of the costs to recipient organisations. The picture is less clear when we come to evaluate effectiveness. Although many countries and some international organisations have set up their own survey catalogues based on the IHSN National Data Archive (NADA) application, this has not, as far as the evaluation can tell, led to an increased use of existing survey data. Access to microdata appears to have increased in a number of countries. Figures collected in 2013 indicate that in many countries significant numbers of users are registering with NADAs, but there is no information as to what microdata sets they are downloading and how they are using them. In the countries with the highest number of registered users, it was previously possible to access microdata but the introduction of a NADA has made access easier. Many countries in Africa are not using those elements of the software which permit almost automatic download of microdata and also register usage. Undoubtedly, IHSN has created greater awareness of what microdata are available. The IHSN catalogue has been a very useful reference tool, but has had inadequate promotion. At the country level, National Statistical Organisations (NSOs) are not doing as much as they could to improve access, and in some cases, see microdata management tools as more important, in that they have helped them document and organise their own data sets better. At present, IHSN/ADP has not been effective in improving the coordination of surveys. Better documentation may have led to more emphasis on quality control of surveys, but overall, there has been little documented effect on survey quality. IHSN has not operated as a network in the usual sense of the term, with the secretariat tending to engage with its partners on a one-to-one basis. In order to make progress towards achieving greater coordination of surveys, there will have to be more effort towards working collaboratively with partners. In terms of impact, IHSN/ADP has improved data availability, both through the IHSN catalogue and through assisting countries to establish their own NADAs. It has had limited impact on the collaboration between data producers and data users and has not had any impact that the - 60 - evaluation could find on the coordination of internationally sponsored survey programmes. It has contributed to an environment of more open data, but it remains to be seen what impact it may have on the use of household survey data beyond internationally renowned academic institutions and the research departments of international organisations. IHSN could do more in terms of advocacy and promotion, but could also strengthen collaboration with partner organisations whose mandate focuses more on data usage. The sustainability of the outputs achieved by IHSN/ADP has quite limited financial implications. There are three established outputs; the maintenance of the software (including the NADA and the microdata toolkit), the use of the microdata toolkit to archive data at national and organisational level, and the maintenance of country NADAs. The first of these is a matter of programmer time and institutional commitment. It is important as there has been an implicit, and at times explicit, commitment by the IHSN to the countries that have adopted the toolkit and established NADAs, but the financial implications of this are not high. For the other two outputs, the sustainability is very much dependent on the capacity and commitment of NSOs. In the countries where the NSO has reasonable staffing levels, a good IT capacity and a degree of financial autonomy, the NADA has been maintained and surveys have been documented and added to the archive as they have been carried out. Sustainability does seem a major issue, however, in countries where the NSO is more dependent on external funding and where there is relatively high staff turn-over. It is important that data archiving and the maintenance of a national data archive become institutionalised within national procedures if the achievements of the IHSN and the ADP are to be sustained. The governance of IHSN has been quite light touch since its inception. The Management Group has met annually at most, with gaps in some years. It discusses a proposed work plan and provides guidance on IHSN priorities. IHSN and ADP develop annual work plans and provide six-monthly financial and progress reports, which are submitted to the World Bank. However the level of detail required is not high. There are quite limited opportunities for partner organisations to participate in management decisions, and no formal role for other stakeholders, such as heads of NSOs. There is no formal M&E system in place. It is not clear that anyone has the obligation to match work plans against progress reports, and there are no measurable performance indicators as far as the evaluation is aware. Reporting against the funding recently provided by DFID has required IHSN to report against targets and milestones, but this is new and it is not clear as yet what, if any, effect this will have on programme management. The evaluation feels that the identification of and reporting against some higher level indicators would bring about a greater strategic focus for the programme, which would be appropriate as it reaches maturity and can no longer be regarded as a pilot programme. Overall, the evaluation team concludes that both IHSN and ADP have made considerable achievements since their inception in 2004 and 2006. The initial mandate was very ambitious and has been narrowed down to a more practical set of objectives. Software tools have been developed for data archiving and for setting up NADAs and ADP has helped train NSO staff to use these, and to archive surveys more effectively. Some progress has been made on improving access to microdata, though there are still legal, political and, in some cases, technical issues to address at national level. A start has been made to improving data and survey quality but progress has been slow. There are still challenges of cooperation, both at national and at global levels. - 61 - The future programme Looking forward, the report makes recommendations for the future development and IHSN and ADP. It also recognises that this development will need to take into account the evolving international statistical architecture, the implementation of the Busan Action Plan for Statistics (BAPS) and, especially the evolving development agenda for the period from 2015 onwards, when the current Millennium Development Goals will come to an end. While IHSN and ADP have interacted with many of the main actors in the international statistical system, so far there has been no direct involvement with the United Nations Statistical Commission (UNSC). The UNSC brings together heads of national statistical agencies from UN member states from all regions of the world and it is the decision making body for the setting of statistical standards, the development of concepts and methods and their implementation at the national and international levels. If the standards, guidelines and recommendations that IHSN develops are to continue to be widely used then it will be important for UNSC to be involved. The international statistical architecture has evolved over time, typically in response to a specific need or concern and as a result is complicated and extensive. It is continuing to change, although not always in a planned or consistent way. One, not always intended, outcome of this complexity is the actual and potential duplication of both activities and statistical products, especially software tools. Many developing countries have expressed their frustration with this situation and it is an area where the IHSN or its successor could make an important contribution. There are a number of implications for IHSN and ADP that come directly from the implementation arrangements for BAPS. Most importantly, BAPS updates and strengthens the mandate for IHSN and ADP activities, requiring that work to develop and implement standards for data preservation, documentation and dissemination continues and is extended. In addition, the link with open data, providing for full public access to statistics, is emphasised. In the discussions around the post-2015 development agenda, there are also important implications for IHSN and ADP. In particular the emphasis on better data and the call for a data revolution present an important opportunity. It will be important to ensure that the development of IHSN is linked directly with the proposal for a Global Partnership on Development Data. The evaluation report proposes that the work of IHSN/ADP should be continued after 2015 at least until 2020. The proposals are divided into two distinct components: the short term from now until the end of 2015; and the period from 2015 onwards. Short-term recommendations Between now and the end of 2015, the assumption is that the work programme will proceed largely in line with existing priorities and objectives. The changes that are proposed are largely incremental and it is anticipated that they can be implemented within the current resource envelope. The main short-term recommendations are: (i) to reaffirm the involvement of partners in the Network and to strengthen the Management Group; (ii) to initiate the involvement of IHSN with the United Nations Statistical Commission; (iii) to promote raised awareness of the importance and availability of microdata through improved advocacy; (iv) to ensure that IHSN-type activities are included in national and regional strategies for the development of statistics; (v) to develop a strategy - 62 - for engagement and disengagement with ADP partner countries; (vi) to facilitate greater experience sharing between countries; and (vii) to develop a more coherent programme of publications. Longer-term proposal For the longer term, in response to the terms of reference, the evaluation team has developed a scenario for the development of the statistical environment post-2015. The team believes that the most likely scenario for post-2015 is one where the demand for data will continue to increase, national statistical systems will continue to be the main source of development data, but there will be an increasing focus on and interest in alternative data sources. The scenario suggests that increased demand for statistics will continue to be translated into demand for the products and services of national statistical systems, especially surveys and censuses. At the same time though, the limitations of these data sources will mean that other kinds of data will be more widely used. NSOs will need to improve the efficiency and effectiveness of their survey programmes and also find ways to make use of other kinds of data. In particular there is likely to be an increased interest in exploiting “big data” to provide a much more disaggregated analysis of the changes that are taking place, in as close to real time as possible. Based on the findings from the evaluation and the expected scenario for post-2015, the evaluation team recommends that the Network be extended to operate over the period from 2015 to 2020, and proposes some key adjustments for the way forward. The aim is to build on the achievements of the first eight years, but with a number of important changes to respond to the evolving environment for statistics. In particular, the network should be positioned as a component of the Global Partnership on Development Data. It is proposed that the network will support the maintenance and development of standards, tools and guidelines for the documentation and use of microdata and will promote their adoption and use by countries and international agencies. The proposed institutional structure of the proposed programme consists of a small secretariat, a small permanent technical team, and temporary technical teams established to undertake specific tasks as required. Members of the network would carry out much of the work programme based on in-kind contributions and some financial support from the network. Where required the inputs from network members will be supplemented by inputs from the secretariat and the permanent technical team. The outputs and activities to be carried out under the proposed programme may be considered under two main categories. First are those activities related to the assessment and analysis of existing data and the improvement of national data programmes, including, but not necessarily limited to, household surveys and censuses. Second, those technical activities such as the inventory, documentation and dissemination of microdata; these include the development of software tools and guidance within the framework of appropriate standards, including the DDI. The network should aim to remain engaged with the UN Statistical Commission. For some or all parts of the technical work programme this could be done by setting up an Inter-Sectoral Working Group, following the model used to prepare SNA2008. Another possibility would be to establish a “Friends of the Chair” (FOC) Group. The purpose of this group would be to review and reach agreement on the use of tools and standards. - 63 - The evaluation team recommends that the continuation of ADP as a separate technical assistance programme does not really make sense after 2015. It is recognised that there is likely to be a need to promote outreach and the widespread adoption of the tools and guidelines as well as advocacy generally. It is clear, though, that the existence of separate technical assistance funds targeted at a specific set of actions does not really fit in with the logic of BAPS or the approach of supporting countries to identify their own priorities through the NSDS process. In the longer term, technical assistance should be provided through general support to NSDS implementation rather than through separate funds under the control of different technical programmes. Where technical assistance needs to be provided by the network, it should not be provided by the secretariat directly. Rather we recommend that technical assistance is provided from consultants with the required expertise and experience or by members of the network. As far as possible, technical assistance should aim to have a multiplier effect, for example by the training of trainers. Direct support to countries should be minimised and should generally be provided through existing TA projects and programmes. The secretariat of the network should be established within an existing institution with a global mandate and a clear interest in microdata management. Options for the location of the network secretariat include: the Development Data Group in the World Bank; the PARIS21 Secretariat; or the UN Statistics Division. The permanent technical team of the network does not necessarily need to be in the same location as the secretariat and the decision as to where it should be located should be made independently. The possible options for the location of the permanent technical team include: the Development Data Group in the World Bank; the PARIS21 Secretariat; the UN Statistics Division; and an existing research centre or agency, outside the traditional international statistical architecture, with an international mandate and expertise in microdata management and use. Governance arrangements should not be very complicated or heavy and should build on existing processes. It is suggested that the work of the network and accountability to the funding agencies would be managed through a Network Management Group. This Group would approve work programmes and budgets, provide advice on priorities and strategies and decide on the allocation of resources. Membership of the network should be extended to include national statistical agencies as well as international agencies and other organisations with an interest in its work. The resource requirements for the proposed programme are of the order of just under $22 million in 2013 prices for a period of 5 years. A second option, based on a smaller budget, would eliminate the technical assistance budget, increase the commissioning fund and reduce the size of the secretariat. The estimated cost of the second option over five years is $14.5 million in 2013 prices. The evaluation team looked at several possible funding models and recommends that the most appropriate funding model would be to identify a group of donor agencies and establish a multi-donor trust fund. It will be important to ensure that management of the trust fund does not unduly constrain the work of the network. It would need to allow multi-year financial commitments to be made and be able to transfer funds on the basis of agreed work programmes approved by the Management Group. It is recommended that consideration be given to modifying the name of the network to indicate that household surveys are not the only focus of the work programme. One important factor in favour of - 64 - retaining the International Household Survey Network name is that it is well recognised and has a good reputation for the quality of its work and the tools. Against this is the need for the name to reflect the breadth and depth of the network’s activities. The report was presented and discussed at the IHSN Management Group meeting held on 18 September 2013. The formal response by the IHSN Secretariat provided in next attachment. These two reports are also available through the following links: http://www.ihsn.org/HOME/sites/default/files/news/IHSN ADP Evaluation Final report.pdf ihsn.org/HOME/sites/default/files/news/IHSN_Response_to_independent_evaluation_Final.p df - 65 - Annex D: IHSN Management Group - Response to the Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP) October 2013 This document summarizes the response of the IHSN Management Group to the Independent Evaluation of the International Household Survey Network (IHSN) and Accelerated Data Program (ADP). The evaluation was conducted by Oxford Policy Management Ltd from January to August 2013 (i) to assess the impact, effectiveness, efficiency and sustainability of the IHSN and ADP, (ii) to assess their relevance in the context of the evolving statistical capacity development agenda, and (iii) to provide guidance on how the programs could be improved in the future. The complete evaluation report (http://www.ihsn.org/home/node/627) was presented and discussed at the IHSN Management Group meeting held in Rome on 18 th September 2013. 1. General 1.1. We appreciate the usefulness of the Independent Evaluation of the IHSN and ADP. The evaluation was conducted on the basis of broad consultation and it provides a comprehensive review of the performance of IHSN and ADP since their inception. The report helps to identify the strengths and challenges of the programs. The findings and recommendations are relevant and useful for developing future strategies and work programs. 1.2. We recognize the need to think strategically about the future of the IHSN to foster its relevance and its impact in a changing data environment. 2. Response to the evaluation findings 2.1. In general, we concur with the overall findings on the relevance, efficiency, effectiveness, impact, sustainability, and governance of IHSN and ADP. However, we would like to clarify a few areas in the effectiveness and impact of the programs: a. Regarding the limited effectiveness and impact of IHSN on survey coordination, it should be noted that although better survey coordination is a widely acknowledged and much desired objective, how to achieve it at the international level remains the greatest challenge. We recognize the need for further information sharing between our Agencies and we are committed to be more ambitious on this. We also stress that the regional level institutions and the national statistical offices need to play greater - 66 - roles themselves. We should encourage all relevant partners to engage with the IHSN Secretariat to leverage its impact. b. We agree with the overall findings on the effectiveness of IHSN/ADP activities. We believe that the effectiveness of the IHSN activities lies in their integration in the IHSN member agencies’ and ADP client agencies’ core work programs. We also stress the importance for the IHSN/ADP Secretariat to work directly with agencies (notably national statistics offices - NSOs) at country level, in particular for the piloting of technical assistance programs. c. In its broad definition, the evaluation report has underestimated the effects of IHSN/ADP activities on improving data quality produced by National Statistical Offices and development agencies. Although IHSN/ADP are not directly involved in the implementation of surveys, their activities have indirectly contributed to improved relevance, usability, accuracy and timeliness of survey data. Improving metadata enables further work in data harmonization and reconciliation. The on- going work on Question Banks and assessments of survey designs also aims to improve the relevance and quality of data collection. The assistance provided to NSOs in the formulation of microdata dissemination policies further improves the timeliness of survey data. Importantly, promoting best practices in data documentation and dissemination has led to a cultural change in several national agencies that directs more attention to data quality. d. While the impact on data use is hard to measure, there is abundant evidence of an increasing demand for microdata, such as the number of registered users and downloads on National Data Archives (NADAs), World Bank Microdata Library, WHO and IPUMS microdata catalogues. While it is difficult to measure how much of the data use in developing countries has led to policy-making or to map policy decisions to a particular data source, microdata made available has been used in publications of national and international partners. Furthermore, the on-going Open Data movement is further broadening the user community and diversifying the range of analytical output. e. The report found limited impact on the collaboration between data producers and data users. We agree that more needs to be done to foster this collaboration. The Microdata Outreach Workshops that ADP has supported in Nigeria, Guinea and Rwanda and are currently supporting in a number of other countries are precisely aimed at promoting user-producer dialogues and strengthening the user-producer relationship at country level. - 67 - f. We would also like to call attention to the important impact of IHSN/ADP on forming an informal, previously non-existent network that goes beyond the IHSN member agencies; composed of heads and staff of NSOs and line ministries, regional statistical communities, user organizations, and other groups such as the DDI Alliance, ICPSR, NSD Norway, etc. At country level, the ADP activities also promoted the culture of different stakeholders working together. 3. Response to the recommendations Outputs and activities 3.1. Extend the scope of activities beyond household survey data. We will continue to focus on surveys and survey data (although not exclusively on household surveys) as they will remain a critical component of the upcoming data revolution and a critical source for monitoring the post-2015 agenda. IHSN/ADP will continue to play a key role in promoting microdata access and improving the quality of surveys. In the meantime, we will explore possibilities of extending the scope of activities to include other data sources and types. 3.2. Raise awareness of the importance and availability of microdata through improved advocacy. We agree with the need to invest more in improving outreach and advocacy. 3.3. Ensure IHSN activities are included in the national and regional strategies for development of statistics. We acknowledge the importance of strengthening links between IHSN activities with strategic approaches to capacity building at national and regional levels. We also stress the relevance of dedicated and easy to mobilize financial resources to spearhead innovation in specific statistical areas. 3.4. Develop a strategy for engagement and disengagement with ADP partner countries. A formal disengagement strategy is not a priority as the ADP is not involved in the long term in any country. Technical assistance will remain a critical component of IHSN and will evolve to include new areas such as anonymization and survey methodology harmonization. We agree that leveraging on the technical assistance programs of the IHSN Agencies should be systematic for mature activities, such as survey metadata management. Governance and institutional arrangement 3.5. Reaffirm the involvement of partners in the Network and strengthen the Management Group. We would like to open our membership to all types of interested - 68 - organizations, in order to engage a diverse range of stakeholders interested in the IHSN focus. 3.6. We agree with the need to strengthen the Network and the Management Group. We need to keep the network informal, non-legal, preserving its flexibility, while at the same time, strengthening the governance and advisory role of the Management Group. The Management Group should be composed of 10-12 agencies with a rotating chair to guide the IHSN work program, advise on budget and resource allocation and support fundraising if needed. We agree with the proposed composition of the Management Group. The Management Group will hold two meetings per year as recommended by the Evaluation: one open to all IHSN member agencies in the margins of the United Nations Statistics Commission (around February), and one meeting of the core Management Group around September. 3.7. Institutional arrangements. We believe that network, technical work and technical assistance should be integrated. We agree with the model of specific ad hoc task teams proposed, which are composed of IHSN agencies staff and supported by external experts and IHSN financial resources. The organization and financing must be flexible and responsive, leaving space for innovation and a “right to fail” in the “incubation of innovation” projects. 3.8. Initiate the involvement of IHSN with UNSC. We have already started engaging with UNSC and the broader development data community. It is important to establish the link to the Global Partnership on Development Data, and generate meaningful responses for the call for a data revolution. Reporting 3.9. Progress against the recommendations will be recorded in the periodic progress reports produced by the IHSN/ADP secretariat. - 69 - Annex E: Persons Met or Consulted PARIS21: Johannes Jutting Manager El-Iza Mohamedou Deputy Manager Shuang Chen Microdata outreach Geoffrey Greenwell IHSN/ADP Program Coordinator Shelton Kanyanda NSDS Regional Program Coordinator, English-speaking Africa Rudolphe Petras NSDS Regional Program Coordinator, French-speaking Africa Leslie Rae Administration Gay Tejada NSDS Regional Program Coordinator, Asia-Pacific UNSD: Francesca Grum Chief, Social and Housing Statistics Section. Srdjan Mrkic Chief, Demographic Statistics Section DECDG: Olivier Dupriez IHSN/ADP Masako Hiraga Gender Thomas Danielewitz NSDS Barbro E. Hexeberg Evaluation coordinator Annette Kinitz MAPS Unit FAO: François Fonteneau Formerly in charge of IHSN/ADP at PARIS21 AfDB: Samuel Blazyk Formerly in charge of NSDS (Africa) at PARIS21 External consultant: Jean Le Nay Consultant on NSDSs, PARIS21 - 70 - Annex F: Terms of Reference Independent Evaluation of the Marrakech Action Plan for Statistics Introduction The Marrakech Action Plan for Statistics (MAPS) was agreed at the Second International Roundtable for Managing for Development Results in February 2004, as a result of shortcomings in data quality and availability at global and national levels, for monitoring and measuring development outcomes identified at the first Roundtable in 2002. The plan focused on six actions, designed to be implemented through partnerships with countries and international agencies, to build on existing work programs, and to make short and medium-term improvements in data provision and statistical capacity. The actions and principles of MAPS were embraced by the international statistical community and many bilateral donors. Important partnerships were developed with UN Agencies and PARIS21, with financial support from the World Bank’s Development Grant Facility, and advisory oversight for MAPS from a high-level Advisory Board. Building on and updating the priorities of the MAPS, a new action plan for statistics - the Busan Action Plan for Statistics - was agreed at the Second International Roundtable on Managing for Development Results in November 2011. A first independent evaluation of the MAPS program to assess the partnerships performance and to obtain advice on how its effectiveness, efficiency, and relevance could be improved was undertaken in 2008. The Development Economics Data Group of the World Bank is now commissioning a second independent evaluation to assess MAPS performance over the past five years, and in particular to obtain insights on effectiveness, efficiency, and relevance of the funded programs. Background The second International Roundtable on Managing for Development Results in Marrakech, sponsored by the multilateral development banks in cooperation with the Development Assistance Committee of the OECD, placed high priority on supporting countries to improve their capacity to better manage for development results. Improving national statistical systems was recognized as one of the key areas for action: managing for results requires timely and reliable statistics at the country and global level. Serious problems beset measurement of many key indicators, including the MDG indicators; many countries need greater capacity to produce reliable statistics and make better use of them for effective decision-making. As part of the Marrakech Roundtable, representatives from the international statistical community took part in a series of meetings to identify the main priority areas that need to be addressed if the objectives of better managing for development results are to be - 71 - achieved. The resulting Marrakech Action Plan for Statistics (MAPS) was adopted, and sat out six key actions (Box 1). MAPS intended to present a comprehensive, coherent and costed plan of action focusing on short and medium-term recommendations, aiming to prepare the way for long-term sustainable improvements in national and international statistical capacity. The actions fell into two groups: those directed at improving national statistical systems and those directed at the international statistical systems – although interdependent. Improvements in national statistical systems lead to improved international statistics, while a more effective international system could better coordinate and mobilize resources for improving national statistics. Box 1. The Marrakech Action Plan for Statistics 1. Mainstream strategic planning of statistical systems and prepare national strategies for the development of statistics (NSDS) for all low-income countries by 2006 2. Begin preparations for the 2010 census round 3. Increase financing for statistical capacity building 4. Set up an International Household Survey Network 5. Undertake urgent improvements needed for monitoring the Millennium Development Goals 6. Increase accountability of the international statistical system To support the actions addressing national needs, existing instruments and approaches have been utilized. These include the Bank’s STATCAP financing vehicle (part of regular operational activities) and the Trust Fund for Statistical Capacity Building. Actions addressing international responsibilities have required a partnership effort and have been supported by the Bank’s Development Grant Facility as follows: i) International Household Survey Network (IHSN) – The overall aim of the IHSN was to improve the coordination of the activities of international organizations engaged in household surveys. The IHSN objectives were to improve the availability, accessibility and quality of survey data, to avoid duplication of data collection activities, to improve cost-effectiveness of surveys, and to reduce the burden of international survey programs on national statistical systems. Priority areas of work were (i) coordinating international survey programs, by fostering better timing, sequencing and frequency of internationally-sponsored surveys; (ii) fostering adoption of international standards and best practices by harmonizing data collection instruments; (iii) developing and maintaining a central survey data repository; and (iv) developing tools and guidelines for improving survey data documentation, dissemination and preservation. Grants have been made available to PARIS21, who have supported this activity as a “satellite” program. - 72 - ii) The (pilot) Accelerated Data Program (ADP) – The objective was to make rapid improvements in the quality and availability of key statistics in pilot developing countries, particularly those derived from household surveys. The ADP would provide policy makers and other stakeholders with better data and analysis for policy design, monitoring, and evaluation. Intermediate objectives of the Program were to (i) test the ability of national statistical systems to respond to emerging data needs, (ii) assist countries that do not have a coherent long-term survey program to improve the planning and financing of data collection activities, (iii) foster better inter-agency cooperation in country activities, under the umbrella of the IHSN, (iv) build national capacity in micro-data preservation, analysis, and dissemination, and (v) generate updated estimates of key indicators by collecting new data. In achieving its objectives, the Accelerated Data Program should make use of the tools developed through the IHSN (MAPS action 4). At country level, activities aimed to be consistent with the National Strategy for the Development of Statistics, where available. Grants have been made available to PARIS21, who have supported this activity as a “satellite” program. iii) Preparation of National Strategies for the Development of Statistics (NSDSs) - Statistical development strategies and master plans have proved to be a powerful tool for guiding the development of national statistical programs and increasing political and financial support for investments in statistics; thus, supporting statistical development through good NSDSs was seen as a main pillar of MAPS. Activities which need support included the compilation and dissemination of guidelines for producing an NSDS and direct assistance to developing countries to help them access trust funds and other instruments for providing financial and technical assistance for statistics. Grants were made available to PARIS21 to support these activities as part of their regular work program. iv) Urgent improvement in key statistics, including Gender Statistics: Women's empowerment and the promotion of gender equality are key to achieving sustainable development. Greater gender equality can enhance economic efficiency and improve other development outcomes by removing barriers that prevent women from having the same access as men to human resource endowments, rights, and economic opportunities. Grants were made available to UNSD to improve Gender Statistics. v) Help countries participate in the 2010 census round - Population censuses are essential tools for policy and planning purposes; data and indicators derived from the census are extensively used as inputs for result-based management and tracking of progress towards national goals (such as those set in PRSPs) and international goals such as MDGs. No other data source provides the level of detail available in the census on location, age and gender, and family size. Developing countries need support to participate in UN 2010 World Program on Population and Housing Censuses. The goals of the World Program were for all countries and areas to agree - 73 - on a set of accepted international principles and recommendations governing the conduct of censuses; to conduct a census during the period 2005 -2014; and to disseminate census results in a timely manner. Grants were made available to UN Statistics Division to support the World Program in developing countries. For more information see the website: http://go.worldbank.org/PRTR3BCNE0 Purpose and Scope of the Evaluation The purpose of the evaluation is to (1) Assess the performance (effectiveness, efficiency, and relevance) of the major MAPS Partnership programs28 in implementing actions of MAPS, and provide guidance on how to improve this performance where relevant; and (2) Assess the extent to which MAPS actions remains relevant to the statistical capacity improvement agenda, assess sustainability, and advise on/make recommendations as to what a successor program such as the Busan Action Plan for Statistics (BAPS) could do differently to be most effective29. The evaluation will involve a review of the relevant work programs and outcomes of the DGF funded programs that have received the most of the funds over the last five years (the IHSN, the ADP, PARIS21’s work on NSDSs, and UNSDs work to prepare for the 2010 Census round and improve gender statistics). Moreover, the evaluation should build on the results of the evaluation of the ADP/IHSN program completed in August 2013. The evaluation should also consult the IEG Global Program Review in 2011: (http://lnweb90.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/43EC8 D4E9B1D3A64852578E2001CED11/$file/STATS-GPR2011.pdf) The evaluator’s assessment shall address the questions noted below, but need not be limited to these. Where appropriate, illustrations of good (or bad) practice/outcomes could be provided either in the body of the evaluation report or in an annex. Efficacy: Outcomes, impacts and their sustainability Assess the performance of the MAPS partnership programs in achieving its desired results. 28 The MAPS Partnership is taken here to mean the recipients of grants provided by the Development Grant Facility 29 PARIS21’s work program was revised following the approval of your the BAPS and the PARIS21 Board’s decision to assign to PARIS21 Secretariat the responsibility of assuring the Secretariat of the BAPS, and to incorporate new work streams emerging since Busan. - 74 - Outcomes and Impacts: To what extent have the MAPS partnership programs/DGF funded activities supported actions at the national and the international level? To what extent have the MAPS partnership programs/DGF funded activities contributed to increased statistical capacity in countries? To what extent have the MAPS partnership programs/DGF funded activities contributed to an improvement in the availability of key statistical data? Are these outcomes and impacts likely to be sustainable? Did they trigger more local awareness and coordinated efforts? Efficiency or cost effectiveness of the programs Have the programs delivered as promised and cost more or less as planned? Methodology and outputs The evaluation will be conducted by one individual consultant. He/she should be an expert evaluator with some statistical background. The consultant will produce a short report describing the evaluation methodology. This will be circulated to the partners/DGF recipients for comments. The methodology should include but not be limited to: - A desk review of MAPS DGF-funded activities of the largest programs of the past 5 years which include OECD/PARIS21 (ADP, IHSN, NSDS), and UN Statistics Division (Census and Gender). This may include interviews of task managers via email, telephone, and video conference. - Review outcomes of the independent IHSN/ADP evaluation - Interviews with MAPS Unit staff at its Washington, D.C. office. - Quantitative methods where feasible. - Any additional sources of information or procedures to obtain views and feedback on the MAPS partners that the reviewer feels to be necessary in order to accomplish the tasks set forth in these Terms of Reference. The consultant will report the findings and recommendations to the MAPS Secretariat in a report not to exceed 15 pages, excluding appendixes. The evaluation consultant shall provide other outputs as noted in the time-plan below. The outcome of the evaluation will be shared with partners (format yet to be decided) in a forthcoming PARIS21 - 75 - meeting. Timing/Calendar Early September 2013 Review of TOR by DGF Secretariat End June 2014 Circulation of TOR to MAPS partners/DGF grant recipients noted above June 2014 Selection and contracting of consultant August 2014 Inception report that includes the detailed work plan, to be approved by the MAPS Unit in consultation with the Advisory Board September 2014 Presentation of draft report to MAPS Unit. This will also provide a chance to redirect the work of the consultant in case there has been any misunderstanding of the content or emphasis of the ToRs or any other information Mid-October 2014 Completion of draft report. Debriefings with the MAPS Unit. End-October 2014 Feedback on draft evaluation report by MAPS Unit and partners/DGF grant recipients November 2014 Final report submitted to MAPS Unit. Shared with/ presented to MAPS and BAPS partners, and DGF recipients. It is anticipated that the consultant will need to devote 10-20 working days to fulfill the task. Role of MAPS Unit The MAPS Unit will manage the procurement of the evaluation. The MAPS Unit will provide consultant with key documents, facilitate contacts with MAPS partners/DGF grant recipients and other relevant partners, as appropriate, facilitate access to World Bank video conference facilities, and ensure independence of the evaluation. Obligations of the consultant The consultants shall: - Inform the MAPS Unit in a timely fashion of all contacts made with MAPS partners and constituents. - Treat documents in confidential manner. - Not publish evaluation results or output without permission of the MAPS Unit. - 76 - - Return all MAPS partnership documents used in the evaluation. - Report on a timely basis any possible conflicts of interest. Required Qualifications The evaluation expert should be a senior and an experienced evaluator with at least 15 years’ experience in the design and implementation of evaluations of projects and programs, with substantial experience in the evaluation of global development programs. Specific experience of the evaluation of programs and projects in developing countries would be an advantage. The statistician should be a qualified and experienced statistician or economist with at least 15 years’ experience in the design and implementation of statistical capacity building programs and projects, or the management of statistical systems, especially in developing countries. Experience in managing international statistical programs and agencies would be an advantage. The evaluation will be carried out in English and excellent written and spoken English is an absolute requirement for both posts. More information Further background information on MAPS and the MAPS partnership can be obtained from the World Bank website: http://go.worldbank.org/PRTR3BCNE0. - 77 -