70639 Crisis Monitoring and Response System (CMRS) Detailed Report September 2010 Table of Contents Abbreviations, Acronyms and Terminology .................................................................... 4 1 Purpose and Content of Document ........................................................................... 6 PART A: CMRS DESIGN, DEVELOPMENT AND IMPLEMENTATION ................. 7 2 CMRS Initiation and Organisational Structure ......................................................... 7 2.1 Origin and General Objectives ...................................................................... 7 2.2 Terminology .................................................................................................. 7 2.3 Scope.............................................................................................................. 8 2.4 Funding and Budget ....................................................................................... 8 2.5 Data Requirements: Content, Frequency and Timing ................................... 8 2.6 Existing Sources of Data ............................................................................... 9 2.7 CMRS Components ....................................................................................... 9 2.8 Complementary Qualitative Crisis Monitoring and Analysis ....................... 9 3 Management and Roles ........................................................................................... 11 3.1 Introductory Remark .................................................................................... 11 3.2 Client............................................................................................................ 11 3.3 Development and Analysis Team ................................................................ 11 3.4 Data Collection and Capture Contractor...................................................... 11 3.5 Consultant .................................................................................................... 12 3.6 Project Manager ........................................................................................... 13 3.7 Data Analyst and IT Specialist .................................................................... 14 3.8 Funding Agency........................................................................................... 14 3.9 Core Analysis Team and Analysis Review Group ...................................... 15 3.10 Other Stakeholders....................................................................................... 15 3.11 Summary of Roles ....................................................................................... 15 4 CMRSS General Design Considerations................................................................. 17 4.1 Specific Objectives ...................................................................................... 17 4.2 Coverage Considerations ............................................................................. 17 4.3 Content Considerations................................................................................ 18 4.4 Frequency and Timing Considerations ........................................................ 18 4.5 Relationship to Existing BPS Surveys ......................................................... 19 4.6 Choice of Panel Design ............................................................................... 20 4.7 Survey Collection Methods ......................................................................... 20 4.8 Obtaining Measures of Change ................................................................... 20 4.9 Approach Modelled on Lot Quality Assurance Sampling (LQAS)............. 21 5 CMRSS Sample Design .......................................................................................... 23 5.1 Sakernas Sample Design.............................................................................. 23 5.2 CMRSS Sample Design ............................................................................... 25 6 CMRSS Questionnaire and Indicators .................................................................... 27 6.1 Sakernas Questionnaire ............................................................................... 27 6.2 CMRSS Questionnaire ................................................................................ 27 6.3 Definition and Derivation of Indicators ....................................................... 28 CMRS Detailed Report Page 1 of 91 7 Data Collection ........................................................................................................ 32 7.1 Three Rounds of Data Covering Four Quarters ........................................... 32 7.2 Interviewer Assignment and Training ......................................................... 32 7.3 Handling Non-Response .............................................................................. 32 7.4 Transmission of Completed Questionnaires to Head Office ....................... 33 8 Data Capture, Storage and Preparation for Analysis............................................... 34 8.1 Data Capture, Storage and Transmission..................................................... 34 8.2 Data Preparation .......................................................................................... 34 9 Aggregation and Estimation .................................................................................... 36 9.1 Introductory Remarks .................................................................................. 36 9.2 Basic Probability Theory Assuming Simple Random Sampling ................. 36 9.3 Basic Probability Theory with Complex Survey Design Effects ................ 37 9.4 Sakernas Estimation Formulae .................................................................... 39 9.5 CMRSS District Level Estimation............................................................... 40 9.6 CMRSS District Change Estimation ........................................................... 41 9.7 CMRSS Provincial Level Estimation Formulae .......................................... 44 9.8 CMRSS National Estimation Formulae ...................................................... 44 9.9 Computation of Other Statistics Relating to Changes at District Level ...... 45 10 Analytical Framework ............................................................................................. 47 10.1 Aims of Analysis ......................................................................................... 47 10.2 CMRSS Data ............................................................................................... 48 10.3 Grouping of Indicators ................................................................................. 49 10.4 Analysis Procedures for National and Provincial Data ............................... 49 10.5 Analysis Procedures for District Data ......................................................... 51 10.6 Distinguishing Crisis Effects from Other Effects ........................................ 56 10.7 Data Quality Considerations ........................................................................ 57 11 Collection and Analysis of Health Data .................................................................. 57 11.1 Data Sources and Collection........................................................................ 57 11.2 Data Capture and Preparation ...................................................................... 57 11.3 Data Preparation .......................................................................................... 57 11.4 Data Analysis ............................................................................................... 58 12 Dissemination and Use of Results........................................................................... 59 12.1 Transfer of Outputs to Clients ..................................................................... 59 12.2 Response Identification ............................................................................... 59 12.3 Dissemination .............................................................................................. 59 PART B: RESULTS OF ANALYSIS ........................................................................... 61 13 General Effects of Crisis: Context for Analysis ...................................................... 61 14 Impacts of Crisis at National Level ......................................................................... 62 14.1 Labour Impacts ............................................................................................ 62 14.2 Household Economics ................................................................................. 65 14.3 Coping Strategies ......................................................................................... 67 15 Impact of Crisis at Provincial and District Levels .................................................. 71 15.1 Impact at Provincial Level ........................................................................... 71 15.2 Impact of Crisis at District Level................................................................. 72 CMRS Detailed Report Page 2 of 91 PART C: EVALUATION AND INSTITUTIONALIZATION ..................................... 73 16 CMRS Evaluation ................................................................................................... 73 16.1 Evaluation of Processes ............................................................................... 73 16.2 Quality Considerations ................................................................................ 74 16.3 Summary of Strengths and Weaknesses ...................................................... 76 17 Future Crisis Monitoring and Response System ..................................................... 77 17.1 Introduction.................................................................................................. 77 17.2 Objectives, Clients and Concepts ................................................................ 77 17.3 Previous Crisis Monitoring Experiences ..................................................... 80 17.4 System Design Considerations .................................................................... 82 17.5 Sakernas and Susenas as Potential Data Sources ........................................ 83 17.6 Data Analysis and Dissemination ................................................................ 85 17.7 Crisis Monitoring by Phase ......................................................................... 85 Annex I: Reference Documents ...................................................................................... 88 CMRS Detailed Report Page 3 of 91 Abbreviations, Acronyms and Terminology As applied to an indicator and district; a movement that is in the Adverse direction of a district being more at risk, for example, reduction movement in average hours worked by head of household. (Also referred to as a negative change.) As applied to a district, as evidenced by an adverse movement in At risk an indicator due to the effects of a crisis, caused by, for example, a global financial crisis or a tsunami. Australian Government agency responsible for managing AusAID Australia's overseas aid program Badan Perencanaan Pembangunan Nasional - Bappenas National Development Planning Agency. Baseline Same as benchmark. The value of an indicators obtained from a more reliable source Benchmark than the CMRSS, for example Sakernas, to which a value obtained from the CMRSS can be compared. As applied to an indicator, meaning having two possible values; Binary particular case of a categorical indicator. BPS Badan Pusat Statistik – Central Bureau of Statistics. As applied to an indicator, meaning having two or more possible Categorical values but not quantitative; aggregation over a population leads to counts and proportions. Change As applied to an indicator, a measure of difference over time. CMRS Crisis Monitoring and Response System. CMRSS Crisis Monitoring and Response System Survey. Effect of shock or more gradually deteriorating situation. Can be caused by a natural event such as disease or earthquake, or be human-related, for example, financial or political turmoil, or Crisis conflict. It can develop suddenly, following a shock, or over a longer period as conditions gradually worsen, for example, as result of a prolonged drought. Data item Same as indicator. Dichotomous (As applied to an indicator) same as binary. Dinas District health centre. Kabupaten - in this document the term will also be used to refer District to the kota, the urban equivalent of a district. GEC Global economic crisis beginning in 2008. GOI Government of Indonesia. CMRS Detailed Report Page 4 of 91 A district in crisis is defined as one adversely affected by the In-crisis GEC or other crisis, as determined by adverse changes in a number of indicators Characteristic of interest belonging to members of a population Indicator of interest. As applied to an indicator, the value of the indicator for given Level time reference period or point. Lot quality assurance sampling – procedure for taking small samples from subpopulations of a large population, with the aim of measuring a binary characteristic of interest and deducing from the sample results whether or not the proportion in each LQAS subset falls short or meets a specified target. For example: the population could be children within country; the subpopulations the children within each province: the characteristic, vaccination against smallpox: and the target 80%. Negative As applied to an indicator and district; the same as an adverse change movement. Group of entities of interest; in CMRSS context, populations of Population interest are persons, households, census blocks, districts, and provinces. PPS Probability proportional to size (sample) Puskesmas Pusat Kesehatan Masyarakat - community health centre. As applied to an indicator, meaning having value set comprising Quantitative a range of integers or real numbers. For an indicator and a district, indicating whether or not district is considered to be at risk based on the value of the indicator. Risk Flag For a district, indicating whether or not the district is considered to be in crisis. Survei Angkatan Kerja Nasional – national labor force survey Sakernas conducted by BPS. Sudden, unanticipated event, such as current financial crisis or Shock tsunami, that is not considered a part of a business trend or cycle, or seasonal, trading day or random effect. SMERU Independent institution for research and public policy. Survei Sosial Ekonomi Nasional - National Socio-Economic Susenas Survey conducted by BPS. Variable Same as indicator. Crisis and Vulnerability Monitoring and Response System VSMRS (possible successor to CMRS) World Bank staff and consultants involved in CMRS WB Team development. CMRS Detailed Report Page 5 of 91 1 Purpose and Content of Document This document contains a detailed description of the design, development, and operations of the Crisis Monitoring and Response System (CMRS). It covers the period from January 2009, when the CMRS was initiated, until September 2010 when the final CMRS reports (including this one) were completed. In addition to providing a definitive record of CMRS design and operational decisions, the document describes the results of the analyses based on the data collected. The document is intended to be the definitive source of reference material. It should prove particularly useful if and when consideration is given to the design, development and implementation of crisis monitoring system in the future. Part A: CMRS Design, Development and Implementation Chapter 2 describes the origin of the CMRS, its basic objectives and parameters Chapter 3 details how and by whom it was developed and managed. Chapter 4 covers the general aspects of the CMRS design. Chapter 5 describes the sample design. Chapter 6 details the questionnaire design. Chapter 7 documents the field operations - data collection and follow-up Chapter 8 describes capture and processing operations. Chapter 9 describes estimation procedures. Chapter 10 details the framework for analysis of the data collected by the CMRS at national, provincial and district levels. Chapter 11 describes the acquisition and analysis of health data. Chapter 12 refers to dissemination and use of the data for crisis monitoring. Part B: Results of Analyses Chapter 13 summarises the general effects of the crisis as reflected in data from the CMRS and other sources. Chapter 14 summarises the effects of the crisis as reflected in CMRS national level data. Chapter 15 summarises the effects of the crisis as reflected in CMRS provincial and district level data. Part C: Evaluation and Institutionalisation Chapter 16 summarises the key aspects of the design, development and results, and contains an evaluation. Chapter 17 completes the main body of the report, summarising the possibilities for institutionalising a crisis monitoring system. The appendices include additional references. CMRS Detailed Report Page 6 of 91 PART A: CMRS DESIGN, DEVELOPMENT AND IMPLEMENTATION 2 CMRS Initiation and Organisational Structure 2.1 Origin and General Objectives In January 2009, the Government of Indonesia (GOI), through its National Development planning Agency (Bappenas), decided it would develop a Crisis Monitoring and Response System (CMRS) to determine the impact of the global economic crisis (GEC) on Indonesia over the course of the following year. The CMRS was to generate data to assess the impact of the crisis and to identify the policy responses appropriate to alleviating the effects of the crisis on the poor and vulnerable. The ultimate objective was to ensure timely and appropriate policy responses in those districts identified as adversely affected as result of the global economic crisis (or any other shock). The output was to be an operational CMRS, comprising a data collection system and analyses that facilitated identification of appropriate responses. 2.2 Terminology In general terms, a crisis is a time of severe difficulty or danger. It is the effect of shock or more gradually deteriorating situation. It can be caused by a natural event such as disease or earthquake, or be human-related, for example caused by financial or political turmoil, or conflict. It can develop quickly, following a shock, as in the case of a tsunami, or over a longer period as conditions gradually worsen, for example, as result of a prolonged drought. The particular crisis that initiated the CMRS was the GEC. The effects of a crisis can be confined to a certain region, for example affected by flooding, landslide, or earthquake, or felt country-wide, such as food or fuel price increases. (Individual households may experience a crisis, for example resulting from death of the main income earner, but this sort of crisis is not being considered here.) For the CMRS a district in crisis was defined as one adversely affected by the crisis as determined by changes in a number of relevant indicators. When a district was identified as in crisis it was assigned a red risk flag. The households in such a district were referred to as stressed or struggling. An adverse movement (or negative change) in an indicator in a district was defined as one that was in the direction of the district being more at risk. Examples of adverse changes are reduction in rice consumption, reduction in average hours worked, and increase in unemployment. If an adverse movement in an indicator in a district was sufficiently large it led to the assignment of a red risk flag for that indicator for that district. A slightly smaller but still significant adverse movement resulted in assignment of an orange risk flag for that indicator for that district. A sufficient number of risk flags resulted in the district being declared in crisis. CMRS Detailed Report Page 7 of 91 2.3 Scope Although the CMRSS collected levels (i.e., current values) of indicators, movements (i.e., changes in level) of indicators were the main goal. For some indicators, for example employment, benchmark levels were available for earlier periods from more reliable sources. They were used as a basis for computing changes relative to a benchmark. However, there were no benchmark estimates of quarterly change for any indicator. Thus, the level of change was considered sufficiently large to result in the assignment of a red or orange risk flag was determined empirically. A distinction was drawn between crisis and vulnerability. Vulnerability refers to the propensity of a household or group of households to sink below the poverty level or to enter a period of extreme hardship. It is a state in which a household’s current situation is fine, but the household is on the borderline of hardship and can easily be pushed there. Vulnerability is a structural/chronic condition, typically measured in terms of low average levels of indicators such as household income, household assets, and household employment. A vulnerable district is more likely to become in crisis following a shock, but this is not inevitably the case as a district that is not vulnerable can become in crisis, and conversely. In summary, the term in crisis was used to describe a district that experienced negative changes in a number of relevant indicators over a recent period whereas a vulnerable district was defined as one that had generally low levels of indicators. Vulnerability is the subject of annual and semi-annual surveys, It was not the subject of the CMRS. 2.4 Funding and Budget Funding for the CMRS was provided by AusAID. The level of funding determined the scale of the CMRS. The budget was sufficient to support three rounds of a survey collecting data via a relatively short questionnaire from about 15,000 households. 2.5 Data Requirements: Content, Frequency and Timing Early in the design phase, a group of stakeholders came together for at a two day workshop (March 31– April 1, 2009). The general role of the CMRS and the specific data requirements were discussed. A program was proposed that comprised three rounds of quarterly data collection with initial results being available within a month. Subsequently this target period was extended to six weeks. One of the main workshop outputs was a comprehensive list of indicators that could be used for crisis monitoring purposes. As the list was far too long for practical implementation, it was reviewed and refined by the WB Team over the following months. The result was a shorter and operationally more feasible list (copy embedded below) which formed the basis for decided what data were required in addition to those already available from surveys and administrative sources List of Crisis Indicators (as of April 30, 2009) Crisis Indicators April 2009 CMRS Detailed Report Page 8 of 91 2.6 Existing Sources of Data Two existing, regular sources of data were identified as potentially useful for crisis monitoring: ï‚· data produced by BPS, in particular data from the labour force survey (Sakernas) and the national socio-economic survey (Susenas); and ï‚· administrative data available from other government departments and agencies, in particular, data related to the provision of health related government services provided by community (Puskesmas) and district (Dinas Kesehatan) health centres. Sakernas and Susenas data were (and are) semi-annual and thus not of sufficient frequency to meet CMRS data needs but useful in providing benchmarks. Health data were (and are) collected monthly and hence were seen as a very useful potential source. 2.7 CMRS Components The indicators required for monitoring that were not available from one or other of these sources became the target of a new quarterly household survey referred to as the CMRS Survey (CMRSS). The distinguishing features of the CMRSS were that it was quarterly, that it collected indicators not otherwise available, and that was conducted by the BPS under contract in conjunction with the existing BPS semi-annual labour and employment survey (Sakernas) but with much faster processing. The CMRSS was the first, and major, component of the CMRS. The second component of the CMRS was the collection of health data from the community (Puskesmas) and district (Dinas Kesehatan) health centres. 2.8 Complementary Qualitative Crisis Monitoring and Analysis Complementary to the CMRS was a qualitative analysis by conducted by SMERU. The intention was that, via regularly conducted rapid assessments, qualitative crisis monitoring would provide a deeper understanding of the phenomena causing the changes. In particular, the assessments would confirm (or not) the CMRS results in specific areas, would provide the possible causes and trends for observed changes in the quantitative data, and would help in determining the potential effectiveness of various policy responses. The main objective of combining the quantitative and qualitative crisis monitoring is to increase the trustworthiness of the results from both types of assessments. Such an increase may occur in one or more of four possible ways. ï‚· Internal validity or credibility. This is linked to the “truthâ€? of the results. The results of the qualitative assessments may confirm the quantitative survey results, or vice versa. Quantitative and qualitative assessments that largely correspond with each other give greater confidence that both sets of the results are credible. ï‚· External validity of transferability. This refers to the situation in which the results are also valid for other locations or other groups of people. For the current crisis monitoring, this could mean that, based on similarities in impact patterns of the GEC as identified by the quantitative surveys, the understanding of crisis impact and recovery pathways gained through the qualitative assessments in specific locations would help in understanding the crisis impact elsewhere. CMRS Detailed Report Page 9 of 91 ï‚· Reliability or dependability. This relates to obtaining the same or similar results when the assessment is repeated with the same or similar respondents in the same or a similar context. For example, the results from the two types of assessments for groups or locations that share similar characteristics (e.g. small fishing communities, or rice-growing agricultural communities) could be checked for similar behaviour over time. However, because the number of qualitative studies was relatively small and the locations where they were conducted were selected for their unique characteristics, such confirmation of reliability between the qualitative and quantitative assessments does not actually occur. ï‚· Objectivity or confirmability. This relates to increasing the certainty that the results are not influenced, or only marginally influenced, by the biases due to inadequacies and/or individual motivations or perspectives of the quantitative data collectors/ qualitative study investigators. Given that the data collectors and study investigators were quite different, coincidence or near coincidence of results is an indication of lack of bias. In summary, the qualitative assessment was aimed at strengthening the credibility and objectivity of the CMRS results, and, conversely, the CMRS results were expected to increase the confidence that the findings of the qualitative assessments could be generalized. CMRS Detailed Report Page 10 of 91 3 Management and Roles 3.1 Introductory Remark The organizational structure for CMRS development and implementation was never officially formulated and documented. The de facto situation was as outlined in the following paragraphs. 3.2 Client Bappenas was the CMRS client. At the time the CMRS was initiated in 2009, Dr Bambang, Deputy Minister Evaluation of Development Performance, played the leading role in establishing the objectives and outlining the data requirements. When Dr Bambang became the Deputy Secretary for Social Welfare to the Vice President in late, 2009, the role of client was shared between the division for Evaluation of Development Performance and the corresponding operational division. Dedi Masykur Riyadi and Endah Murniningtyas assumed the client role. In addition, in March 2010, a number of more junior Bappenas staff joined the CMRS Core Analysis Team (as described later). 3.3 Development and Analysis Team The CMRS was developed by a World Bank (WB) Team from the Poverty Section, initially comprising Dr Matthew Wai-poi, Dr Ririn Purnamasari and Ms Lina Marliani. (Ms Marliani left the Poverty Team in mid 2009). The Team reported to Dr Vivi Alatas. The Team took overall responsibility for the design and development activities, for planning, budgeting and managing the contracts, and for data analysis and presentation of results to the client. Bappenas and the WB Team determined the data required to support an operational CMRS in terms of coverage, content, frequency, timeliness and number of rounds. In conjunction with BPS methodologists and labour force and social survey specialists, the WB Team designed the CMRSS, as detailed in the following chapters. 3.4 Data Collection and Capture Contractor BPS was contracted to collect and capture the CMRSS data in accordance with the sample and questionnaire design and to collect health data. The BPS CMRSS Team, lead by Pak Happy Harjo, Head of Social Surveys was responsible for planning and designing the data collection, capture and processing activities. The contract involved three rounds of data collection by CMRSS questionnaire and from the district and sub-district health offices. The three survey rounds were conducted, in August 2009, November 2009 and February 2010. The August and February rounds accompanied the biannual Sakernas; the November round was standalone. Each survey round aimed to cover all 471 districts, with 5 census blocks of 6 households each being surveyed in each district, for a total of 471 x 5 x 6 = 14,130 households. There were also three quarterly rounds of health data collection. Each round collected the values of 39 indicators for each of the three months in the quarter. In each district, five community health centers (puskesmas) were surveyed, along with the district health office. If the number of community health centers in a district was less or equal than CMRS Detailed Report Page 11 of 91 five, then all community health centers in that district were surveyed. Data was to be processed within one month of collection. Further details are in the BPS Financial Proposal and in the Terms of Reference for the BPS contract embedded below Financial Proposal for CMRSS Data Collection by BPS CMRS Financial Proposal Terms of Reference for BPS Data Collection TOR for BPS Data Collection 3.5 Consultant In June 2009, Mr Michael Colledge was appointed consultant with the aim of guiding development and implementation of CMRSS design requirements, questionnaire, sampling plan, formulating the data analysis framework, and providing quality assurance. The consultant’s terms of reference stated The short-term consultant (STC) will be part of the Technical Assistance Team assisting the Government of Indonesia in establishing a Crisis Monitoring and Response system (CMRS). The STC will have primary responsibility for all aspects of the new household survey: (i) questionnaire design and testing; (ii) sampling design; and (iii) survey analysis. Tasks will include: ï‚· Revising the questionnaire household survey if necessary ï‚· Identifying sampling design if necessary ï‚· survey which is incorporated into existing BPS survey data collection ï‚· stand alone survey ï‚· Developing analysis framework for survey data ï‚· Working with Bappenas to set up survey analysis on an ongoing basis ï‚· Providing quality assurance for the CMRS in general and the CMRS Survey in particular. The outputs were specified as: ï‚· Field-testing of new household survey questionnaire, if necessary ï‚· New sample design for survey rounds done in conjunction with existing BPS surveys and stand-alone survey rounds, if necessary ï‚· Analysis framework for survey data ï‚· Technical assistance to Bappenas to support analysis of data ï‚· Reports on quality assurance, along with reviews on the current CMRS and CMRS Survey and recommendations. More details are in the Consultant’s Terms of Reference copy embedded below. CMRS Detailed Report Page 12 of 91 Terms of Reference for CMRS Consultant - Final August 2009 Consultant's Terms of Reference - Final August 2009 3.6 Project Manager In November 2009, Mr Luc Spyckerelle was appointed CMRS Project Manager, taking over the day to day planning and administration of the CMRS and monitoring operations. He was responsible for: ï‚· Providing strong leadership in functional CMS management and co-ordination to ensure that the project activities and processes operate to achieve the objectives. Ensuring timely preparation and implementation of project plans, procurement plan, monitoring and evaluation plan and stakeholder participation plan; ï‚· Developing an organizational framework to ensure coordination of activities across stakeholders and government counterparts. Liaising effectively with government counterparts and other relevant stakeholders. Ensure consultative and participatory processes with stakeholders; ï‚· Providing practical and effective guidance for members of the World Bank Technical Assistance Team to support planning and operations of the project; ï‚· Consulting closely with BPS on the preparation and implementation of the planned household and facility surveys; ï‚· Coordinating and providing technical direction on the development of analytical framework; ï‚· Coordinating and providing technical direction on the development of desktop reporting system; ï‚· Facilitating effective overall project operations and organizing regular meetings with government counterparts and other stakeholders, preparing minutes and implementing agreed actions/decisions and action plans; ï‚· Ensuring that outputs of CMS activities are delivered on time, to a high quality and within agreed budget; ï‚· Reviewing consultancy reports and deliverables of members of the Technical Assistant Team against intended outputs and regularly advising the World Bank’s Poverty Group Task Team Leader and Bappenas of progress; ï‚· Producing comprehensive reports after each survey round and a final report - the reports to provide an overall picture of the impact of the crisis at the district, provincial and national level, synthesizing the analysis of the activities and processes and discussing the effectiveness/challenges of the project with recommendations; ï‚· Developing a monitoring and evaluation framework for project performance; ï‚· Developing institutional and organizational frameworks for the development of a sustainable monitoring system, beyond a crisis monitoring system and lessons learns from establishing such system. This included identifying relevant counterparts and stakeholders, their particular roles and responsibilities, overall project activities and processes required. CMRS Detailed Report Page 13 of 91 More details are in the Project Manager’s terms of reference, copy embedded below. Terms of Reference for CMRS Manager TOR STC CMRS PROJECT MANAGER - 2009-08-07_final.pdf 3.7 Data Analyst and IT Specialist In November 2009 Pak Taufik Hidayat was appointed data analyst, taking over the editing, weighting and aggregation programs developed by Mr Matt Wai-poi for the first round. His tasks included: ï‚· Familiarization with the DesInventar methodology and its adaptation for DiBi and PNPM in Indonesia by working closely with the UNDP technical assistance teams in Indonesia and at the Regional Centre Bangkok; ï‚· Development of a dissemination prototype and sharing with key stakeholders for feedback and further enhancement of the system; ï‚· Building of linkages or system integration with other similar systems (DiBi, Simpadu-PNPM and others) to enhance the analytical capabilities of the system; ï‚· Redesign of the CMRS processing system and for easy administration and maintenance; ï‚· Identification and integration of related data and information that supports crisis and vulnerability monitoring and continuous updating of data; ï‚· Development of required user manuals and system instructions, and provision of trainings to key personnel as required; ï‚· Assistance with data analysis; ï‚· Preparation for institutionalization of the information management system at Bappenas or other relevant partner government agencies. In July 2010 an IT specialist was hired to develop the dissemination systems for CMRS and other vulnerability and crisis monitoring programs and studies. More details are in the IT Specialists’ terms of reference, copies embedded below. Terms of Reference for Data Analyst TOR Data Analyst Workplan for Second IT Specialist Workplan 3.8 Funding Agency As previously noted, AusAID provided the funds for the program. The WB Team reported to AusAID after each round of data was analysed. For illustration, a copy of the slides for the June presentation is embedded below. CMRS Detailed Report Page 14 of 91 Presentation to Aus AID June, 2010 For AusAID June 2010 3.9 Core Analysis Team and Analysis Review Group For the purposes of analysing second and third round data the Core Analysis Team (CAT) and Analysis Review Group (ARG) were constituted in February 2010. The role of the CAT was to develop analysis methods, to analyse the data and to present the results. The role of the ARG was to review the analysis methods and results. Member of the CAT included Dwi Ratih, Grace Manalu, Fisca Aulia, Yudhie Hatmadji and Hamid Rizali (from Bappenas) and Ardi Adji, Windy Prabowo (from BPS) as well as Luc Spyckerelle, Michael Colledge and Taufik Hidayat (from WB Team). Members of the ARG included Endah Murniningtyas, Dadang Solihin, and Vivi Yulaswati (from Bappenas), Wiwiek Arumwaty, S. Happy Harjo, Hamonangan Ritonga, Gautjang Amannulah, Purwanto Ruslam, Kadarmanto, and Margo Yuwono (from Bappenas) and Matthew Wai-Poi and Ririn Purnamasari (from the WB). In total the CAT met some 12 times, and the ARG three times. Further details are in the CAT/ARG Work Plan, copy embedded below. Core Analysis Team/ Analysis Review Group Initial Work Plan CAT/ARG Work Plan 3.10 Other Stakeholders Other CMRS stakeholders included the international and national organisations interested in poverty reduction and growth of the economy, in particular SMERU (which was conducting the qualitative study) and the UNDP. In addition to the March 2009 Workshop which focused on elaborating the objectives and content of the CMRS, several other meetings and workshops were organised for stakeholders In particular there were workshops in March and April 2010 at which methods and results to date were discussed. In May the WB Team presented a training session in analysis methods. 3.11 Summary of Roles Table 3.1 summarises the roles and participants in CMRS development and operations. Table 3.1 Participants in CMRS Development and Operations Role Responsible Support No formal steering Bappenas, SMERU, BPS, Steering Committee committee UNDP representatives Bambang Widianto/ Dedi Masykur Rivadi and Dadang Solihin and Vivi Client Representative Endah Murniningtyas Yulaswati (Bappenas) (Bappenas) CMRS Detailed Report Page 15 of 91 Table 3.1 Participants in CMRS Development and Operations Role Responsible Support Coordinator Vivi Alatas (WB) WB Team Luc Spykerelle (WB Management, Planning Team) (initially Ririn and Administration Purnamasari, WB) Collection and Michael Colledge (WB Estimation Matthew Wai-poi (WB) Consultant) Methodology BPS Methodology Data Collection and Happy Harjo (Director of Directorates of Population and Processing Social Welfare Statistics, Labor Statistics and Social Organisation BPS) Welfare Statistics, BPS Directorate of Population and Data Collection Field Operations, BPS Labor Statistics, BPS Data Capture and BPS Head Office Preliminary Editing Operations and IT Data Cleaning, Editing, Taufik Hidayat (WB Imputation, Tabulation Team) Michael Colledge (WB Weighting, Estimation Taufik Hidayat (WB Team) Consultant) Matthew Wai-poi, (WB) Core Analysis Team Analysis Michael Colledge (WB Analysis Review Group Consultant) Dissemination Systems Luc Spykerelle (WB Quality Assurance and Team) Evaluation Michael Colledge (WB Consultant) CMRS Detailed Report Page 16 of 91 4 CMRSS General Design Considerations 4.1 Specific Objectives The objectives of the CMRSS were succinctly summarized in the paper “Crisis Monitoring and Response Using Lot Quality Assurance Sampling as a Household Survey Designâ€? by Matthew Wai-poi as follows. The objective is to implement a low-burden household survey collecting data on household education, health and employment to identify districts requiring a specialized policy response to alleviate impacts of the current economic crisis. Requirements are that the survey be frequent (say, quarterly), low burden (low cost to put into the field, low technical capability required in the field, low processing cost), quick to turn around from fielding the survey to having the indicators available and digestible, and having national coverage but being representative at the kabupaten level. The intention is for a short, one to two page questionnaire. A key obstacle is that it is difficult using traditional cluster sampling techniques to get national coverage and be kabupaten representative, while also being low cost. 4.2 Coverage Considerations Specifying the required coverage of the CMRS in terms of the level and number of administrative units for which data were collected was a critical aspect of the CMRS design. It was decided that the level of unit at which crisis monitoring was to take place was the district. Province would have been too coarse a unit and sub-district would have been too fine. As there were 471 districts in 2009 (the number increased to 494 in 2010), a total CMRSS sample size of about 15,000 households, provided options of covering: ï‚· 500 districts with a sample of 30 households in each; ï‚· 200 districts with a sample of 30 households, 90 with a sample of 100; ï‚· 100 districts with a sample of 30 households, 60 with a sample of 200; ï‚· 100 districts with a sample of 150 households; ï‚· 50 districts with a sample of 300 households; ï‚· 25 districts with a sample of 600 households. The list of options showed that, if the number of districts of potential interest could have been reduced, then the sample sizes within each district could have been significantly increased. 600 households per district would have produced estimates of about the same precision as the Sakernas main annual sample. At the other extreme, a sample size of less than 30 would have been unlikely to reveal any information at all with acceptable precision. The client expressed a desire to cover all districts equally. The justification was as follows. Districts that could be reasonably expected to be in crisis on account of their dependence on exports (or other factor likely to have been affected by the GEC) could readily be identified and examined through qualitative research. To complement such research a system was required that could identify other districts that were in crisis. CMRS Detailed Report Page 17 of 91 Even after pressure to reconsider this viewpoint and to focus the sample on a sample of districts thought to be at greatest risk, the client reiterated the need to cover all districts equally. 4.3 Content Considerations It was assumed that structural differences between districts, as manifested in differences in levels of the various indicators across districts, were already known based on existing sources and that these differences had already been taken into account in formulating and implementing the ongoing policies. Thus, the main focus of the CMRS was negative changes at district level in the indicators identified as useful for crisis monitoring. Production of levels and changes at provincial and national levels was acknowledged as a valuable by-product, and a guide to district analysis, but not the primary target. 4.4 Frequency and Timing Considerations The fact that the CMRS aimed to measure effects of the GEC as it evolved implied a requirement for indicators on a monthly or quarterly basis. As the budget was sufficient for only three rounds of the CMRS, and in view of the time required to process the data, it was decided that the indicators would be collected quarterly. Even if there had been a budget for nine monthly rounds, it would likely have been better spent in conducting a quarterly survey with three times the sample size, or with twice the sample size for four quarters. As the aim was to obtain the earliest possible warning of districts at risk, production of estimates within one month from the start of data collection would have been ideal. However, this was (rightly) considered unrealistic and a target of six weeks was set, with an indicative schedule as shown in the following table. Indicative Schedule (in working days) CMRSS Activity Start Finish Develop questionnaire and instructions T Print and distribute questionnaire and instructions T+1 T+2 Select sample of CBs T+2 T+2 Select samples of households T+2 T+2 Collect data from households T+3 T+4 Transmit questionnaires to Head Office T+5 T+5 Follow-up non-responses T+5 T+5 Transmit completed questionnaires to Head Office T+14 T+14 Capture and check data T+6 T+26 Edit and impute data T+11 T+26 Weight data and generate national and provincial T+16 T+16 estimates and district risk flags - preliminary Check results and fine tune estimation procedures T+16 T+26 CMRS Detailed Report Page 18 of 91 Indicative Schedule (in working days) Weight data and generate national and provincial T+27 T+27 estimates and district risk flags - first production run Check results and fine tune estimation procedures T+27 T+32 Weight data and generate national and provincial T+33 T+34 estimates and district risk flags – final production run Send data outputs to clients T+35 T+35 Analyse data outputs T+35 Prepare analysis presentations and reports T+45 T+50 Based on this indicative schedule the following target schedule for the three rounds was developed. 2009 2010 Activity May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul CMR Survey Round 1 Survey design Questionnaire tryout Preparation for implementation Implementation and supervision Data sent to BPS for data entry and validation Data analysis Report writing and presentation of results CMR Survey Round 2 Questionnaire revision and survey preparation Implementation and supervision Data sent to BPS for data entry and validation Data analysis Report writing and presentation of results CMR Survey Round 3 Questionnaire revision and survey preparation Implementation and supervision Data sent to BPS for data entry and validation Data analysis Report writing and presentation of results As a result of the teething problems that had to be solved, and the staff available, there was some slippage in the schedule in all rounds, particularly in data analysis. 4.5 Relationship to Existing BPS Surveys A standalone survey would have been expensive and wasteful of resources, hence the CMRSS was conducted as a subsample of, and piggy-backed on, an ongoing BPS household survey. There were two BPS surveys that could have been carriers, namely, Susenas and Sakernas, both of which were (and are) conducted semi-annually. Of these two, Susenas would have been the more natural vehicle in terms of data content. However, the June 2009 round of Susenas was in the field before the CMRSS design could be completed, which meant that Susenas was not a feasible option. Thus Sakernas was selected as the carrier survey, with the first round being August 2009. Sakernas is a semi-annual survey conducted in February and August. It was partially redesigned in August 2007 and a rotating sample was established for use until August 2010. (The future sample design is discussed in Chapter 17.) The sample actually comprises two samples: CMRS Detailed Report Page 19 of 91 ï‚· the provincial (semi-annual) sample producing national and provincial level data for February and August; and ï‚· the district (annual) sample producing national, provincial and district level data for August. The semi-annual sample is a subset of the annual sample. The CMRSS was designed as a subsample of the semi-annual sample to the extent possible, as further discussed below. 4.6 Choice of Panel Design The benefits of cross sectional sample design for the CMRSS would have been: ï‚· data could have been added or averaged across quarters to produce annual level estimates at district level of acceptable precision; ï‚· flexibility to adjust to new coverage needs, for example larger sample sizes in selected districts, on a quarterly basis; ï‚· more scope for selecting a sample that was a subset of Sakernas provincial sample in February quarter. The benefits of a panel were: ï‚· simpler field operations; and ï‚· higher precision estimates of quarterly change. Given that the focus of the CMRSS was producing estimates of change, a panel design was selected as being far and away preferable to a cross-sectional design. This did not preclude the possibility of making corrections or minor improvements to the questionnaire between quarters, or excluding some districts from the sample, based on data quality as the data were analysed. 4.7 Survey Collection Methods The sort of indicators required could be effectively collected only by personal interview as an insufficient proportion of households had landlines for telephone interview and a mail questionnaire would not have elicited sufficiently quick or accurate responses. 4.8 Obtaining Measures of Change The CMRSS was aimed at measuring short term changes rather than levels. Whilst there were benchmark data available for levels of some indicators, in particular the labour force indicators, there were no benchmark data available for quarterly changes. Thus, for the first (August) round, measures of quarterly change were obtained by asking the respondent to report values of the indicators for a reference period one quarter earlier (May) as well as for the current period. This was not ideal as it involved respondent recall and the likelihood of increased reporting error. However, it was the only solution. With the exception of a small number of districts not covered in the first round, there were no questions requiring respondent recall in the second and subsequent rounds as data from the previous round were available. CMRS Detailed Report Page 20 of 91 Consideration was given to asking the respondent for data for the previous month. This would have involved a shorter recall period but was rejected for three reasons. ï‚· First, it would have implied maintaining two sets of changes – monthly and quarterly – which would have been confusing. ï‚· Second, monthly changes might have proved too small to measure given the district sample sizes. ï‚· Finally, it was quite difficult to rephrase the employment related questions in such a way as to indicate accurately and consistently what was meant by “last monthâ€? whereas “three months agoâ€? is sufficiently vague that subtle distinctions between reporting periods became irrelevant. Another approach that was considered for handling respondent recall error in the first quarter was to ask for change in categorical terms. For example in place of asking for number of days absent from school over a one month period three months ago, an alternative would have been to have asked whether the number of days absent from school (a) increased, (b) is about the same, (c) decreased over the last quarter. This coarser response might well have been just as accurate. However, the option was rejected as the indicators would have been different in the first quarter than in subsequent quarters. 4.9 Approach Modelled on Lot Quality Assurance Sampling (LQAS) Lot quality assurance sampling (LQAS) is an appropriate technique for determining whether the coverage (expressed in the form of a rate of some particular binary indicator) in a population of units of interest is acceptable or not based on a small sample (referred to as a lot) drawn from the population. As described in Matthew Wai- poi’s paper Crisis Monitoring and Response Using Lot Quality Assurance Sampling as a Household Survey Design (and in more detail in other references, see Annex), the basis of the LQAS approach is as follows. ï‚· Take a small random sample of units in a lot. In public policy this usually involves sampling households or individuals within a geographic district. If the number of households failing in terms of a certain indicator exceeds a predetermined allowable number, then the district is found to have an unacceptable level for this indicator. The number of allowable households is based on a set standard and a predetermined sample size. The sample size is chosen so that the evaluator has a high probability of designating a district in which the proportion of households that actually pass on the indicator as acceptable, and has a high probability of rejecting districts that fail to reach the set standard. ï‚· An example is an evaluation of malnutrition within a district. Children not gaining weight in a month are deemed to fail the nutrition indicator, and districts with less than, say, 50% of children gaining weight would be flagged as requiring intervention or further investigation. Consideration was given to applying the LQAS approach directly to classification of districts as in crisis or not. However classifying districts based on CMRSS binary indicators would have meant interpreting percentage change as coverage and, for each indicator, identifying acceptable and unacceptable levels of adverse change as the basis for determining a decision rule that designated districts as acceptable or not in terms of the indicator. There are two reasons why this would not have been easy. First, change CMRS Detailed Report Page 21 of 91 in a binary indicator is not itself a binary indicator as it can have three values (-1, 0, +1). Second, there was no basis for specifying acceptable or unacceptable levels of change. Thus, LQAS was not directly used. However the approach adopted was in the spirit of LQAS in the sense that, given the very small sample sizes and consequent low level of reliability of estimates at district level, district risk flags were set in preference to disseminating estimates. CMRS Detailed Report Page 22 of 91 5 CMRSS Sample Design 5.1 Sakernas Sample Design As the CMRSS sample was a subsample of the Sakernas sample, it was vital to understand the Sakernas sample design, which, up to and including August 2010, was a stratified two stage cluster design as described in the following paragraphs. Sampling Frame The primary sampling units were census blocks (CBs) as defined for the 2000 Population Census and updated in 2005 for the elections. CBs were classified as urban or rural and carried population and household counts, which were used as measures of size for sample selection and estimation. There were in the order of 500,000 CBs in total across 471 districts in 33 provinces. For Sakernas sampling purposes, the CBs were divided into just two strata – urban and rural. Within each stratum, CBs were systematically ordered in a standard sequence within each district according to their relative geographic locations. The districts themselves were systematically ordered in a standard sequence within provinces, which themselves were ordered in a standard sequence. As all sampling is done using systematic selection, the net result of the ordering is to produce implicit stratification by province and district and by geographic location within district. The second stage sampling units are households. (The BPS does not define or list dwellings.) There were some 60,000,000 households in total which on average contained about 4 persons per household for a total population in the order of 240,000,000. Annual Sample Allocation and Selection The annual sample was designed to produce district level estimates. It contained 18,318 CBs of which 6,712 were urban and 11,606 are rural. They were allocated to districts such that there were between 30 and 60 CBs in almost all districts, thus giving estimates of similar precision across districts. Within this range the larger districts tended to have larger sample sizes however the actual numbers of CBs per district were not determined by any particular rule (such as a square root allocation by size). Within each stratum, within each district, the first stage sample of CBs was independently, systematically selected with probability proportional size (pps), the size measure being number of households in the CB (as originally collected in the 2000 Census and updated for the electoral purposes in 2005). Within each selected CB, households were identified and listed in a standard order according to their location. A second stage sample of 16 households was then selected using systematic random sampling. This produced about 293,088 households in total across all districts in the complete sample. Semi-Annual Sample Allocation and Selection The semi-annual sample was a subset of the district sample of CBs and households and was designed to produce provincial level estimates. The sample size was 4364 CBs of which 1772 were urban and 2592 are rural. The sample was allocated to provinces in proportion their size, rounded to the nearest multiple of four for rotation purposes. CMRS Detailed Report Page 23 of 91 Within each stratum, within each province, the first stage semi-annual sample of CBs was independently, systematically selected from the district sample of CBs. As a random sample of a pps sample is itself a pps sample, the provincial sample was a pps sample within each district. Within each selected CB, the second stage sample comprised the 16 households as selected in the district sample, giving 69,824 households in total across all districts in the complete sample. Sample Rotation and Household List Maintenance For rotation purposes, all CBs in the annual (and hence semi-annual) sample were assigned to one of six packets labelled 1 to 6. The allocation began with the CBs in the semi-annual sample, which were assigned to the packets labelled 1 to 4 using a systematic assignment. As the number of CBs within each stratum within each province was a multiple of four, the number of CBs within each panel, within each stratum within each province was the same. This meant that rotation of the semi-annual sample (as described below) was balanced across strata within province. The remaining CBs in the annual sample were allocated to the packets labelled 5 and 6 using a systematic assignment. As the number of CBs within each stratum within each province was a multiple of two, the number of CBs within each panel, within each stratum within each district was the same. This meant that rotation of the annual sample (as described below) was also balanced across strata within district. The initial samples of households selected within Packets 1 and 4 were referred to as household clusters A, B, C, and D respectively. ï‚· Within Packet 1 two further samples of 16 different households were selected in each CB. These were collectively referred to as household clusters E and I. ï‚· Within each of Packets 2, 3, and 4, one further sample of 16 different households were selected in each CB. These were collectively referred to as household clusters F, G, and H, respectively. The initial samples of households selected within Packets 5 and 6 were referred to as household clusters J and K respectively. ï‚· Within each of these packets two further samples of 16 different households were selected in each CB. These were collectively referred to as household clusters L and M. The resulting packets and rotation pattern are shown in the following table The net result of this approach is that no household remained in sample for more than 2 years – that is 4 rounds for the semi-annual sample and 2 rounds for annual sample. Of particular note is the fact that between August 2009 and February 2010, the cluster sample in Packet 4 was rotated. Thus only the households in Packets 1-3 were utilised in forming the CMRSS panel. Sakernas Packets and 2008 2009 2010 Packets Rotation Patterns Feb Aug Feb Aug Feb Aug CBs in CBs in Semi- 1 A E E E E I Annual Annual Sample 2 B B F F F F CMRS Detailed Report Page 24 of 91 Sakernas Packets and 2008 2009 2010 Packets Rotation Patterns Feb Aug Feb Aug Feb Aug Sample 3 C C C G G G 4 D D D D H H Remaining CBs 5 I L L in Annual Sample 6 J K M Household List Maintenance Prior to data collection the household listings within the sampled CBs were checked and updated. 5.2 CMRSS Sample Design Sampling Frame The CMRSS sampling frame was the list of CBs in the Sakernas annual sample. This meant that no additional listing work needed to be done for the CMRSS and that the CMRSS sample was bound to be a subset of Sakernas annual sample. Sample Size and Allocation Designing the sample for district level estimates of equal precision required an equal number of households (say m) in each district. For simplicity of implementation, the number of CBs per district (say n) had to be the same. Given a budget related total of 15,000 households, the values that m and n could have taken are shown below. Option 3X10 4X8 5X6 6X5 7X4 8X4 9X3 10X3 Households 3 4 5 6 7 8 9 10 per CB (m) CBs per 10 8 6 5 4 4 3 3 district (n) Households per district 30 32 30 30 28 32 27 30 (mn) Households 14130 15072 14130 14130 13188 15072 12617 14130 total Considerations in selecting m and n were as follows. ï‚· The higher m, the higher the design effect (DEFF) resulting from the clustering, which meant the less precise the estimates. ï‚· The higher n, the more CBs that would not be in the Sakernas semi-annual sample and hence the greater the costs incurred in collecting data from the February sample (as further discussed in the following section). ï‚· As CMRSS data were to be collected with priority compared with Sakernas data, for operational purposes m should roughly equal to a day’s work load. Analysis of the options suggested that a sample allocation of n=5 CBs by m=6 households provided an optimum balance between the desirability of a low m for maximum precision, an m that reflected a day’s work in the field, and a low n that minimised the cost of collecting data from CBs not in the semi-annual sample in February 2010. CMRS Detailed Report Page 25 of 91 Selection of Sample of CBs within Each District The aim of CMRSS sample selection was to create a panel sample that: ï‚· was a subsample of the Sakernas annual sample; ï‚· was a pps sample at district level and hence self-weighting at district level; ï‚· contained 5 CBs in each district having maximum overlap with the February semi- annual Sakernas sample, thereby minimising the number of households to be visited in addition to those in the semi-annual sample; With these conditions in mind the CMRSS sample was selected as follows. 1. In each stratum (urban/rural) in each district a list of the CBs in the Sakernas semi- annual sample in clusters Packets 1, 2, and 3 was extracted and sorted in the usual order of CBs in that district. A single list for the district was then created by simply adjoining the lists for the two strata (urban, rural) in the usual order. 2. When the resulting list of CBs in a district contained less than m=5 CBs, then, for each stratum in the district, the list of the CBs in the Sakernas annual sample was extracted and sorted in the usual order of CBs in that district. A single list for the district was then created by simply adjoining the lists for the two strata in the usual order. 3. In each district, from the list of CBs in the semi-annual sample thus formed a sample of m CBs was systematically selected. If there were less 5 CBs in the list, they were all selected. 4. If in step 3, r=35 hours per week, and as 0 otherwise. CMRS Detailed Report Page 29 of 91 A binary risk flag based on the quantitative indicator weekly hours worked would be defined along the lines: ï‚· 0 if no evidence of significant reduction in weekly hours worked; ï‚· 1 if significant reduction in weekly hours worked; whereas a binary risk flag based on the derived binary indicator full employment would be defined along the lines: ï‚· 0 if no evidence of significant reduction in full employment rate; ï‚· 1 if significant reduction in full employment rate. These two risk flags would obviously be highly correlated but not necessarily identical. For example average hours worked could reduce significantly because part time employees worked fewer hours, whereas the number and proportion of full time employees remained the same. In this case the weekly hour worked risk flag would be set but the full employment risk flag would not. Alternatively a significant proportion of full time employees could become part time but without a significant impact on average hours worked due to an unchanged and relatively large contribution of part time employees. In this case the weekly hour worked risk flag would be not set but the full employment risk flag would be set. A categorical indicator with m possible values can be replaced by m-1 binary indicators with no loss of information at all. For example, the categorical indicator employment status (employed, unemployed, not economically active) could be replaced by two binary indicators: ï‚· a employed status indicator - defined as 0 if not employed, 1 if employed (leading to employment rate in accordance with the standard definition); and ï‚· a economically active status indicator - defined as 0 if not economically active and 1 if economically active. The particular indicators chosen were those that appeared most appropriate in terms of displaying the effects of a crisis. Reference Periods There were slightly differing reference periods for different indicators: ï‚· labour market indicators were for the first weeks of May, August, November, and February, ï‚· most other indicators were for the months of April, July, October and January. For the sake of simplicity when describing the data, however, reference is made to May, August, November and February quarters for all indicators, even though, for some indicators, the data actually refer to one month earlier, or to the quarter ending in these months. Definition of Underlying Population In defining the proportions associated with binary indicators, the relevant population (and hence the denominator and in calculating proportions) must be appropriately defined. For example: CMRS Detailed Report Page 30 of 91 ï‚· the population for a dropped out of secondary school indicator is the set of households with children of secondary school age (i.e., not all households); ï‚· the population for an employed/not employed indicator is the set of households with an economically active head of household (i.e., not all households). Thus, there were several indicators for which the sample of 30 households yields less than 30 relevant households, as further discussed below. Metadata The precise meanings and derivations of all indicators (including those obtained from the health data sources) and the changes made in those indicators between rounds are given in the document embedded below. CMRS Questionnaires: Metadata CMRS Questionnaire Metadata CMRS Detailed Report Page 31 of 91 7 Data Collection 7.1 Three Rounds of Data Covering Four Quarters The first round of the CMRSS was conducted in August 2009, collecting data for August 2009 and May 2009 reference periods. The interviewed households were part of the Sakernas sample, and the survey was conducted in the form of additional data collection for the 30 households that belonged to the CMRSS sample within each district. CMRSS data were collected for households in the CMRS sample after obtaining the Sakernas data. The second round of the CMRSS was conducted in November/December 2009, collecting data for the November 2009 reference period. This round was conducted as a stand-alone survey. The third round of the CMRSS was conducted in February 2010, collecting data for the February 2010 reference period. This survey was conducted for the most part as a supplement to the Sakernas semi-annual survey. A small number of households not in the Sakerbas sample had to be separately covered. The BPS Head Office notified the BPS district offices well in advance of the three CMRSS rounds. Copies of the questionnaire were printed at Head Office and distributed to district office representatives via the provincial offices. 7.2 Interviewer Assignment and Training The BPS used three person interviewer teams for the household visits. This seemed expensive but was considered necessary to ensure that the interviews were properly conducted. There was no time nor funds for interviewer training in any round. Thus, interviewers collected CMRSS data without practice. This required the questionnaire to be relatively simple to understand and administer and the interviewer teams to include one or more experienced interviewers. The CMRSS data were collected by a single team of interviewers, or at most two teams, in each office. As data were required from 6 households in each CB, in principle data from all the households in a CB could be collected in a single day by a single interviewer team. Data collection from households in the CMRSS sample was given priority over data collection from other households reporting to Sakernas. 7.3 Handling Non-Response Interviewers followed up as many non-responses as possible during the first week. Even so, non-response usually prolonged data collection activities into the following weeks. Some of the reasons for non-response and the corresponding actions taken during Sakernas field operations and (if still persisting) at the estimation stage are indicated in the following table. CMRS Detailed Report Page 32 of 91 Non-Response Reasons and Treatments Treatment by District Treatment in Estimates Reason Office (if still non-response) Household has moved out of Replace with one of the CB two alternate households Household has moved within Locate household at new CB address Interviewer has not managed to Ignore record Keep trying until end of find anyone at home and able data collection period to respond No one in the household is Replace with one of the willing to respond two alternate households Achieved Response Rates At national level the response rates were good (85% or more). There were some districts and provinces where response rates were poor. 7.4 Transmission of Completed Questionnaires to Head Office For each completed Sakernas questionnaire with a CMRSS supplement, the interviewer team copied relevant data from the Sakernas questionnaire to the CMRSS questionnaire. The completed questionnaires were sent to Head Office by regular mail. In principle, a copy was made of the entire completed CMRSS questionnaire and retained in the district office for use in the event of loss of the original completed questionnaire. In view of non-response and the need to follow-up those households, most districts made a second mailing to Head Office, several weeks after the first. CMRS Detailed Report Page 33 of 91 8 Data Capture, Storage and Preparation for Analysis 8.1 Data Capture, Storage and Transmission Data from the completed questionnaires received at Head Office were captured by teams of BPS clerical staff. These data were handled well in advance of, and quite separately from, Sakernas data. Data storage did not pose any particular problems as the data had a simple structure and there were only 15,000 or so records for each round. Data for each round were stored by BPS in a database and supplied to the WB Team on CDs. 8.2 Data Preparation The captured data were brought into a STATA database, linked across rounds, reformatted, edited, aggregated, and output into Excel spreadsheets for analysis by the WB Team. Data for the same household across rounds were linked using head of household name. This was a tedious and time consuming process requiring manual intervention. It could have been easily avoided if the raw data had included a unique household identification code across all rounds, Reformatting included derivation of indicators such as employment status and conversion of all of the indicators subject to analysis to numeric. Editing was required as the data received were certain to contain errors: ï‚· the original questionnaire was put together very quickly; ï‚· there was no formal interviewer training; and ï‚· the BPS conducted only very limited checks during data entry. In the case of quantitative indicators it was easy to spot some unlikely values, for example: ï‚· values of 0 meals per day and 6-8 meals per day; ï‚· clustering of hours worked around high multiples of seven like 91 and 98; ï‚· very different values from one quarter to the next that comprised reversed digits or some very unlikely differences, for example transportation costs decreasing by a factor of 4; and ï‚· huge consumptions of rice by value (suggesting values recorded in units rather than thousands of rupiah) and by volume (suggesting values recorded in litres or grams rather than kilograms); In the case of the binary variables, errors in the data for any given quarter were not so obvious. However, quarter to quarter comparisons indicated some anomalies. For example, some districts showed incredible quarterly swings in the numbers of households with females working, and these swings were in both directions. Time did not allow comprehensive editing. However, two obvious types of problems were addressed by automated editing. Unreasonably large values of rice value and rice CMRS Detailed Report Page 34 of 91 volume (actually measured in kilograms) were replaced by plausible maximums. The adjustment method and results are described in more detail in the document embedded below. Editing rice volume and rice value Editing Rice Value and Rice Volume CMRS Detailed Report Page 35 of 91 9 Aggregation and Estimation 9.1 Introductory Remarks In formulating the estimation formulae two possible approaches were considered. First, as the CMRSS sample was a subset of Sakernas sample, the Sakernas estimation methods were taken into account. Second, given the small district sample sizes, there was the case for using lot quality assurance sampling (LQAS) at district level. This chapter describes the general approach and the actual methods used to obtain estimates. Section 9.2 outlines the general probability theory under the assumption of simple random sampling. Section 9.3 describes the effects of the survey design, in particular the two stage cluster sample at district level, and the weighting of district estimates to obtain provincial and national estimates. Section 9.4 presents the formulae used for the Sakernas on which the CMRSS was piggy-backed. The following sections give the estimation formulae on which district, provincial and national estimates of level and change were based. 9.2 Basic Probability Theory Assuming Simple Random Sampling For Quantitative Indicators Assuming for ease of explanation: ï‚· the value of a quantitative indicator is represented by X; ï‚· E(X) = µ; Var(X) = σ2; ï‚· when a probability sample of n units is drawn from the population and the values of X measured, the sample mean is represented by . If the sample is a simple random sample (SRS) with replacement then E( ) = µ, Var( ) = σ2/n and has approximately a normal distribution. This is denoted by ~ Normal (µ, σ2/n). In most situations, sampling is without replacement, and then has approximately a normal distribution with E( ) = µ, and Var( ) = {(N-n)/(N-1)}σ2/n, where N is the population size. The term {(N-n)/(N-1)}is referred to as the finite population correction factor. It can be ignored when N>>n. In a case such as the CMRSS where, at district level n=30 and N is in the range 10,000- 200,000, the finite population correction factor is ignored. If ~ Normal (µ, σ2/n) then Z ≡ ( - µ)/ (σ*√n) ~ Normal (0,1) and normal cumulative frequency distribution tables provide: ï‚· the set of values {zα ,1-α}for which Prob (Z≤ zα) = 1-α, ï‚· also, by symmetry of the N(0,1) distribution, Prob (Z≤ -zα) = α. Thus normal cumulative frequency distribution tables can be used as the basis for probability statements about the observed sample mean. CMRS Detailed Report Page 36 of 91 Counts and Proportions Based on Binary Indicators The terminology can be a bit confusing as it is sourced from two different applications of the same binomial distribution theory, namely, quality control, and lot quality assurance sampling (LQAS). Both applications start with the concept of a population of individual units (households in the CMRSS context) each of which may or may not have a particular property. The proportion of units having the property is usually denoted by p, with q defined as 1-p being the proportion not having the property. A probability sample of n units is drawn from the population and the number of units, X, not having the property, is counted. Under the assumption of a simple random sample with replacement X is distributed binomially with parameters n and q. This is denoted by X ~ Binomial (n,q). In quality control theory the property of interest is that of being defective, and the focus is on the proportion of defectives in the population, which is estimated by the number of defectives in the sample X as a proportion of the sample size n. In LQAS the property of interest is that of being covered, for example by a vaccination program, and the focus is on the proportion covered in the population which is referred to as the coverage, typically denoted by p. For sets of values of n, d and p, LQAS tables give the Probability (X≤d) where X is the count of units not covered in a sample of size n. Intuitively, the quality control notion of defective is equivalent to the LQAS notion of not covered. However, whereas in quality control the focus is on the proportion defective, in LQAS it is on the proportion covered. This can lead to confusion in using the tables. In the CMRSS context neither covered nor defective were strictly appropriate terms as the binary indicators being analysed did not refer to coverage or to defects. Given the focus on the use of LQAS, the following convention was adopted for the CMRSS in describing how the binomial theory could be applied. ï‚· A property of interest was expressed in such a way that having the property was economically desirable, for example being employed, not dropping out of school. ï‚· A unit that had the property was referred to as covered or not defective. ï‚· A unit that did not have the property was referred to as not covered or defective. ï‚· The proportion of units having the property in the population was denoted by p. ï‚· The proportion of units in the population not having the property was denoted by q defined as 1-p. ï‚· X denoted the count of units not having the property in a sample of size n. Under this convention, assuming the sample was a simple random sample (SRS) without replacement, and knowing that sample size was much smaller than the population size, X ~ Binomial (n,q) and, for sets of p and d, {Prob (X≤d)} could be obtained from LQAS style binomial tables. 9.3 Basic Probability Theory with Complex Survey Design Effects In the CMRSS context there was a sample design consideration that could not be ignored. The CMRSS did not actually involve a simple random sample (SRS) within CMRS Detailed Report Page 37 of 91 each district. Rather the CMRSS was a particular case of a so called complex survey design. Specifically, the CMRSS had a two stage cluster sample design in which the first stage selection of 5 census blocks per district was with probability proportion to size and the second stage was systematic selection of a sample of 6 households from each census block,. The effects of the differences between this cluster sample design and a SRS outlined in the following paragraphs. For Quantitative Indicators Suppose that for some population and quantitative indicator, X, E(X) = µ; Var(X) = σ2. As noted above, if simple random sample of n units is drawn from the population, the sample mean ~ Normal(µ,σ2/n). If on the other hand, the sample of n units is drawn using a complex design, then, depending upon the design, E( ) may still equal µ, but Var( ) = (σ2/n)*DEFF, where DEFF is the design effect corresponding to the particular design. In other words, from perspective of the variability expected in the sample mean (or other sample estimate) the effective sample size, n* = actual sample size n /DEFF. For each district the CMRSS was a two stage cluster design which was self weighting thus E(X) = µ. At the second stage of sampling as same number of households (m) was selected from each census block, the DEFF = 1+(m-1)Ï? where Ï? is the intracluster correlation coefficient. As m=6 for the CMRSS, the DEFF = 1+5Ï?. For example, if Ï?=0.1, the DEFF=1.5; if Ï?=0.2, the DEFF=2.0; if Ï?=0.3, the DEFF=2.5. Thus, if for a particular indicator Ï?=0.1, with an actual sample size of 30, the effective sample size was 20. For Counts and Proportions The situation is similar for counts and proportions corresponding to a binary indicator. Suppose for some population and property of interest, the proportion of units having the property is q. As noted above, if a probability sample of size n is selected and the number of units X having the property is counted, X ~ Binomial (n,q), For the CMRSS cluster sample, E(X) = nq, but Var(X) = npq*DEFF, where, as noted above, DEFF = 1+5Ï?. For the sample mean , E( ) = q, Var( ) = (pq/n)*DEFF = pq/n* where the effective sample size n*=n/DEFF. Thus, intuitively, it is not appropriate to use the Binomial (n,q) frequency table to compute probabilities with a cluster sample as the effective sample size n*= 20 there is little difference between the N(0,1) and tn-1 distributions.) In calculating the z-score the main issue is the calculation of the variance. As previously noted the options are: ï‚· low variance estimator – using paired values. In the case where some observations are not paired this almost invariably gives an underestimate. ï‚· high variance estimator – taking the two estimates of level of which the change estimate is the difference, and adding together their variance estimators. This would be exact under the assumption that the corresponding samples of households for the two periods were completely independent. As the samples were, in fact, heavily overlapping, this almost invariably gives an overestimate. Both sets of estimates were computed. P-value (paired) The p-value was the probability that the change estimate with the observed or a more extreme value could have been derived at random from a population of changes having an average of zero. It was computed on the basis of the paired values using the appropriate options in STATA SVY option to take the cluster design into account. Relationship between z-score and p-value (paired) Where all the observations were paired, or where only where paired observations are used in the calculations, the z-score and p-value were in 1-1 correspondence as the p- value was simply the probability of observing a value equal to or more extreme than the z-score. For example, assuming a Normal (0,1) distribution, a z-score of 1.96 corresponded to a p-value of .025. However, where not all observations were paired and the z-score estimate was computed using the higher variance estimator the z-score was typically smaller than the value corresponding to the p-value assuming a Normal (0,1) distribution. CMRS Detailed Report Page 46 of 91 10 Analytical Framework 10.1 Aims of Analysis As the basis aim of the CMRSS was to pick up impacts as soon as they occurred, the primary focus of the analysis was quarter to quarter changes, in particular changes in an adverse direction. However, given the data were experimental, some attention was also paid to levels, particularly as they provided a basis for assessing subsequent changes. Aims of Analysis at National Level ï‚· To identify indicators that showed significant adverse movements between the current quarter and earlier quarter, to quantify those movements, and to suggest possible causes and consequences. ï‚· To quantify the current quarter movements in indicators that in the previous quarter showed significant adverse movements, and to suggest possible causes and consequences. ï‚· To identify indicators that showed significant adverse movements relative to earlier benchmark data, and to quantify these movements, and to suggest possible causes and consequences. ï‚· To determine variations in current quarter movements according to rural/urban breakdown, poor/non-poor breakdown, and sex breakdown (where sex is determined by the sex of the head of household). ï‚· To summarise the levels of and movements in indicators for which data were not recently available from other sources. Aims of Analysis at Provincial Level At provincial level the sample sizes were much smaller so no attempt was made to explore rural/urban, poor/non-poor, and sex breakdowns. Also, in view of the large number of indicators, analysis was focused on those indicators that showed significant quarterly movements at national level, in particular: ï‚· Labor Market: HoH labor force participation rate, HoH unemployment rate, HoH weekly hours worked; ï‚· Household Economics/Coping: Household income higher/lower, Difficulty meeting everyday living costs, Reduction in cost/quality of food accompanying staple The analytical aims were: ï‚· To classify provinces into groups reflecting the extent to which they reflected the adverse movements identified at national level. ï‚· To identify provinces that appeared to be at risk in the sense of having adverse levels of indicators relative to national averages. ï‚· To summarise the levels of and movements in indicators for which data were not recently available from other sources. CMRS Detailed Report Page 47 of 91 Aims of Analysis at District Level The primary focus of the analysis was the impact of the crisis, not the subsequent recovery or improvement. The ultimate aim was to identify districts that were in crisis as evidenced by significant negative changes occurring in several indicators. For each district and indicator a significant negative change was highlighted by setting the corresponding risk flag to red or orange. In this context, as previously noted a negative change referred to a change in an adverse direction. It did not necessarily mean a numerical decrease in an indicator. It could be associated with an increase, depending upon the particular indicator. 10.2 CMRSS Data The data produced by the CMRSS on which the analyses were based were: ï‚· national estimates of level for each indicator, by urban/rural, poor/non-poor, and male/female HoH breakdowns, together with variance estimates; ï‚· national estimates of period to period change for each indicator, by urban/rural, poor/non-poor, and male/female HoH breakdowns, together with variance estimates, z-scores and p-values; ï‚· provincial level estimates of level for each indicator together with variance estimates; ï‚· provincial level estimates of period to period change for each indicator together with variance estimates, z-scores and p-values; ï‚· district level estimates of level for each indicator together with variance estimates; ï‚· district level estimates of period to period change for each indicator together with variance estimates, z-scores and p-values; ï‚· red or orange risk flag for each indicator based on the value of the period to period change in the indicator in an adverse direction; ï‚· orange or red in crisis flags for each district based on the number of indicators (if any) with adverse period to period changes. Because of the small sample sizes and resulting large standard errors, the district level estimates were not considered to be of any value to clients. Instead the focus was on measures of change at district level as reflected in the risk flags. In essence, the value of the risk flag (red, orange, or not set) for each indicator for each district was determined using: ï‚· a one sided hypothesis test (with some particular level of certainty), the null hypothesis being no significant change in an adverse direction; ï‚· supplemented by the requirement that the (normalised) change exceeded some particular threshold. The choices of the particular level of certainty and threshold value were determined empirically for each indicator. CMRS Detailed Report Page 48 of 91 10.3 Grouping of Indicators National and provincial indicators were analysed in four groups – labour market, household economics, coping, and outcomes as shown in Figure 10.1. The figure illustrates how the GEC might have impacted on labour market conditions and household economics, how individuals and households might have coped with adversity, and what possible impacts could have been on outcomes. These groupings were not carried through to the district analysis because there were not enough indicators showing significant adverse movements to warrant a subdivision. Figure 10.1 Grouping of Indicators in Context of Crisis Monitoring 10.4 Analysis Procedures for National and Provincial Data National Data The data analysed were: ï‚· national estimates of level for each indicator, by urban/rural, poor/non-poor, and male/female HoH breakdowns, together with variance estimates; ï‚· national estimates of period to period change for each indicator, by urban/rural, poor/non-poor, and male/female HoH breakdowns, together with variance estimates, z-scores and p-values. All indicators were analysed, focusing on those that showed significant adverse period to period changes. CMRS Detailed Report Page 49 of 91 Provincial Data The data analysed were: ï‚· provincial level estimates of level for each indicator together with variance estimates; ï‚· provincial level estimates of period to period change for each indicator together with variance estimates, z-scores and p-values. In view of the large number of indicator-province combinations, analysis was focused on those indicators that showed significant period to period changes at national level. The selected indicators were grouped in two broad categories (dimensions) reflecting the overall extent of the movements relating to labour market impact and household hardship. The labour market dimension included: ï‚· HoH working hours; ï‚· HoH unemployment; and ï‚· HoH labour force participation. The household hardship dimension included: ï‚· household income; ï‚· difficulty meeting consumption costs; ï‚· substitution of lower quality or cost lauk-pauk. For each consecutive pair of quarters, for each dimension, each province was classified as: ï‚· reflecting the national average (average); ï‚· displaying a more adverse effect than the national average (bad); or ï‚· displaying a less adverse effect than the national average (good). The provinces were divided into nine cells according to whether labour market changes were generally good, average or bad and whether the household economics/coping situation was generally good, average or bad. For the purposes of summarising differences in provincial patterns for the period July- October in a reasonably succinct way, the provinces in the 9 cells were then further clustered into 4 groups: ï‚· Group 1: Labour Market good; Household Economics/ Coping good or average; ï‚· Group 2: Labour Market average; ï‚· Group 3: Labour Market bad; Household Economics/ Coping good or average; ï‚· Group 4: Labour Market bad; Household Economics/ Coping bad. Thus, as illustrated in Figure 10.2, the output of the provincial analysis was four groups of provinces roughly summarising the extent (if at all) to which they had been adversely affected over the July-October period. CMRS Detailed Report Page 50 of 91 Figure 10.2 Groupings of Province Based on July-October Changes 10.5 Analysis Procedures for District Data Indicators Used as Basis for Setting Risk Flags Given the very large number of district-indicator combinations, attention was focused on those indicators that: ï‚· that gave rise to interesting period to period results at national and provincial levels; and/or ï‚· that were believed to be reliable; and/or ï‚· that referred to all or most households in each district. The indicators satisfying these criteria were the ones on the basis of which the individual district-indicator risk flags were set and from which district in crisis flags were computed. They are listed in Figure 10.3, together with their type and direction of adverse change. Figure 10.3 Indicators For Which District Risk Flags Computed Type Short Adverse (Negative) Indicator [derivation] Name Direction (unit) Labour Market Proportion HoH unemployed in Increase (positive value of unemptrue [derived from reference month Q-Q difference) binary] HoH has formal Proportion Decrease (negative value of employment in formal [derived from Q-Q difference) reference month binary] Hours worked Quantitative Decrease (negative value of hrs during last week (hours) Q-Q difference) CMRS Detailed Report Page 51 of 91 Figure 10.3 Indicators For Which District Risk Flags Computed Type Short Adverse (Negative) Indicator [derivation] Name Direction (unit) Household Economics Household income Categorical on Larger Q value (larger in reference month scale of 1-6 hhinc value implies smaller compared with a [treated as income)* quarter ago quantitative] Difficulty meeting Categorical on Larger Q value level (larger education costs scale of 1-6 afford_educ value implies more compared with a [treated as difficulty)* year ago quantitative] Difficulty meeting Proportion everyday cost of Increase (positive value of consdiff [derived from living in reference Q-Q difference) binary] month Coping Strategies/Outcomes Volume of rice Quantitative Increase (positive value of consumed in ricevol (kilograms) Q-Q difference) previous week Value of rice Quantitative Increase (positive value of consumed in riceval (Rupiah) Q-Q difference) previous week Proportion Substitution of staple Increase (positive value of staple [derived from in reference month Q-Q difference) binary] Substitution of side Proportion Increase (positive value of (lauk-pauk) in side [derived from Q-Q difference) reference month binary] Tranportation decrease Quantitative expenditure in constrans (negative value of (Rupiah) reference month difference estimate) Adult females Proportion Increase (positive value of working or looking fwrk [derived from Q-Q difference) for work last week binary] Children (under 15 Proportion Increase (positive value of years old) working chwrk [derived from Q-Q difference) or looking for work binary] *Element of change built into indicator Other indicators for which data were available but that were not used to compute risk flags are listed in the following embedded document, together with the reasons why were not were not used. Indicators for which district risk flags computed/ not computed with reasons Indicators Analysed CMRS Detailed Report Page 52 of 91 Setting Individual Indicator Risk Flags For each district, for each indicator for each period to period change, an individual risk flag was defined. ï‚· It was set to red if the estimate of change of the indicator exceeded some specified threshold(s). ï‚· It was set to orange if it exceeded some specified lesser threshold(s). ï‚· Otherwise it was not set (represented by blank in district analysis spreadsheet). The rule for setting an individual indicator risk flag was referred to as a risk identification rule. There were many possible rules. A rule could be expressed in terms of the estimate of change directly and/or its p-value or z-score. In essence every rule involved a ranking of districts: ï‚· by the size of the estimate of change for the indicator; ï‚· or by its z-score; or ï‚· by it p-value: ï‚· and designation of those districts with the most extreme negative (meaning adverse) values as red or orange, according to the value. There was no theoretically optimum method for determining the most appropriate rule for setting risk flags. Figure 10.4 indicates the types of rules that were actually used. Figure 10.4 Individual Risk Identification Rules Type of risk identification Indicator Type Negative direction rule* upper p-value of level estimate hhinc categorical larger value < specified p-value (treated as afford_educ quantitative) (of level estimate) under assumption true value = indicator specific threshold ricevol lower p-value of difference decrease riceval estimate < specified p-value quantitative (negative value of constrans and difference estimate < difference estimate) indicator specific threshold hrs staple lower p-value of difference proportion increase side estimate >1-specified p-value (derived from (positive value of consdiff and difference estimate > binary) difference estimate) indicator specific threshold umemptrue lower p-value of difference proportion decrease estimate < specified p-value formal (derived from (negative value of and difference estimate < binary) difference estimate) indicator specific threshold chwrk proportion increase difference estimate >= CMRS Detailed Report Page 53 of 91 Figure 10.4 Individual Risk Identification Rules Type of risk identification Indicator Type Negative direction rule* fwrk (derived from (positive value of indicator specific threshold binary) difference estimate) * specified p-value and indicator specific threshold were parameters that determined the actual rule in any particular case. Different settings of the parameters (p-value and indicator specific thresholds) resulted in different rules. Three particular different sets of rules were used: ï‚· p-value = 0.05 and a conservative set (“Set 1â€?) of threshold settings; ï‚· p-value = 0.05 and a more liberal set (“Set 2â€?) of threshold settings; ï‚· p-value = 0.07 and the conservative set (“Set 1â€?) of threshold settings. Defining the In-Crisis Composition Rule The in crisis flag was applied to a district as a whole. It is a composite flag whose value (red, orange, or not set) was determined by a crisis composition rule applied to the values of the individual indicator risk flags for the district. Thus district in crisis identification depended upon the combination of the crisis composition rule and individual risk identification rules. The simplest type of crisis composition rule was of the form: ï‚· set district in crisis flag to red if number of red (or red + orange) risk flags for district exceeds a specified threshold; ï‚· set district in crisis flag to orange if it has not been set to red and if the number of red (or red + orange) risk flags for district exceeds a smaller specified threshold ï‚· otherwise do not set the district in crisis flag. More sophisticated variants of this rule were derived by supplementing the rule with one or more further conditions of the form: ï‚· risk flag(s) must be set for a specified indicator, or composite group of indicators, in order for the district in crisis indicator to be set; ï‚· individual risk flag settings were weighted according to the particular indicators; for example, a rule could involve assigning the working hours risk flag double the significance of the difficulty meeting consumption needs risk flag; ï‚· for an in crisis indicator to be set to red (or orange) the district had to belong to a specified group of provinces as defined in the provincial data analysis. There was no way of deducing a best or optimum rule, thus, over the three rounds of data, a number of different rules were tried. Furthermore, although it would have been ideal to use a single rule for all pairs of reference periods, there was no reason to suppose that the most appropriate rule would be the same in all cases. For the Round 1 various alternative rules were developed that depended upon assigning different weights to risk flags for different indicators. CMRS Detailed Report Page 54 of 91 For Round 2, given the relatively small numbers of districts and indicators for which the indicator risk flags were set to red or orange, only simple crisis identification rules with the fourth variant were used. Based on the provincial analysis of July-October data, districts were divided into the four groups listed in Section 10.4 (Figure 10.2). ï‚· Group 1: Labour Market good; Household Economics/Coping good or average ï‚· Group 2: Labour Market average ï‚· Group 3: Labour Market bad; Household Economics/Coping good or average ï‚· Group 4: Labour Market bad; Household Economics/Coping bad. Taking into account these provincial groupings, three alternative crisis composition rules were developed. 1. District in crisis flag was set to red if number of red + orange risk flags for district exceeded a specified threshold; 2. District in crisis flag was set to red if number of red + orange risk flags for district exceeded a specified threshold, and if district belonged to a province in Group 4. 3. District crisis flag was set to red if number of red + orange risk flags for district exceeded a specified threshold and if district did not belong to a province in Group 1. Based on the three variants of the individual risk flag identification rules (defined above in terms of p-value setting and indicator threshold set) and these three variants of the district crisis indicator rules, nine different sets of district in crisis flags were generated according to rules shown in the following table. Figure 10.5 Rules for Setting District In-crisis Flags Individual indicator risk identification District crisis Rule rule composition rule id P-value Indicator threshold Inclusion of districts setting settings by provincial group 1.1.1 0.05 Set I (conservative) Any Group 1.1.2 0.05 Set I (conservative) Group 4 only 1.1.3 0.05 Set I (conservative) Not Group 1 1.2.1 0.05 Set II (liberal) Any Group 1.2.2 0.05 Set II (liberal) Group 4 only 1.2.3 0.05 Set II (liberal) Not Group 1 2.1.1 0.07 Set I (conservative) Any Group 2.1.2 0.07 Set I (conservative) Group 4 only 2.1.3 0.07 Set I (conservative) Not Group 1 The results of applying each of these rules were compared to see how many districts (if any) were red or orange under most/all of the rules. CMRS Detailed Report Page 55 of 91 10.6 Distinguishing Crisis Effects from Other Effects Given the data were obtained by observation and not from a controlled experiment, care was taken in attributing causality. There was often insufficient information to distinguish between two or more possible causes. In particular, there was often no way of knowing whether changes were due to crisis after effects, seasonal effects, or other trend-cycle effects. Also, as prices steadily increase, measurements of income and expenditures were subject to overall quarter to quarter increases due to prices alone. Types of Effects A typical time series model for monthly or quarterly indicators allows for the following systematic effects: ï‚· Trend: long term movement in one direction, usually growth; ï‚· Business cycle: often combined with trend into trend-cycle; ï‚· Seasonality: cyclical movement following seasonal pattern; ï‚· Trading day: effect of weekends, national holidays, numbers of trading days per month. These systematic effects are accompanied by a random effect, being the net result of many different little effects, including measurement errors, that average out to zero. On top of these, the CMRS introduced two additional effects: ï‚· In-Crisis: being defined as the effect of a shock or more slowly deteriorating situation, resulting in significant change, usually negative, in economic situation; ï‚· Crisis Recovery: restoration to more normal situation following a crisis; can be short or long term. In concept, based on these definitions, the effects of a crisis can be distinguished from other effects by being too abrupt to be trend or business cycle, by not being connected with seasonal patterns or trading days, and by being too large and uni-directional to be considered random. In practice identifying the effects of a crisis (in particular the GEC) using the CMRS was not easy as three rounds of data were nowhere near sufficient to determine trend cycle, seasonal or trading day effects for any indicator. Identifying crisis recovery was even more difficult as recovery may be short and/or long term and may blend in imperceptibly with trend, business cycle or seasonal effects The commonsense approach adopted by the CMRS in attempting to identify crisis related impacts was as follows. ï‚· Identification of general prevailing trend-cycle and seasonal effects as well as possible using other information, for example Sakernas and Susenas data ï‚· Taking these general systematic effects into account in the analysis of CMRS indicators, for example, expecting a growth in household income of, say, 3% per annum, expecting a decline in unemployment of, say, 5%, in, say, the wet season. ï‚· Attempting to distinguish crisis effects from random effects by designating as red or orange only those indicator values that seemed unlikely to have occurred by chance, making use of relevant frequency distribution tables. CMRS Detailed Report Page 56 of 91 10.7 Data Quality Considerations Data also had to be interpreted cautiously in the light of the possible sources of error, especially as the CMRSS was a new survey with very limited time for testing the questionnaire and no time or resources for training the interviewers. Evidently there were measurement errors because some questions were not precisely worded, the interviewer failed to ask the right questions, the respondents did not understand the questions, or did not know the answers, or did not wish to provide the answers, or because there was no true “rightâ€? answer in the case of an opinion. Furthermore, to obtain quarterly changes for the first round, respondents were asked to provide data for two quarters, April and July. The April data, depending on respondent recall over three month period, were obviously more subject to error. Processing errors may have occurred during data recording or capture. Where estimates of change were the primary target, as in the CMRSS, and these changes were based on data from a relatively small number of households, processing errors may have had disproportionally large effects. Sampling errors occurred because data were collected from a sample rather than from the whole population. To minimise costs and respondent burden, the sample was restricted to 30 households per district and was piggy-backed on Sakernas. Because the sample was stretched over all districts equally, and used a two stage cluster design in order to be compatible with Sakernas, the standard errors associated with the estimates were quite large. 11 Collection and Analysis of Health Data 11.1 Data Sources and Collection For each of the three quarterly CMRS rounds, data were collected from the district health centre (Dinas Kesehatan and a sample of five community health centres (Puskesmas) in each of the 471 districts. The sample of community health centres was not random. It comprised the centres that were most easy to reach by BPS field office staff given their CMRSS data collection duties. BPS field office staff usually dropped off questionnaires at the health office and community health centres and picked up the completed forms a day or so later. Each questionnaire asked for data for each of the preceding three months. The data items including supply side data such as numbers of doctors, paramedics, equipment, as well as usage and health conditions. In total, the questionnaires generated 76 indicators. 11.2 Data Capture and Preparation BPS Head Office staff captured the data. Data for the district health centres and for the community health offices were captured separately 11.3 Data Preparation The BPS members of the CAT were responsible for preparation and analysis of the data. Data for the district health centres and for the community health offices were analysed separately, but the procedures were the same in each case Preparation involved: CMRS Detailed Report Page 57 of 91 ï‚· data consistency checking; ï‚· missing value imputation; ï‚· identifying clean records to be used in the analysis. On examination of health centre data for the first two rounds it was soon evident that there were considerable data inconsistencies between the two quarters. For many indicators, the average across the three months covered by the second round was more than half as large again, or less than half as large, as the average for the three months covered by the first round. This applied even to indicators such as number of doctors, which could reasonably be expected to be very stable over six months. On the other hand, there was relatively little variation within each of the three month periods. It was conjectured that the main reason for the inconsistencies in the data were that the numbers of reports received from the community health centres that were included in the district report differed from quarter to quarter. Under this hypothesis data from individual community health centres should have been much more consistent from quarter to quarter. It was found that they were not. The conclusion was that both the district health centre data and community health centre data were of exceptionally poor quality. 11.4 Data Analysis In view of the poor quality of the data, the analysis was limited. Five indicators were analysed in some detail: ï‚· Number of doctors in kabupaten; ï‚· Number of puskesmas with doctor; ï‚· Number of kader ; ï‚· Number of active posyandu; ï‚· Number of under-fives attending posyandu. At national level, the results looked plausible, but at district level the data were obviously inconsistent from quarter to quarter and thus of no value whatever for crisis monitoring. The data of most interest were those relating to in health conditions. They were too unreliable to be analysed even at national level. More details are contained in the BPS presentation embedded below. CAT (BPS) DINAS Data Analysis - Presentation CAT (BPS) DINAS Presentation In summary, the health data did not yield any useful results from the perspective of crisis monitoring. CMRS Detailed Report Page 58 of 91 12 Dissemination and Use of Results 12.1 Transfer of Outputs to Clients The mechanisms for the transfer of national and provincial estimates and district risk flags from the WB Team to Bappenas were never formalised. 12.2 Response Identification During the period of CMRS operation there were three broad categories of possible policy responses to handle districts that appeared to be in crisis: ï‚· application of elements of the program for support of households and individuals – examples are scholarships, free health care, cash transfers; ï‚· application of the program for community improvements; ï‚· application of the program for support of micro-businesses. However, as the impact of the GEC took place before the CMRSS was introduced (as elaborated in the following chapters) very few districts were identified as being in crisis and the range of possible responses and prioritisation of these responses was never fully developed or tested. 12.3 Dissemination As of the time this report is being written, CMRS output data have not yet been broadly disseminated. They will be disseminated using an information dashboard. In the first instance at least, the dashboard will be based on a software platform known as DesInventar and will be jointly owned and administered by Bappenas, the UNDP, and the World Bank. The system will provide: ï‚· a description of the impact of the GEC on vulnerable households and individuals in Indonesia; ï‚· identification who, where, how deep, and through what channels, the GEC manifested itself in Indonesia; ï‚· support for the formulation of appropriate policy responses in a targeted and effective manner. The system will enable: ï‚· dynamic access to baseline profiles comprising various types of socio-economic data for crisis vulnerability analysis; ï‚· use of information for policy-making in response to crisis vulnerability through a spatial/national vulnerability index; ï‚· access to multiple crisis vulnerability studies/reports. The initial prototype will be based on the conceptual framework, data and results from the completed CMRS. System development will involve: ï‚· identifying potential and committed partners; ï‚· identifying the technical requirement (sub-domains/servers) for deployment of prototype on the Internet; CMRS Detailed Report Page 59 of 91 ï‚· agreeing on the Internet deployment of the prototype; ï‚· establishing a schedule for review of the prototype, and development and Internet deployment of the final product; ï‚· developing guidelines and procedures for maintenance; ï‚· training users and system maintainers. More details are available in the presentation embedded below. Disinventar for Crisis Vulnerability Monitoring System, August 2010 Disinventar Dissemination System CMRS Detailed Report Page 60 of 91 PART B: RESULTS OF ANALYSIS 13 General Effects of Crisis: Context for Analysis CMRS data had to be interpreted within the context of the prevailing economic environment. The Indonesian economy had enjoyed a solid recovery from the global economic crisis (GEC) beginning late 2008. Robust domestic consumption helped the Indonesian macro-economy to weather the storm. ï‚· The effects of the GEC on the Indonesian economy began in late 2008. A slow recovery was underway by the second quarter of 2009. ï‚· Exports fell sharply in the fourth quarter of 2008 but recovered through 2009. ï‚· Growth in GDP also slowed in the fourth quarter of 2008 and into the first quarter of 2009. ï‚· Financial markets were also affected but recovered strongly. ï‚· Since July 2009, households have faced increasing food prices. BPS data show increases in a number of food staples over the second half of 2009. This put considerable pressure on household expenditures, particularly for the poor, for whom food represents nearly three quarters of their consumption. ï‚· The labor market was expanding through to near the end of 2008. Much of the growth was in casual and unpaid work. The trend of gradual recovery is expected to continue to the end of 2011. In summary, it is certain that the initial impact of the crisis took place well before April 2009, the first quarter for which data have been collected by the CMRSS. Thus the CMRSS results could not be expected to show (and did not show) the initial impact of the financial crisis. However, they could be expected to show some after effects. These could be continuation, perhaps even worsening, of difficult times, or signs of recovery. CMRS Detailed Report Page 61 of 91 14 Impacts of Crisis at National Level 14.1 Labour Impacts Labour Force Changes Fig. 14.1: Proportion of households with economically active head of household The CMRSS data show a small decrease in the proportion of economically active heads of household (HoHs) over May-August 2009, an increase over the August-November 2009 period, but a decrease again over November 2009 - February 2010. Over the period July 2009 - January 2010, the net change is not significant. Unemployment Fig. 14.2: Head of household unemployment The Sakernas data in Figure 14.2 indicate that unemployment of HoHs was highest in the first half of 2008, and was more pronounced in urban centres than in rural areas. The CMRS Detailed Report Page 62 of 91 GEC did apparently not lead to an increase in HoH unemployment; in fact, unemployment largely fell over the crisis period. The CMRSS data show an increase in unemployment of HoHs between November 2009 and February 2010. This may be a seasonal effect. Changes in Working Hours Fig. 14.3: Head of household weekly working hours Historical Sakernas data show that weekly working hours are generally higher in August than in February. Figure 14.3 illustrates the situation in 2008 and 2009, also indicating that the 2009 difference is smaller than for 2008. The graph with the CMRSS data shows that there were significant movements between the August 2009, November 2009 and February 2010 quarters for the HoH weekly working hours. For the quarter May-August 2009, the figures were down but there are no seasonal benchmarks for this. The quarter August-November saw an increase in working hours, but without November historical data, it cannot be determined how much of this might be seasonal. However, since hours in August are usually higher than in February, it is unlikely that all of this increase is seasonal, and this increase may well be a partial recovery from the decline in the previous quarter. November 2009 to February 2010 saw another decline, but one which is probably in line with a seasonal decline, given that February hours are generally lower than August. Thus the November-February quarter indicates that the partial recovery of the previous quarter has been sustained, or that at least no further deterioration is evident. Nonetheless, February 2010 hours had yet to recover to previous February levels. Changes in Wages/Income in the Formal Sector As illustrated in Figure 14.4, HoH wages in the formal sector have remained relatively stable. The February 2009 data, at the height of the crisis, actually show an increase in formal wages rather than a stagnation or decrease, especially for the urban areas. The CMRSS data for the period May 2009 to February 2010 also show a stable to increasing trend, except for households with a female HoH where wages showed a negative trend over the quarters August-November 2009 and November 2009 - January 2010. CMRS Detailed Report Page 63 of 91 Fig. 14.4: Head of household wages in the formal sector Formal/Informal Sector Changes Fig. 14.5: Head of household working in the formal sector The data on HoHs working in the formal sector show a slight but significant decline in proportion of heads of households working in the formal sector. This seems to correspond to the longer term trend, where formal sector employment generation does not keep pace with work force growth, and new employment generation occurs in the informal sector. CMRS Detailed Report Page 64 of 91 14.2 Household Economics Reduced Household Income Fig. 14.6: Quarterly household income comparison, Jul-Oct 2009 and Oct 2009 - Jan 2010 Households were asked to compare their current household income with that of a quarter earlier on a six point scale ranging from “much higherâ€? to “much lowerâ€?. The first survey round data, providing the April-July 20091 comparison, are skewed to the “much lowerâ€? end of the scale, indicating that HoHs perceived a loss of household income, especially in rural areas and for poor households.. This viewpoint is consistent with the reduction in working hours that was reported over that period. For the second and third rounds of the survey, Figure 14.6 indicates that the rural and the poor reported a worsening of their household incomes, and that percentage actually increased over the period October 2009 to January 2010, compared to the previous quarter. Difficulty in Meeting Consumption Needs Households were asked whether they had difficulty meeting consumption needs (“yesâ€? or “noâ€?). As indicated in Figure 14.7, the number of households that reported difficulty affording consumption increased from April to July 2009. This is consistent with an increase in food prices and fall in working hours over the same period. The increase disappeared over the July to October period as conditions improved. All categories of households experienced the April to July increase, with the increase being larger for the poor than non-poor. However, only the part of the increase that relates to decreased working hours is likely to have been crisis-related. With food being around two-thirds of the poverty basket, non-crisis-related inflation in food prices will also have been a significant cause. 1 As mentioned before, there are differences in timing for different indicators: labour market indicators are for the first week of May/August/November/February, while most other indicators are for the month of April/July/ October/January. CMRS Detailed Report Page 65 of 91 Fig. 14.7: Households having difficulty meeting consumption needs Costs for Transport and Food Fig. 14.8: Transportation costs, rice costs and rice volume Households were asked to report actual transportation costs, and volume and cost of rice consumed for a one week period. Reported transportation costs remained largely the same between April and July 2009, but increased over the quarter July to October 2009 especially in rural areas, possibly corresponding to the Idul Fitri holiday period, followed by a slight decrease over the period October 2009 - January 2010. Rice volumes remained essentially the same, but the cost of rice increased along with the increase in the price of rice. CMRS Detailed Report Page 66 of 91 Affordability of Education Fig. 14.9: Affordability of education Figure 14.9 illustrates that as regards affordability of education, 53% of the respondents nationally reported that it was much easier, easier or somewhat easier in October 2009 than in July 2009. This could be linked to July being the start of the new school year. In January 2010, nationally 52% reported that meeting education costs was somewhat harder, harder or much harder than in October 2009. What is somewhat surprising is that the perception of the poor differs from the non-poor; 54% of them found it harder in October 2009 than in July 2009, compared to 47% nationally, and 44% of them found it harder in January 2010 than in October 2009, compared to 52% nationally. 14.3 Coping Strategies Coping by Seeking Employment During the first round of the survey, the proportion of households with child workers was just under 3 percent, being more than twice as high in rural areas than in urban ones. The number of child workers per household was 1.3 for these households, indicating that 70 percent or more of them had just one child working. However, in 2009 the proportion of households with one or more females in the labour force (excluding the head of household) was around 36 percent, with little difference between urban and rural, poor and non-poor. In January 2010, there was a slight increase in the number. In summary there is no evidence of increasing attempts to look for employment either by child worker or female entry into the labour force CMRS Detailed Report Page 67 of 91 Fig. 3.10: Child and female labour force participation It is worth noting that interpretation of the impact of an increase in female participation in the labour force is not entirely straightforward. In the long term, an increase in female labour force participation may well be a desired outcome. However, short-term, unplanned entrances, due, for example, to unexpectedly low household income, may be indicative of adverse effects, for example if accompanied by a reduction in the planned caring for children. Coping by Reducing Consumption Costs The following figure and observations refer only to households who expressed difficulty in meeting consumption needs, referred to as “struggling householdsâ€?. Fig. 14.11: Food substitution by households having difficulty in meeting consumption needs CMRS Detailed Report Page 68 of 91 As illustrated in Figure 14.11, in May 2009, just under six percent of these households were substituting their staple food (generally rice) for one of lower quality or cost, and this was twice as common in rural as in urban households. The situation remained essentially unchanged in the following three quarters. This is quite consistent with unchanged rice volume (Fig. 14.8). On the other hand, the proportion of households substituting their lauk-pauk (main food accompanying rice, generally a protein such as meat or fish) to one of lower quality or cost increased from 14 percent in May 2009 to 16 percent in August 2009. This substitution corresponds to falling working hours, increased food prices, and higher difficulty affording daily consumption needs over the same period. As for staple, lauk- pauk substitution was more common in rural than in urban households, with poor households seeing a 3 percentage point increase. The increase was reversed in the following quarter with lauk-pauk substitution reverting to May level and this was followed by a slight increase in January 2010. In summary, the data over the three survey rounds provide evidence that households used reduction in food expenditures by substitution of their lauk-pauk for one of lower cost or quality as a coping strategy. Coping by Financing Proportion of Respondents 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Relig./Social Money-lender Relatives Relatives Assets Other Financial Inst. Friends Friends Savings Borrowing Receiving Money or Goods National Rural Urban Non-poor Poor Fig. 14.12: Usage of Financing Mechanisms by Households Expressing Difficulty Meeting Consumption Needs, July 2009 The data on financing consumption needs have three limitations. First, they are self- reported. Second, for the first round survey they refer only to the subset of households who stated they had difficulty in meeting consumption needs. (In the second and third round surveys, the question was asked of all respondents). Third, they refer to incidence of the use of the various financing mechanisms, not the value of the corresponding CMRS Detailed Report Page 69 of 91 transactions. Thus the principal coping mechanisms by value cannot be determined from the data2. The following observations refer only to struggling households, i.e., those who expressed difficulty in meeting consumption needs. In July 2009, around 20 percent of struggling households used savings to meet living expenses (the figure being significantly lower for the poor, presumably because fewer of them have savings). A similar proportion sold assets for the same reason. Urban and rural households who were struggling in both April and July made significantly less use of savings and sold significantly more property and assets in July than April. One explanation is that their savings were becoming exhausted. Around 60 percent and 50 percent, respectively, of households borrowed from relatives/family and from friends/neighbours. The proportion of rural households struggling in April and July who borrowed from relatives and family increased significantly. In July, borrowing from non-institutional lenders was at about 8 percent nationally, being twice as common in urban households than rural ones. There was little increase in this form of borrowing from April to July 2009. Borrowing from financial institutions was also around 8 percent nationally in July, and twice as common in struggling urban households than rural ones, with a significant increase in rural households over the quarter. The proportion of struggling households receiving assistance by way of money or goods from relatives was around 45 percent and increased significantly over the period April to July in both rural and urban households, whereas the proportion of households receiving assistance from friends or neighbours, running at about 25 percent, decreased significantly. The proportion of households receiving assistance from religious or social or other organisations was at around 12 percent, and did not change significantly in either urban or rural areas from April to July 2009. In summary, in mid-2009 household borrowing by households experiencing difficulty went up, whereas use of other financial mechanisms remained much the same. Coping by Migration In the first survey round, around five percent of households indicated some outward migration over the quarter, more from rural households than urban ones. Around one percent indicated inward migration, roughly the same in rural households as urban ones. Thus, there is no evidence of increasing attempts to look for employment by migration. 2 This applies to most indicators other than the labour market ones for which data are available from Sakernas. CMRS Detailed Report Page 70 of 91 15 Impact of Crisis at Provincial and District Levels 15.1 Impact at Provincial Level Not surprisingly, there were substantial variations in the values the 60 or so CMRSS indicators across the 33 provinces. The challenge was to analyse the data without becoming overwhelmed by the number of data points. As noted in Section 10.4, for each round, analysis was focused on those indicators that showed significant quarterly changes at the national level and that were considered reliable. The availability of benchmark data (for labour force indicators) was also a consideration. The selected indicators were grouped in two broad categories (dimensions) reflecting the general extent of the movements relating to labour market impact and household hardship. The labour market dimension included HoH working hours, HoH unemployment, and HoH labour force participation. The household hardship dimension included household income, difficulty meeting consumption costs and substitution of lower quality or cost lauk-pauk. For each consecutive pair of quarters, for each dimension, each province was classified as average (reflecting the national average), good (displaying a less adverse effect than the national average), or bad (displaying a more adverse effect than the national average). Based on these two dimensions each province was allocated to a cell in a 3x3 matrix. For the purposes of summarising differences in provincial patterns for the period July- October in a reasonably succinct way, the provinces in the 9 cells were then further clustered into 4 groups. Thus, as illustrated in Figure 10.2 (for ease of reference copied below as Figure 15.1) the output of the provincial analysis was four groups of provinces roughly summarising the extent (if at all) to which they had been adversely affected over the July-October period. Figure 15.1: Provincial Groupings (Round 2) CMRS Detailed Report Page 71 of 91 The actual indicators selected, the way in which they were combined to reflect labour and hardship dimensions, and the final groupings, differed slightly for each pair of quarters. 15.2 Impact of Crisis at District Level Following the first round of data, nine districts were identified as being in crisis (red status) namely: ï‚· Bener Meriah (1117) ï‚· Tapanuli Utara (1205) ï‚· Payakumbuh (1376) ï‚· Oku Selatan(1608) ï‚· Batang (3325) ï‚· Timor Tengah Utara (5305) ï‚· Barito Utara (6205) ï‚· Tapin (6305) ï‚· Asmat (9415) 25 additional district were assigned orange in-crisis status Given the absence of evidence of crisis impacts in subsequent months no further districts were considered in crisis. CMRS Detailed Report Page 72 of 91 PART C: EVALUATION AND INSTITUTIONALIZATION 16 CMRS Evaluation 16.1 Evaluation of Processes Questionnaire There were several of ways in which the questionnaire could have been improved. ï‚· Data from relevant parts of the Sakernas questionnaire were transcribed from the Sakernas to the CMRS questionnaire by the interviewer. This approach carried a risk of transcription errors. A more reliable approach would have been to photocopied the relevant parts. ï‚· Response categories for questions asking for current value by comparison with previous value were typically of the form much higher, higher, somewhat higher, somewhat lower, lower, much lower. It would have been preferable to have included a neutral response category, for example much higher, higher, about the same, lower, much lower. ï‚· The question sequencing was somewhat erratic. Pilot Test Conducting a small scale pilot test in two districts - preferably in an urban district and a rural district - would have highlighted some problems before production. The aim would have been to check respondents’ understanding of the questions and readiness to respond, and time taken to obtain all the data required. The usual three person BPS interviewer team would have been replaced by a BPS interviewer, a WB representative, and a Bappenas representative. Data Collection Capture and Processing There was not time to create comprehensive terms of reference (TOR) for data collection and capture contractor, i.e., BPS. Some omissions in the TOR included: ï‚· requirement for discussions of the questionnaire by focus groups comprised of experienced interviewers, held in at least one predominantly urban and one predominantly rural district; ï‚· pilot testing of procedures; ï‚· training of interviewers; ï‚· quality assurance of data collection procedures; ï‚· quality control of data capture; ï‚· specification of target response rates. Editing and Imputation As time and resources were not sufficient for comprehensive micro-level editing, it would have been good to have macro-edited the data. This would have meant identifying anomalous aggregate values and for each one investigating the micro-level CMRS Detailed Report Page 73 of 91 data contributing to the aggregate to look for obvious errors. In particular, for each district designated as in crisis, values for indicators that indicated high levels of risk should have been checked at micro-level. Analysis Identifying Districts Doing Well Given that the national results in Rounds 2 and 3 indicated no negative widespread effects of the GEC, in fact rather the reverse, it would have been appropriate to look, at least briefly, at districts at the other end of the spectrum, i.e., districts doing well in terms of several indicators. Similar tests could have been used to identify good performance (“greenâ€?) flag settings as were used for risk (“redâ€?) flag settings for each indicator, the difference being that the districts thus identified had large positive changes rather than large negative (adverse) ones. 16.2 Quality Considerations The various sources of error are described in the following paragraphs. Measurement Errors – Respondents Errors may have occurred because the respondent did not understand the question, did not know the answer, or did not wish to provide the correct answer. Furthermore, in the case of questions soliciting an opinion, there is no “correctâ€? answer. A respondent may give a different answer if asked the same question on another day or in other circumstances. In addition, the April data are based on respondent recall from August and cannot be regarded as accurate as if they had been obtained in April. Measurement Errors - Non-response A certain effect of non-response on the estimates is to increase their variance. A possible effect is to introduce bias, which occurs if and only if the non-respondents are significantly different from the respondents. The extent of the bias depends upon difference and the non-response rate. As noted above there were 15 districts in Papua for which no data are available for the first round. Elsewhere the unit response rates by district were very good, thus, except in districts with low responses rates, it may be assumed that the response bias is negligible. Measurement Errors - Interviewers Errors may occur because the interviewer fails to ask the right question or to record the answer received. In the case of questions involving monetary values in Rupiah a particularly common problem is that interviewers enter data in units instead of thousands according to the instructions. Processing Errors Errors may occur during data capture. Typically they are random and can be expected to balance out to a large extent. However, as estimates of change are based on changes in a relatively small number of households it is important to ensure that these data are checked for possible processing errors that could have disproportionally large effects on the estimates. CMRS Detailed Report Page 74 of 91 Dealing with Measurement and Processing Errors In cleaning the files received from the BPS obvious errors have been removed. Subsequently the data have been macro-edited, meaning that some errors have been identified and removed by looking for anomalies in the aggregates. This is not sufficient to eliminate all errors. These will be further investigated on a case by case basis for households in districts that are identified as at risk. A full assessment of the extent of interviewer errors would require a program of re- interview, which is not feasible. Likewise data capture errors could be fully assessed only by recapture, which again is not feasible. Sampling Errors Sampling errors occur because data are collected from a sample rather than from the whole population. Conceptually they may be divided into two types. ï‚· Bias, which can occur because the estimation formulae systematically produce a biased estimate, for example if the data are used without weighting, or because there are problems with sampling weights due to poor or out of date population estimates. Bias remains constant whatever the sample. ï‚· Sampling variance, which reflects the variation that can occur in selecting the sample. Sampling variance is the only type of error that can be readily quantified, though with some difficulties as discussed in the Technical Report appended (Annex 1). Typically this sort of error is presented in the form of a standard error (square root of variance), or of α% confidence interval, meaning an interval includes the true value (1-α)% of the time. As noted in the previous subsection, because the sample is stretched over all districts equally using a two stage cluster design, standard errors are quite large. Model and Analysis Errors Approximations made in deriving standard error estimates or in measuring quarterly changes can lead to errors in confidence intervals and tests of significance. In particular this occurs in the case of quarterly changes. They typically have two components: 4. changes represented by households reporting both quarters; and 5. changes represented by households reporting for one quarter only. The standard error of the difference based on paired observations (i.e., from the same households) for the two quarters refers only to the first component and is an under estimate of the standard error associated with the total change. However, for simplicity of analysis it has been assumed to apply to the total change. Summary of Effects of Errors The data are subject to a wide range of errors. The most important are: ï‚· Respondent errors in questions of opinion – levels are unreliable, trends are more reliable; ï‚· Respondent errors in recalling data for April; ï‚· Sampling errors, especially for indicators based on small numbers of households. CMRS Detailed Report Page 75 of 91 16.3 Summary of Strengths and Weaknesses Strengths: The survey covered all districts in the whole country. Thus there was no need to guess where the impacts of a crisis were most felt. Some monitoring efforts in previous crises used ex-ante estimates of likely crisis-affected locations, but with hindsight these proved only partly accurate. Country-wide crisis monitoring is preferable. The CMRSS was attached to a well established survey (Sakernas), with some additional data collection. Thus the enumerators and supervisors were already familiar with the data collection and data entry procedures. This is likely to have resulted in better quality survey data than could have been expected from new interviewers. The CMRSS collected a relatively small number of indicators. The workload for data entry was therefore manageable by a small team. Weaknesses The small sample size of 30 households per district required the use of non- conventional statistical analysis methods. Because such methods were improvised, analysts required training. Interpretation of results required great care as the small sample sizes increased the likelihood that random changes (noise) could give wrong signals regarding the districts at risk. There was, in part due to the lack of baseline (pre-crisis) data for most indicators. This made it difficult to distinguish between crisis impacts, seasonal effects and adverse changes due to other, non-crisis related factors. In contrast to the situation with Sakernas, detailed labour-related data was collected only from the head of household (HoH). It was difficult to generalise HoH results to the broader labour force, or to interpret the level of hardship in certain households since data on employment and earnings of other members of the household were not available. Data from the Puskesmas and from the District Health Offices were intended to be the main source of information to assess health outcomes. However, they were not consistent and highly unreliable. CMRS Detailed Report Page 76 of 91 17 Future Crisis Monitoring and Response System 17.1 Introduction Not all the initial CMRS expectations were met. Results took longer to produce than expected; a fully definitive identification of at risk districts was not achieved due to the limited impact of the crisis during the measurement period; and only one response was based on the data collected. Nevertheless there is a general consensus among the key stakeholders that the approach showed promise and there is an interest in developing and implementing a crisis monitoring and response system. It would enable the Government to assess rapidly the impacts of future crises as they unfold and to take measures to alleviate their adverse effects. There are many other considerations in the development, operation and maintenance of crisis monitoring and response system. This chapter outlines the issues. It discusses the potential objectives and scope of the system, the users of system outputs, the required characteristics of the system, the relationship to other systems (for example systems for monitoring vulnerability and/or progress towards millennium development goals), the sources of input data, the development of the system, and its mode of operation. One of the concluding activities of the CMRS Project was the production of a comprehensive paper that deals with this very topic. It is entitled Preparing for the Next Crisis:Establishing a Vulnerability and Shock Monitoring and Response System in Indonesia (VSMRS). It proposes that a system be developed that monitors vulnerability on an ongoing basis and that can be ramped up to monitor the effects of a crisis when one occurs. The contents of this chapter are largely extracted from the paper. 17.2 Objectives, Clients and Concepts What is required? In designing an ongoing crisis monitoring and response system the key questions to be to be addressed are: ï‚· what are the scope and objectives of the system? ï‚· for what clients is the system intended? ï‚· to what uses will the clients put the system? The following paragraphs address these issues. Scope and Objectives The system is unlikely to be the first source of identification of the onset of a crisis. The crisis will be manifestly obvious from other sources. Rather the system will be a crisis confirmation tool that provides information on the spread and intensity of crisis impacts and that guides crisis alleviation measures to regions and sectors where they are most needed. In essence the objectives of the system are likely to be: 1. to produce relevant, reliable, timely, accessible, understandable and coherent data to enable monitoring of how a crisis is unfolding, how it is affecting Indonesian society, especially vulnerable groups, what the impacts of on affected households are, and the broader socio-economic outcomes; CMRS Detailed Report Page 77 of 91 2. to make such information broadly available to government agencies and other stakeholders to support decision-making on how to design and target policy responses to the crisis; 3. to put rapid and effective response mechanisms in place, to address crisis impacts; and 4. to monitor and evaluate the effectiveness of such mechanisms. Outputs should be targeted to the information needs of the various programs that are most likely to be the channels of additional support to regions and sectors in event of a crisis, for example the PNPM, health and education sector programs. Outputs should be provided in formats that closely match those used by the programs and involved ministries. Examples of broad categories of possible responses at district level are: ï‚· increased application of the program for support of households and individuals – examples are scholarships, free health care, cash transfers; ï‚· increased application of the program for community improvements; ï‚· increased application of the program for support of micro-businesses. Clients The main user of the information produced by the system will almost certainly be BAPENNAS. Other potential users are: ï‚· other ministries that have funds designated for crisis response and that seek information on locations where such support would be most effective; ï‚· provincial and district governments who need to plan crisis response activities funded by various organizations; ï‚· donor agencies and NGOs searching for information in support of their activities. Phases of a Crisis In order to design an effective system it is useful to consider the way in which a crisis typically evolves. A crisis can be modelled in terms of five phases, as illustrated in Figure 17.1. ï‚· Pre-crisis. A normal state where some households face crisis situations, but these are primarily due to individual or local circumstances, for example, illness, accident, landslide, etc. ï‚· Onset of crisis. Where natural or human-induced events trigger changes that result in severely deteriorating living conditions or increased hardship for a significant and growing number of households. ï‚· Peak of crisis. Worsening conditions and increases in hardship start to level off and some early signs of recovery are noticeable. ï‚· Recovery. Decreases in hardship and steady improvements for increasing numbers of households affected by the crisis. ï‚· Post-crisis. Return to the pre-crisis state where crisis situations are confined to individual or local circumstances. CMRS Detailed Report Page 78 of 91 Mild crisis impact No. of people affected by the crisis Moderate crisis impact Severe crisis impact 1 2 3 4 5 Pre-crisis Onset Peak Recovery Post-crisis Fig. 17.1: Phases of a crisis Impact of Crisis The impact of a crisis is unlikely to be the same for all households. Richer households are usually better able to cope with crises than poorer ones as they have more assets. These can be physical assets, for example, agricultural land, livestock, car, motorbike, TV, and jewelry, and/or labour related assets, in particular household members who are capable of working and have skills. When a crisis occurs, household income may decline and assets may be sold in order to cope with the impact. For a richer household the decline in assets may be a relatively smaller and short lived. For the poorer household, the impact may be more dramatic and sustained. However, the impact of any particular crisis is not necessarily more profound for a poor household than a richer one. It could be that the channels through which the worst impacts of a crisis are transmitted largely bypass a poor household. For example, a rural household which lives largely of what it produces on its own plot of land and from the surplus it sells may be little affected by a fall in exports whereas an urban household that was relatively well off before the crisis but for which the main source of income was via the employment of the head of household in an export-oriented industry may go through a difficult period. Relationship to vulnerability Vulnerability refers to the propensity of a household to sink below the poverty level or to enter a period of extreme hardship. It is a state in which a household’s current situation is fine, but the household is on the borderline of hardship and can easily be pushed there. Vulnerability is a structural condition. Typically it is measured in terms of indicators such as household income, household assets, and household employment. Vulnerability is applied to collections of households comprising regions (villages, districts, provinces, cities, etc). A vulnerable region is one that has low average levels of the relevant household vulnerability indicators. In general, a household or region that is vulnerable is more likely to be seriously affected by a crisis. However, as noted above, this is not always the case. CMRS Detailed Report Page 79 of 91 In summary, vulnerability refers to the state of a household or collection of households measured in terms of a number of relevant indicators, whereas a crisis is an event which impacts upon a household or set of households and is measured in terms of changes in these indicators. Typically the Government’s initial concern will be the impact of a crisis on vulnerable groups. 17.3 Previous Crisis Monitoring Experiences In designing a crisis monitoring a response system it is vital to take into account previous experiences. In addition to the recent GEC, Indonesia has experienced several large scale crises over the last 15 years that have affected parts, or the whole, of the country. ï‚· The Asian Financial Crisis started in 1997 and lasted until around 2000. ï‚· The tsunami of 26 December 2004 hit Aceh and North Sumatra causing the death of 132,000 people with a further 37,000 people missing. ï‚· In 2008, large increases food and fuel price increases had an impact throughout the country. ï‚· Other natural disasters, primarily earthquakes, had severe impacts on specific regions, for example Yogyakarta, and Padang. The monitoring and response systems that were developed as these crises were unfolding, or in their aftermath, provide some insights into the design of a monitoring system for the future. They should be thoroughly analysed. In this context, it would be good to establish an easily accessible database in which all crisis monitoring systems have well documented together with lessons learned - including evaluations of how the Government and other interested parties gained information about the crisis and responded to it, and what worked reasonably well and what did not. Some notes on monitoring and response systems that predate the CMRS follow. 100 Village Survey The 100 Village Survey (Survei Seratus Desa) was a collaborative effort between BPS and UNICEF that was first conducted in May 1994 and again in May 1997 prior to the Asian financial crisis. As the impact of the crisis spread, additional rounds of the survey were conducted in August 1998, December 1998, May 1999 and October 1999. The purpose of the survey was to monitor changes in health, nutrition, education and socio-economic status at the household level. The survey was conducted in 100 purposely selected villages with between 500-1000 households each, spread out over 10 districts in eight provinces. In each of the 100 villages, 120 households were selected, resulting in a total sample size of 12,000 households. To the extent possible the same households were re-interviewed in successive rounds. Of the 12,000 households interviewed in May 1997, 8,142 were interviewed in August 1998 and 6,201 in December 1998. Data was gathered on the demographic attributes of household members, education, health and fertility, migration, labor market activity, socio-economic status and crime. The surveys of 1998 and 1999 had greater focus on living standards and coping mechanisms. CMRS Detailed Report Page 80 of 91 The survey was not designed to be a nationally representative. It was concentrated on rural and relatively poor areas. Thus, the findings were indicative of what might have occurred in similar villages elsewhere, but could not be generalized to the country as a whole. Kecamatan Crisis Impact Survey 1998 The Kecamatan Crisis Impact Survey was designed as a quick response survey to obtain country wide, up-to-date information on the impact of the financial crisis in 1998. It was conducted by the BPS. In each of the (then) 4,025 kecamatan (sub-districts) three key respondents were asked a series of qualitative questions about the extent of various kinds of impacts (migration, access to health and education, food availability, etc.), about the frequency of different types of coping strategies (selling assets, reducing frequency and quality of meals, etc) and about the most severe impacts in each area. The questions were designed to measure changes relative to the same time a year earlier, thereby eliminating seasonal effects. The respondents were asked to rate their answers to the 21 qualitative questions on a five-point scale: 1) somewhat improved; 2) about the same; 3) somewhat worse; 4) much worse; and 5) very much worse. Summary indices were constructed from a combination of indicators in each of five dimensions: 1) use of coping strategies in response to crisis impacts; 2) food security; 3) employment; 4) education; 5) health. Preliminary findings of the survey became available in October 1998. They indicated that – contrary to some initial predictions – the impact of the crisis was quite uneven, with some regions actually benefitting through higher export earnings from the higher exchange rate. The preliminary findings were later found to be consistent with those from other GOI and donor surveys. An important finding of the survey was that there was little correlation between pre- crisis poverty levels and crisis impacts. Some relatively poor areas were not hard hit while the impacts of the crisis were quite severe in some relatively well off areas. The survey also indicated that, for the 1997 crisis at least, crisis impact targeting and poverty program targeting were quite different. The survey indicated that it was possible to quickly obtain useful policy information at relatively low cost with a quick turnaround. The survey gave a good indication of extent of crisis impact and of overall trends. Nutrition and Health Surveillance System The Nutrition and Health Surveillance System (NSS) was established by the international NGO Helen Keller International (HKI) in collaboration with the Ministry of Health in 1995 to evaluate a program in Central Java that promoted vitamin A-rich foods. When the economic crisis hit Indonesia in 1997, the NSS was quickly restarted and expanded to monitor the impact of the crisis on nutrition. In late 2000, the NSS was operational in seven rural provinces and in poor areas of four major cities, thus covering nearly 70% of the population. The data was collected quarterly from more than 44,000 households (see Fig. 17.2). In the rural areas, a multi- stage cluster sampling design was used to obtain the random sample. In each province, ecological zones were identified (Central Java, e.g. had six zones) and from each zone, 30 villages were selected by PPS sampling technique. From each village a list of CMRS Detailed Report Page 81 of 91 households with at least one child younger than five years of age was obtained, and from that list, 40 households were selected by fixed interval systematic sampling. For the urban areas the sample households came from slum areas. In Jakarta and Surabaya, 40 kelurahan with slums were selected by simple random sampling from all kelurahan with slums. Within each of the selected kelurahan, 2-3 RWs with slums were selected, within which 30 households with at least one child younger than five years old were selected. In both Semarang and Makassar, 80 RWs with people living in slums were randomly selected, and from each RW 30 households with at least one child younger than five years old. The NSS was designed to monitor Fig. 17.2: Sampling framework of the HKI/GOI malnutrition, including micro- NSS nutrient deficiencies and protein-energy malnutrition of women and children. This enabled identification of an increase in anaemia and night blindness for which no data were currently collected by the existing Ministry of Health surveillance systems One key achievement of the NSS is that it regularly issued the “Indonesia Crisis Bulletinâ€? and press reports with key information for policy makers and program managers as soon as such information became available. Between October 1998 and September 2000, 24 issues of the bulleting were released, most frequentlybetween September 1999 and May 2000 when a bulletin was released every month. . 17.4 System Design Considerations Elements of System An effective, evidence-based, decision-making monitoring and response system is one in which: ï‚· the demand for comprehensive, reliable and timely data on how a crisis affects vulnerable groups and individuals is met through a well organized data collection process; ï‚· the data are thoroughly analyzed; ï‚· relevant information becomes available and is disseminated to decision-makers in formats that are easy to comprehend; thereby supporting ï‚· the provision of adequate and timely responses for impact alleviation where they are most needed. Determining Data Demand When the impact of the GEC started to be felt in Indonesia in 2008, many questions needed to be addressed, such as the following. CMRS Detailed Report Page 82 of 91 ï‚· How many people would lose employment, and in what sectors? ï‚· Would only export oriented industries be affected, or would the impact of the crisis also spread to non export oriented industries? ï‚· How many workers would move from full employment to part-time employment? ï‚· What would be the impacts on levels of poverty? ï‚· How would people try to cope with increased hardship? It is evident that the information needed to answer such questions is diverse. Furthermore, another type of crisis would involve different information needs. So the question is what data should be collected in pre-crisis mode or order to anticipate the data needs associated with the next crisis? There is a risk that, in the search for completeness of coverage, too much complexity is built into the system and it is too costly to operate. It is not realistic to suppose that a full range of data can be collected on an ongoing basis just on the off chance of crisis occurring. This suggests that, in the absence of a crisis, the system should collect and analyse a just sufficient data to detect the onset of a crisis, but should be capable of being quickly expanded in scope should a crisis occur. Data Supply and Relationships to Other Systems The system should integrate data from other systems and sources that can help in crisis monitoring. ï‚· Two very obvious potential sources of data are Sakernas and Susenas, especially in view of the impending redesign of both surveys (as discussed in the next section). ï‚· The information systems of the social safety net programs are potential sources of information. ï‚· Signals of a crisis as its effects develop, or during the recovery period will come from regular macroeconomic data sources, such as trade and price indicators, also media reports on items such as the fall or increase in orders from abroad, factory closures, etc. ï‚· The system should be coordinated if not integrated with other monitoring systems, for example disaster monitoring, food and nutrition security monitoring, MDG achievement monitoring. Only those data items not available from any other source, or not available in a sufficiently timely fashion should be collected specifically by the system. The paper Vulnerability and Shock Monitoring and Response System (VSMRS), specifically proposes that a single system simultaneously monitors vulnerability and crisis impacts. 17.5 Sakernas and Susenas as Potential Data Sources Sakernas and Susenas for 2011 and onwards The BPS is planning to redesign Sakernas and Susenas from 2011 onwards. The two surveys will have the same sample design. ï‚· The sample will be redrawn each year. CMRS Detailed Report Page 83 of 91 ï‚· Data for one quarter of the sample will be collected each quarter. ï‚· Each quarterly sample will be representative of the country as a whole and capable of supporting unbiased estimates at national, provincial and district levels. ï‚· National and provincial data will be published quarterly and district data annually. ï‚· Provisional thinking is that Susenas data will be collected in February, May, August and November, and Sakernas data in March, June, September and December. ï‚· The questionnaires will remain much the same as for 2010. Quarterly data collection for Susenas and Sakernas will enable use of a smaller, better trained, permanent (or near permanent) workforce of interviewers. The same workforce will collect Susenas and Sakernas data. This will need to be carefully managed as the Susenas questionnaire is much larger than that of Sakernas. At times when they are not engaged on these surveys the interviewers may collect data from other smaller surveys. Data collection will be more computerised. Provisional thinking is that data will be captured at district or level on laptops at the time the supervisor is checking the questionnaires that have been completed by the interviewers. This will enable data verification before the data leave the district offices. Implications for Crisis Monitoring From 2011, the full range of Susenas and Sakernas data items will be available for sample of about 75,000 households on a quarterly basis. These will serve most of the crisis monitoring needs. Additional data regarded as critical for crisis monitoring can be collected by a supplementary module if and when required, for example in the event of emerging crisis. The data will be derived from an entirely new sample each quarter, which means increased standard errors relative to the panel design used for the CMRS. However, the large Susenas and Sakernas sample sizes will more than compensate for the non-panel design and the standard errors of the estimates will substantially lower than for the CMRS. The cost of crisis monitoring based on Susenas an Sakernas will be basically that of aggregating and analysing data at district level on a quarterly basis, as this would not be part of normal Susenas or Sakernas production. It should be noted that the current BPS plan to switch to the new survey design and collection arrangements for the first quarter of 2011 is quite ambitious given all the other ongoing work, in particular the Population Census, and associated PES. So the possibility of some delay to the current schedule and/or of a rather rocky start should be factored into planning. Further details are contained in the documents embedded below. Notes on the design of Sakernas and Susenas for 2011 and beyond SAKERNAS and SUSENAS Future Design CMRS Detailed Report Page 84 of 91 Improvements to Susenas – as documented by the ABS SUSENAS Redesign - ABS Paper Susenas Sample Design and Estimation – as documented by the ABS SUSENAS Sample Design and Estimation Issues 17.6 Data Analysis and Dissemination In non-crisis situations data analysis should be automated to the extent possible. In the event of a crisis, additional data may well be needed and the use of more complex data analysis methods will be justified. Information produced by the system should be made available in in the media and formats that fit the users. As different users may have different preferences, it may be desirable to have outputs available both electronically and in hard copy format. A management information system like DesInventar can help in dissemination. It gives users flexibility in tailoring analyses and reports to their specific needs. 17.7 Crisis Monitoring by Phase The manner in which the system operates will vary according to the crisis phase, as illustrated in Figure 17.3. This implies having clear operational definitions to determine for the country as a whole, and for specific regions, in what phase a crisis is. Mild crisis impact No. of people affected by the crisis Moderate crisis impact Severe crisis impact 1 2 3 4 5 Pre-crisis Onset Peak Recovery Post-crisis On-going, basic monitoring Existing social safety programs Crisis-specific monitoring Crisis-specific response measures Fig. 17.3: Stages in crisis monitoring and response CMRS Detailed Report Page 85 of 91 Pre- and Post- Crisis In pre- and post-crisis modes, the system should operate at a minimal level of effort, confined to basic data collection and analysis enabling ongoing monitoring and detection of early signals of an emerging crisis. Maximum use should be made of readily available data, in particular from Sakernas and Susenas. Because any part of the country can be impacted by a crisis, such monitoring should be nationwide at district level. The monitoring unit should build up a reference library with systematic documentation of monitoring approaches that have been used in past crises. The documentation should explain the essence of each approach, what instruments were used, their strengths and weaknesses. Onset of Crisis If and when signals of a developing crisis are detected, more intensive crisis monitoring procedures should be activated, including the following. ï‚· More intensive monitoring – depending on the numbers of households and/or regions affected by crisis impacts and/or the changes in levels of indicators. ï‚· Additional data collection - additional data from regions and/or economic sectors that are thought to be the most affected by the developing crisis, complemented possibly by qualitative studies. ï‚· Increased analysis - There should be an analysis manual that specifies the routine analyses to be undertaken when there is no crisis, and the recommended additional analyses when there is a crisis. ï‚· Faster data processing and analysis - especially if the crisis is widespread and has a severe impact. ï‚· Involvement of local governments and other agencies - local government agencies in the affected areas should be more involved in local monitoring and in initiating measures for crisis alleviation. ï‚· Increased financing – financing is required not only for basic on-going monitoring but also in the form of a quick access reserve for more intensive monitoring when a crisis occurs. Crisis Response There should be clear guidance on crisis response procedures. Some measures will be better suited to specific crisis conditions than others, and there will also be differences in how long it takes for the measures to be approved, and when they start to have effect for crisis alleviation. Increasing the allocation of ongoing safety programs is likely to give quicker results than initiating completely new crisis impact alleviation mechanisms. ï‚· The trigger for crisis response. Determining how many households and/or regions must be affected, or the change in levels of crisis indicators that must occur in order to initiate a crisis response. ï‚· Agreed crisis response activation plans that are known to all key stakeholders. Most crisis response measures require extra funds. Because public expenditure arrangements are slow to change, the key stakeholders likely to be involved in the CMRS Detailed Report Page 86 of 91 activation of response measures should decide upon crisis response activation plans in advance, so that time is not lost when such measures are needed. ï‚· Involvement of local governments in crisis response. The implementation of crisis alleviation measures usually requires the active participation and support of local governments. There needs to be guidance to local governments on how they can actively participate in monitoring in an integrated manner ï‚· Response scenarios. There should be a catalogue of crisis response scenarios, with information on how they can be expected to soften crisis impacts, and how they can be activated on a national or a regional scale. CMRS Detailed Report Page 87 of 91 Annex I: Reference Documents For the smaller documents, electronic copies are embedded below or in the main text. Electronic and paper copies of all documents are available from the WB Poverty Team. CMRS Initial Proposals Crisis Monitoring and Response Using Lot Quality Assurance Sampling as a Household Survey Design, Matthew Wai-poi, Using LQAS for World Bank, April 2009 CMRSS Crisis Monitoring and Response Using LQAS, April 29, 2009 Use of LQAS in CMRS Presentation April 2009 CMRS Technical Specifications List of Crisis Indicators (as of April 30, 2009) Embedded in text CMRS Questionnaires: Metadata Embedded in text CMRSS Questionnaire (Bahasa) August 2009 Embedded in text CMRSS Questionnaire (Bahasa) November 2009 Embedded in text CMRSS Questionnaire (Bahasa) February 2010 Embedded in text Lot Quality Assurance Sampling (LQAS) – General References Sample Size Determination: A Users Manual, Epidemiological and Methodology Unit, World Health Organization, 1986 Sampling Techniques for Evaluating Health Parameters in Developing Countries, Stanley Lemeshow and George Stroh, Academy Press, 1988 Manual for Conducting Lot Quality Assessments, Mark Wolff and Robert Black, The John Hopkins University, 1989 Sample Size Determination in Health Studies, S K Lwanga and S Lemeshow, Epidemiological and Methodology Unit, World Health Organization, 1991 Assessing Child Survival Programs in Developing Countries, Joseph Valdez, Harvard School of Public Health, 1991 Description and comparison of the methods of cluster sampling and lot quality assurance sampling to assess immunization coverage, Stacy Hoshaw-Woodard, Department of Vaccines and Biologicals, World Health Organization, 2001 Global review of health care surveys using lot quality assurance sampling (LQAS), 1984–2004, Susan E. Robertsona, Joseph J. Valadezb, Social Science & Medicine 63 (2006) 1648– 1660 CMRS Detailed Report Page 88 of 91 Large Country - Lot Quality Assurance Sampling, Bethany Hedt et al, Health Nutrition and Population, 2008 Sakernas documentation Sakernas Questionnaire Embedded in text Informal Sector and Its Measurement, Indonesian Country Paper presented at Workshop on Measuring Informal Sector, Asian Development Bank, Manila 20-23 May 2008 Methodology of National Labor Force Survey 2008-2010, BPS Subdirectorate of Developing Master Frames, BPS Contractors and Consultants Terms of Reference Financial Proposal for CMRSS Data Collection by BPS Embedded in text Terms of Reference for BPS Data Collection Embedded in text Terms of Reference for CMRS Consultant - Final August Embedded in text 2009 Terms of Reference for CMRS Manager Embedded in text Terms of Reference for IT Specialist Embedded in text Work plan for Second IT Specialist Embedded in text Core Analysis Team/ Analysis Review Committee Core Analysis Team/ Analysis Review Committee Initial Embedded in text Work Plan CAT (BPS) DINAS Data Analysis - Presentation Embedded in text Analysis Results and Presentations Indicators for which district risk flags computed/ not Embedded in text computed with reasons Presentation to Aus AID June, 2010 Embedded in text Disinventar for Crisis Vulnerability Monitoring System, Embedded in text August 2010 12.3 Contractor’s Reports (to be inserted) Consultant’s Reports Consultant’s June Mission Report Design Report, June 2009 Consultant’s October Mission Report Analysis Framework, October 2009 Consultant’s February Mission Report CMRS Detailed Report Page 89 of 91 Analysis Notes Round 3, June 2010 CMRS Manager’s Reports from districts Report on Field Visit to South Sulawesi, December 2009 Report on CMRS Conduct December 2009 (to be inserted) CMRS Manager’s Other Reports Establishing a Crisis and Vulnerability Monitoring and Response System in Indonesia Microsoft Office Word Macro-Enabled Document Work plan for Establising a Crisis and Vulnerability Monitoring and Response System Institutionalising the CMRS (to be inserted) Data Analyst’s Reports Editing rice volume and rice value Embedded in text (to be inserted) CMRS Detailed Report Page 90 of 91