Report No. 27 52784 South Asia Human Development Sector Data Utilization and Evidence-Based Decision Making in the Health Sector Survey of Three Indian States. April 2009 Discussion Paper Series Data Utilization and Evidence-Based Decision Making in the Health Sector Survey of Three Indian States April 2009 ACKNOWLEDGMENTS This report was originally conceived as part of a larger study that aimed to address two related issues of data quality and data use in the health sector. These issues were to be explored through three interrelated activities: (a) a survey of health managers to assess the level of data utilization and explore ways of enhancing evidence-based decision making; (b) an assessment of the quality of different sources of data, mainly through a triangulation exercise; and (c) a series of case studies to examine efforts of strengthening the results focus among managers. This report covers the first activity and the other two will hopefully follow. Earlier draft of this report was shared with MOHFW. The current version incorporates comments from MOHFW. The report was prepared by a team led by Tekabe Belay and including Nkosinathi Mbuya and Vikram Rajan. Data was collected by the Constella Futures group. The team would like to thank officials of the national Ministry of Health and Family Welfare (MOHFW), and officials in the states of Rajasthan, Maharashtra, and Uttar Pradesh for their assistance and support during the course of the report preparation. The team would like to specially thank Mr. G. C. Chaturvedi, ex-Additional Secretary & Mission Director (NRHM), Mr. S. K. Das, ex-Additional Director General (Statistics) and Mr. Pravin Srivastava, Director (Statistics) from the MOHFW for their constant support to the study. Consultations with local NGOs in Uttar Pradesh and Rajasthan and stakeholders at central level provided valuable inputs to the report. The team would like to acknowledge the valuable comments and contributions of various people. Especially comments from Jerry La Forgia, Eduard Bos, Patricio Marquez, Ajay Tandon and colleagues in the HNP unit are very much appreciated. Benjamin Loevinsohn has contributed to the work at different levels including the design of the questionnaire and providing valuable comments on the draft report. The team would like to thank Julie McLaughlin, Sector Manager, HNP South Asia, for her support and guidance. ABBREVIATIONS AND ACRONYMS MDG Millennium Development Goal DUI Data Utilization Index NRHM National Rural Health Mission MOHFW Ministry of Health and Family Welfare PHC Primary Health Center CHC Community Health Center SC Sub Center RBF Results Based Financing ii Table of Contents ACKNOWLEDGMENTS ................................................................................................... i ABBREVIATIONS AND ACRONYMS ........................................................................... ii Executive summary ............................................................................................................. v 1. Introduction ................................................................................................................. 1 2. Study Methodology..................................................................................................... 2 2.1. Survey Design, Sampling and Coverage .............................................................. 2 2.2. Questionnaire Development and Survey Team .................................................... 3 3. Findings and Analysis ................................................................................................. 4 3.1. Measuring Current Levels of Data Use ................................................................ 4 3.2. Perceived Challenges and Constraints to Data Use ............................................. 6 3.3. Skills Assessment ................................................................................................. 8 3.4. Data Sources and Use ......................................................................................... 10 3.5. Ranges of Actual Decision Making ................................................................... 10 3.6. Decision Making Process: Who and How.......................................................... 11 4. Possible Interventions ............................................................................................... 13 4.1. Agreement on Performance Indicators ............................................................... 13 4.2. Training and Capacity Building ......................................................................... 14 4.3. Introducing a New Cadre of Health Managers................................................... 16 4.4. Incentives and Accountability ............................................................................ 17 4.5. Results Based Financing .................................................................................... 18 4.6. Sensitivity Analysis ............................................................................................ 19 5. Conclusion and Way Forward .................................................................................. 19 6. Update on progress made by MOHFW .................................................................... 20 List of Figures Figure 1: Perceived Challenges and Constraints to Data Use............................................. 7 Figure 2: Percentage of Respondent Reporting a Source Very Useful ............................. 10 Figure 3: Perception of Basis for Decision Making in the Health Sector ......................... 12 Figure 4: Use of Computers by Level of Government ..................................................... 15 List of Annexes Annex 1: Health indicators in the states of Rajasthan, UP and Maharashtra ................... 22 Annex 2: Sample Selection ............................................................................................... 23 Annex 3: Questionnaire for India Health Data Use Study ................................................ 25 Annex 4: Constructing the Indices and Sensitivity Analysis ............................................ 46 iii List of Tables Table 1: Mean Data Utilization Index by Level ................................................................ 6 Table 2: Perceived Challenges and Constraints to Data Use by Level .............................. 7 Table 3: Perceived Challenges and Constraints to Data Use by Level .............................. 8 Table 4: Mean Skills Test Scores by Level ....................................................................... 9 Table 5: Percentage Distribution of Decision Categories ................................................ 11 Table 6: Who Influences Decisions: Percentage Distribution ......................................... 12 Table 7: Knowledge of NRHM Performance Indicators ................................................. 13 Table 8: Mean DUI by Knowledge of NRHM Performance Indicators .......................... 14 Table 9: Data Utilization Index by Type of Training ...................................................... 15 Table 10: Mean DUI by Use of Computers ...................................................................... 16 Table 11: Skills score and DUI ......................................................................................... 16 Table 12: Mean DUI by Educational Attainment ............................................................. 16 Table 13: Mean DUI by In-service Training in Health Management ............................... 17 Table 14: Mean DUI by Explicit Recognition for Performance on the Job ..................... 17 Table 15: Mean DUI by Assessment of Departmental Performance During Management Meetings ............................................................................................................................ 18 Table 16: Mean DUI by Respondents' Attitude Towards Introduction of RBF............... 18 List of Plots and Box Plot 1: Distribution of Data Use Index ............................................................................... 5 Plot 2: Distribution of Skills Test Scores............................................................................ 9 Box 1: How to Improve Data Use...................................................................................... 2 iv EXECUTIVE SUMMARY Studies on data use are very limited. Despite increasing discussion on the need for strengthening accountability and the results culture, little is concretely known about the impediments to using health sector data in the Indian context. The current study provides a glimpse of the situation in India. It provides an empirical assessment of the level of data use in the health sector. Though it is limited in its geographic coverage, it provides insight into the level of data use at the different levels of government. Data use in the health sector is very low. The average data utilization score of the respondents is well below the half point mark. This average hides considerable variation among central, state, district and below levels. Data use increases as one moves from the central level to state, district and below. A number of factors contribute to such low level of data utilization. The factors range from organizational to behavioral to skills. At the organizational level, a work culture that is less focused on results, and resource allocation decisions that are based on normative practices provide little incentive for evidence-based decision making. At the individual level, impediments include lack of skills to analyze and use data, absence of incentives for data use and lack of recognition for performance at work place. Having an agreed set of performance indicators is strongly associated with data use. A common set of indicators upon which performance is measured promotes accountability and transparency. Equally important is ensuring that these indicators are widely known to managers and implementers. As shown by the strong association between knowledge of National Rural Health Mission (NRHM) indicators and use of data for decision making, disseminating knowledge about indicators is as important as defining appropriate indicators. An effective way of making sure managers' know about indicators is indeed using them to assess performance. Analytical and interpretation skills are necessary to improve data use in the sector. The current levels of data analysis skills are modest overall, but are strongly related to the level of use of data. What is striking is the perception of respondents about their own skills. About 88% believe that they have the necessary data analytical and interpretation skills they need for their job. This self-assessment overstates the actual skills managers possess as measured by the skills tests. Some types of trainings are particularly effective. Formal schooling -- with the exception of post graduate in medicine -- has no association with of the use of data. However, in-service training in health management is highly positively associated with the level data use. Those with such training scored high on the data use index created for this study. Interventions that provide such training to managers or introducing new cadre v of health managers with health management training would thus likely improve data utilization in India's health sector. Incentives might be the most effective intervention to improve data use. The results show that those who received either financial or any other form of explicit recognition for their performance demonstrate high data use index. One form of incentive is making financing dependent on performance. The other form would be a more specific type that targets either a group's or individual's performance. Replacing the existing norm base resource allocation with a scheme where financing in part depends on performance could also improve data use. Promoting accountability and regular performance assessments are also highly associated with data use. Data use is significantly higher among employees that were regularly assessed on their performance and were held accountable. This requires a management style whereby managers assess the performance of their staff on a regular basis. In summary, data use for decision making in the health sector can be improved through: (a) having an agreed set of indicators of success and making sure they are widely known; (b) rewarding individual or group achievement; (c) promoting accountability and regular performance assessments; (d) in service training in health management; and (e) providing training and capacity building, especially at district and sub-district levels. vi 1. Introduction Recently, there has been increased attention to data use in the international public health community. At the global level there have been major investments in data collection for public health programs. However, there is concern that such data are not being used. Data and information lack value unless they are used to inform decisions. Interventions that increase local demand for information and facilitate its use enhance evidence-based decision making. Data use therefore, is critical to improving the effectiveness and sustainability of the health system. Data quality is poor and use of available data is low in India's health sector. In order for India to improve the performance of its health sector and know if it is achieving the MDGs, it will have to do a better job of tracking its performance on key indicators. Tracking performance more effectively will require that: (a) the weaknesses in data quality are understood and the quality of data collected is improved; and (b) the data produced are actually used to inform decision making. Currently, there are serious issues of quality affecting many of the sources of data. The result has often been confusion and less-than-optimal decision making. Unfortunately, even when reasonably good data are available, it is rarely used for management purposes. Knowledge on impediments to data use is limited. While there is increasing discussion on the need for strengthening accountability and the results culture, little is known about the impediments to collecting, analyzing and using data in the Indian context. A preliminary review of the global literature found very few articles dealing with these issues, suggesting that India is not alone in its lack of understanding of this challenging area. The study reported here is part of a larger study that involves: (a) assessing and improving the quality of the different types of data that are currently available to decision makers; and (b) increasing the use of such data so as to improve the performance of the health sector. To address the issue of data quality, the Ministry of Health and Family Welfare (MOHFW) has initiated the process to establish a data triangulation unit. The unit will be tasked among other things to assess the quality, reliability, timeliness and relevance of data to enhance evidence-based decision-making at different levels of the public health service delivery system. The current study focuses on issues of data use. It aims to shed light on how data are used for decision-making at different levels of the health sector in India and to highlight impediments to improved data utilization. The study is among few that attempts to examine data use issues in India's health sector. Methodologically, the study is not designed to establish causal relationships. Rather it is mainly descriptive in nature and attempts to identify associations between data use and the factors accounting for higher or lower use of data. The set of actionable recommendations (see Box 1) emerging from such analyses are worth testing at a smaller scale. The results show that analytical and interpretation skills are necessary to improve data use in the sector. But skills by themselves are not sufficient. Linking financing to results achieved would create strong incentives for data use. Incentives are perhaps the most effective intervention that improves data use. Promoting accountability and regular performance assessments as well as having an agreed set of indicators of success on which performances are assessed is also highly associated with data use This report is organized in five sections. The survey design, instruments and the data collection process will be described in the next section. Findings including measurement of data use, perceived data use issues, skills assessment of respondents and ranges of actual decision making are presented in Section III. Section IV presents a number of hypotheses and discusses their implications. The last section provides conclusions and recommendations. Box 1: How to Improve Data Use Set clear performance indicators at national and departmental level. Ensure the performance indicators are widely known at all levels. Introduce incentives for data use. Incorporate regular performance assessments & recognize good performers. Provide in-service trainings (e.g. health management) Provide practical data-related training & the necessary hardware at district and lower level. 2. Study Methodology 2.1. Survey Design, Sampling and Coverage Three states with different levels of performance in health outcomes were included so that the potential for cross fertilization could be exploited. The selected three states were Maharashtra, Rajasthan, and Uttar Pradesh. While the first belongs to the group of high performing states, the latter two are among the low performing states. Within these two, Rajasthan appears to do better than Uttar Pradesh (see Annex 1). The survey also included some of the policy makers and health officials at the central government. To optimize the coverage, within each state, four districts were selected based on a composite index1 of socio economic indicators. Districts with low, high and average 1 The composite Index is based on the following indicators: Female Literacy, Male Literacy, and Gender Disparity in Literacy, Sex Ratio 0-6 population, Proportion of Births of Order 3 and above, % Girls Married below 18 years, CPR, Coverage of Safe Delivery, Coverage of Complete ANC, and Coverage of Complete Immunizations. 2 scores were selected. Districts where the state capitals are located were also selected. The list of districts covered from each of the state is presented in the Annex 2. In terms of respondents, the survey covered officials and health managers from the central, state, district and below district levels. It was intended to explore both the supply and demand side issues of data use. On the supply side, issues including the accuracy, quality and timeliness were explored. On the demand side, incentive structures surrounding data use, accountability issues, analytical skills and perceived training needs were assessed. A total of 270 interviewees were randomly selected from a stratified sample. Of these, 30 were from the central level and 80 from each state. At state level, 20 respondents were from the state headquarters, including policy makers, heads from medical colleges, and 15 managers from each of the selected four districts (see Annex 2 for detail). Out of the 270 sampled, a total of 242 respondents were available for the interviews. Five senior policy level officers at the central level and 21 state and district level health staff could not be interviewed; two state level officers declined to be included in the study. Of the 242 respondents included 25 (10.3 percent) were from the central level in Delhi, 72 (29.8 percent) from Rajasthan, 84 (34.7 percent) from Uttar Pradesh, and 61 (25.2 percent) from Maharashtra. In terms of the levels, 25 (10.3 percent) of the respondents were from the central level, 47 (19.4 percent) from the state, 132 (54.5 percent) from the district and 38 (15.7 percent) from sub-district levels (PHC/CHC/SC). In terms of education, the majority of the respondents had a MBBS (51 percent) or post graduate training (28 percent). Undergraduate and graduate diplomas constituted about 17 percent of the respondents. Respondents' job experiences ranges from less than a year to more than 30 years with more than half of the respondents having 11-30 years of experience. In sum, the respondents represent a mix of senior policy decision makers at the central and state levels and middle and junior level health staff at all levels. They include civil service officers from the Indian Administrative Services (IAS) and allied cadres, clinical and public health doctors, nurses, data managers from the health departments, and high ranking persons from the media. 2.2. Questionnaire Development and Survey Team As there were few examples of similar surveys in India or in comparable environments, the team developed the survey instruments from scratch. Initial drafts were revised several times with inputs from the MOHFW and other stakeholders. The final version was pre-tested in the field and modified. The single questionnaire was designed to accommodate both senior level policy makers as well as mid- and lower level decision makers, although some parts of the questionnaire (e.g., questions on training needs, skills assessment and computer use) were not administered for senior policy level officials including Joint Director and above at the central and state levels and District Collectors/Magistrates and Chief District Medical Officers at the district levels. The questionnaire was developed to include a combination of questions that were pre-coded 3 and designed to generate quantitative data, as well as some "open-ended" questions capable of generating more nuanced, qualitative information. (Refer to Annex 3 for the final questionnaire.) A survey team consisted of a team leader, survey manager and survey design expert at the central level. The actual interviews were conducted by three three-person teams consisting of a state team leader and two data collectors in each state. Interviewers and state team leaders were given day training and provided with a manual and list of the respondents to be covered in their respective states/districts. The Team Leader and Survey Manager shared the responsibility of carrying out the interviews at the central level, as for the most part the senior policy managers were interviewed in Delhi. Data quality: responses to open-ended questions were recorded and then recoded after using a code list for each open-ended question. The completed questionnaires were then given to a professional data entry firm for double data entry with built-in error checks. After the data entry, a consistency check was conducted on each of the responses as a second layer of data quality. Study constraints: The study is limited in its geographic coverage and the states covered by the study may not represent the whole of India. But they were intended to represent states that are under various stages of health systems development. Another possible weakness of the study concerns the way two central variables are calculated: Data Use Index and Skills Score (see Annex 4 for a description of the calculations). These two variables are constructed by combining a number of responses. While each of these responses might carry a different level of importance to the respondent, the computation assigns equal weight to each of the responses. 3. Findings and Analysis This section describes the survey findings, which are disaggregated by level of government, whenever possible. It begins with the status of data use in the health sector followed by a description of perceived explanations for the observed level of data use. Next, analytical and interpretation skills are assessed using results of the test administered to respondents. Finally, sources of preferred data, ranges of actual decision makings, and the decision making process are discussed. 3.1. Measuring Current Levels of Data Use Having an objective measure of the level of data utilization not only serves as a baseline for future interventions, but also allows analysis of potential interventions to improve data use for decision making. Hence, a composite index of data use is constructed to determine the extent to which health managers at the different levels of the Indian Government use data for decision making. This Data Use Index (DUI) is constructed from a number of questions that were meant to capture data utilization. Some of these questions include whether managers know their own performance on a number of indicators, whether tables/charts/graphs, etc. were displayed, whether data were used for 4 key decisions such as management decisions, monitoring key objectives, budget preparation and planning. The variables used to construct the DUI are of two types: (a) a "Self assessment": respondent's subjective assessment of own data use; and (b) an "Objective assessment": the observation by the interviewer and specific questions meant to objectively measure respondents' knowledge of their own performance. The variables included in each of these groups and the way scores are calculated to construct the DUI index is discussed in Annex 4. In addition, a sensitivity analysis is presented to assess the robustness of the DUI for the inclusion and exclusion of these variables. The range of possible DUI scores is between zero and eleven. Overall data utilization is very low with a mean DUI for all managers of 4.9 and a median of 4.63 (Plot 1). The distribution of the DUI is close to symmetric with few outliers. Managers at the two extreme ends of the level of government have the highest data use index (Table 1). The difference in DUI between these two groups is statistically significant. Low level of data use at district level is consistent with managers' response that decisions are often influenced by higher level administrative officials (see section 3.6). Of all government levels, the influence of administrative officials is the highest at district level. Given the importance districts have in a number of decisions in the context of NRHM, such low level of data utilization at the district level should be of concern. Plot 1: Distribution of Data Use Index .25 .2 .15 Density .1 .05 0 0 5 10 DUI 5 Table 1: Mean Data Utilization Index by Level Mean (SD) Central 5.97 (0.55) State 4.64 (0.37)* District 4.62 (0.23)* Sub-district 5.49 (0.40) * The differences are significant at less than 5%. A breakdown by state shows that managers in Maharashtra have the highest level of data use (5.34) followed by Rajasthan (4.65), while those in Uttar Pradesh have the lowest (4.45). The level of data use seems to correlate with the performance of the health system, although the level of data use is low across all three states. 3.2. Perceived Challenges and Constraints to Data Use One of the first issues the survey attempted to identify was the perceived constraints to data use. Respondents were provided with a list of thirteen constraints to improving data use that had been found in other contexts and were asked the extent to which they agreed that these were also constraints in their setting. Overall, the lack of skills to analyze and use data was identified as a major constraint to data use (Figure 1). About 64 percent of the respondents either agreed or strongly agreed that it was a potential impediment to data use. Absence of a work culture that focuses on results, lack of agreement on key indicators of performance, lack of incentives for data utilization, delays in the transmission of data to relevant levels, as well as too much unfiltered information were also impediments to data use identified by over half of the respondents. Surprisingly, data quality was not perceived to be as much a constraint to data use. Impediments by level: The constraints to data use differ across state, district and sub- district levels. In fact, there is little overlap in constraints identified by respondents at different levels. For instance, lack of agreement on performance indicators is identified by the majority of respondents (63.6 percent) at sub-district level as an impediment to data use, whereas the majority of district (71.2 percent) and state (81.8 percent) respondents, respectively, identified lack of skills to analyze and use data and absence of work culture that focuses on results as the main impediments (Table 2). 6 Figure 1: Perceived Challenges and Constraints to Data Use Percentage of Respondents Agreeing on Constraints to Data Use Poor quality data 37 Data not used for program monitoring (just record keeping) 42.2 Gov't decisions not based on information 38.5 No analysis or feedback from supervisors on data 45.9 Unwillingness to accept shortcomings in data 46.7 No set criteria for data collection and analysis 49.6 Frequent data duplication and inconsistency 49.6 Too much information 52.6 Little agreement on key indicators of performance 52.6 Data slow to reach relevant levels 53.3 No incentives for data utilization 55.6 No culture of focusing on outputs and outcomes 59.3 Lack of skills to analyze and use data 63.7 0 10 20 30 40 50 60 70 Table 2: Perceived Challenges and Constraints to Data Use by Level Perceived Challenges State District Sub-district Poor quality data 27.3 41.1 59.1 Data not used for programmed monitoring 27.3 50.7 54.5 Government decisions not based information (but on 18.2 49.3 31.8 No analysis or feedback from supervisors on data 27.3 52.1 45.4 Unwillingness to accept shortcomings in data 44.5 52.1 45.4 No set criteria for data collection and analysis 27.3 60.3 54.5 Frequent data duplication and inconsistency 54.5 47.9 50.0 Often too much information 54.5 58.9 50.0 Little agreement on key indicators of performance 36.4 60.3 63.6 Data slow to reach relevant levels 45.5 60.3 54.5 No incentives for data utilization 63.6 56.2 50.0 No culture of focusing on outputs and outcomes 81.8 64.4 54.5 Lack of skills to analyze and use data 63.6 71.2 54.5 Impediments by state: Data use issues differ significantly across the three states. In Maharashtra, for example, it is notable that none of the responses are identified by more than 45 percent of the respondents as constraints to data use. In contrast, all the 13 constraints are identified by over 50 percent of respondents from Rajasthan as impediments to data use (Table 3). Differences in magnitude aside, there are some commonalities in the constraints identified in all three states. Whereas perceptions of data quality appear to be less of an issue in all three states, there seems to be consensus that lack of skills for data analysis and use, lack of agreed sets of performance indicators, slow data flow, too much 7 unfiltered information, lack incentive for data use, and absence of guidance for data collection and analysis are all challenges for data use. Table 3: Perceived Challenges and Constraints to Data Use by Level Perceived Challenges Rajasthan Uttar Maharashtra Poor quality data 64.5 22.2 22.2 Data not used for program monitoring 65.4 44.4 27.8 Government decisions not based information (but 53.8 61.1 16.7 No analysis or feedback from supervisors on data 67.3 38.9 25.0 Unwillingness to accept shortcomings in data 67.3 50.0 25.0 No set criteria for data collection and analysis 71.2 72.2 25.0 Frequent data duplication and inconsistency 63.5 44.4 30.6 Often too much information 67.3 50.0 44.4 Little agreement on key indicators of performance 80.8 55.6 27.8 Data slow to reach relevant levels 80.8 33.3 36.1 No incentives for data utilization 65.4 66.8 36.1 No culture of focusing on outputs and outcomes 90.4 50.0 33.3 Lack of data analysis and use skills 78.8 83.3 41.7 In summary, whatever way the data are analyzed, a few impediments emerge as major constraints, though with varying degree of importance; the lack of data analysis and use skills, lack of agreed performance indicators, a work culture that does not emphasize results, lack of incentives to use data, and absence of guidance for data collection and analysis emerge consistently. In the latter part of the paper, these constraints are partially tested to consider if addressing them could indeed improve data use. 3.3. Skills Assessment If decision-makers are to improve data use, they need to have a basic understanding and appreciation of how to interpret and use quantitative information and tools (e.g., proportions, rates, and ratios, graphs and charts) for solving problems. Because a lack of such skills was perceived by the respondents as a key impediment to data use, the study attempted to assess skills in this regard. Two types of assessment were done: (a) respondents' self-assessment of their skills; and (b) an objective assessment of their skills using a skills test. First, respondents were asked whether they felt they had the skills necessary to use the data and information for decision making. Almost 88 percent of the respondents thought that they had the necessary skills. This sounds inconsistent with the results in the previous section where managers identified lack of analytical skills as one of the major constraints to data use and a reason why data use remains low. Second, respondents were given a short skills test comprising nine questions designed to assess the extent to which they understood data presented in different formats, their understanding of basic epidemiologic principles (e.g., prevalence vs. incidence), and 8 performance of simple calculations (see Annex 3). The distributions of scores from this test are presented in Plot 2. The mean score for all respondents who completed at least one skills question was 5.5 out of 9 with 24 percent of the respondents scoring between 1 and 3; 43 percent between 4 and 6; and 33 percent between 7 and 9. More than 55 percent of the respondents scored well above the 66th percentile indicating the relatively high skill level of managers. Plot 2: Distribution of Skills Test Scores There was no discernable pattern in the skills scores test for the different levels of government employee (Table 4). Respondents from the central level of government have the highest test scores and the state level respondents the lowest, but with the small sample size, the differences were not statistically significant. Table 4: Mean Skills Test Scores by Level Level Mean (SD) Central 6.67 (1.21) State 4.17 (2.63) District 5.79 (2.40) Sub-district 4.58 (2.28) An assessment of the skills test scores by state found that respondents from Rajasthan scored the highest in the skills test (mean score of 6.14) followed by those from Uttar Pradesh (5.72). Respondents from Maharashtra had the lowest test scores (3.72). The differences in test scores between respondents from Rajasthan and Maharashtra 9 (p=0.0002) and between Uttar Pradesh and Maharashtra (p=0.0015) were both statistically significant. 3.4. Data Sources and Use Respondents were also asked about the types and sources of data they found useful (Figure 2). It is clear that respondents value routinely collected data (e.g., HMIS, reports) over survey data. Such dependency on administrative data makes the case for improving the quality of these data sources strong and urgent. Figure 2: Percentage of Respondent Reporting a Source Very Useful 3.5. Ranges of Actual Decision Making To understand better the context in which data use occurs, it is useful to ascertain the range of decisions that the respondents make in the course of their duties. Managers were asked whether they were involved in making or influencing decisions as part of their routine job. Sixty-nine percent responded affirmatively to this question. These respondents were then asked to indicate, from a list of eight categories, the types of decisions that they were involved in making or influencing (Table 5). 10 Table 5: Percentage Distribution of Decision Categories Decision Categories Central State District Sub-district Day-to-day program management 100.0 (5) 100.0 (12) 86.4 (58) 80.0 (16) Medical supply and drug management 100.0 (3) 57.1 (4) 66.2 (49) 26.2 (17) Formulating plans 100.0 (5) 62.5 (9) 86.2 (56) 62.5 (10) Budget preparation 100.0 (4) 75.0 (10) 81.1 (49) 58.8 (10) Deciding budget reallocation 100.0 (4) 71.4 (8) 78.8 (47) 56.3 (9) Human resource management 100.0 (4) 87.5 (10) 78.6 (49) 68.7 (11) Monitoring key objectives 100.0 (4) 87.5 (11) 86.2 (56) 87.5 (14) Identification of emerging issues including 100.0 (3) 83.3 (6) 84.9 (51) 89.5 (17) Epidemics * Total number of respondents is indicated in bracket. Among this sixty-nine percent, most respondents are involved in making key decisions in program and management, monitoring, and identification of emerging issues. Not surprisingly, respondents at the central level had more decision-making role than those at state, district and lower levels. Decisions regarding medical supply and drug, and budget reallocation are the least decentralized. In almost all categories, district level respondents seemed to be more involved in decision making than state level respondents. In decisions related to human resource management and day to day program management, central and state levels are much more involved that the lower levels. The sub-district levels were least involved on all decision categories, except for decisions regarding identification of emerging issues and monitoring key objectives. Within state, district and lower levels, however, it is not clear who is more involved in decision making. 3.6. Decision Making Process: Who and How (a) Who Influences Decisions Once the types of decisions each level is authorized to make has been identified, respondents were probed to indicate who influences the decisions they make (Table 6). Higher level administrative officials are by far the most frequently cited groups that influence decisions followed by district level elected officials. Civil society and Panchayats also have clear roles in influencing decisions, though much less than administrative officials. The influence of Panchayats and district level leaders decreases as one moves to state level and above. Within a district, administrative officials are far more influential than elected officials (i.e., Zila Parshad & MP). The same holds true for sub-district levels in that administrative officials are more influential than Panchayats. It is also notable that the influence of civil society groups declines as one moves from state to district and sub-district levels. 11 Table 6: Who Influences Decisions: Percentage Distribution Category Central State District Sub-district Total Panchayat 0.0 (21) 14.29 (35) 8.18 (110) 22.58 (31) 10.66 (197) Zila Parshad & 0.0 (21) 8.57 (35) 20.00 (110) 12.90 (31) 14.72 (197) MP Higher level 14.29 (21) 40.00 (35) 43.64 (110) 35.48 (31) 38.58 (197) Administration Civil society 4.76 (21) 17.14 (35) 13.64 (110) 9.68 (31) 12.69 (197) * Total number of respondents is indicated in brackets. Note that equal number of respondents responded under each category. (b) How Decisions are Made Knowing who makes decisions and influences decisions by itself tells us little about whether the decisions are based on evidence. What is important is whether data are used to make those decisions. Figure 3 plots the response of managers on how decisions are made in their respective departments. In general, respondents believe that both evidence/data and political considerations are the basis of decisions making. This is not however consistent with the observed level of data use. Figure 3: Perception of Basis for Decision Making in the Health Sector % agree % disagree 100 90 80 70 60 proportions 50 40 30 20 10 0 Personal liking Superiors' directives Evidence/facts Political interference Decisions are based on 12 4. Possible Interventions This section provides findings of preliminary tests on a set of hypotheses on how the utilization of data could be improved in the health sector. The hypotheses tested are generated from respondents' assessment of key impediments and also from issues identified during consultations with civil society in UP and Rajasthan. 4.1. Agreement on Performance Indicators Indicator-driven planning is one of many institutional management approaches that can be used to create a culture of evidence-based decision making. This type of planning requires a shared strategic vision for a sector that enables managers and their subordinates to understand better what is expected from them and how they can determine whether they are making progress towards their objectives. Linking data on the specific indicators to the different planning processes creates demand for data and its utilization. In order to determine whether having a shared strategic vision could lead to better use of data among health managers in the India context, the respondents were asked about their knowledge of NRHM indicators and whether they could recall three of them (evidencing a global shared vision). Among those respondents who attempted to recall at least one NRHM indicator, 77 percent recalled at least one indicator correctly and 23 percent failed to recall any NRHM indicator in their three attempts. Only 11 percent of the respondents recalled all 3 NRHM indicators correctly in their three attempts, 326 percent recalled two and 34 percent recalled one. Knowledge of NRHM indicators varied across the different levels of government (Table 7). Only 40 percent of the managers at state level could recall at least one NRHM indicator in three attempts; whereas 87.5 percent of managers at sub-district level could do so. Table 7: Knowledge of NRHM Performance Indicators Level % Who Can Recall at Least One in Three Attempts Central 66.7 State 40.0 District 76.6 Sub-district 87.5 The mean DUI of those who do not know any of the performance indicators of NRHM was significantly lower (p=0.000) than that of managers who knew about NRHM 13 indicators and could recall at least one of them (Table 8)2. Given the importance of NRHM in terms of size, expected impact and resources allocated to its implementation, such a low level of knowledge about its performance indicators among managers is worrying. One expects NRHM indicators to be widely known among managers in the health sector. Table 8: Mean DUI by Knowledge of NRHM Performance Indicators Knowledge of NRHM Mean DUI (SE) Know NRHM performance 5.80 (.24) ttest: -4.41 indicators and can recall at least for Diff=0 one of them (p=0.000) Does not know of NRHM 4.24 (0.21) performance indicators and/or rho=0.33 cannot recall correctly at least (p=0.000) 3 f h As an indicator of shared strategic objective at departmental level (superior-subordinate shared vision), respondents were asked whether superiors explain what they expect from staff. The effect of shared vision on data utilization was assessed by comparing the DUI of those respondents who have a shared expectation about their job with their managers with those who do not. The mean DUI of the former group is 5.03 (s.e.=0.57) which is almost a point higher than that of the comparison group (DUI = 4.16; s.e.=0.22). Hence, having an agreed set of indicators of success both at national and departmental level is strongly related with improved data utilization. Establishing a shared understanding of the objectives of the sector/department, and how performance towards its achievement is tracked, improves data utilization through improved accountability. 4.2. Training and Capacity Building Improvements in health information systems require investments in the training and capacity development of human resources at all levels of government. For example, at national level, skilled personnel are needed to oversee data quality and standards for collection, as well as ensure the appropriate analysis and utilization of data. The capacity for data analysis is often lacking thereby impeding the ability of health staff to analyze, interpret and use data for planning and management. Managers were asked about the different types of formal trainings they had received and simple statistical tests were done to ascertain whether receipt of this training is related to use of data. Table 9 shows that receiving any form of data-related training and increased 2 About 48% of the respondents did not make any attempt to answer the question about knowledge of NRHM indicators. It was then assumed that their failure to respond to the question was due to lack of knowledge about the indicators. 3 The significance test is against the null that the two variables (DUI and knowledge of NRHM indicators) are independent. 14 data use go in tandem. In almost all cases the differences in DUI between those with the training and those without are statistically significant. Though training is important, it is the least viable intervention to improve data use. This is mainly due to limited scope for doing more training (Table 9). The current level of training among managers in all categories is large and there is little scope for doing more training. Table 9: Data Utilization Index by Type of Training Type of Formal Training % had any training Mean DUI (SE) ttest for Diff=0 HMIS 74.6 Yes: 4.75 (0.19) -1.61 Survey 80.2 Yes: 4.71 (0.19) -2.34** Data Analysis 83.8 Yes: 4.69 (0.18) -2.95** Data Utilization 84.2 Yes: 4.73 (0.18) -2.49** Planning 85.2 Yes: 4.78 (0.18) -1.75* * Significant at less than 10%; ** significant at less than 5%. To apply the trainings in HMIS, surveys, data analysis and or/data utilization as well as to make use of the trainings in other areas, the availability and use of appropriate hardware is necessary. As such, respondents were asked about challenges they face in this regard. Fifty nine percent of the respondents reported that they did use computers, but the proportion declines as one goes down the level of government (Figure 4.1). Data utilization among computer users is significantly larger than non-users (Table 4.4). Figure 4: Use of Computers by Level of Government 15 Table 10: Mean DUI by Use of Computers Mean DUI (SE) Use computers 5.49 (0.24) ttest: 3.36 for Diff=0 (p=0.005) Do not use computers 4.18 (0.32) rho=0.27 (p=0.001) Building the data analysis and interpretation skills of managers also have an impact on their level of data use. This is consistent with managers' response that lack of skills for data analysis is one of the main constraints to data use (Table 11). DUI scores are consistently and positively correlated to skills level (rho=0.39). Respondents with high skills test scores have higher DUI compared to those with medium and low skill test scores. The differences in DUI between low and highly skilled managers are statistically different. Table 11: Skills score and DUI Skills Test Score Mean DUI (SD) Low (1-3) 3.32 (0.50) rho=0.39 Medium (4-6) 4.72 (0.45) (p=0.001) High (7-9) 5.41 (0.37)*** *** The difference between low and high is significant at less than 1% 4.3. Introducing a New Cadre of Health Managers Introducing a new cadre of trained health managers could be a viable option for improving the performance of the health sector as well as for enhancing evidence-based decision making. In the present study, an attempt was made to examine whether educational attainment and in-service training in health management could improve data utilization. In general, the results show that, except post graduate diploma in medicine, the level of formal education had no effect on the reported use of data (Table 12). The level of data use of those with the highest level of education (e.g., PhD) is not statistically different from that of the lowest level of education. Table 12: Mean DUI by Educational Attainment Highest Level of Education Mean (SD) Secondary/Intermediate 4.29 (0.19)* Bachelors 4.30 (0.66)* MPH/MSc/MA/MBA 4.26 (0.35)** Post grad. Diploma in Medicine 5.33 (0.21) or MBBS or MD/MS PhD 4.63 (0.96) The difference with MBBS MD/MS is significant at less than 10% (*) and 5% (**). 16 In contrast, participating in in-service training in Health Management (HM) is associated with increased use of data (rho=0.19). Those respondents who reported that they had received HM training have a DUI of 5.46 compared to a DUI of 4.55 for those who had not received the training and the difference is statistically significant (Table 13). Hence while formal education has little effect, introducing new managers with health management skills or training existing managers in health management would increase evidence-based decision making in the sector. Table 13: Mean DUI by In-service Training in Health Management Mean DUI (SE) Received HM training 5.46 (0.27) test: 2.35 for Diff=0 ( p=0.020) Did not receive HM training 4.55 (0.26) rho=0.19 (p=0.018) 4.4. Incentives and Accountability Here the hypothesis that performance assessments and recognition of performance on the job is associated with increased data utilization is tested. Respondents were asked whether they had ever received any kind of explicit recognition for their performance on the job. About 60 percent reported that they had received some explicit recognition. Those who had received explicit recognition have a significantly higher level of data use than those who did not (Table 14). Moreover, a test of whether such recognition is associated with increased data utilization shows that a strong association did exist (rho=0.20). Table 14: Mean DUI by Explicit Recognition for Performance on the Job Mean DUI (SE) Received explicit 5.39 (0.22) test: 3.44 for Diff=0 ( p=0.001) recognition for performance rho=0.20 (p=0.007) on the job Did not receive any explicit 4.20 (0.25) recognition for performance on the job Another dimension of performance assessments examined is whether regular discussions about the unit's performance are related to more data usage. Respondents were also asked whether they held regular (at least once a month) management meetings at which the performance of their unit is assessed. Seventy four percent reported that they had held regular meetings with their subordinates at which the performance of the unit was discussed, and 85 percent had held regular meetings with their superiors at which the performance of the unit was discussed. Holding these regular meetings to review unit performance is related to increased data utilization (Table 15). 17 Table 15: Mean DUI by Assessment of Departmental Performance During Management Meetings Mean DUI (SE) test for Diff=0 (p value) Management meetings with subordinates held at Yes: 5.54 (0.17) -6.71 least monthly and at which unit performance is No: 3.16 (30) (0.0000) reviewed. Management meetings with superiors held at Yes: 5.22 (18) -3.07 least monthly and at which unit performance is No: 3.88 (0.43) (0.0021) reviewed. The results above suggest that interventions such as introducing incentives for data use, and changes in management style whereby performance assessment becomes part of the departmental routine could be highly effective. 4.5. Results Based Financing The underlying principle in results based financing (RBF) is to introduce financial incentives for managers, staff and service delivery personnel and service users to improve performance. In India, there are few examples of RBF schemes. In order for RBF to work, performance indicators have to be established and sound M&E systems implemented to provide the necessary data. Therefore to the extent that such schemes are introduced the level of data use would be expected to increase. It would have been more realistic to test the DUI difference of those under RBF system and those under a non-RBF system. Due to lack of data to so do, a second best test based on a hypothetical situation is performed; respondents were asked to express their opinion about the introduction of results-based financing in their system. Table 16: Mean DUI by Respondents' Attitude Towards Introduction of RBF Mean DUI (SE) Enthusiastic about 5.03 (0.22) test: 0.31 for Diff=0 introduction of results- (p=0.7566) based financing Against or indifferent to 4.88 (0.36) introduction of results- based financing Although respondents who were enthusiastic about the introduction of results-based financing have higher data use indices (Table 16), the difference is not statistically significant. Moreover the rank correlation coefficient between the two groups is small (rho=0.002) and not significant. Nevertheless, it would be worthwhile to collect data on actual RBF implementation to make a grounded conclusion. 18 4.6. Sensitivity Analysis The sensitivity analysis shows that the DUI is robust as regards the inclusion and exclusion of sets of variables and the results presented so far remain robust to alternative ways of constructing the DUI (see Annex 4). The DUI is constructed using a set of variables that are arguably diverse. In particular, some of the variables used imply data use only indirectly. Hence, in order to convince the reader and better understand the robustness of the index, a sensitivity analysis was done on DUI. The sensitivity analysis assesses the robustness of the DUI by including/excluding three sets of variables: i) excluding those variables that imply data use only indirectly; ii) excluding those variables that measure data use based on the respondents own assessment; and iii) the third sets of estimates include only objective measures of data use. The estimates are presented in Annex 4. 5. Conclusion and Way Forward The current study provides the first empirical measure of the level of data use in the health sector. It is among the few that have tried to measure data use in the health sector. Though it is limited in its geographic coverage, it provides insight into the level of data use at the different levels of government. Data use in the health sector is very low. The average score of the respondents is below the half point mark. This average hides a considerable variation among central, state, district and below levels. Data use is lowest at district and state levels and highest at sub- district and central levels. A number of factors contribute to such low level of data use. The factors range from organizational to behavioral to skills. At organizational level, a work culture that is less focused on results, and resource allocation decisions that are based on normative practices provide little incentives for evidence-based decision making. At individual level, impediments include lack of skills to analyze and use data, absence of incentives for data use and lack recognition for performance at work place. The next paragraphs present three sets of interventions to improve data use. The findings clearly suggest several effective interventions including: (a) setting performance indicators, (b) creating incentives and accountability; and (c) skills development and capacity building. (a) Setting performance indicators. Setting clear sets of performance indicators and making them widely known promotes evidence-based decision making. A common set of indicators upon which performance is measured promotes accountability and transparency. Making sure that managers know the performance indicators of NRHM has proven to promote evidence-based decision making in its implementation. Likewise, setting clear indicators for all other programs and ensuring managers' performances are measured against those indicators will improve data use. This may mean setting indicators at 19 various levels including at sectoral, program, and even departmental level and ensuring that they are widely known among managers and implementers at all levels. Such an exercise may mean establishing a common understanding about the objectives of the sector, program or department. It involves standardization of data collection formats, guidelines on tabulation and calculation of specific indicators, etc. (b) Incentives and accountability. Incentives and accountability are central to enhancing evidence-based decision making. The findings show that data use is significantly higher among employees who have received explicit recognition from employers. Such incentives address motivational factors which otherwise undermine evidence-based decision making. Those who are regularly assessed on their performance and held accountable have significantly higher levels of data use. This requires a change in management style where managers assess the performance of their staff on a regular basis and explicitly recognize good performers. It also requires transparent performance measures reflecting the shared understanding of managers and staff. Finally, budgetary processes that link resource allocation to performance through performance based financing would promote evidence-based decision making. (c) Skills development and capacity building. Trainings and building capacity create enabling environments for evidence-based decision making. The results revealed little association between formal level of education and data use. Those with higher levels of formal schooling had statistically similar levels of data use as those with lower levels of education. Providing managers with formal schooling or hiring managers with advanced degrees will do little in the way of enhancing data utilization. On the other hand, trainings that are focused on practical issues that managers care about do improve data use. Trainings in data analytical and interpretation skills; in-service trainings in health management, etc. are more useful. Motivating existing managers to go through these trainings and attracting new mangers with these skills will go long way to enhance evidence based decision making. Capacity building goes hand in hand with training. Those who have access to computers and reported using them have higher levels of data use. Providing the hardware and software would facilitate data storage, analysis and retrieval system; it improves data quality and facilitates standardization of data collection, tabulation and reporting formats. It also enables the optimum utilization of skills acquired through trainings. Finally, it should be stressed that training and capacity building provide the necessary ground for evidence-based decision making. 6. Update on progress made by MOHFW The preliminary results of the study were shared with representatives of the MOHFW, development partners and other stakeholders at a national consultation held on July 12, 20 2008 in New Delhi. The comments received during this consultation have been incorporated in final draft. The MOHFW has taken into account the findings and recommendation made by the study and has since rolled out an improved Health Management Information System (HMIS) throughout the country. This HMIS has rationalized the indicator list and has built in features towards improving data quality. The HMIS is operational at the district level and data is now being received from almost all districts (over 600). The system will gradually be extended downwards to the block and individual facility levels. The MOHFW is planning to develop and roll out a large scale training program to sensitize program managers at all levels in the entry, analysis and use of data for program planning and implementation with the assistance of the National Health System Resource Centre (NSHRC) and the development partners. The information available on the HMIS portal is in the public domain and the MOHFW is continuously sensitizing the states on the importance and usage of information related to the Health Sector available through the HMIS and other sources. 21 Annex 1: Health indicators in the states of Rajasthan, UP and Maharashtra Indicator All India Uttar Pradesh Rajasthan Maharashtra CPR (%) 56 26.7 54.0 62.6 ANC( first 43 25.1 32.7 61.6 visit) % Institutional 41 24.5 45.5 63.6 Delivery (%) Fully 44 30.3 48.8 74.0 immunized children (%) Note: Data for the states is from DLHS-III (2007-08) and data for all India is from NFHS-3 (2005-06) 22 Annex 2: Sample Selection 1. Selection of Districts The selection of districts was based on the Composite Index4 for each district. The Composite Index is based on the following indicators: 1. Female Literacy 2. Male Literacy 3. Gender disparity in literacy 4. Sex ratio 0-6 population 5. Proportion of Births of Order 3 and above 6. % Girls Married below 18 years 7. % current user of family planning 8. Coverage of Safe Delivery 9. Coverage of Complete ANC 10. Coverage of Complete Immunizations Table of Selected Districts Maharashtra Uttar Pradesh Rajasthan District Index District Index District Index Jalna 47.14 Shravasti 17.49 Jaisalmer 21.24 Gadchiroli 61.56 Moradabad 31.54 Bharatpur 35.32 Sindhudurg 73.83 Kanpur 58.13 Bhilwara- 44.99 Nagar Mumbai 78.63 Lucknow 50.67 Jaipur 41.68 4 Ram F, Mohanty SK. State of Human Development in states and districts of India. International Institute for Population Sciences, 2003 23 2. Selection of Respondents Number of Sites Sampled State State District Urban CHCs PHCs Sub Total Offices Offices Health Centers Sampled Centers Rajasthan 1 4 4 4 4 4 21 Uttar Pradesh 1 4 4 4 4 4 21 Maharashtra 1 4 4 4 4 4 21 Central 1 1 Total 64 Number of Respondents per District District Urban CHCs PHCs Sub Total Offices Health Centers Centers 11 1 1 1 1 15 Total Number of Respondents State State District Urban CHCs PHCs Sub Total Offices level Health Centers Sampled facilities Centers Rajasthan 20 44 4 4 4 4 80 Uttar 20 44 4 4 4 4 Pradesh 80 Maharashtra 20 44 4 4 4 4 80 Sub total 60 132 12 12 12 12 240 Central Level 30 Total 270 24 Annex 3: Questionnaire for India Health Data Use Study 1. ID ------/------/------/------ 2. (FOR CENTRAL USE) READ: This survey is part of a World Bank Study being conducted at the request of the Government of India. The objective of this is to improve decision making in the health sector through the use of data. Please express your opinion with honesty. Your responses will remain confidential and will not be shared with anyone, except presented in tables where your responses have been tabulated with others who are participating in this survey. We appreciate your assistance and co-operation in completing this study. Read above paragraph word for word. Are you willing to participate? No 1 Yes 2 IF "NO" STOP INTERVIEW AND THANK RESPONDENT. Before beginning the interview, confirm if the respondent is willing to participate in the interview. If the person is not willing to participate, stop the interview and circle code 1 for "NO" then thank him or her for their time. If s/she is willing to participate, circle code 2 for "YES" and continue with the interview. No. Questions Coding categories I.1 Name of office or facility Write in the name of the office or facility where the respondent is working. I.2 District: Write the name of district as given to you by your field coordinator. I.3 State: Write the name of the state in which the respondent is working. 25 Levels : I.3a Level: Central ........................................1 State ...........................................2 District ........................................3 Block level-CHC/PHC ...............4 Subcenter ....................................5 First, circle the code which corresponds to the level at which the respondent works. Programs : I.3b Do you work in any vertical disease control HIV/AIDS ..................................1 program TB...............................................2 Malaria .......................................3 Leprosy.......................................4 Blindness ....................................5 Others(Specify) ..........................6 None ...........................................7 . Secretary .....................................1 I.4 What best describes your current title? Additional/Joint Secretary .......... (what is the nature of your daily work ) Director General .......................2 Director ......................................3 Additional Director ....................4 Joint Director..............................5 Deputy Director..........................6 Principal/Dean ...........................7 Chief Medical Officer (CMO)....8 Deputy CMOs ............................9 Hospital/Medical Super- Intendent...................................10 District Health Officer................11 Program Officers ........................12 Medical Supervisor ....................13 PHC Medical Officer .................14 LHV/ANM/MPW ......................15 Statistical Assistant/Computer at district level ..............................16 Statistical Assistant/Computer at block.level ................................17 Others (specify) In the right-hand column circle the correct code that best describes the nature of the respondent's daily work. For the "Other" response option, allow respondent to specify their title and record the answer verbatim in the space provided. 26 How long have you been in your current I.5 job? Years Write the number of years the respondent has been in their current position. If less than one year, record as "less than one." What is the total number of years you have I.6 worked in the health system? Years Write the number of years the respondent has worked in the health system. If less than one year, record as "less than one". How many more years do you expect to stay in I.7 your current position? Years Write the average number of years that a person in the same position has worked in the health system. If less than one year, record as "less than one." No Yes I.8 a. Secondary 1 2 b.Intermediate 1 2 c. Bachelors 1 2 d.Masters 1 2 What is your level of education? e. Master in Public Health 1 2 1 2 (Multiple responses allowed) f. MBA 1 2 g.MBBS 1 2 h.MD/MS 1 2 i. Post Graduate Diploma in Medicine 1 2 j. PhD 1 2 k.Others (specify) Circle the appropriate code for each response option. I.9 Have you ever worked in an NGO/the No 1 private sector? Yes 2 IF "NO", SKIP TO QUESTION II.1 I.10 If Yes, in what position? I.11 If Yes, for how long (in years)? For question I.9, if the respondent answers "No" then skip to question II.1. If the respondent answers "Yes" to question I.9, record the position in which they work in the space provided for I.10 and record the duration (in years) in the space provided for I.11. 27 I.12 Do you feel that there are major No 1 differences in program monitoring in Yes 2 public and private sector such as IF "NO", SKIP TO QUESTION II.1 supervision, service delivery etc? I.13 If Yes, can you specify what are major differences between the two sectors 1. (Max up to 2 answers): 2. SKIP SECTION II. FOR SENIOR POLICY LEVEL INTERVIEWS SECTION II. TRAINING AND CAPACITY IN DATA AND INFORMATION PRESENTATION II.1 In general do you feel you have the skills Yes ............................................. 1 necessary to use data and information in order t No............................................... 2 help with the kinds of decisions that you are involved in? II.2 Have you ever received any Types of training Yes No If yes, If yes, formal training in the following areas? Duratio When (in week (year) If Yes, for how long and when? HMIS a) 1 2 Survey b) 1 2 Data analysis c) 1 2 Data utilization 1 d) 2 Planning e) 1 2 other data related 1 f) 2 (specify)______ ___ _____________ Read the question and each response option one at a time circling "Yes" or "No" for each type of training received. If the respondent answers "Yes" to any of the questions, then ask how the long the training lasted and record the duration in number of weeks. Also ask when the training occurred and record the date as a four-digit number (year) in the last column. II.3 What kind of further training would you or Circle the response your staff benefit from? A Statistical analysis ............................... 1 2 3 B Understanding current HMIS data ...... 1 2 3 Legend C How to communicate and present You ............................................ 1 Data ..................................................... 1 2 3 28 Staff............................................ 2 D How to insure good quality data ......... 1 2 3 Don't need.................................. 3 E Using data base software ..................... 1 2 3 F Other (specify) __________________1 2 3 (Multiple response possible) II.4 Have you ever had any formal training in Epi Info ...................................... A computer software packages like: STATA....................................... B SPSS........................................... C (Multiple response possible) SAS ............................................ D Power Point ................................ E Excel .......................................... F Data base systems (e.g.. Access) .................................... G None ........................................... H II.5 Have you received in-service training in health Yes .......................................................... 1 Skip to II.7 Management No............................................................ 2 II.6 If yes, when and for how long did you receive? When (year)?_____________________ How long (in days)?______________ SKIP FOR SENIOR POLICY LEVEL INTERVIEWS SECTION II.A USE OF COMPUTERS S.No Questions Code Category Skip II.7 Now we would like to know about the Yes .......................................................... 1 computers you have. Do you have computers in No............................................................ 2 your unit? II.8 Do you use computers? Yes .......................................................... 1 No............................................................ 2 II.9 We would like to know how frequently you use a computer for certain tasks. Please tell us how frequently you use a computer for: Circle the response A Word processing ......................... 1 2 3 4 Legend B. Email and Internet ...................... 1 2 3 4 Daily ................................................ ...... 1 C. Data analysis .............................. 1 2 3 4 Once or more per week but NOT daily .. 2 D. Presentation of data and Less than once per week .................. ...... 3 Information.. ......................... .... 1 2 3 4 Never ............................................... ...... 4 II.10 What are the difficulties you are facing in using No access to computers ........................... A a Do not have enough computers............... B Computers? No training in computer. ......................... C Computers are old and are in continual (Multiple response possible) need of service .................................... D Computer speed is too slow .................... E Problem in power supply ........................ F Others (Specify) ...................................... G 29 II.11 Do the computers in your unit belong to your Belong to the unit .................................... 1 unit or are they rented? Rented on a contract basis....................... 2 Do not have any ...................................... 3 II.13 Do you have any provision for maintenance Yes .......................................................... 1 on a regular basis? No............................................................ 2 II.14 Do you have a full time IT person who Yes .......................................................... 1 is available? No............................................................ 2 II.15 Do you have a contract with an IT firm or Yes .......................................................... 1 consultant? No............................................................ 2 SKIP FOR SENIOR POLICY LEVEL INTERVIEWS SECTION II.B SKILLS In general, do you feel you have the skills No ............................................................ 1 II.B necessary to use data and information in Yes .......................................................... 2 order to help with the kinds of decisions that you are involved in? Circle the appropriate code for the response given. 30 FOR THE NEXT SET OF QUESTIONS, GIVE THE RESPONDENT APPENDIX A AND READ THE INSTRUCTIONS BELOW. (Provide a separate sheet and a calculator if needed for the respondent. This section should be on a separate page to be filled in by the respondent without help) In order to better plan training on data utilization, it would be helpful if you could answer the following written questions to the best of your ability. The answers you provide are anonymous and the data will only be analyzed in aggregate. There is no identifying information linked to you personally. You don't have to answer these questions if you don't want. If you do agree, we appreciate your participation. II.B1 In March 2006, 2500 cases of measles were reported while in April, 2006 only 2000 were reported in Rajasthan. What was the % change in reported cases of measles between (please check the answer and circle one) March and April? Correct............1 % change ----------in reported cases Not correct........2 II.B2 A Program Officer in UP had received reports from only four out of eight districts and will (please check the answers and write one in be presenting results at a conference. From the space provided) the data below: Correct............1 District Measles coverage Target Pop. Not correct.......2 1 60% 4000 2 50% 3000 3 80% 6000 4 90% 8000 a) What is the average coverage in the 4 districts without considering target population? _______________ II.B3 On her way to the conference, the programme officer realizes that she also has the target Correct............1 population for each district. They are: 4000, Not correct........2 3000, 7000, and 10,000 for districts 1, 2, 3, and 4 respectively. Now what should she report as the average coverage in the four districts? _____________ II.B4 The cost distributions for polio coverage in 1st District two districts were as per the pie charts attached. Which district has incurred higher proportion on: a) Drug supplies____________ b) Staff salaries_____________ Drugs Salary Transport Training Others 31 2nd District (please check the answers and write one in the space provided) a) Correct ................................................ 1 Not correct .......................................... 2 b) Correct ................................................ 1 Not correct .......................................... 2 II.B5 A malaria program officer calculated that he had achieved 35% of his annual target houses sprayed by the end of October. Assuming that (please check the answers and write one) performance continues at the same rate what Correct .................................................... 1 percentage of his annual target will he have Not-correct.............................................. 2 accomplished by the end of the year (the fiscal year April-March)? % achieved _________________ II.B6 In a school of Block headquarters the growth of 100 600 children were to be monitored on monthly basis. The graph shows the cumulative 80 percentage of students actually monitored in a 60 year 2007. % 40 What is the number of children whose growth 20 are monitored by the end of 0 2007?_____________ a M u A S N D F ar pr O ct ul e M J J u e o e A a J Correct............1 Not correct........2 II.B7 A survey is carried out among randomly selecte a) Incidence rate ....................... 1 high risk behaviour groups on whether or not b) Prevalence rate ..................... 2 they are HIV positive. Out of 1000 individuals c) Case fatality rate ................... 3 interviewed, 50 were found HIV positive. The d) Attack rate ............................ 4 number 50/1000 referred to: II.B8 Using data from notified disease forms, a state a) Incidence rate ....................... 1 immunization officer calculated that there were b) Prevalence rate ..................... 2 450 new cases of measles in the last year in a c) Case fatality rate ................... 3 block population of 130,000. The rate d) Attack rate ............................ 4 450/130 000 is referred to as: 32 SECTION III. OUTSOURCING Read: The next set of questions that I would like to ask you is about outsourcing your data needs. Do you "outsource" any of your data needs to Response III.1 consultants or other organizations on a (Circle that applies) contractual basis for: A Data analysis .............................. 1 2 3 B Preparation of district plans ....... 1 2 3 Legend Never .......................................................1 Sometimes ...............................................2 Always ....................................................3 Read the question including the category (e.g., "data collection") followed by the response options (never, sometimes, always). In the right-hand column, circle the responses that apply. If you do outsource, why do you? Response III.2 READ OPTIONS (Circle that applies) A Data analysis .............................. 1 2 3 Legend B Preparation of district plans ....... 1 2 3 We don't have enough staff ....................1 We have the staff but they are not trained to do this ...............................................2 We prefer specialized groups to do this job because they do a good job .............3 Other reason (Specify)___________ ......4 Read the question including the category (e.g., "data collection") followed by the response options (we don't have enough staff, etc). In the right-hand column, circle the responses that apply. How satisfied are you with the outsourcing of: III.3 READ OPTIONS Response (Circle that applies) Legend A Data analysis.............................. 1 2 3 Not satisfied at all ................................... 1 B Preparation of district plans ....... 1 2 3 Somewhat satisfied ................................. 2 Very satisfied .......................................... 3 33 Read the question including the category (e.g., "data collection") followed by the response options (not satisfied at all, somewhat satisfied, etc). In the right-hand column, circle the responses that apply. If you are not satisfied No Yes III.4 please tell us why. A. Work is of poor quality 1 2 READ OPTIONS B. Work is not performed on time 1 2 C. Work is not performed to 1 2 specifications D. Too expensive 1 2 E. Other [Specify] _____________ 1 2 Circle the appropriate codes for the response given. SECTION IV. DECISION MAKING AND USE OF INFORMATION READ: Now we would like to know about the ways in which you use data especially in making decisions IV.1 Now we would like to ask you about the kind No ............................................................1 of decisions that you make or influence. Yes ..........................................................2 Are you involved in making or influencing IF NO, SKIP TO IV.4 OTHERWISE any kind of decision? CONTINUE For question IV.1, if the respondent answers "No" then skip to question IV.4. otherwise continue to IV.2 SKIP FOR SENIOR POLICY LEVEL A. Day-to-day program management ......1 2 3 4 INTERVIEWS B. Medical supply and drug Management .......................................1 2 3 4 If yes, then we would like to know the C. Formulating plans ...............................1 2 3 4 extent that data are used in making the D. Budget preparation .............................1 2 3 4 decision in the following categories E. Deciding budget reallocation ..............1 2 3 4 F. Human resources management ..........1 2 3 4 Legend G. Monitoring key objectives .................. 1 2 3 4 Don't influence or make this decision.....1 H. Identification of emerging issues such as Always use data.......................................2 emerging epidemics ...........................1 2 3 4 Sometimes use data .................................3 Never use data .........................................4 If the respondent answered "yes" to IV.1, read the question from IV.2 including the category (e.g., "day-to-day program management") followed by the response options (don't influence or make this decision, etc). In the right- hand column, circle the responses that apply to each category. (APPENDIX B) 34 IV.3 SKIP FOR SENIOR POLICY LEVEL Politicians (General ......................... 1 ..... 2 3 4 5 INTERVIEWS Panchayat Leaders .......................... 1 ..... 2 3 4 5 Panchayat Samiti (block leaders) .... 1 ..... 2 3 4 5 For the following categories of decisions, Zila Parishad (district leaders) ........ 1 ..... 2 3 4 5 which other parties (stakeholders) are MLA/MP ......................................... 1 ..... 2 3 4 5 influential in decisions that you are involved Commercial sector .......................... 1 ..... 2 3 4 5 with: Indian civil society .......................... 1 ..... 2 3 4 5 Legend Community groups.......................... 1 ..... 2 3 4 5 Very influential.......................1 International NGOs ......................... 1 ..... 2 3 4 5 Influential................................2 International donors ........................ 1 ..... 2 3 4 5 Somewhat influential..............3 Higher level administrations ........... 1 ..... 2 3 4 5 Rarely influential....................4 Others specify ................................. 1 ..... 2 3 4 5 Not influential at all...............5 Read the question including the category (e.g., day to day program management) followed by the response options. For each category, circle the number that corresponds to the option in the legend. IV.4 Do you have access to the Yes No DK Rating following types of data and Types of data information? HMIS ...............................................A 1 2 3 1 2 3 READ RESPONSE State/District department reports .....B 1 2 3 1 2 3 OPTIONS Diseases surveillance.......................C 1 2 3 1 2 3 Sample Registration System............D 1 2 3 1 2 3 Please also rate each type Vital registration system. .................E 1 2 3 1 2 3 of data as to its Population census ............................F 1 2 3 1 2 3 usefulness to you. National surveys like the NFHS ......G 1 2 3 1 2 3 District level surveys (e.g., 1= not useful or used by DLHS/RCH). ................................H 1 2 3 1 2 3 me, National Sample Survey 2= somewhat useful, Organisation ..................................I 1 2 3 1 2 3 3 = very useful Specially commissioned survey ......J 1 2 3 1 2 3 Financial reports ..............................K 1 2 3 1 2 3 Published research ...........................L First, read the question then read the list of options one at a time. Circle "yes" or "no" for each type of data. Second, ask the respondent to rate each type of data from 1 to 3 where 1=being not useful or used by me, 2=somewhat useful, and 3=very useful. Read each type of data and circle the number corresponding to the respondent's response. 35 IV.5 Do you or your staff use any of the following to analyze and present data? Epi Info ......................... 1 2 3 STATA ......................... 1 2 3 Legend SPSS ............................. 1 2 3 You.......................................................... 1 SAS ............................... 1 2 3 Staff ......................................................... 2 Power Point .................. 1 2 3 Don't use ................................................. 3 Excel ............................. 1 2 3 Data base systems (Multiple response possible) (e.g., Access) .............. 1 2 3 Other (specify) .............. 1 2 3 IV.6 In your current position, do you feel data are used for Yes ................................................ 1 program monitoring? No ................................................. 2 2 IV.7 If yes, for which program? A. Family planning acceptance Related .................................... 1 2 Yes .......................................................... 1 B. Maternal health related ............ 1 2 No............................................................ 2 C. Child health related .................. 1 2 D. Disease specific ....................... 1 2 E. Patient related (consultation?) .. 1 2 IV.8 Do you use any kind of data for Yes ................................................ 1 management purposes? No ................................................. 2 IV.9 For what purpose you made use of the data? Monitor my work ....................... A (Multiple response possible) Monitor others work .................. B Monitoring programme output .. C Budgeting .................................. D Modification in program strategy ................................... E Plan preparation ......................... F Others (Specify)......................... G IV.10 Do you prepare reports that involve data? Yes ................................................ 1 No ................................................. 2 IV.11 In your report do you frequently Yes ................................................ 1 use data? No ................................................. 2 IV.12 For the year 2007, do you know the following in your a)________________ catchments area (facility/block/district/ b)________________ state as appropriate)? (we want to know if he recalls without checking his records) a. DPT3 coverage? 36 b. TB case detection rate IV.13 For the year 2007, what are the recorded figures of a) ________________ following in b)________________ your catchments area (facility/block/district/state as appropriate) (we want to know the actual recorded figures on his records) a. DPT3 coverage? b. TB case detection rate IV.14 Are there graphs/charts displayed in Yes ................................................ 1 the office? No ................................................. 2 (we want to know if there are displays) IV.15 Is the data used for the graphs/charts up to date?? Yes ................................................ 1 (we want to know whether the data used include the las No ................................................. 2 completed month/year/quarter as the case may be) IV.16 Are there tables displayed in the office? Yes ................................................ 1 No ................................................. 2 IV.17 Is the data used in the tables up to date?? Yes ................................................ 1 (we want to know whether the data used include the las No ................................................. 2 completed month/year/quarter as the case may be) IV.18 For the following health areas what are the two most A important indicators you use to judge progress? B (Please Provide up to two indicators for each) C Primary health care (General) ................. A Maternal health ....................................... B D Child health ............................................. C Family Planning ...................................... D Tuberculosis (TB) ................................... G E Malaria .................................................... H F G H IV.19 During 2007 What was your performance on the first Primary health care listed indicator for? (General)__________ ( we want to know if he recalls without checking his Maternal records) health_____________________ 37 IV.20 During 2007, what was your performance on these Primary health care indicators? (General)__________ Maternal (we want to know the actual recorded figures on his health_____________________ records) IV.21 Do you know any performance indicators for Yes ................................................ 1 NRHM ? No ................................................. 2 IV.22 From your perspective, what are 1. _______________________________ the three most important 2. ________________________________ performance indicators for NRHM? 3. ________________________________ IV.23 Have you prepared any district NRHM Yes ................................................ 1 plan? No ................................................. 2 IV.24 If yes, who prepared it? Your office................................. 1 Hired consultant......................... 2 Prepared by the state .................. 3 Other (specify) ........................... 4 Not applicable ............................ 5 IV.25 In terms of the data and information that you A Raw data on the computer ..... A use, in what format do you usually receive it? B Reports/completed paper (Circle all that apply): Data forms ............................. B C. Summary reports................... C D. Graphs and charts ................. D E. Others (specify) ..................... E IV.26 Please rate the stated each types of data as to its Ratings usefulness to you. A Raw data on the computer .. 1 23 9 (multiple response possible) B Reports/completed paper data forms ................................... 1 2 3 9 Legend C. Summary reports ................ 1 2 3 9 Very useful ................................. 1 D. Graphs and charts .............. 1 2 3 9 Somewhat useful ........................ 2 E. Others (specify) .................. 1 2 3 9 Not useful ................................... 3 Not applicable ............................ 9 IV.27 Do you have management meetings Yes ................................................ 1 at which decisions are made with your subordinates? No ................................................. 2 IV.28 When was the last one? Within last week ........................... 1 Within last month ......................... 2 Within last quarter ........................ 3 More than 3 months ago ............... 4 IV.29 Do these meetings involve Yes ................................................ 1 reviewing the performance of your unit? No ................................................. 2 IV.30 Are data presented at the meetings? Yes ................................................ 1 No ................................................. 2 38 IV.31 During these meetings how often Never ............................................ 1 are data used to support decisions? Sometimes .................................... 2 Frequently ..................................... 3 Always .......................................... 4 IV.32 Are there an agreed set of Yes ................................................ 1 indicators used? No ................................................. 2 IV.33 What are the three most important indicators used in 1. reviewing performance? 2. 3. IV.34 Are there any follow up of the decisions at the meeting? Yes ................................................ 1 No ................................................. 2 IV.35 What kind of follow-up is done of Submission of progress on the review meetings? specific time............................... 1 Supervisor visits to see performance ............................... 2 Progress is reviewed in next meeting ...................................... 3 Reminders are sent........................ .4 Action is taken for non- performance ............................... 5 Others (Specify)____________ IV.36 Do you have regular management meetings at which Yes ................................................ 1 decisions are No ................................................. 2 made with your superiors? IV.37 When was the last one? Within last week ........................... 1 Within last month ......................... 2 Within last quarter ........................ 3 More than 3 months ago ............... 4 IV.38 Do these meetings involve Yes ................................................ 1 reviewing the performance of your unit? No ................................................. 2 IV.39 Are data presented at the meetings? Yes ................................................ 1 No ................................................. 2 IV.40 During these meetings how often are data used to Never ............................................ 1 support a decision? Sometimes .................................... 2 Frequently ..................................... 3 Always .......................................... 4 IV.41 Are there an agreed set of Yes ................................................ 1 indicators used? No ................................................. 2 IV.42 What are the indicators used in reviewing performance? 1. 2. 3. IV.43 Are there any follow up of the decisions at the meeting? Yes ................................................ 1 No ................................................. 2 39 IV.44 What kind of follow-up is done of the review meetings? Submission of progress on specific time ............................................ 1 Supervisor visits to see per- formance .................................... 2 Progress is reviewed in next meeting ...................................... 3 Reminders are sent........................ 4 Action is taken for non- performance ............................... 5 Others (Specify)............................ 6 Civil Society Organizations IV.45 To what extent does civil society influence decisions in Not at all ....................................... 1 your area (such as state/district/block/facility)? Some what .................................... 2 Quite a bit ..................................... 3 IV.46 What are the main civil society groups that NGO ............................................. 1 Influence your decision? Swasthya Mahila Samities ............ 2 Youth groups ................................ 3 Community influentials ................ 4 Religious leaders........................... 5 IV.47 How much contact do you or your Never ............................................ 1 organizations have with civil society groups? Infrequently .................................. 2 Regularly ...................................... 3 IV.50 What is the nature of your interaction? Occasional meetings as needed .... 1 Frequent meetings......................... 2 Regular meetings .......................... 3 IV.51 Have you ever heard of "results based Yes ................................................ 1 Financing?" No ................................................. 2 IV.52 If yes, please describe it in you own words? IV.53 Do you know of any examples of this in Yes (SPECIFY) ............................ 1 India? No ................................................. 2 IV.54 How would you feel about it if "results I would be against it...................... 1 based financing" were introduced into your system? I would be indifferent ................... 2 I would be enthusiastic about it .... 3 Don't know ................................... 4 IV.55 Have you ever received any kind of explicit recognition Yes ................................................ 1 for performance on the job? No ................................................. 2 IV.56 If Yes what kind? Verbal recognition from my (Multiple responses possible) superior ....................................... 1 Letter from my superior ................ 2 Certificate ..................................... 3 Cash reward or other material benefit ......................................... 4 40 IV.57 In some places incentive systems have been implemented No, I would be strongly against it 1 to encourage better Yes, I would support it ................. 2 performance by health personnel. Yes, I would be enthusiastic about it ........................................ 3 A. Would you support introducing employee recognition Don't know ................................... 4 programs as a system of incentives in your organization? B. Would you support introducing cash No, I would be strongly against it. 1 rewards to employees as a system of incentives in your Yes, I would support it ................. 2 organization? Yes, I would be enthusiastic about it ........................................ 3 Don't know ................................... 4 V. DATA QUALITY V.1 For each of the following please rate the quality on a scale of (1) poor to (5) excellent Very Some Some Good Excelle Do not poor what what nt know Poor good 1 2 3 4 5 6 Types of data A. Population census- B. National surveys like the NFHS 1 2 3 4 5 6 C. District level surveys like the DLHS/RCH 1 2 3 4 5 6 D. Sample Registration System 1 2 3 4 5 6 E. Vital registration system 1 2 3 4 5 6 F. National Sample Survey Organisation (NSSO 1 2 3 4 5 6 G. Diseases surveillance 1 2 3 4 5 6 H. Health Management Information System 1 2 3 4 5 6 J. Financial reports 1 2 3 4 5 6 K. State/District department reports 1 2 3 4 5 6 L. Specially commissioned surveys 1 2 3 4 5 6 M. Published research 1 2 3 4 5 6 V.2 In your work, are there systems in place to No ...................................................... 1 assure data quality from HMIS? Yes .................................................... 2 IF "NO", SKIP TO V.4 If the respondent answers "Yes" to question V.2, ask each question in V.3 V.3 If Yes, ask: No Yes A. Are data regularly checked for accuracy? 1 2 B. Are HMIS compared with other data sources (e.g. 1 2 surveys) 1 2 C. Are staff trained in data quality control? Read the question then read the list of options circling the appropriate response for each system of data quality. 41 V.4 Are there cases you avoided using data No ...................................................... 1 because of its poor quality? Yes .................................................... 2 IF "NO", SKIP TO SECTION VI V.5 If yes, specify:__________________ For question V.4, if the respondent answers "No" then skip to Section VI. If the respondent answers "Yes" to question V.4, ask him/her to explain why s/he thinks the data is of poor quality and record the response verbatim in the space provided for V.5. SECTION VI: BEHAVIOURAL FACTORS READ: In the next section, I would like you to read a series of statements and rate the extent to which you agree or disagree using a scale from 1 to 5 with 1=strongly disagree, 2=somewhat disagree, 3=neither agree nor disagree, 4=somewhat agree and 5=strongly agree. HAND RESPONDENT APPENDIX C VI.1 In your organization, Strongly Somewhat Neither Somewhat Strongly decisions are based Disagree Disagree agree nor Agree Agree on... disagree a. Personal liking 1 2 3 4 5 b. Superiors' directives 1 2 3 4 5 c. Evidence/facts 1 2 3 4 5 d. Political interference 1 2 3 4 5 VI.2 In your organization, 1 2 3 4 5 superiors a. Seek feedback from 1 2 3 4 5 concerned persons b. Emphasize data quality 1 2 3 4 5 in regular reports c. Are open to 1 2 3 4 5 alternative views d. Allow disagreements 1 2 3 4 5 before reaching a decision e. Explain what they 1 2 3 4 5 expect from workers VI.3 In your organization, 1 2 3 4 5 staff a. Are aware of their 1 2 3 4 5 responsibilities b. Are rewarded for 1 2 3 4 5 good work 42 c. Feel that promotion 1 2 3 4 5 is based on merit d. Rely on data for 1 2 3 4 5 planning and monitoring set target e. Are given 1 2 3 4 5 appropriate training for data activities f Facilities receive 1 2 3 4 5 timely monthly feedback on their submitted report Read the instructions prior to the series of questions. Hand the respondent Appendix D and allow him/her to rate the statements using a scale from 1 to 5 with 1=strongly disagree, 2=somewhat disagree, 3=neither agree nor disagree, 4=somewhat agree and 5=strongly agree. Respondent should record responses in the table. VI.4 Collecting information is appreciated 1 2 3 4 5 6 by Co-workers and superiors VI.5 Are you told that your performance will be evaluated based on the following criteria: a. Changes in service delivery 1 2 3 4 5 6 indicators such as immunization rates, FP, health service coverage rates, morbidity, etc.? b. Improvements in quality of care 1 2 3 4 5 6 c. Predetermined career advancement 1 2 3 4 5 6 criteria d. Work ethics/values are the basis for 1 2 3 4 5 6 evaluating performance. VI.6 To what extent do you feel satisfied with Very dissatisfied ................................ 1 your job? Somewhat dissatisfied ....................... 2 Neither dissatisfied nor satisfied ....... 3 Somewhat satisfied............................ 4 Very satisfied .................................... 5 43 READ: In the next section, I would like to you to read a series of statements and rate the level of agreement or disagreement of each issue using a scale from 1 to 5 HAND RESPONDENT APPENDIX D VII. PERCIEVED DATA USE ISSUES AND REMEDIES VII.1 Please rate the following potential Strongly Somewhat Neutral Somew Strongly Disagree Disagree hat Agree impediments to data use on a scale of agree a. There is little culture of looking 1 2 3 4 5 at outputs and outcomes b. There little agreement on what 1 2 3 4 5 the key indicators of performance are c. There is often data duplication 1 2 3 4 5 and confusion as to the real figures d. There is too much information 1 2 3 4 5 e. There is no analysis and 1 2 3 4 5 feedback from supervisors on data that are collected f. Government discussions are 1 2 3 4 5 based on political issues, not information g. There are no set criteria for data 1 2 3 4 5 collection and analysis h. No incentives for data 1 2 3 4 5 utilization; too much trouble i. Data are of poor quality 1 2 3 4 5 j. There is an unwillingness to 1 2 3 4 5 accept shortcomings in data k. There is a general lack of skills 1 2 3 4 5 to analyze and use data collected l. Collected data take too long to 1 2 3 4 5 reach at relevant level m. Use of data is only for keeping 1 2 3 4 5 records and not for program support/ monitoring. VII.2 Please rate the following strategies for Strongly Somewhat Neutral Somewhat Strongly Disagree Disagree agree Agree improving data use on a scale of a) Have independent data collection 1 2 3 4 5 and compare it to service (HMIS data) 44 b) Make results available to the 1 2 3 4 5 public c) Improve the timeliness of data 1 2 3 4 5 d) Implement simple to use 1 2 3 4 5 software for data presentation at all levels e) Establish uniform data reporting 1 2 3 4 5 /feedback systems across all levels f) Ensure that data reports are 1 2 3 4 5 available for all appropriate levels g) Encourage top level leadership to 1 2 3 4 5 use evidence-based decision making h) Provide incentives for results, 1 2 3 4 5 linked to performance measurement i) Ensure that data needs are 1 2 3 4 5 clearly identified and linked to needs at all levels j) Improve the quality of data 1 2 3 4 5 k) Provide training of health care 1 2 3 4 5 providers in the importance of data collection, analysis and use. l) Provide training for management 1 2 3 4 5 on the use of data for policy and program management m) Regular reviews of health sector 1 2 3 4 5 performance by politicians n) Regular reviews of health 1 2 3 4 5 performance by advocacy groups o) 1 2 3 4 5 Read the instructions prior to the series of questions. Hand the respondent Appendix D and allow him/her to rate the statements using a scale from 1 to 5 Respondent should record responses in the table. Thank the respondent: "Thank you for your participation!" Date ­ Duration of Interview- 45 Annex 4: Constructing the Indices and Sensitivity Analysis A. Constructing the Data Use Index (DUI) A composite index of data use was constructed based on a number of questions. These variables can be grouped in to two: (a) "Self assessment": these are sets of questions where the respondent is asked to assess herself about data use; and (a) "Objective assessment": these are sets of questions where the interviewer makes observation and asks specific questions meant to measure objectively the respondent's knowledge of his/her own performance. Self assessment: Respondents were asked to assess their own data use. First, respondents were asked a general question whether they used data to manage their programs. Those who responded affirmatively were asked subsequent questions regarding for what specific management purpose data is used. Second, respondents who were involved in report preparation were asked whether they used data in the preparation. Third, respondents were given sets of decision categories and they were asked to select the extent to which data is used in making those decisions. The choices range from "I do not make decisions in this category" to "always use data to make those decisions". The three self assessment questions correspond to questions IV.8, IV.11 and IV.2 of the attached questionnaire. The score is given in such a way than each question has one point. The total score then ranges between 0 and 3. Objective assessment: This includes two sets of questions. The first set involved looking at the register as well as soliciting response. In these sets of questions, interviewers were instructed to look the records of the respondent and complete the questions and ask the respondent related question and register the answer. These include four sets of questions and observations. First, respondents were asked to state their performance in DPT3, TB case detection rate, one primary health care coverage indicator and one maternal and child health indicator. After recording the response, interviewers looked at the registers and copied the figure from the register. During the analysis, the respondent is given a point for each record that matches.5 The maximum score one can get is 4. These questions correspond to questions IV.12, IV.13, IV.19 and IV.20 in the questionnaire. The second group involves observing the place of work to indentify signs of data use and soliciting response. Interviewers check if there are charts/graphs/tables etc. displayed in the office/facility/etc. and record the answer. Then check if the data used are up-to-date and record the answer. During analysis, a point is given to those who had charts/graphs/tables displayed and the data used in constructing them is up-to- date. If either the charts, etc. are not displayed or the data used are not current, then no point is given. Thus a maximum of two points can be scored. These questions correspond to IV.14, IV.15, IV6 and IV.17. 5 The figures do not necessarily have to match 100%. A 10% up and down is tolerated. 46 Finally, respondents were asked whether their unit holds management meetings, whether performances are discussed and whether data are presented to support the discussion. These questions were asked for meetings with subordinates as well as with superiors. If the answer is affirmative to all, a point is given. The total score ranges between 0 and 2. These questions correspond to questions IV.27, IV.29, IV.30, IV.36, IV.38 and IV.39 in the questionnaire. The maximum DUI score one can get thus the sum of the `objective' and `subjective' assessments which adds up to eleven. B. Sensitivity Analysis The DUI is based on a set of variables that are arguably varied. In particular, data use can only be indirectly inferred from some of the variables used to construct the DUI. In order to convince the reader and better understand the robustness of the index, a sensitivity analysis was done on the DUI. The sensitivity analysis assesses the robustness of the DUI by including/excluding three sets of variables: (a) First those variables that imply data use only indirectly are excluded- these are presented as Group A estimates; (b) The second set of estimates exclude those variables that measure data use based on the respondents own assessment- these are presented as Group B estimates; (c) Finally, the third sets of estimates include only objective measures of data use- these are presented as Group C estimates. In what follows the tables for each of these estimates are presented. For ease of reference estimates are presented in order in which the tables appear in the text. 1. Mean DUI Table 1a. Estimates (excluding variable IV2) Mean (SD) Central 5.81 (2.38) State 4.40 (2.20) District 4.27 (2.24) Sub-district 5.23 (2.17) Table 1b. Estimates (excluding variables IV8 and IV11) Mean (SD) Central 4.16 (2.30) State 2.02 (1.86) District 3.10 (2.05) Sub-district 3.87 (2.02) 47 Table 1c: Estimates (Including objective measures only) Mean (SD) Central 4.00 (2.17) State 2.69 (1.89) District 2.75 (1.93) Sub-district 3.61 (1.99) 2. Mean DUI by knowledge of NRHM performance indicators Table 8a: Estimates (excluding variable IV2) Knowledge of NRHM Mean DUI (SE) Know NRHM Performance indicators 5.46 (.23) ttest: -4.78 and can recall at least one of them for Diff=0 ( p=0.000) Does not know of NRHM 3.97 (0.20) Performance Indicators and/or cannot recall correctly at least one of them Table 8b: Estimates (excluding variables IV8 and IV11) Knowledge of NRHM Mean DUI (SE) Know NRHM Performance indicators 4.03 (.21) ttest: -4.50 and can recall at least one of them for Diff=0 ( p=0.000) Does not know of NRHM 2.75 (0.18) Performance Indicators and/or cannot recall correctly at least one of them Table 8c: Estimates (including objective measures only) Knowledge of NRHM Mean DUI (SE) Know NRHM Performance indicators 3.70 (.22) ttest: -4.41 and can recall at least one of them for Diff=0 ( p=0.000) Does not know of NRHM 2.48 (0.17) Performance Indicators and/or cannot recall correctly at least one of them 48 3. DUI by type of training Table 9a: Estimates (excluding variable IV2) Type of Formal Training % had any training Mean DUI (SE) ttest for Diff=0 HMIS 74.6 Yes: 5.04 (0.33) -1.55 Survey 80.2 Yes: 5.43 (0.36) -2.56** Data Analysis 83.8 Yes: 5.69 (0.40) -2.98** Data Utilization 84.2 Yes: 5.51 (0.42) -2.44** Planning 85.2 Yes: 5.31 (0.44) -1.80* Table 9b: Estimates (excluding variables IV8 and IV11) Type of Formal Training % had any training Mean DUI (SE) ttest for Diff=0 HMIS 74.6 Yes: 3.69 (0.31) -1.54 Survey 80.2 Yes: 3.94 (0.34) -2.19** Data Analysis 83.8 Yes: 4.39 (0.37) -3.33** Data Utilization 84.2 Yes: 4.12 (0.40) -2.44** Planning 85.2 Yes: 3.85 (0.43) -1.55 Table 9c: Estimates (Including objective measures only) Type of Formal Training % had any training Mean DUI (SE) ttest for Diff=0 HMIS 74.6 Yes: 3.36 (0.31) -1.46 Survey 80.2 Yes: 3.69 (0.33) -2.42** Data Analysis 83.8 Yes: 4.06 (0.36) -3.35** Data Utilization 84.2 Yes: 3.77 (0.41) -2.36** Planning 85.2 Yes: 3.85 (0.43) -1.56 4. Mean DUI by use of computers Table 10a: Estimates (excluding variable IV2) Mean DUI (SE) Use Computers 5.09 (0.23) ttest: 3.25 for Diff=0 (p=0.002) Do not use computers 3.87 (0.30) Table 10b: Estimates (excluding variables IV8 and IV11) Mean DUI (SE) Use Computers 3.78 (0.22) ttest: 3.03 for Diff=0 (p=0.003) Do not use computers 2.74 (0.26) 49 Table 10c: Estimates (Including objective measures only) Mean DUI (SE) Use Computers 3.38 (0.22) ttest: 2.85 for Diff=0 (p=0.005) Do not use computers 2.44 (0.25) 5. Skills score and DUI Table 11a. Estimates (excluding variable IV2) Skills Test Score Mean DUI (SD) Low (1-3) 4.87 (2.38) Medium (4-6) 3.96 (2.38) High (7-9) 4.63 (1.67) Table 11b. Estimates (excluding variables IV8 and IV11) Skills Test Score Mean DUI (SD) Low (1-3) 3.48 (2.15) Medium (4-6) 2.81 (2.16) High (7-9) 3.37 (1.61) Table 11c. Estimates (Including objective measures only) Skills Test Score Mean DUI (SD) Low (1-3) 3.26 (2.12) Medium (4-6) 2.43 (1.95) High (7-9) 2.92 (1.49) 6. Mean DUI by educational attainment Table 12a: Estimates (excluding variable IV2) Highest Level of Education Mean (SD) Secondary/Intermediate 4.18 (2.81) Bachelors 4.00 (2.57) MPH/MSc/MA/MBA 3.98 (2.08) Post grad. Diploma in Medicine 5.00 (2.18) or MBBS or MD/MS PhD 4.42 (2.37) 50 Table 12b: Estimates (excluding variables IV8 and IV11) Highest Level of Education Mean (SD) Secondary/Intermediate 2.94 (2.48) Bachelors 3.07 (2.27) MPH/MSc/MA/MBA 2.65 (1.93) Post grad. Diploma in Medicine 3.63 (1.99) or MBBS or MD/MS PhD 3.20 (1.97) Table 12c: Estimates (Including objective measures only) Highest Level of Education Mean (SD) Secondary/Intermediate 2.82 (2.46) Bachelors 2.76 (2.14) MPH/MSc/MA/MBA 2.37 (1.81) Post grad. Diploma in Medicine 3.30 (1.96) or MBBS or MD/MS PhD 3.00 (1.83) 7. Mean DUI by in-service training in health management Table 13a: Estimates (excluding variable IV2) Mean DUI (SE) Received HM training 5.04 (0.26) ttest: 2.13 for Diff=0 ( p=0.034) Did not receive HM training 4.24 (0.25) Table 13b: Estimates (excluding variables IV8 and IV11) Mean DUI (SE) Received HM training 3.67 (0.26) ttest: 1.67 for Diff=0 ( p=0.096) Did not receive HM training 3.10 (0.22) Table 13c: Estimates (Including objective measures only) Mean DUI (SE) Received HM training 3.26 (0.25) ttest: 1.39 for Diff=0 ( p=0.166) Did not receive HM training 2.80 (0.21) 51 8. Mean DUI by explicit recognition for performance on the job Table 14a: Estimates (excluding variable IV2 Mean DUI (SE) Received explicit 5.00 (0.34) ttest: 3.87 for Diff=0 ( p=0.000) recognition for performance rho=0.26 (p=0.000) on the job Did not receive any explicit 2.99 (0.35) recognition for performance on the job Table 14b: Estimates (excluding variables IV8 and IV11) Mean DUI (SE) Received explicit 2.03 (0.18) ttest: 2.97 for Diff=0 ( p=0.003) recognition for performance rho=0.16 (p=0.025) on the job Did not receive any explicit 1.24(0.16) recognition for performance on the job Table 14c: (Including objective measures only) Mean DUI (SE) Received explicit 1.75 (0.17) ttest: 3.21 for Diff=0 ( p=0.002) recognition for performance rho=0.21 (p=0.004) on the job Did not receive any explicit 0.93 (0.16) recognition for performance on the job 9. Mean DUI by assessment of departmental performance during Management Meetings Table 15a: Estimates (excluding variable IV2) Mean DUI (SE) ttest for Diff=0 (p value) Management meetings with subordinates held at Yes: 5.20 (0.17) -6.42 least monthly and at which unit performance is No: 2.97 (0.29) (0.0000) reviewed. Management meetings with superiors held at Yes: 4.91 (0.17) -2.95 least monthly and at which unit performance is No: 3.66 (0.42) (0.0036) reviewed. 52 Table 15b: Estimates (excluding variables IV8 and IV11) Mean DUI (SE) ttest for Diff=0 (p value) Management meetings with subordinates held at Yes: 3.83 (0.16) -6.30 least monthly and at which unit performance is No: 1.82 (0.23) (0.0000) reviewed. Management meetings with superiors held at Yes: 3.55 (0.16) -2.79 least monthly and at which unit performance is No: 2.47 (0.36) (0.0058) reviewed. Table 15c: Estimates (Including objective measures only) Mean DUI (SE) ttest for Diff=0 (p value) Management meetings with subordinates held at Yes: 3.49 (0.16) -5.93 least monthly and at which unit performance is No: 1.64 (0.23) (0.0000) reviewed. Management meetings with superiors held at Yes: 3.24 (0.15) -2.62 least monthly and at which unit performance is No: 2.25 (0.35) (0.0096) reviewed. 10. Mean DUI by respondents' attitude towards introduction of RBF Table 16a: Estimates (excluding variable IV2) Mean DUI (SE) Enthusiastic about 5.23 (0.26) ttest: -1.92 for Diff=0 introduction of results- (p=0.0575) based financing Against or indifferent to 4.33 (0.39) introduction of results- based financing Table 16b Estimates (excluding variables IV8 and IV11) Mean DUI (SE) Enthusiastic about 3.80 (0.23) ttest: -1.99 for Diff=0 introduction of results- (p=0.0497) based financing Against or indifferent to 2.98 (0.33) introduction of results- based financing 53 Table 16c Estimates (Including objective measures only) Mean DUI (SE) Enthusiastic about 3.56 (0.22) ttest: -2.09 for Diff=0 introduction of results- (p=0.0395) based financing Against or indifferent to 2.73 (0.36) introduction of results- based financing C. Skills Score Respondents were given nine questions meant to test their ability in interpreting and manipulating simple data. The questions are in the attached questionnaire. Each of the nine questions were given equal weight and the maximum score a respondent can score is 9. 54