46617 WORKING PAPER NO. 6 Donors and planners often have unrealistic expectations about what can be achieved in a short time period and underestimate the challenges facing EMIS. Donors have often tended to assume that EMIS outputs would play an important role in helping managers to administer their education system in a more efficient and effective manner, while, governments have assumed that EMIS output would help them identify priority areas for targeting resources and helping them monitoring progress of Rethinking Education strategies towards defined objectives. A review of four Management Information country cases reveals that this Systems: does not always occur and significant problems are often Lessons from and Options experienced with the operation of EMIS at all levels of the for Less Developed education system, and in the vast majority of instances Countries systems are unsustainable without a considerable amount of donor support. Similarly, the utilisation and November 2006 dissemination of EMIS outputs are often lower than anticipated. WORKING PAPER NO. 6, 2006 About infoDev's working papers series: These short papers represent work commissioned by infoDev that has not been through infoDev's peer review and formal publication processes. infoDev makes such reports available as informal 'working papers' to share information or ideas about a given topic, in case doing so may be of any interest or utility to practitioners working in related fields. WORKING PAPER NO. 6, 2006 Rethinking Education Management Information Systems: Lessons from and Options for Less Developed Countries October, 2006 Marcus Powell, Cambridge Education1 Prepared by: Edited by: Michael Trucano (infoDev). 1Dr Theo Sparreboom, Mr Jim Shoobridge and Dr David Smith provided valuable support in collecting empirical data for the country case study reports upon which the present study is based. 2 Contents 1. Introduction................................................................................................................4 2. The purpose and changing functions of EMIS ......................................................7 2.1. The management and operation of EMIS ....................................................10 2.3 Utilisation and dissemination of EMIS outputs...........................................11 2.4 The important and changing role of ICT infrastructure...............................12 2.5 Institutional building and capacity development .........................................13 3. Lessons Learnt and Best Practice: The management and operation of EMIS in Bangladesh, Ghana. Mozambique and Nigeria............................................................14 3.1 Data collection and Analysis Processes.......................................................14 3.3 Utilisation of EMIS outputs.........................................................................17 3.4 Dissemination of Outputs ............................................................................19 3.6 Institutional Development and Capacity Building.......................................22 4. Re-thinking EMIS? ..............................................................................................25 5. Concrete steps to develop a responsive and sustainable EMIS ...........................26 5.1 Strategies for driving EMIS and the importance of commitment................26 5.2 Determining data needs and strategies for data collection...........................27 5.3 Choices about ICT infrastructure.................................................................29 5.4 Utilisation of EMIS outputs.........................................................................31 5.5 Dissemination of EMIS outputs...................................................................33 5.6 Institutional building and capacity development .........................................34 6. Conclusion .......................................................................................................35 References....................................................................................................................38 Appendix II: Template used to collect research data..................................................40 3 1. Introduction This study was commissioned by Info Dev as part of their ongoing programme to understand how the widespread use of ICT can help countries meet Education Millennium Development Goals (MDGs) and to provide policy makers and donors with a coherent understanding of EMIS performance, especially with regard to how data is collected, disseminated, utilised along with issues associated with managing these processes and transparency. Substantial resources have been invested in EMIS-related activities over the past two decades, yet limited evidence exists about what constitutes best practice and why certain systems are more effective than others. During the past four years the World Bank has supported over 40 projects containing an EMIS element (see: infoDev 2005; Haiyan, 2003). Understandably, donors and decision makers are concerned about how wisely resources have been spent and the degree to which outputs are responding to new information demands. Much of the existing evidence is anecdotal and comes from project documents; typical problems include considerable time delays in establishing the information systems, lack of internal capacity to utilise outputs and problems of long-term sustainability2. The present study builds on this work by providing analytical research on the EMIS experience of four different countries, identifying concrete steps that planners can take to improve the management and operation of their EMIS. Research for the present study took place between October 2005 and May 2006 using an eclectic approach, combining a literature review and four empirical case studies (Yin, 2002). Through literature review, relevant issues were identified and subsequently refined at a workshop held with programme officials in Washington. Following the workshop a research template was developed to guide the data collection processes in the following country case studies: Bangladesh, Mozambique, Nigeria and Ghana3. This resulted in the production of four country case study reports, the contents of which formed the basis for the present analysis. The four countries selected met two sets of criteria: first that they are tracking progress towards MDGs or receiving FTI funds, and second that researchers could have an adequate degree of access to education officials and project documents, evaluations and studies. Complementing the country case studies, a number of interviews were held with experts who had supported EMIS developments in other parts of the world, allowing our study to identify common problems and innovative solutions being implemented elsewhere. There are four sections to this paper, the contents of which are guided by the analytical approach outlined in Table 14. The first section focuses on the purpose and functions of EMIS. Essentially, an Education Management Information System is a data collection, storage, retrieval, processing and dissemination system that is 2There are a number of papers which helped inform the present study (see: Hua and Herstein, 2003; Cassidy, 2005). 3A copy of the research template can be found in annex I 4An analytical approach provides the mechanism for linking issues to be investigated, with the production of the final results 4 specifically designed for use by decision makers and administrators to plan and manage education systems in an efficient, effective and sustainable manner (AEPM, 2007). Normally, EMIS is an information centre in a Ministry of Education and responsible for the use of information for policy planning, planning and implementation, decision making, monitoring and evaluation of the education of the education system.(Wako, 2003). Having defined the purpose and location of EMIS, the second section identifies important issues for managing and operating an EMIS, including strategies for data collection and analysis, the significant role that ICT infrastructure can play in supporting these processes and the imperative of ensuring that outputs are disseminated and utilised. Then, the third section investigates the degree to which such practices are occurring in Bangladesh, Ghana, Mozambique and Nigeria, and what lessons can be learnt from the way each manages and operates their EMIS, including what difficulties were experienced and how these were overcome. Finally, we elaborate ways forward` for EMIS and concrete steps that can be taken by donors and planners to ensure greater sustainability. 5 Table 1: Analytical Approach Guiding EMIS study Theoretical issues to be investigated5 Empirical Evidence from case studies6 Concrete Steps Forward7 The purpose and changing functions of Data collection and functions of EMIS Strategies for driving EMIS and the EMIS importance of commitment Utilisation of EMIS outputs The management and operation of Determining data needs and strategies EMIS Dissemination of outputs for data collection The management of data collection The role played by ICT in supporting Choices about ICT infrastructure and analysis EMIS Capacity development and institutional Utilisation and dissemination of EMIS Institutional development and capacity building outputs building Utilisation of EMIS outputs The importance and changing role of Re-thinking EMIS ICT infrastructure Dissemination of EMIS outputs Don`t neglect institutional building and capacity development 5 The theoretical issues to be investigated were obtained from the study`s initial terms of reference and existing literature on EMIS. These were organised to address EMIS issues in a practical manner, particularly with regard to the conditions necessary for the effective and efficient management of an EMIS. 6Having defined the conditions necessary for best practice, the study investigated the degree to which such practices occur in developing countries. This was achieved through undertaking empirical case study research in Bangladesh, Ghana, Nigeria and Mozambique. The content of case studies enabled us to understand the comparative differences between each of the countries. 7The third stage, the production of results, involved a review of our empirical findings in the light of the issues raised in the theoretical approach and identifying concrete steps forward, particularly in relation to how EMIS could operate more effectively and efficiently. 6 Our research found that donors and planners had unrealistic expectations about what could be achieved in a short time period and underestimated the challenges facing EMIS. Donors tended to assume that EMIS outputs would play an important role in helping managers to administer their education system in a more efficient and effective manner. While, governments assumed that EMIS output would help them identify priority areas for targeting resources and helping them monitoring progress of strategies towards defined objectives. However, this did not always occur and significant problems were experienced with the operation of EMIS at all levels of the education system, and in the vast majority of instances systems were unsustainable without a considerable amount of donor support. Similarly, the utilisation and dissemination of EMIS outputs were lower than anticipated. Poor performance and lack of sustainability, combined with a low utilisation of EMIS outputs, suggests a need to re-think some of the assumptions upon which EMIS are based. In the past donors assumed that a top down approach to EMIS, starting with the information requirements of the Ministry, would respond to the needs of most developing countries. This assumption is probably derived from the EMIS experiences of North America and Western European countries where the development characteristics are totally different to those in the developing world. Donors must move away from the perspective of one-approach fits all contexts. Clearly, this is wrong and developing countries require EMIS strategies that respond to their own specific characteristics and needs. Existing approaches to EMIS also tend to focus on systems and procedures for data collection, including technical issues associated with ICT infrastructure, but neglect institutional building and capacity development. Unless the latter processes are given sufficient attention, it will be difficult for countries to encourage a sustainable demand for EMIS outputs. Nevertheless, it must also be remembered that it takes a considerable amount of time and commitment to develop an organisational culture that supports data sharing, dissemination and utilisation, a fact often neglected by donors and their partners. Despite the fact that significant difficulties were experienced in the setting-up of EMIS, it is not all bad news. In the majority of instances governments in the four countries studied were developing innovative strategies to deal with the problems associated with the management and operation of their EMIS. Often this involved rejecting initial approaches and developing strategies based on their own needs, all of which provides valuable lessons for other developing countries who are reforming their EMIS. 2. The purpose and changing functions of EMIS A key question facing government and decision makers, especially in the light of increased competition and resource constraints, is how to maximise individual learning opportunities and encourage human capital formation using minimal resources. In order to achieve these objectives decision makers must understand how resources are translated into learning outcomes, particularly with regard to the efficiency and effectiveness of existing processes. EMIS can play an important part in these processes by providing analysts and decision makers with the information to understand how educational inputs are transformed into educational outputs. Access to quality and timely data can improve decision-making and ensure that resources are targeted at areas in most need and where returns will be highest. Conversely, a lack of 7 education data can act as a constraint on decision-making and lead to poor resource use. However, it is important to understand that the demands for information are not static or limited to measuring certain variables. Policy makers are under pressure to respond to new policy demands and it is important that they have the appropriate information to make informed decisions. EMIS must respond to such demands and those working in this area must understand how this impacts on the demands for information, particularly in relation to the following: Changes in the nature and objectives of education policy ­ Until recently, EMIS was primarily concerned with providing information on education inputs, such as the number of schools, enrolment levels and the number of teachers. However, as a result of the recent drive towards increased transparency, combined with the need to demonstrate value for money, information systems are increasingly required to produce more complex information about educational processes and outputs, resource utilisation, the effectiveness of new learning techniques and the responsiveness of existing provision. The demand for more analytical information has been given momentum following EFA and MDG commitments which require countries to gather data which measures equity and quality across schools in different regions. Decentralisation of education provision ­ In the past, EMIS provided information to support planning processes at the national level. However, with the process of educational decentralisation across the world, regional and district offices increasingly require disaggregated data in order to formulate their own operational plans and to develop budgets for their implementation. Similarly, schools require improved administrative data and improved record-keeping in order to manage resources in a more transparent, effective and efficient manner. The growth of private schools ­ During the 1980s and 1990s the majority of education data focused on public sector provision. However, over the past 10 years there has been an expansion in the number of private or non-governmental organisations involved in educational provision, yet limited information has been collected on this area. The extent of growth of non-governmental and private schools should not be underestimated; analysts argue that in some countries non-governmental provision amounts to over 50% of all primary and secondary education (Chapman & MahlkcK, 1997)8. Failure to record such information could undermine the legitimacy of existing information systems and result in Ministries of Education becoming Public Ministries of Education (Haivan et Herstein, 2003). Civil Society ­ Over the past five years, the movement towards democratic societies and widespread market reforms have increased the pressure on all public institutions to provide more accurate records on resource allocation. This is particularly evident in the field of education, where parents are expecting more from their schools in terms of effectiveness and results. Pressure also comes from community groups, who demand increased accountability and transparency in education spending and management. Ministries of Education are normally responsible for helping to define 8With the increasingly important role played by different organisations in supporting and providing education, the distinction between governmental and non-governmental is becoming blurred in certain areas. However, in this paper non-governmental provision refers to institutions or services that are not funded directly by the state or channelled through state institutions. 8 ways of measuring performance and resource utilisation, as well as for putting in place the appropriate systems for understanding transparency and accountability. The movement towards using qualitative data ­ Quantitative data helps analysts and policy makers understand progress towards achieving targets or pre-defined objectives. However, quantitative data does not explain why difficulties were experienced in achieving a particular target or exploring the context in which learning takes place. EMIS must therefore evolve to capture complementary qualitative information. Transition of young people onto the labour market ­ Demands are also being placed on EMIS to help measure the performance and appropriateness of existing education provision in helping young people find employment. This stems from attempts to introduce vocational subjects into the school curriculum and the demands from parents that education provides their children with the skills to obtain productive employment. Increasingly, it will become important for policymakers to understand what happens to young people once they have completed their compulsory education, especially in the light of the increased access to primary education as result of EFA. Need for information from other education sectors ­ Other sectors of education, including non-formal and higher education, also require timely and accurate information for planning purposes. For instance, in the case of higher education, countries need to know the efficiency of a particular programme measured as the degree to which it is helping young people obtain employment. 9 2.1. The management and operation of EMIS In order to respond to changing information demands, managers must have the appropriate systems and procedures to ensure that valid data is collected and analysed in a timely manner to support decision making (Wako, 2003). However, even if these systems are put in place, one of the other challenges facing government departments is how to develop a culture of informed decision-making based on the information provided by sound EMIS rather than for political reasons or hunches. The relationship between the various institutions and structures involved in data collection and analysis, including the information flows and the organisations which are assumed to utilise EMIS outputs, are shown in figure 1. 2.2 The management of data collection and analysis One of the first task for an EMIS manager is to identify what information is required by analyst or planners to help them understand how education resources are transformed into educational outputs, particularly in relation to whether resources are being used efficiently or effectively in the education process. EMIS managers must not only think about information is require to answer specific questions, but also how it can be analysed in conjunction with other data to produce the desired results. Often, government ministries fail to address such issues and continue to undertake large scale expensive surveys without taking into account how the resulting data will be used in the policy process. Clearly, managers must think about the needs of the end user and whether data will help analysts answer specific questions or understand specific processes. Stakeholders and end users of information should be involved in this process, not only as passive recipients for EMIS outputs, but also as owners of the process, to ensure that [process, efforts] support regularly (re)examined purposes. When answers to these questions have been identified it is necessary to see what other information exists in neighbouring planning units or departments. Often planners fail to make best use of existing information and assume that collecting new data will solve their information requirements. Clearly, this is not always the case and senior managers must support information-sharing between units or departments. This can be achieved through informal data-sharing agreements or the signing of memoranda of understanding (MOUs). These measures will also facilitate the merging of data bases as well as data integration, all of which is useful for addressing issues of efficiency and effectiveness, including how resources are allocated and the performance of individual schools. The starting point for collecting new data is the design of the template or questionnaire. Unless the perspectives of schools are taken on board they are less likely to be committed to the data collection since they are unable to see the potential benefit. This bottom-up approach to data collection should by supported by other structures and institutions within the education system to ensure minimum standards are achieved. Normally, district education officers will ensure that the questionnaires or templates are completed correctly by the schools and passed on to the provincial offices in a timely manner. The provincial offices tend to be responsible for collating this information and forwarding it to the EMIS in the Ministry, who in turn are responsible for analysis and subsequent dissemination. 10 Once the roles and responsibilities for data collection have been identified, it is important to determine the operational plan for data collection, or as it is more commonly called The data production cycle. Essentially this plan or cycle identifies the time period in which data activity take places and who is responsible for this process. Obviously, this will vary from country to country and depend on the structure of a country`s education system and the type of policies they are implementing. Nevertheless, it will be important to determine the exact time period and format in which data needs to be produced, all of which provides the basis for working back and identifying when specific activities need to take place. Figure 1: Management and operation of an EMIS Base-line data Evaluation of policy Formulation of policy Ministry/Central government Monitoring of policy Base-line data Evaluation of policy Formulation of policy Collection and Regional/Provincial Dissemination analysis of data level of data Monitoring of policy Base-line data Evaluation of policy Formulation of policy District level Monitoring of policy Base-line data Evaluation of policy Formulation of policy School level Monitoring of policy 2.3 Utilisation and dissemination of EMIS outputs Another important management question facing those managing EMIS is the degree to which data is disseminated and utilised in the policy process. Dissemination refers to the action of distributing information using a variety of media, ranging from the traditional annual abstract of statistics to the use of the world-wide-web. Whereas 11 utilisation refers to how data is used in the policy process. Normally, the policy process consists of four inter-connected phases, including: the formulation of policies or targets, the monitoring of progress towards targets and the measuring of impact, and the formulation of adjusted or new policies. In many ways dissemination and utilisation are closely linked, but the former is more concerned with the information needs of external users and the later with the information needs of internal users. Obviously, dissemination and utilisation activities must fit into the data production cycle outlined previously. One of the best means of supporting dissemination is to develop a dissemination strategy, which should identify the information requirements of the target audience, the medium in which the document needs to be produced (paper or electronic) and the time period in which circulation occurs. The development of a comprehensive dissemination strategy is only part of the solution. Parents and external stakeholders should also be provided with EMIS data in a format that they can understand and be given training to act on this information. Trade unions and other civil society groups can play an important role in this process, especially with regard to using data to improve accountability and transparency in relation to how education resources are utilised. Only through supporting such activities is it possible to encourage an external demand for EMIS outputs. Within the education system it is equally important to ensure the demand for EMIS outputs are translated into utilisation. There is little point in having a demand for information unless it is used in the policy process. One of the most important mechanisms for translating demand into utilisation is to support institutional development and ensure that neighbouring units within the ministry have the capacity to utilise EMIS outputs. For instance, planning units should understand how to use EMIS data in order to monitor the progress of policies towards defined objectives. Demand can also be translated into utilisation lower down the education hierarchy through providing decentralised offices with the capacity to develop operational plans and by linking these plans to the disbursement of funds. Utilisation of information will also depend on whether end users have confidence in the data and do not resort to using other information sources to make decisions. 2.4 The important and changing role of ICT infrastructure In order to ensure that data collection, analysis and dissemination processes occur in an efficient and effective manner, it is important to choose the appropriate ICT for the context in which EMIS operates. When choices are made about ICT it is vital that decisions are driven by information requirements and not hardware or software issues. In the past, and to a limited extent today, donors began projects by defining the ICT architecture, instead of determining the information requirements or how an EMIS would support the policy process. Clearly, this is wrong and government departments must ensure that choices about ICT infrastructure are determined by their own information requirements and are compatible with systems being used by other units or government departments. Another decision concerning ICT is the degree to which they are sustainable over the longer term. All ICT infrastructures supporting EMIS should also be housed in an environment which will prevent the negative impact of heat, dust and humidity. 12 Decisions about how ICT infrastructure will support EMIS have changed radically over the past couple of years and will continue to do so, partly in response to new software which reduces the problems of compatibility between platforms and the merging of data bases. Similarly, software packages have become easier to adapt to the specific needs of the end user. However, perhaps the greatest impact on future decisions about ICT will come from recent advances in cellular technology, including digital and satellite phones. In some parts of the world this will reduce the costs associated with networking systems and help ensure a more effective flow of information between schools, provincial offices and government departments. According to Cassidy (2005) digital and cellular phones are already being used by some countries in Latin America and the Caribbean for data collection and dissemination. 2.5 Institutional building and capacity development Besides ensuring that timely and valid data are collected and appropriate choices are made about ICT infrastructure, it is equally important to ensure that sufficient attention is given to institutional building and capacity development. Institutional building focuses on whether institutions have the appropriate systems, procedures and structures to carry out their functions and mandates, while, capacity development is concerned with the professional and continual development of those working in the field of EMIS and related activities. Often donors neglect to invest enough resources or time in these two areas and this could impact negatively on EMIS sustainability over the long term. Developing the institutional capacity to support EMIS operations is complex due to the political nature of bureaucracies. Improving long-term institutional capacity involves cultural changes, especially with regard to how information is used and how different institutions depend on each other`s activities. For instance, EMIS are normally responsible for producing data in order to meet the planning requirements of other units, such as those involved in planning, budgeting or monitoring type activities. The management decisions within such units rely heavily on data produced by EMIS in order to carry out their functions and mandate. This data has to be produced within specific time frames and be based on the business cycles of these different units. Under such an arrangement, a Budget unit will require data on anticipated enrolment levels in order to calculate annual operational budgets for education. Similarly, an M&E unit will need data to measure the performance of a strategy towards pre-defined objectives, or the impact of a particular learning initiative on examination pass rates. Correspondingly, if there is no capacity within such units the demand for EMIS outputs will fall, all of which could impact negatively on data quality. Institutional development is about building this inter- dependency between such units and supporting a culture of information use and sharing. This requires professionals to understand the way in which their unit interacts with other units and how this impacts on the performance of their country`s education system. In addition to institutional building, it is also important to ensure people have the appropriate skills to operate and maintain EMIS at all levels of the education system. This can pose a dilemma to government departments since professionals often leave their posts in search of higher paying jobs once they have completed their training. Government departments could solve the problem through sub-contracting out certain 13 activities to the private sector. This might involve sub contracting the process of data collection or analysis to a local university. However, this does not totally solve the problem since government officials would require new skills in order to understand contract management and tendering procedures. 3. Lessons Learnt and Best Practice: The management and operation of EMIS in Bangladesh, Ghana. Mozambique and Nigeria So what lessons can we learn about the management and operation of EMIS in Bangladesh, Ghana, Mozambique and Nigeria? Obviously, the context in which EMIS operate will shape the type of problems experienced. However, what is more important is how they respond to such problems and what lessons can be gained from the comparative experience of the different countries. 3.1 Data collection and Analysis Processes In all four countries, significant problems were experienced with the processes of data collection, collation and analysis. Many of these problems could have been easily ameliorated through the development of certain procedures or systems. However, what appears more difficult to tackle is how to improve the commitment of senior management to the whole process of data collection and analysis. The majority of technical problems associated with data collection appear to originate from the design of the data collection instruments and the processes associated with their implementation. One of the most common instruments for data collection in all four countries consisted of a census form, varying in length from 10 to 20 pages. These are normally administered from a central or regional office and involve asking the head teacher a series of questions. However, the research found that in Ghana, Mozambique and Nigeria lack of compatibility between census forms and school record keeping prevented head teachers from responding to requests from provincial offices or ministries. Even when information was available the head-teacher would have to undertake complex calculations in order to produce data in a format required by the census form. Under such circumstances there are incentives for teachers to either leave the form blank or guess, both of which could have a negative impact on the type of information being recorded. When designing questions for the census form it is vital that common definitions or terminology are used. At first glance the use of common terminology may appear obvious, but the research found that changes in the nature of a policy could have a significant impact on whether appropriate or valid data is collected. For instance, under decentralisation a new school may contain the facilities and resources of three schools due to the merging of neighbouring institutions. Unless the census form anticipates such changes, it is difficult for a head teacher to supply the appropriate information in a format requested by head-office. Given these reasons it is important that census forms ask questions about data that can be readily collected and collated by head-teachers. 14 A related problem with data collection can also occur if two different sets of results are produced due to the use of contrasting methodologies and sampling frameworks. This occurred in Ghana when UNESCO supported the Ministry to implement a nation-wide school census, but at the same-time UNICEF and JICA funded a national survey to map primary schools and collect data on the wider community. Not only can this be viewed as a waste of resources due to the duplication of activities, it can also lead to a lack of confidence in the data. However, there is also another valuable lesson to be gained from this experience. Under the UNESCO survey, district offices and schools were not committed to the process of data collection since it was viewed as being imposed from head-office. In contrast the school mapping approach actively involved district offices in the development of questionnaires and the data collection process, enabling them to use the outputs to develop their own operational plans. This bottom-up approach ensures that those lower down the educational hierarchy are more likely to be committed to this process. Another added benefit of the school mapping exercise was that it collected data on private schools, all of which had been neglected under the school census. A further issue to arise from this experience is the apparent lack of co-ordination between donor activities. It is equally important to ensure that commitment is obtained from senior management and donors to the data collection process. Unfortunately, this does not always occur and in Nigeria when donor funding for EMIS stopped in the late 1990s all data collection activities ceased, but when donor funding became available a couple of years later data collection activities also started again. Lack of continuality in data collection prevents time series analysis from being conducted and can also lead to reduce confidence in that data set. Further problems with data collection can be experienced if the process is sub- contracted to an external agent, as occurred in Nigeria (see box 1). The lesson to emerge from the Nigerian experience is the nature of the complexities associated with sub-contracting and the importance of having the appropriate skills to manage this process. Obviously, the type of management skills associated with tendering and procurement are very different from those involved in operating an EMIS. Box 1: Difficulties associated with sub-contracting activities in the private sector During the years 2001, 2002 and 2003, the Federal Ministry of Education in Nigeria, with project support from the World Bank, conducted a series of base-line studies. These surveys were put out to tender and a private company was awarded the contract. There was nothing wrong with contracting a private company to conduct this research, but the project resources were intended to develop capacity within the Federal and state Ministries of Education as opposed to supporting the implementation of the survey by a private company. Second, some of the funds were provided to support the analysis of data. Third, despite Microsoft Access being installed on government computers, government officials were unable to use the software because they received minimal training. Fourth, since all the survey data was stored on a series of different data bases and lacked consistent codes, it was not possible to compare or analyse this data (see Shoobridge, 2005). 15 In contrast to the other three countries Bangladesh has a long history of successful data collection, beginning in the early 1990s with the implementation of the first school census. A significant proportion of work in this area has been supported by donor aid, enabling the country to build up a significant amount of analytical data to measure the quality of educational inputs (physical facilities, materials, numbers of text books, numbers and training status of teachers) and also the internal efficiency of the education system. However, dependency on overseas aid has created unforeseen difficulties for data collection processes. For instance, under the data collection process head teachers and the chair of the school management committees are responsible for ensuring that returns are accurate. Subsequent levels of the education system are also expected to check a sample of the census forms in order to ensure consistency and validity. This validation of data rarely occurs due to the project-led approach in which officials perceive data collection activities as additional to their normal duties and expect to receive extra remuneration for the work involved. While it is possible to meet such payments during the implementation of donor projects, it is not possible once they have ended. There are significant problems in creating a dependency culture, especially once donor funding is withdrawn. Another option for encouraging data collection is to introduce some form of incentives for those involved with this process. This occurred in Ghana through the use of a capitation grant. Under this process primary schools submit enrolment figures based on an assumption that pupils who completed a particular level of education would also enter the system the next academic year. In return for submitting this information, a school will receive 50% of their capitation grant. The remainder of the grant will be received at the start of the new term when they submit actual enrolment figures. Clearly, there are merits in linking financial remuneration to the data collection, but this process requires careful checks and balances to ensure enrolment levels are not inflated to obtain higher payments. The quality of collected data in all countries could be questioned on a number of grounds and some of the reasons have already been touched upon, including contrasting methodologies and the problems associated with questionnaire design. However, in some of the countries questions were raised about the data on learning outcomes, especially the degree to which qualifications reflected a pupil`s learning achievements. In Ghana, for instance, government said that examination results were not always a true reflection of the pupil`s performance due to the way in which they were assessed using a combination of continual assessment and a final examination. Under this process teachers did not appreciate the additional working load associated with continual assessment and in a large number of instances the examination results were made up, resulting in higher examination passes than were actually achieved. This demonstrates the importance of taking care when interpreting data and the need to understanding the context in which data collection takes place. Even if valid and timely data are collected, those analysing EMIS outputs experienced difficulties due to the fact that data was stored on different databases with incomplete codes. This posed a significant problem if attempts were made to analyse trends over time, or to use data obtained from different government ministries. Under such conditions it was impossible to understand how enrolment levels responded to the introduction of new funding arrangements, or how increments in teachers` pay can impact on learning outcomes. In all four countries the lack of compatibility between data bases or limited cooperation between government departments made it very difficult to produce the type of analytical data that can be used by policy makers. 16 However, the situation is beginning to change and Ministries in all four countries were starting to produce analytical type information, especially in Mozambique where planners have used EMIS outputs to understand how expansion in enrolment levels have been achieved at the expense of quality. A final information problem experienced by all countries in varying degrees, related to the length of the time period between the initial data collection activities and the publication of final results. A combination of factors can account for this delay, but the most significant appears to relate to the lack of commitment from those involved in the data collection, or due to the bureaucratic procedures that are supposed to support this process. In the majority of circumstances lack of commitment stems from the fact that those involved in data collection do not see how they can benefit from this process. Evidence from Nigeria, Ghana and Mozambique showed that at the school level head-teachers are reluctant to spend their time completing forms that will only be used by the Ministry to track progress towards national targets. Similarly, officials working at the district or state level, in Ghana and Nigeria couldn`t see the benefit of collating forms for similar reasons. The research found that in most countries delays were experienced in forwarding the census forms to the Ministry due to the nature their bureaucracies. Often, this process would involve going through a centralised bureaucracy in which officials have to comply with complex and official procedures in order to undertake simple or routine tasks, such as those associated with arranging transport to follow up an incomplete census form, or to hire a part-time clerk to assist with the process of data entry. It could even be argued that some bureaucracies have developed their own rationales and often become dysfunctional, especially with regard to tasks they are supposed to perform. This process is sometimes compounded by what appeared to be lack of commitment from senior management. Due to this alleged lack of commitment and what could be viewed as dysfunctional bureaucracies, information is sometimes only ever published two or three years after the data was collected. Out of date statistics are of limited use, especially to those involved in the planning or policy process. The evidence presented in this section suggests that the processes of data collection and analysis are plagued by a number of difficulties, including those associated with: the construction of questionnaires, the co-ordination of donors and the lack of commitment from those involved in the process. However, there is also some good news. The experience of the four countries also helped identify conditions necessary for supporting effective and efficient data collection. In larger developing countries, especially those containing a significant number of states or provinces, there is little point in developing ambitious national plans for data collection that are incapable of being implemented. A bottom-up approach to data collection involving schools and decentralised offices may have more merits than a top down approach imposed by the Ministry. Similarly, there are benefits of adopting an incremental approach to data collection in which resources and effort are focused on a small geographical area, instead of beginning the process by implementing a nation wide survey. 3.3 Utilisation of EMIS outputs The most important question facing those managing EMIS is the degree to which data is utilised in the policy process. As mentioned earlier, the utilisation of data occurs in the formulation of polices, the monitoring of progress towards targets and the measuring of impact. In all four countries there was clear evidence that EMIS outputs 17 were being used in the policy process, although with varying degrees of success. There was also a general trend for EMIS to play an increasingly important role in supporting the process of education decentralisation by providing information to help state, provincial and district offices to formulate operational plans and in some cases develop annual budgets to fund their implementation. Evidence from Ghana demonstrates the important role EMIS can play in supporting operational planning at the national and district levels (see Box 2). The research found that EMIS data is also helping district offices to formulate their annual operational budgets, according to the following funding criteria: 30% is awarded for pupil enrolment levels, 10% is based on the number of schools and 60% according to the disadvantage levels. This disadvantaged criterion is measured according to the following variables: number of untrained teachers, pass marks for science and maths, the type of classroom structures, and availability of water and seating capacity. At present this approach has only been piloted in a number of districts but there are plans to roll it out across the whole of Ghana. The benefits of using such funding mechanisms are that they ensure provision at the decentralised level, are responsive to local needs and that districts offices have a significant role in developing their own operational plans, instead of having them imposed by head-office. Box 2: The Role of EMIS in supporting planning in Ghana At the national level the EMIS unit plays an important role in helping the Ministry to formulate operational plans and also monitor progress towards achieving such plans. There are a number of stages to this process, each of which is heavily dependent on EMIS data. Prior to the preparation of the annual operational plan a preliminary sector performance report is produced and a review meeting is held in order to obtain inputs from stakeholders and donors. This results in a series of recommendations which are fed- back into the operational plan. Moreover, in Ghana the EMIS is also beginning to play an important role in supporting the process of decentralisation. The outputs from the EMIS are being used to support the development of operational plans and budgets at the district level. It is hoped that this will help improve operational efficiency, promote responsiveness and improve service delivery. Under these changes district offices will now have more autonomy in developing their plans, as well as some discretion over spending their annual budgets (see Neumann, 2005). In Nigeria the lessons are not so positive in that EMIS does not play a significant role in supporting planning processes at the national level due to lack of valid or timely base-line data. The Federal Ministry of Education recognises the difficulty of implementing a national top-down approach and has resorted to supporting data collection at the state level. Under this process all states are provided with access to common software and a centralised depository to store data. At the state level stakeholders have been involved in construction of data collection instruments and there has been a deliberate attempt to ensure that private schools are involved in the process. This strategy has only been piloted in the state of Kano and the success can be gauged by the fact that the EMIS outputs were used directly in the development of operational plans and as a basis to justify their 2005 budget submission. Following this initial pilot, the Kano state level government has now provided funds to support the process of data collection and analysis, indicating their commitment to this process at the decentralised level. There are plans to extend this approach to a further 19 states, but only time will tell if this process is effective at supporting decentralised planning across Nigeria. 18 The empirical evidence from Mozambique is mixed. At the national level, EMIS outputs play a significant role in generating targets for national policy frameworks, as well as assisting in the process of monitoring progress. This involves a participatory process in which progress towards targets is discussed in review meetings. Moreover, as outlined earlier, analytical work undertaken by the Ministry has enabled them to identify that some of the recent expansion in enrolment levels has been achieved at the expense of quality. However, at the decentralised level EMIS outputs have not played significant roles in planning due to the lack of funds and capacity constraints. Moreover, at the decentralised level operational plans are developed separately from strategic plans, and as a consequence both have different sets of indicators. In Bangladesh the research found that EMIS has not played a significant role in planning at the national level due to the lack of capacity and commitment in this area. Nevertheless, recently EMIS outputs are beginning to play an important planning role at the district level. In Bangladesh a pilot project has been implemented to support decentralised education planning in 20 sub-districts across the country. This involved each sub district (Upazilla) collecting accurate and up-to-date information from each school and providing a basic set of indicators. On the basis of this data a list of challenges was drawn up and a set of objectives developed. The benefit of this approach is that it involves identifying local issues and local responses. Utilisation of EMIS data varies form one country to another, but one of the important messages to emerge from the experience of the four countries is the need to ensure close links between data collection and utilisation, particularly at the decentralised level. Unless managers at the decentralised level can see the utility of data they are collecting, they are not likely to be committed to this process. Linking operational plans to budgets can also encourage utilisation of EMIS data. Not only does this ensure managers at decentralised levels appreciate the importance of using EMIS data to formulate strategies, it also provides a mechanism for linking plans to local needs. The evidence in this section also pointed to the important facilitation role played by the Ministry in supporting the collection and utilisation of data at the state level. This provides states and districts with the technical knowledge to determine what data is collected and to a limited extent how it is used, as opposed to having a top down approach imposed by head office. Only through adopting such an approach is it possible to ensure that different layers of the education system have a sense of ownership and commitment to using EMIS outputs in the policy process. 3.4 Dissemination of Outputs What does the empirical evidence reveal about dissemination of EMIS outputs in Bangladesh, Ghana, Nigeria and Mozambique? Dissemination of EMIS outputs occurred to varying degrees in all four countries, but in none of them was there a deliberate attempt to develop a dissemination strategy. In nearly all countries the Ministry of Education produced an annual digest of statistics, but this tended to be distributed in a format not readily understandable by people working outside of the education system. Only in Ghana was there an attempt to produce census data in an electronic format and distribute this to the stakeholders. However, the census data was saved in Acrobat Reader (pdf) format and as a consequence users were not able to readily utilise this data. 19 All four countries also produced detailed plans to use the World Wide Web as a medium for disseminating and collecting EMIS data, but only in Mozambique had concrete steps been taken to implement such plans. In many ways Mozambique could be viewed as an example of best practice in that it produced timely information in an analytical manner and made this available on the web. However, the extent to which stakeholder always have access to the internet is questionable, particularly those living in remote or rural areas. This suggests careful attention needs to be given to the type of mediums used to disseminate EMIS outputs. Another important dimension of dissemination concerns the degree to which stakeholders and people external to the education system can utilise this data. Unfortunately, there was no concrete evidence on the external utilisation of data. However, in both Ghana and Nigeria there were plans to involve the media in future dissemination activities, but once again only time will tell if these plans are ever implemented. The lessons from dissemination are not encouraging. Without a wide-spread dissemination strategy it is not possible to start creating an external demand for EMIS outputs. It is important that Ministries identify the target groups for dissemination, determine in what format data should be produced and the time period in which it should distributed. 3.5 The role played by ICT in supporting EMIS To what extent has ICT infrastructure facilitated the management and operation of EMIS in each of the four countries and what lessons can be learned from their comparative experiences? Once again there appears to be a common pattern of early development, involving attempts to network stand-alone PCs and the widespread use of spread sheets. Commonly experienced problems included limited storage capacity, the usage of outdated and fragmented software, and the inability to link data sets produced by other departments or to analyse trends over time. However, recent patterns of ICT developments have been different in each country and they offer lessons for countries embarking on the process of ICT reform. At one end of the spectrum is the minimalist approach used by the Ministry in Mozambique, involving the use of simple databases, spreadsheets and transfer of data via email. Under this strategy provincial offices are responsible for data collection from schools and for the forwarding of collated information to the Ministry. All of these processes are fully funded and managed by government staff at the central and the provincial level directorates. The benefits of such an approach are that it is sustainable and facilitates the transfer of data in a timely manner, all of which may appeal to other less developed countries who are keen to keep costs to a minimum and not depend on skilled experts to maintain their system. However, on the downside the collated information is only available at the head-office and provincial offices or other stakeholders cannot readily obtain access. In contrast, Education Ministries in the other three countries have taken advantage of recent advances in technology and falling costs. The strategy varies from one country to another, but one common influence has been the role of donor funds. Bangladesh has benefited from over 15 years of successive donor projects, providing the opportunity to expand systems and upgrade specifications and software. Initially the 20 system in Bangladesh relied on spreadsheets and relationship databases for data entry and storage. However, with the spread of internet service provision the different layers of the education system are turning to web-based solutions for data entry and storage. The benefit of such an approach is that data can become accessible at all levels of the education system and cost effective. Nevertheless, due to the extensive donor involvement in the procurement of ICT equipment there have been expectations by governments that development projects would fund the procurement and maintenance of hardware and software. This has resulted in a situation where limited resources are available for post-project maintenance. Under such circumstances virus software quickly becomes out-of-date and component failure renders hardware inoperable. Once again the lesson from Bangladesh illustrates the problems of creating a dependency culture and the possible negative impact on longer-term sustainability. The experiences of Nigeria are more positive and provide lessons for larger developing countries, which have complex political systems and are about to reform their ICT infrastructure. Part of the explanation for the success was that an incremental approach was used. Thus, instead of introducing a wide-scale national reform of the whole ICT system, a small scale pilot was implemented at the state of Kano. This helped identify problems and ensure they were ironed out prior to going national. However, before implementing the pilot, a needs analysis was undertaken to identify the most appropriate architecture for the EMIS, resulting in the decision to develop a web-enabled system to support the collection, collation and reporting of school level data. Advantages of a web-enabled system are ease of access, low maintenance cost and the fact that one good computer, the server, is required. Understandably, it is easier to maintain such a system and virus protection is only needed for one computer. Moreover, because local programmers and analysts are readily available this system can be adapted to new demands and be sustainable. The early use of ICT infrastructure in Ghana experienced similar problems to those experienced in other countries namely lack of compatibility between systems and problems of sustainability. However, the Ministry has started to address these issues through an Enhancement and Expansion Plan for EMIS. This has involved the introduction of new platforms to ensure synergy occurs with other government departments, enabling data sets to be merged and compared. The Ministry is also at the stage of introducing new software, with plans to go live over the next four years. This would connect all district regional data bases to a web-enabled system at head- office, enabling information to be available on demand at all levels of the education systems. However, the existing system faces a number of serious constraints which are yet to be resolved. Lack of computer equipment is a major problem facing the country, particularly at the district office where around half of the offices lack computer facilities. Equally significant is the lack of job descriptions for staff working in these offices and their lack of understanding about their role in this process. In Ghana, Mozambique and Nigeria the evidence suggests that appropriate choices had been made to match ICT infrastructure with a particular environment. As mentioned earlier the minimalist approach could provide a cost effective and efficient means of data collection, especially in countries that lack skills to maintain complex systems. However, the experience of Nigeria is important since it demonstrates the importance of adopting an incremental approach in order to iron out potential difficulties. There are even more important lessons to be learnt from Bangladesh, 21 especially with regard to the negative impact of creating a dependency culture. As each of these four countries develop, and technology becomes more advanced, managers will have to ensure that appropriate ICT choices are made to support EMIS related developments, taking into account issues surrounding sustainability. 3.6 Institutional Development and Capacity Building So far the empirical evidence has shown a mixed picture of EMIS performance across the four countries. Governments working with donors have supported the development of complex systems and procedures for data collection, and also invested considerable resources in ICT infrastructure to facilitate these processes. Despite advances being made in the former areas, relatively limited attention has been given to how EMIS outputs are disseminated or utilised in the policy process. This suggests that donors and their partners automatically assume that EMIS outputs will be utilised in the policy process. This assumption is misguided, especially in the context of developing countries that are characterised by resource constraints, lack of institutional and human capacity, and where decision-making is often driven by short- term political gain. The challenge for developing countries is how to reverse the above situation and ensure that decision making becomes more data driven. This can only be achieved by improving the institutional capacity of units that produce and utilise EMIS outputs, and also by ensuring that those working in these units have the appropriate skills and support to carry out their functions. When such conditions have been established it is possible to create a culture of information sharing, dissemination and utilisation, all of which will help encourage a sustained demand for EMIS outputs. This raises the important question of to what extent has this occurred in each of the four countries and has this helped improved the legitimacy of EMIS? Institutional building and capacity development have occurred in all four countries to varying degrees, but only in Ghana and Nigeria have there been any coherent attempts to support these two processes. Once again Ghana provides an example of best practice with institutional building and capacity development occurring at all levels of the education hierarchy. This was made possible by a comprehensive programme of reform, whose specific purpose was to develop decision making processes at all levels of the education system. At the Ministry level, a recent initiative supported institutional development of the Planning, Budget, Monitoring and Evaluation Unit, the significance of which being that this unit uses EMIS outputs to formulate the country`s education sector plan, a tool which enables senior officials to monitor progress towards goals defined in the country`s Annual Education Sector Plan. There are also plans to establish an Education Sector Technical Advisory Committee, a structure which will enable donors and stakeholders to have an input into the policy process when targets are not being achieved. All of these institutional structures are important means of generating demand for EMIS outputs. At other levels of the education system institutional developments in Ghana have encouraged a utilisation of EMIS outputs through encouraging a sense of ownership and commitment to the data collection processes. This was achieved by the setting up of a new institutional structure for EMIS, involving a centralised EMIS unit responsible for managing and co-ordinating all EMIS related activities. Under the centralised EMIS unit are sub EMIS units for the following areas: TVET, non-formal 22 education, higher education and general education. Each of these sub units is responsible for data collection and utilisation for their own sector. Similar reforms have occurred vertically within the education system and involved devolving responsibility for data collection and utilisation to district offices (see previous sections for detailed discussions). Prior to these institutional reforms, a single structure was responsible for data collection and validation across all education sectors, including data from district offices. Under such an arrangement the different units within the Ministry, including the district offices, felt no sense of ownership and had little commitment to the data collection process. However, as a result of the Enhancement and Expansion Plan, attempts have been made through institutional reform to ensure closer links are made between data collection and utilisation, all of which helps improve demand for EMIS outputs. Institutional reforms in Ghana have also been supported by a comprehensive package of capacity building activities across horizontal and vertical levels of the education system. The first stage involved a needs analysis to identify skill requirements, followed by the development of a plan for human resource development. This is in the stages of being implemented and four officials have been appointed at each district office to support this process. Already, a series of workshops has been implemented across the country to train people in basic data analysis and in how to develop operational plans. Attempts were made to train more people in IT skills than are employed at district offices. The rational behind such an approach is that they can create a pool of skilled workers who can be drawn upon should the district officials leave in search of other employment. The approach to institutional and capacity building in Ghana provides examples on how to encourage demands for EMIS outputs at different levels of the education system, but it has not been without difficulties. Perhaps the most significant of which is the fact that the operation of the Planning, Budget, Monitoring and Evaluation Unit relies on expatriate and part-time staff, bringing into question the long term sustainability of this unit. In addition, the attempt to encourage stakeholder involvement in the policy process through a Technical Advisory Committee is another positive move, but lack of commitment from senior management has prevented this from occurring. Nevertheless, these are only minor difficulties and could be tackled with minimal resources and more political commitment from senior management. There are important lessons to be learnt from the institutional developments that have recently occurred in Nigeria, especially with regard to addressing institutional constraints and developing the capacity of EMIS managers. One of the constraints facing EMIS was the fact that two institutional structures existed for data collection. This created a complex situation in which reporting lines and responsibilities between these different structures became blurred, all of which reduced confidence in EMIS outputs. In an attempt to improve this situation a National EMIS Task Force was established to make recommendations on this issue, and this resulted in a decision to stream-line existing structures and provide support for only one institutional structure at the state level. It is important to point out that this institutional reform was accompanied by an extensive training programme under which managers were trained on how to use EMIS generated reports to develop operational plans. This decentralised approach to institutional building offers a solution for dealing with complicated bureaucracies and for bringing data collection closer to the end user. 23 In the case of Mozambique, limited attention has been given to the process of institutional building or capacity development for EMIS. When training is provided for using education data, it occurs in an ad-hoc manner and appears not to be linked to any institutional change or policy reform. At the district level managers have relatively little understanding of indicators nor do they have access to collated or analysed data since information only flows from district offices to the Ministry. In Mozambique more needs to be done to encourage a culture of information-use at the district offices. Donors recognise this problem and GTZ attempted to introduce a culture of information utilisation, but in practice efforts focused on the monitoring of work-plans (that is on the systematic checking of whether activities listed on plans had been undertaken) and not on higher level objectives. A number of reasons have been identified for the lack of interest in data usage at the district level, ranging from lack of commitment to the process, limited capacity, to the lack of access to appropriate data. While some of these issues are important and need addressing, it must be remembered that it takes several years to establish the necessary institutional structures to support a culture of information use. Most donors neglect this issue and attempt to develop institutions or human capacity over a period of three to four years, instead of a more appropriate time-frame of six to seven years. The experience of Bangladesh highlights the problems that can occur due to lack of institutional linkages and limited capability of those working with EMIS. Earlier sections of this report highlighted that Bangladesh had made considerable progress in establishing EMIS, particularly with regard to the collection and analysis of data. Much of this development has been possible through donor support, but donor aid has not been so effective in supporting institutional reform, and training has not always had its intended impact. In Bangladesh institutional structures within the education system are still in need of vital reform. A case in point is the various units involved in data collection and analysis, all of which have no formalised links or reporting lines between them. This is compounded by the fact that those working in neighbouring units lack the appropriate skills to make use of EMIS outputs because they are recruited into their posts from schools and colleges. These people only remain in their position for a short period of time. Clearly, unfamiliarity with how to use data, combined with short tenure, prevents closer links occurring between the EMIS unit and other divisions. Understandably, this impacts negatively on the demand for EMIS outputs. With regard to capacity building in Bangladesh a lot of donor support has gone into this area, but there is a tendency to focus on either technical areas or data collection, while neglecting issues related to data utilisation. A further problem with relying on donor support is that the government has tended to leave capacity building and HR issues to donors, confidently expecting donors to provide the required funding. During project implementation this approach may appear fine, but once implementation ceases those working in technical fields leave in order to find higher paid jobs in the private sector. The lessons from this section are complex and it is clear that institutional building and capacity development are closely associated with all stages of data collection, analysis and utilisation. It is also clear that demand for EMIS can only be encouraged through paying more attention to institutional building and capacity development at all levels of the education system. However, this is not a simple process and involves working closely with different units and institutions, both horizontally and vertically within the 24 education system. This cannot be achieved overnight and requires sustained donor support over several years, along with commitment of senior government officials to ensure that such institutions have the capacity and skills to utilise EMIS outputs. 4. Re-thinking EMIS? The case study evidence suggests donors and planners not only had unrealistic expectations about what could be achieved in a short-time period, but also underestimated the challenges facing EMIS. In all four countries significant problems were experienced in the management and operation of EMIS at all levels of the education system, and in the vast majority of instances, systems were unsustainable without a considerable amount of donor support. Similarly, the utilisation and dissemination of EMIS outputs were lower than anticipated. Poor performance and lack of sustainability, combined with a low utilisation of EMIS outputs, suggest a need to re-think some of the assumptions upon which EMIS are based. Since donors have played, and continue to play, a pivotal role in establishing EMIS in different countries around the globe, they need to start re-thinking what can be realistically achieved using existing resources, particularly with regard to how funds and technical assistance are allocated. Similarly, governments must also become more committed to EMIS developments, and provide longer term financial support for institutional building and capacity development. The former actions will help EMIS to become more sustainable, ensuring that EMIS outputs help managers to administer their education systems in a more effective and efficient manner. The starting point for re-thinking EMIS is to move away from the assumption that one size fits all. Top-down approaches to EMIS don`t always work and countries need different strategies to respond to their own specific needs and development characteristics. There is no point in implementing a top down approach to EMIS, in a large country with a complex political system. Under such circumstances an EMIS is more likely to succeed using an incremental approach, whereby resources are focused on a small geographical area to ensure maximum impact. Nevertheless, if a country has a political consensus and is characterised by good governance, than a bottom up strategy might be more appropriate. Conversely, in a small developing country a top- down approach to EMIS might be more appropriate to ensure that results are achieved as quickly as possible. There will also be instances when a combination of approaches might work best. However, it is important to note that none of these strategies for EMIS is necessarily better than the other, and each represents an attempt by a Ministry or planning unit to respond to different development characteristics. There are a number of other issues important to all countries regardless of their development characteristics. One concerns the changing role and function of the state. In the past an Education Ministry tended to be directly involved in all stages of EMIS development, but now it is more concerned with facilitating processes and ensuring that the appropriate environment exists to support data collection, analysis and utilisation. These changes must be reflected in the type of technical assistance donors provide. Thus, rather than focus efforts on establishing complex systems and procedures at the Ministerial level, technical assistance must also facilitate such developments at the decentralised level through providing access to common templates, software and management advice. This will enable Ministries of Education to help managers at the district and state levels to make decisions about data 25 collection, collation and utilisation (as opposed to having them determined by the Ministry). Another area that donors need to re-think, and which is related to the issues discussed above, concerns support for institutional building and capacity development. In our case studies there were isolated examples of best practice, but on the whole institutional building and capacity development tended to be neglected by donors and their partners. Moreover, when support was provided it failed to have the intended impact. Donors need to re-think support for these areas and remember that it takes several years to change an organisational culture and develop institutions with the type of structures and capacity to effectively utilise EMIS outputs. 5. Concrete steps to develop a responsive and sustainable EMIS What follows is not a comprehensive practical guide, nor is it a review of existing literature, since these issues have already been tackled more thoroughly in other pieces of work. Instead, it is an attempt to identify what lessons can be gained from the experiences of our four countries, and how these can be translated into concrete steps for guiding EMIS reform, especially in relation to areas that have not been given sufficient attention in the past. 5.1 Strategies for driving EMIS and the importance of commitment Guiding Principals for developing an EMIS strategy Identify the development characteristics of your country Determine which EMIS strategy is most appropriate, taking into account possible risks Develop a strategy that is capable of being implemented Ensure that results can have maximum impact as quickly as possible Pilot your strategy to ensure problems are identified prior to going nationally Develop a data production cycle, identifying who does what and when Ensure commitment to this strategy is obtained from educational professional and other stakeholders Prior to embarking on a full scale EMIS reform, it is important for mangers to identify what strategy will drive the management and operation of their EMIS. The composition of this strategy will ultimately determine how data will be collected, what type of investments will be made in ICT infrastructure, what type of institutional and capacity development will occur and how the outputs will be disseminated. Obviously, any strategy for EMIS will have to take into account the specific development characteristics of an individual country, including: demographic issues, levels of socio-economic development and political characteristics. In the past there has been a tendency to neglect these issues and implement a top down strategy for EMIS, but as we have seen the results have not been encouraging. 26 Managers need to choose a strategy that is most suitable for the context in which EMIS developments occur (see table 2). There is no magic formula to ensure a perfect match occurs between an EMIS strategy and a country`s development characteristics, but there are a number of issues that should be taken on board when developing a strategy. Managers should be careful to ensure that a strategy is capable of being implemented and achieve its intended impact within a short space of time. There is no point developing a complex strategy that is incapable of being implemented. Managers might also want to adopt an incremental approach for EMIS reform, ensuring that any perceived difficulties or problems are ironed out prior to going to nationally. Table 2: Relationship between development characteristics and type of EMIS strategy Demographic Development Type of EMIS strategy characteristics characteristics Population size Level of Top down Rural/Urban economic/social approach population ratios development Bottom up Large developing Type of state approach country bureaucracy State level Medium Characteristics of approach developing country political system Incremental Small land locked Status of civil approach Small island state society Combination of the Level of demand above for education data Equally significant to the success of any EMIS strategy is the commitment and support from professionals working at different levels of the education system, as well as from stakeholders outside of the education system. Unless they are committed to EMIS it is unlikely that timely and valid data will be produced. There are no short- cuts for obtaining this commitment, but professionals are more likely to be committed if they have support from senior management, see the potential benefits of the end product and understand their role in the EMIS process, including: what tasks need to carried out and the time period in which such activities have to occur. 5.2 Determining data needs and strategies for data collection Determining data needs and strategies for data collection Carefully dentify data needs of officials at different levels of the education system and stakeholders. Strategies for data collection must involve managers, teachers and other appropriate stakeholders. Ensure data can be easily collected and that synergy occurs between census forms and school record keeping. 27 Investigate use of financial incentives for data collection, but ensure appropriate checks are in place. Ensure close links are made between data collection and data utilisation Sub-contracting of data collection activities can work, provided managers have the appropriate skills. Student report cards are an alternative approach for collecting data. Supplement quantitative data with qualitative data in order to understand the policy process. After the EMIS strategy has been identified, and commitment obtained to implementation, it is imperative to identify information needs for EMIS. However, it is important that managers don`t just develop a shopping list of data they think is needed. Instead, managers must carefully identify the information needs of professionals at different levels of the education system. At the Ministerial level, managers should determine what data planners require to formulate and monitor national policies, as well as what information they need to develop budgets. While at the district and state levels they should ensure that data is collected to help officers formulate operational plans and financial strategies for their implementation. Equally significant, but often neglected, is what type of data could be used in school planning by head teachers and administrators. The information needs of stakeholders could also be taken into account. Finally, it is also important to anticipate future information needs, particularly those relating to the labour market or non-formal education. Increasingly, parents want to know whether existing school provision is helping their young people enter the labour market. Managers also need to think carefully about how they will collect data. The experience from our case studies pointed to the importance of ensuring teachers and other educational professionals are involved in the design of the data collection instruments. Not only does this help improve commitment to the process of data collection, it also ensures that data can be collected easily without having to perform complex calculations. Our case studies also pointed to the importance of ensuring that close links are made between data collection and utilisation. The strategies for data collection also need careful attention. Financial incentives could be used to ensure a higher rate of questionnaire return, but unless checks and balances are introduced questionnaires might be just completed to obtain payments. Sub-contracting might help ensure data is collected in an efficient and effective manner, but unless the managers have a thorough understanding of procurement and tendering procedures they might run into serious difficulties in managing this process. Managers might want to investigate other mechanisms for data collection. For instance, the use of a student report card has benefits over the traditional school census because it provides the opportunity to trace the progress of individual pupils and ensures that data is more likely to be used in the school planning process (see Box 3). 28 Box 3. Information systems based on tracking individual pupils In the Gansu province in China there was limited accurate statistical data on the education system until 2003. Prior to this period data was collected at the provincial and county levels, but in the majority of cases this was not very accurate and was based on projections obtained from previous population censuses. As a result of this problem a decision was taken to pilot an EMIS in four counties. However, instead of the school, the public was used as the basis for data collection. All pupils in these counties were required to complete a record every semester. This covered issues such as age, gender, and attendance and examination record. Under legislation, teachers had to ensure that pupil records were successful completed and forwarded to the county education bureau. In addition to legislation the county education bureau was keen to ensure that accurate data was collected since it demonstrated how well their county had performed compared to other counties. (Cambridge Education, 2004a). Besides quantitative data, managers need to investigate the potential benefits of collecting qualitative data. Until recently, there has been a tendency to focus on quantitative data at the expense of qualitative data. Clearly, qualitative and quantitative data have different, but complementary, functions. Quantitative data can highlight certain trends or performances within certain parts of the education system, but qualitative information helps us to understand why certain initiatives are successful and others unsuccessful. For instance, critics argue that qualitative data helps us to understand whether expansion in enrolment levels have been achieved at the expense of learning quality. Clearly, qualitative research helps inform policy makers about the context in which certain quantitative trends take place. 5.3 Choices about ICT infrastructure Factors influencing the choice of ICT Technology cannot solve all your EMIS problems. Strategies for ICT must be realistic and capable of being implemented. Appropriate choices should be made about software and networking systems (managers must think about whether ICT systems should be developed locally) Managers choice of software and hardware should also be influenced by the availability of local skills to repair and maintain such systems If ICT services are sub-contracted, managers should ensure that they have the knowledge and skills to successfully manage this process. It is important to remember the old adage garbage in, garbage out. Unless the appropriate systems and procedures have been put in place to ensure the collection of valid and timely data, investing resources in ICT will do nothing to improve the quality of future data. Decisions about ICT should be driven by the requirements of data or the needs of the end users, and not by advances in technology. It is important that donors start from the premise of how technology can support data collection and decision-making, instead of thinking about how advances in new technologies can drive these processes. 29 Decisions on ICT infrastructure will be influenced by the type of strategies being used to manage and operate EMIS. For instance, a decentralised EMIS strategy will determine what types of networking systems are used and what choices are made about hardware and software. Developing countries are also in a position to take advantage of recent advances in technology and falling costs to develop their ICT infrastructure. However, it is important that ICT strategies are not too ambitious and are capable of being implemented in a specified time frame and budget. There is no point developing a Rolls Royce when a bicycle will be perfectly suitable and satisfactory. From our country case studies, one of the most successful ICT strategies was the minimalist approach, whereby data was recorded on spreadsheets and transferred via email. The starting point for a successful ICT strategy is for managers to ask a number of simple, but important, questions. First, should the ICT system be developed locally or purchased off the shelf? There are benefits and draw backs with both. It may take time and cost more to develop your own system, but it will be more suited to your specific needs. Secondly, to what extent is the system capable of being upgraded or responding to new demands for information? There is no point in developing a system if it cannot take advantage of new technologies or expand to cover new data requirements. Thirdly, to what extent are skills available locally to ensure that systems can be serviced and repaired? This is a continual problem facing developing countries and managers must think carefully about developing maintenance strategies. Fourthly, does the system meet the needs of managers and end users? The system must be user friendly and enable people to enter data with little difficulty. Only by addressing the former issues is it possible to ensure that ICT facilitates the management and operation of an EMIS, and is sustainable over the medium- to long- term. Managers at Ministries might want to think about sub-contracting their ICT activities to the private sector, or developing a public private partnership. Once again there are benefits in removing complex decision associated with managing ICT infrastructures, but Ministries have to be careful to avoid technological lock in (see Box 4). Box 4: Technological lock in South Africa Technological lock-in normally occurs because a department or ministry uses a specific technology and depends on a certain provider to maintain or service this technology. Under such circumstances the Ministry will spend a lot of resources to acquire such services and may not be able to choose more efficient solutions. For example, in South Africa the Department of Labour constructed their information systems using Oracle as a platform and had to pay around $2.5 million per year in licence fees, despite the fact that only a small number of people used this system. Clearly, Ministries in developing countries need to avoid technological lock-in, especially if this technology is not transferable to another supplier. A possible means of avoiding technological lock-in is to use open source software. This software is normally free, requires no licence and is not dependent on specific sets of technical skills (Powell, 2004). 30 Those involved in the management and maintenance of ICT infrastructure also need to keep ahead of the latest developments. None of the EMIS in our country case studies used cellular devices for the recording or transferring of information, but in the future this may become a reality. There are potential benefits of such advances, including reduced costs and the removal of difficulties associated with managing networking systems in complex environments. 5.4 Utilisation of EMIS outputs Measures for encouraging the utilisation of EMIS outputs Ensure timely and valid data is always produced. Encourage a symbiotic relationship between the EMIS and other units who utilise the EMIS outputs. Support utilisation of EMIS outputs by demonstrating how it can enable people to carry out their jobs more effectively. Facilitate data utilisation at the decentralised level by linking the approval of operational plans to funding. Institutional building and capacity development are vital for encouraging the use of EMIS outputs. Develop the skills of parents and stakeholders to utilise data. Access to the appropriate data can help improve accountability and transparency. The raison d`être for any EMIS is to produce outputs that can be used by managers within the education system and to respond to the information needs of those outside the education system. It goes without saying that EMIS must respond to these different needs by producing timely and accurate data. At the Ministry level EMIS outputs are primarily used to support national policy, but they can also be used as a basis for decision making in other units, especially budgeting and monitoring. One of the most effective mechanisms for encouraging the utilisation of EMIS outputs is to ensure that a demand exists from other units, such as budgeting or monitoring and evaluation. This can be achieved through institutional building and capacity development. For these reasons managers need to understand the importance of this symbiotic relationship between units and also that it depends on EMIS producing accurate and timely information. The failure to produce such data will result in other units relying on different sources of data, all of which will undermine the legitimacy of the EMIS unit. Another means of encouraging demand is through developing the capacity of stakeholders and parents to use data. If stakeholders and parents are provided with the appropriate skills, they can provide an important means of improving transparency and accountability, particularly over how resources are utilised in the education process (see Box 6). 31 Box 6: Links between demand for education data and accountability A campaign undertaken in Uganda provides an illustration of how the dissemination of educational data helped improve accountability, as well as school performance. Under this campaign local newspapers were provided with data on the monthly transfer of national grants to districts. The government supported this campaign due to the fact that, in previous years, schools only received 20% of their grant from central government (the remainder being removed as it filtered down the different government tiers. This information empowered parents and local schools, enabling them to exert pressure on the appropriate institutions, all of which reduced leakage and ensured that more resources were available at the school level. This also had a knock-on effect of increasing enrolment levels and the quality of existing provision (Reinikka, 2004) Utilisation of data at decentralised levels also needs to be encouraged. One of the most important actions to support utilisation is for educational professionals to understand how they can benefit from using such information. Officials at the middle levels of the education system and head teachers need to understand how data can help them carry out their jobs more effectively (see Box 7). Box 7: The role played by data in improving management of a teacher training college A relatively simple small-scale survey of the eight Teacher Education Institutions (TEI) in Lao PDR has delivered unique insights into a complex situation, previously evaluated by anecdote and subjective opinion alone. The study approach required deputy directors of each TEI to organise data collection by all teaching staff on their own time use, students and classroom utilisation throughout a five week period. Leadership by these institution managers had many benefits, including response rates approaching 100% for teachers` work diaries and classroom occupancy charts, new research skills for the managers involved, and genuine ownership of the study outcomes--despite the sometimes difficult findings (see figure on teachers` contact time with students by subject). The results included a well- informed debate at institutional and national level and a shift in the management culture at TEI and ministry levels (Cambridge Education, 2004b). Another strategy for encouraging data utilisation at decentralised levels is through developing links between the improvement of operational plans and budget allocation. Under this strategy district education offices have to submit their annual operational plans, including a financial proposal, to the Ministry for approval. Not only does this improve the commitment of decentralised offices to using data in the planning process, it can also provide a means of ensuring that plans respond to local education needs. 32 5.5 Dissemination of EMIS outputs Characteristics of a successful dissemination strategy Ensure synergy between the outputs of dissemination strategy and the data production cycle. Identify target audiences for the EMIS outputs. Determine what data they require. Carefully determine the format in which the output should be produced. Identify deadlines for sending out promotional literature. Attempt to produce an annual publication that will generate interest in the work of the EMIS unit. Dissemination of EMIS outputs also tends to be an issue that is often neglected by Ministries of Education, both in the developed and developing world. However, dissemination can be an important process by which an EMIS unit can market their products and improve credibility by informing other government bodies and stakeholders about what data is available. The process of dissemination does not automatically occur by just publishing data. Dissemination requires a specific strategy that is closely aligned to the data production strategy for EMIS. The dissemination strategy should carefully identify the target audience; determine what information they require and the format in which this data should be produced, as well as the time period for sending out this data. Each of these stages needs careful attention and planning. For instance, the format or medium for dissemination is very important and information must be presented in a format readily understandable by the target audience. If successful, a dissemination strategy can create circumstances in which supply creates its own demand (see Box 8). Box 8: Publication called "State of Skills in South Africa" In South Africa the Skills Development Planning Unit was given the mandate to collate, analyse and disseminate all information on skills development. To facilitate this process a dissemination strategy was developed, including an identification of annual outputs and target audiences. A number of different publications were produced, the most significance of which was entitled State of Skills in South Africa`. This was a policy orientated document of about 20 pages that outlined trends in education and training, as well as comments on important issues and constraints. After two years of publication this became known as one of the key documents about skills development in South Africa. Reference was even made to this publication by the President in his Address to the Nation speech. Understandably, this gave considerable credibility and legitimacy to the work of the planning unit (SDPU, 2004). 33 5.6 Institutional building and capacity development Measure for effective Institutional building and capacity development Institutional building and capacity development are complementary and must be implemented as together. Institutional building is about improving capacity of EMIS and related units. Institutional development and reform requires the support of senior managers. Institutional reform must not be imposed. Institutional reform can raise the status of EMIS. Capacity development must begin with a needs analysis of EMIS and related units. The needs analysis results. in an HRD strategy, covering short-term and long term goals. The HRD strategy must also cover issues such as job descriptions and how to deal with staff turnover. Institutional building and capacity development are complementary activities and must always be carried out together. Provided these activities are correctly implemented they can have a positive impact on all EMIS related activities. Institutional building is about improving the capacity of EMIS and neighbouring units to carry out their activities in a more effective and efficient manner. In practical terms this involves putting in place the systems, procedures and structures to enable such units to carry out their day-to- day functions. It can be relatively easy to establish such processes within an EMIS unit, but more complex to introduce them into neighbouring units. For this reason it is important that senior management support institutional reforms that cut across different units or departments. However, institutional building is also about improving the working relationship between such units and building a culture of trust. Institutional reform must never be imposed, since building trust between neighbouring units can be a very difficult process. In many senses management should encourage a culture of openness between units and support closer integration, but this process can also be facilitated through the development of memoranda of understanding, identifying what data needs to be produced, by whom and the time period in which this needs to occur. Institutional reform can also help raise the status of EMIS. One of the most effective means of elevating EMIS status is to change reporting lines, so that the managers EMIS report directly to the Director General, instead of the director of planning. This can enable the EMIS to have more influence over its own processes and those of neighbouring units. The establishment of new institutional structures can help improve the commitment of different structures to EMIS related activities. Examples might include the setting-up of sub-EMIS structures within a Ministry, each of which has responsibility for data in their own education sector. Alternatively, it could involve the setting-up of EMIS units at the decentralised levels and devolving responsibility for decision making about data collection and utilisation to such structures. However, it is important to 34 remember that this type of institutional reform cannot be achieved overnight and must be accompanied by a comprehensive programme of capacity development. Capacity development covers not only issues surrounding the continual and professional development of employees, but also wider human resource development issues, such as job descriptions and remuneration packages. One of the first tasks for developing an HR strategies is to undertake a training needs analysis, covering staff working at horizontal and vertical levels of the education system, including those in centralised units in the Ministry and managers working in decentralised offices, as well professionals working in other Ministry Units, such as those in budgeting, monitoring and planning. The audit from the training needs analysis will form the basis for a human resource development strategy, containing immediate short-term actions and longer-term career development goals. When addressing human resource issues it is important for managers to understand how employment activities impact on the performance of the EMIS, including the development of job descriptions and remuneration packages. Professionals working in the unit need to understand their job functions and how they fit into the data production cycle. The Human Resource Development plan must also anticipate difficulties facing the EMIS unit, such as how to deal with the continual turnover of staff once they have completed their training. This is difficult to solve since most EMIS units don`t have funds to compete with the higher pay offered by the private sector. Nevertheless, contingency plans should be developed, such as the creation of a pool of skilled people who can be called upon to deal with the problem of continual turnover. Another possibility could involve the development of capacity manuals. These detail all tasks associated with a particular job and enable a new person with a similar educational background to carry out the activities of a former employee, provided that the appropriate supervision is provided. 6. Conclusion The experience of the four countries has highlighted the complexities involved in establishing EMIS and the difficulties associated with ensuring that the outputs of EMIS are utilised. Without a doubt donors and planners not only had high expectations about what could be achieved in a short-time period, but also underestimated the challenges and obstacles. This mismatch is not confined to the developing world and administrators in North America and Western Europe have the same optimistic expectations about what EMIS can achieve within specified timeframes and budgets. For instance, an assessment of EMIS developments in the state of Ohio in the United States of America it was found that administrators had underestimated the size and complexities of setting-up an EMIS. Indeed, unrealistic budgets and time frames for developing and implementing EMIS in Ohio resulted in the annual budget rising from $3 million to just over $9 million (Ladd et al, 1993). In the UK similar problems have been experienced with ICT projects involving national level information systems. Given the difficulties experienced with EMIS in North America and Western Europe, it is difficult to understand why donors continue to have such high expectations of EMIS in the developing world, especially where the risks associated with EMIS development are much higher due to lack of commitment, under-resourced bureaucracies and underdeveloped infrastructure. 35 Clearly, donors and planners also need to be cautious about what EMIS can achieve and what it cannot achieve. The evidence from our case studies suggest that many of the assumptions underpinning EMIS need to be modified, taking into account the complex operating environment of developing countries. In the past donors gave limited attention to this issue and assumed that top down approaches, involving an identification of the Ministry`s needs, would be most appropriate, regardless of the operating environment. Often, such approaches failed and numerous attempts were made to reform the country`s EMIS, a process which often took several years. Clearly, this is wrong and careful attention should be given to identifying the most suitable strategy, taking into account local contextual issues. In addition the strategy should also be piloted in a small geographical area, ensuring that any problems associated with implementation are ironed out prior going nationally. There is also a need to be more realistic about the role played by quantitative data in the policy process. At present EMIS tend to rely solely on quantitative information and assume that by collecting appropriate data it is possible to understand a situation, and subsequently develop an appropriate response. This is true to a certain extent, but it also raises the question of how much quantitative data you need to understand a particular situation and when you stop collecting quantitative data? For instance, if a large developing country has successfully established an EMIS education managers might be in a position to compare the performance of primary schools in different states or provinces. However, such a comparison could then be criticised for not being valid since it doesn`t take into account the socio-economic context in which the school operates. The response to this criticism would be to collect more background data about the schools environment. However, there are always going to be other influences and it is not always possible to understand how they impact on education outcomes due to the costs and logistics associated with collecting more data. For these reasons it is important to have more realistic expectations about what EMIS can achieve. An analogy can be made between developments in EMIS and approaches to manpower planning. During the 1970s and 1980s it was assumed that through collecting the appropriate data, manpower planning could help planers determine the precise nature and extent of their country`s human resource requirements. However, after decades of experience it is generally recognised that labour markets are more complicated than initially thought and that it is impossible to account for how different factors impact on the supply and demand of labour (Powell, 2005). As a consequence planners have adopted a signalling approach where attempts are made to understand broad trends and specific surveys or evaluations are undertaken to address specific policy issues. This could have lessons or implications for EMIS developments, particularly in terms of focusing resources on understanding broad trends towards targets and implementing stratified surveys to understand specific policy issues. Through adopting such an approach the boundaries for EMIS could become tighter and planners would not have unrealistic assumptions about what can be done with quantitative data. It is equally important for donors and planners to think more carefully about the benefits of using qualitative data The quantitative paradigm has tended to dominate approaches to EMIS, yet the importance of qualitative information to the policy process should not be underestimated. For instance, the drive towards to achieving educational targets and outcomes should not be criticised, but the obsession with learning outcomes has resulted in a neglect of the educational processes. Indeed, focusing on learning outcomes could be viewed as just measuring how successfully 36 teachers have taught pupils to answer questions for a particular test and has little to do with what the child has learnt. For these reasons it is important that qualitative research helps planners understand the processes behind education outcomes, both in relation to the policy process, institutional performance and pupil achievement. However, what is most important is the amount of resources and time required to establish a fully operational EMIS in the developing world. Unless, adequate levels of resources are provided it is impossible to establish the systems and procedures necessary to ensure that a sustainable demand exists for EMIS outputs. Unlike countries in North America or Western Europe those in the developing world often lack the institutional structures and capacity that help generate a demand for EMIS data and for these reasons it can take a considerable amount of time and effort to develop such structures. The links between institutional building, capacity development and data utilisation cannot be underestimated and one which requires time, significant resources and political commitment, without which EMIS will continue to experience difficulties and not be sustainable over the medium term. 37 References AEPM (2007) http://www.aepam.gov.pk/Emis-Purpose.htm, government of Pakistan. Caillods, F (2005) The International Working Group on Education: Supporting Global Education Initiatives`, Journal of Education for International Development ­ 1 (1). Cambridge Education (2004a) Gansu Basic Education Report ­ School Mapping, Government of the People`s Republic of China, Gansu Provincial Education Department. Cambridge Education (2004b) Lao PDR ­ Teacher Training Enhancement and Status of Teacher Project ­ Final Report, MOE,Lao PDR. Cameron, L (2005) Primary Completion Rates ­ technical paper, Education Policy and Data Centre, Washington DC. Chapman, D and Mahlck, K (1997) Changing what happens in schools: central-level initiatives to improve school practices in Chapman,D, Mahlck,L and Smulders, A. (1997) From Planning to Action: Government Initiatives for Improving School-Level Practice, Pergamon: International Institute for Educational Planning, Paris. Chifwepa, V (2003) An Assessment of Educational Management Information Systems in Zambia`, DPMN Bulletin: Vol X, No 1, January Haiyan, H & Herstein, J (2003) Education Management Information System (EMIS): Integrated Data and Information Systems and their Implications`, Paper presented at the Annual Conference of Comparative and International Education Society, New Oleans, LA, InfoDev (2005) Terms of Reference for EMIS Case Studies of Best Practice, Washington DC, World Bank. Kinniburgh, I (2005) Developing Countries and the Millennium Development Goals`, Draft prepared for the Technical Meeting of the Group of 24, Washington DC. Ko-Chin (2005) Interview for the Assessment, Information Systems, Monitoring and Statistics Unit (AIMS), UNESCO. Ladd, L (2003) An Assessment of Ohio`s Education Management Information System, Ohio State Legislative Office of Education Oversight, Columbus. Lindsey, L (1993) An Assessment of Ohio`s Education Management Information System, Ohio State Legislation Office of Education Oversight, Columbus Moses, K (2001) Education Management Information System: What is it and Why do we not have more of it ?` Journal of TechKnowLogia, Jan 2001. 38 Naumann, K (2005) Technical Assistance to the Planning, Budgeting Monitoring and Evaluation (PBMB) Division of the MOE, document 3, Quarter No.3, EC, Ghana. Powell, M (2004) Final Quarterly Report for Project 2 ­ Information Systems for Strategic Planning, Pretoria South Africa, EC. Powell, M and Spareeboom, T (2006) InfoDev study #1253 Education Management Information System: Case Studies and Lessons Learnt, Case Study of Mozambique Powell, M (2005) Economic Restructuring and Human Resource Development, Ashgate, Aldershot. Powell, M (2006) InfoDev study #1253 Education Management Information System: Case Studies and Lessons Learnt, Case Study of Ghana. World Bank (2005) Education Management Information Systems: Guidelines and Referencing for Putting Information Systems to Work in Education, Washington: World Bank Reinikka, R and Svensson (2004) Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda, Research Report, The World Bank and CEPR. Shooebridge, J(2006) InfoDev study #1253 Education Management Information System: Case Studies and Lessons Learnt, Case Study of Nigeria Skills Development Planning Unit (2004) State of Skills in South Africa, Department of Labour, Pretoria, South Africa. Smith, D. (2006) InfoDev study #1253 Education Management Information System: Case Studies and Lessons Learnt, Case Study of Bangladesh. Silva, L (2001) Risky but effective Improvisations in Managing Information Systems, Electronic Journal of Information Systems in Developing Countries ­ Vol.5. UNESCO (2005) EFA Global Monitoring Report ­ The Education for All ­ The Quality Imperative, UNESCO, Paris. USAID, (2005) Meeting EFA: Cost-Effectiveness of Complementary Approaches, Working Paper, Equip 2, Washington. Wako, N (2003) Education Management Information Systems ­ A guide for Young Managers, Harare, Zimbabwe. Yin, R. K. (2002) Case Study Research, Design and Methods, 3rd ed. Newbury Park, Sage Publications. 39 Appendix II: Template used to collect research data The purpose of this template is to facilitate researchers in Bangladesh, Ghana, Mozambique and Turkey with the collection of information. Where possible the lines of questioning should follow the suggestions outlined in this template. By adopting such an approach the project will be in a position to make comparisons between each of the four countries and to identify common patterns, as well as differences. However, we recognise that each of the countries will have their own unique characteristics and it may not be possible to always follow the exact questions. Nevertheless, these should be followed as closely as possible. We anticipate that the complete template should be around 30 pages in length. These templates will enable the development of country reports and will be placed in the annex of the final report. 1. Background Implementation Context Outline the broad education policy framework, including major characteristics and challenges facing the country Broad education policy framework Education strategies being implemented Goals/targets, indicators for different policy areas Projects being funded and role of donors Institutional structures for implementation Location of EMIS and relationship to other structures What information is available? What are the sources of this information? How is it collected/analysed? How does it help tracking of progress towards domestic goals/MDG/EFA? 2. Design of EMIS How do individual countries design their own systems ? How do individual countries design their own systems, including issues associated with gender, regional variation, etc? Do countries use a needs analysis or a strategy driven approach; is EMIS demand or supply driven? How did they identify their own information needs? What determines new information needs? Role of donors, including co-ordination, sector-wide approaches, relationship with local counterparts Role of stakeholders and other civil society groups Assumptions underpinning systems ­ what were the initial expectations about the EMIS and to what degree were end-users satisfied? Costs associated with design ­ were attempts made to identify funding required for sustaining EMIS? Capacity of government to deliver ­ was this taken into account and was training provided? Was it adequate and were recipients retained? Role of private sector/partnerships with universities, both for training and also for data collection/collation Were potential risks identified during the setting-up of systems? 40 3. Technical Issues associated with EMIS Identify the degree to which the appropriate software/hardware were identified and used What technologies were used and how were they funded? How were decisions made about hardware/software? Were licensing agreements put into place and what did this involve ? To what extent are data bases linked? Did the country adopt a complex approach to EMIS (involving complex data bases and network systems) or did they adopt a minimalist approach (excel sheets and emails). How appropriate are the country`s IT systems for the context of the country taking into account, costs, sustainability etc? To what extent are IT systems transparent? (For instance, to what extent can a system enable a district level body to undertake analysis and correspondingly act on this information) To what extent does outsourcing occur in IT and what difficulties have been experienced, such as technical lock-in`? 4. Operations and Management of System Investigate the process involved in data collection, collation, and analysis, including the relationship between central government, regional/provincial offices and schools. What different data sources exist (administrative, payroll, survey etc)? To what extent does data cover all areas (geographical, urban/rural, ethnic groups How often is data collected ? What important data is missing and what proxies are used? Who owns the data ? What are the costs associated with data collection/analysis ? Who is responsible for analysing data (i.e. to what extent does MOE collect data and other bodies analyse or utilise this data?) Who is responsible for managing what? To what extent are quality assurance systems put in place to validate and reconcile data? To what extent is data governed by confidentiality agreements ? How is the EMIS linked into existing planning or financial cycles? What are the organisational and management structures for supporting data collection and analysis, including adequate job descriptions, clearly defined roles and functions of different units/offices? What are the lines of reporting and the position of EMIS in the organisational hierarchy? How is the systems financed and to what degree is the budget adequate/timely. What role do donors play in managing systems? 41 4. The role played by EMIS in strategic planning Identify how outputs from EMIS support policy reform and help decision makers identify areas for intervention, especially relating to resource allocation, equity and areas considered strategic How does EMIS monitor and report on: o provision of physical facilities o provision of materials/equipment o student enrolment and attendance o access/enrolment rates o student flow, cohort analysis o student achievement o school contact hours/days open o teacher demand/supply o teacher/head teacher qualifications o teacher/head teacher performance o in-service training o sources and amounts of school finance? How does EMIS support reform and identify areas for intervention, especially those relating to: o the quality of educational inputs, processes and outputs? o equity of access, participation and outcomes according to gender, ethnicity, poverty, location and disability? To what extent are provincial/regional offices able to analyse the information or do they just receive information from central office (i.e. if regional bodies have ownership they are more likely to utilise information) Links with other areas of strategic planning, such as gender and health, as well as VET systems. 5. Role played by EMIS in operational planning .What role does EMIS play in supporting operational planning at central, regional/provisional and school levels? How does EMIS support operational planning at central, regional/provincial and school levels? To what extent is information in a format that is readily usable by the different levels? To what extent is information used? How is this information used ? 6. The relationship between EMIS and M&E What monitoring and reporting systems exist within the country ? What are the monitoring and reporting systems; do they relate to projects and/or programmes? How do they relate to EMIS ? How do they help countries track progress towards MDG and EFA? 42 7.. The relationship between EMIS and M&E What monitoring and reporting systems exist within the country ? What are the monitoring and reporting systems; do they relate to projects and/or programmes? How do they relate to EMIS ? How do they help countries track progress towards MDG and EFA? 9. Strategies for maintaining EMIS Investigate strategies and procedures for maintaining EMIS What happened to people who received EMIS training, both in relation to specific technical functions and for data collection, data collation and data analysis? To what extent are systems sustainable (including procedures developed for maintaining the system) especially with regard to financial resources? 8. Strategies for dissemination How are outputs from EMIS disseminated ? What format are used to disseminate outputs, e.g. reports, web, newspapers etc? How frequently is information disseminated? How large is the audience? To what extent does demand exist for the outputs of EMIS? Who benefits from utilising the outputs of EMIS ? To what extent is data presented in a useable form? To what degree have stakeholders, teachers and administrators been provided with training to utilise data? Is the information presented in a format appropriate for the target audience 11. Lessons learnt and recommendations What are the lessons learnt and major recommendations ? What factors facilitate best practice (i.e. what is the role of a facilitating/enabling environment)? Why do systems facilitate difficulties? What is the relationship between the system and the local environment, especially in relation to institutional and structural difficulties? How is it possible to develop terms of reference for an EMIS consultant? How is it possible to evaluate whether a consultant is suitable to undertake work? To what extent is the existing system sustainable, effective or efficient ? What changes or reforms should be introduced to make the EMIS more effective, efficient and sustainable 43