E L P GUIDANCE NOTE August 2016 E L P Measuring the quality of early learning programs This brief is intended to provide basic information on measuring the quality of early learning programs using a range of tools and approaches. WHY IS IT IMPORTANT TO MEASURE THE QUALITY OF EARLY LEARNING PROGRAMS? Investment in early childhood programming is increasing in response to convincing evidence on the benefits of supporting the social, emotional, and cognitive development of young children. As countries around the world work to expand access to early learning opportunities, it is critical to ensure the quality of both the services and children’s experiences. High- quality programs can improve outcomes for children and set them on a positive trajectory in life. Low-quality programs, in contrast, are unlikely This brief was produced by the to generate the desired outcomes and can even be detrimental to children’s Early Learning Partnership and reflects significant contributions development.¹ from the following core members of the Measuring Early Learning Quality and Outcomes (MELQO) HOW DO YOU MEASURE THE QUALITY OF EARLY LEARNING? Initiative: Lucy Bassett, Amanda High-quality early learning can be delivered in a range of settings, including Devercelli, Alexandra Solano Rocha, Abbie Raikes, Kate Anderson, Tamar formal preschools, community-based programs, and home-based child Manuelyan Atinc, Pia Britto, Ana care. There is no single blueprint for achieving quality under these different Nieto, and Fabiola Lara. models; however, there are common elements of quality to be considered across preschool settings. For more information or to be It is important to measure both structural and process aspects of added to the ELP mailing list, please write to quality. Process quality refers to a child’s day-to-day experiences in early earlylearningpartnership@ learning settings and encompasses dynamic elements such as interactions worldbank.org with teachers, peers, and materials, the quality of daily routines, and the implementation of the curriculum. Structural elements, in contrast, include a center’s infrastructure and materials, health and safety aspects, characteristics of the relevant groups of children and teachers (such as group size and student/teacher ratio), and caregiver characteristics (such as teachers’ level of education, experience, and salary). (See Table 1.) TABLE 1 Examples of structural and process variables2 Physical environment Infrastructure, availability of equipment and materials, health and safety STRUCTURAL VARIABLES Group characteristics Adult-child ratios, group size Caregiver variables Initial education, training, mentoring/ supervision, salary Interactions Caregiver-child and child-child PROCESS interactions VARIABLES Program characteristics Curriculum, quality of daily routines Some of the common dimensions that are considered when measuring quality include the following (note that the dimensions can be defined and grouped in different ways): Physical environment: The physical space of the classroom and school, including space for each child, the characteristics of play areas and furnishings, the availability of materials, and the safety of the facilities. This dimension often includes cleanliness and access to potable water and toilets. Teaching and learning processes: This dimension, which can be called or include pedagogy, play, or teacher-child interactions, covers the approaches teachers take to engage and teach children (such as individualized learning vs. group learning), and especially the type and BOX 1 quality of interactions among teachers, children, and the environment. It can include A NOTE ON • Play, the emphasis of the program on creating opportunities for TERMINOLOGY children to engage in free and group play; adequate toys and spaces to This brief uses the term play; and “early learning.” Different • Teacher-child interactions, the type and quality of interactions between countries and different institutions may use the teachers and children as well as among children. terms early childhood Curriculum: This dimension measures the extent to which curriculum development (ECD), early content addresses children’s physical, socio-emotional, linguistic, and stimulation, early childhood cognitive development needs and stimulates early literacy and numeracy care and education (ECCE), skills. early childhood education Teacher and school characteristics: This dimension covers the (ECE), preschool, or pre- amount and type of training, professional development, support, and primary school. compensation teachers receive; teachers’ knowledge of child development In this brief, we use the term early learning to refer principles; and whether the number of trained staff is sufficient to to center-based programs maintain appropriate teacher/child ratios for the age groups in the focused on promoting classroom. It also sometimes includes children’s development • Leadership: The leadership skills (such as management, administration, and school readiness, communication, support and guidance) of principals, directors, and generally for children ages administrators; and three to six. 2 EARLY LEARNING PARTNERSHIP Inclusion: The extent to which the classroom is able to support participation by all children, including those with disabilities and special needs. Family and community engagement: This dimension reflects the degree to which families and communities are able to engage in children’s education and the life of the school. Table 2 provides examples of constructs and items used to measure quality early learning. TABLE 2 Examples of quality constructs and items CONSTRUCT SUB-CONSTRUCT ITEM Physical Organization of physical • Does each child have access to his/her own learning materials, like pencils and environment space and materials paper? • Are there places for children to play within classrooms? • Are there enough seats for all children? Cleanliness • Does the environment promote good health practices (e.g., personal hygiene, including hand washing? • Do children have access to clean water and sanitation? Play Materials and • Do all children have access to free time for play? opportunities for play • Do all children have access to materials for play inside the classroom? Teacher-child Positive interactions • Does the teacher smile, clap, or offer positive words of praise for children’s interactions efforts? • Are children allowed/encouraged to interact with one another? Teacher Knowledge, skills, • Do educators/caregivers demonstrate knowledge of child growth, development, characteristics qualifications of personnel and learning, and if so, are they able to apply this knowledge to practice? Family and Engaging families and • Are parent workshops on education matters held? community communities in program engagement Source: These items and variations appear in a number of the assessments included in Annex 1. What are the different methods available to collect data? Because of the complex nature of measuring quality, various forms of data collection are used. There are three main mechanisms: direct observation, self-reporting by qualified informants, and reviews of existing documentation. Utilizing more than one mechanism can provide more information and improve the precision of quality measurement, and some measurement tools combine different mechanisms. Direct observation: Direct observation is the best way to effectively capture the interactions and experiences at a school or center. This method requires more training, resources, and time than other methods, since observers must be well-trained to collect information on specific elements of quality defined in the instrument and usually spend more than an hour in a given school. Videotaping or photography can be used to capture information that can be coded and interpreted at a later time and—in some cases—by another person (often with a higher degree of training and expertise). Measuring the quality of early learning programs 3 Self-reporting by qualified informant: Structured interviews or surveys are used in this method to gather information on the dimensions of interest. Informants can be teachers, school administrators, parents, or children. Examples of information that can be acquired from teachers or administrators in this way include the percentage of students with disabilities and the level of education required to be a teacher in the school. Review of documentation: This kind of review can be useful to gather information on structural aspects of the early learning environment. Examples of documents to review include curricula, classroom guidelines, and safety protocols. How do we score results? A variety of different scoring methods can be used to evaluate the data you have collected using the methods described above. These include checklists, Likert scales, and time sampling. Checklists are fast and easy methods for collecting data. They mainly use yes-or-no options, and can be used to record observations for individuals, groups, or a whole class. Likert scales are rating scales that usually offer four or more response options, allowing enumerators to represent a range of performance levels. Effective Likert scales use descriptors with clear options, such as frequencies (e.g., “frequently, sometimes, rarely, never”), and avoid subjective or relative descriptors, such as “fair” or “good.” Time sampling is a method by which enumerators repeatedly evaluate the setting in specific time increments (e.g., one minute, 10 minutes) in order to measure aspects of quality throughout the day. How do we select an instrument? Defining why you are interested in measuring quality will help you determine the type of measurement tool you need. For example, your purpose could be to assess the quality of an intervention that is being evaluated to better understand its impact. The purpose of measuring quality could also be to identify critical areas of improvement and resources needed, or to inform key policy decisions (including budget, staffing, and licensing), or to contribute to quality enhancement processes (such as for accreditation). The purpose of your measurement will have implications for the resources, time, and stakeholders you will need to conduct the assessment. It is important to keep in mind that the best tools to measure quality will have reliable and valid measures (see Box 1). For example, when adapting items to the cultural context, you cannot assume that the reliability and validity of the original items will be automatically passed on to the adapted items. Instead, you will likely need expert guidance to ensure the reliability and validity of the modified tool. It is recommended to document reliability before and during fieldwork to ensure that scores are consistent over time. Annex 1 provides examples of other commonly used tools, the dimensions they measure, and the countries where they have been used, among other details. All the tools featured there have been validated, used in at least one developing country, and adapted on at least one occasion to be culturally appropriate. They are all used to help identify what improvements are needed in preschool settings and can track improvements in quality when changes are made. Annex 2 shares information on the Measure of Early Learning 4 EARLY LEARNING PARTNERSHIP Environments (MELE) module, which was produced by the Measuring BOX 2 Early Learning Quality and Outcomes (MELQO) initiative. This tool RELIABILITY & includes items and technical guidance to support countries in measuring VALIDITY the quality of early learning. Reliability refers to the extent to which a test What are the key considerations when choosing an instrument?3 will consistently provide Following are the key considerations that matter when one is choosing an similar scores when and if instrument to measure the quality of early learning programs. administered to a child or Purpose. What is the purpose of your measurement exercise? Has the group of children over time. instrument been used before for the purpose you’ve defined? Validity refers to how Adaptation. Was the tool designed for use in the country or region where well a test or assessment you are working, or will you need to adapt the tool to country context, measures what it intends to culture, or language—or all three? measure. Cost. How much does the tool cost? Plan for training, data collection, Concurrent validity refers in-country workshops or meetings, and, sometimes, adapting and to the extent to which the contextualizing tools to fit local settings. Even when the cost of an results of measurement assessment is low or free, implementation costs can add up, especially in correspond to those of the case of direct observation. a previously established measurement for the same Training. What level of training is required to administer the tool? Some construct. tools require that interviewers complete a specific (and sometimes costly) training as well as pass reliability tests (such as the Classroom Assessment Predictive validity refers to Scoring System or CLASS; see Annex 1) whereas others do not require the extent to which results formal training. of a test are related to later Repeatability. Will your project have funds to use the same tool more than performance that the test was designed to predict. once to track progress? Conducting the same assessment repeatedly can yield valuable information on trends in quality. How do we interpret and use the findings? Once you have identified what needs to be improved using the right tools, it is important to discuss your findings with all the relevant stakeholders to explore options for making improvements in early learning. Your findings can be used by policymakers, teachers, and other stakeholders to select priority areas and determine how to improve the quality of settings, for example. In that case, the elements to be improved might be expanding and improving the physical space, providing additional classroom materials, integrating nutrition and health programs, improving the management of early childhood care and education programs, increasing teacher qualifications and training, or for strengthening connections with parents and community members—or some combination of these elements. How can we integrate quality measurements into a national monitoring system? Ideally, early learning quality measurement will become integrated into a national monitoring system to track changes in quality over time; inform standard setting, curricula, and teacher training; and influence decisions about resource allocation for quality improvement. Key considerations include the following. Having a designated government agency. There should be a unit in government that is intimately involved in the design and implementation of the first quality measurement effort, with capacity, willingness and mandate, to maintain ongoing quality measurement efforts. Measuring the quality of early learning programs 5 Having personnel on hand to collect data. Who is available in country to collect data regularly over time? This could be local education officials who already inspect education facilities (e.g. district education officers) or, in some cases, local university graduate students with expertise in child development. Ensuring that new data can be integrated. What are school inspections covering at present and how can additional quality measures or data collection processes be integrated most effectively? Similarly, it is important to know what information system is currently available and in use in the country. Can indicators of preschool classroom quality be integrated into school census efforts that are ongoing and channeled into a national education management information systems (EMIS)? Are other related quality measurement efforts routinely carried out with which you could align yours? How much will measuring quality cost and how long will it take? 4 Budgets and timelines can vary significantly depending on the country context, the tool being used, sample size, data collection methodology, and, especially, the purpose of measurement. For example, measurement for a project evaluation could be less expensive than measurement for a nationally representative study with a larger sample size. In thinking about cost, it is important to consider expenses related to licenses for tools, adaptation workshops and pilot testing, translation, fees for approval of modified instruments (where relevant), and training and salaries of enumerators/assessors (which varies depending on their level of training and qualifications), as well as data collection, analysis, and dissemination. Measuring the quality of settings can take time. Table 3 shows an example of a daily data collection schedule from Ecuador. TABLE 3 Example of daily data collection5 HOUR OBSERVER SUPERVISOR Arrival and preparation, selection 7:30 of group to be studied 8:00-12:00 Filming/coding instrument(s) 12:00-14:00 Caregiver interview Structural quality questionnaire 14:00-15:00 Coordinator interview Table 4 shows the types of budget items needed to prepare for and measure quality, with examples from both a project evaluation and a nationally representative study. 6 EARLY LEARNING PARTNERSHIP TABLE 4 Sample budget COUNTRY B ESTIMATE OF COUNTRY A TASK (NATIONALLY TIME NEEDED (PROJECT EVALUATION) REPRESENTATIVE STUDY) 1–4 days, including PREPARATION Preliminary meeting planning and US$500 US$1,000 execution 10 days (often Expert time for adaptationa ~US$400–600/ US$4,000 – US$6,000 US$5,000 – US$7,000 day) Local meeting(s) (including ADAPTATION per diems, space, pre– 5 days US$1,000 – US$3,000 US$2,000 – US$5,000 piloting) Depends on Translation of tools language, length US$2,000 – US$4,000 US$2,000 – US$4,000 of tool International expert 10 days (often time for training (master ~US$400–600/ US$3,000 – US$5,000 US$4,000 – US$7,000 trainers) day) 1 week mission TRAINING Travel (including experts, for each US$4,000 – US$20,000 US$4,000 – US$20,000 TTL) participant Depends on Space rental, materials, etc. US$500 – US$2,000 US$500 – US$2,000 country Data collection (transport, enumerators’ salary/per US$30,000 – US$45,000 US$120,000 – US$200,000d Depends on diem,b etc.), materials (sample sizec = 80 (sample size = 600 country DATA (tablets, questionnaires, classrooms) classrooms) etc.) COLLECTION, ANALYSIS, AND Data analysise and report ~4–6 weeks US$4,000 – US$10,000 US$5,000 – US$20,000 DISSEMINATION writing 1 event Dissemination (US$1,000– US$1,000 US$2,000 3,000) 1–4 missions in OTHER General TTL travel one year US$5,000 – US$20,000 US$5,000 – US$20,000 TOTAL US$55,000 – US$126,500 US$150,500 – US$288,000 a Includes time to update tools and prepare manual, if needed. b Depends on enumerators’ level of training required. c Note this is NOT a representative sample. d Note this amount can vary greatly depending on the country context, transportation costs, etc. e Can include psychometric analysis, statistical analysis, etc. Measuring the quality of early learning programs 7 Table 5 is an approximate timeline, meant to illustrate the steps needed to undertake the measurement process and how this would be sequenced over time. In reality, depending on the specifics of the quality measurement objective, the steps could be combined or expanded and could happen more or less quickly than noted in the table. TABLE 5 Approximate timeline MONTH 1 MONTH 2 MONTH 3 Hold meeting(s) to identify main research questions of X government (local/regional/global experts) Appoint task force or focal point from government X Review existing curriculum and service delivery X standards and align with assessment domains Propose study design based on government priorities X X LAUNCH AND Map out project timeline and budget X X PLANNING Write and release ToR for data collection Get ethical approval as needed Gather necessary background data for sampling purposes Select contractors and finalize terms of contract Translate and back-translate tools Hold in-person meeting(s) with national experts ADAPTATION AND (including curriculum developers, academics conducting ECD research, school inspectorate (or equivalent)) to PRE-FIELD TESTING adapt items/measures and align with curriculum Test tools on small sample and further revise and adapt tools as needed Train enumerators for data collection (training should include access to children and classrooms so enumerators can practice using the instruments) PILOTING/ DATA Collect data COLLECTION Clean data Analyze findings Synthesize findings in easily digestible reports for different audiences ANALYSIS AND Distill findings for policy makers based on current policy plans SYNTHESIS OF Disseminate findings through meetings and other RESULTS AND events with national policy makers, academics, civil DISSEMINATION society, donor partners, regional/district education officials, teachers/principals/school managers, and parents/general public Ensure findings are accessible/understandable for teachers and parents, etc. 8 EARLY LEARNING PARTNERSHIP MONTH 4 MONTH 5 MONTH 6 MONTH 7 MONTH 8 MONTH 9 MONTH 10 MONTH 11 MONTH 12 X X X X X X X X X X X X X X X X X X X Measuring the quality of early learning programs 9 HELPFUL TIPS Distinguish between minimum and ideal standards. Consider making a distinction between minimum standards and ideal standards. It is critical to identify the minimum health and safety standards to which any center must adhere to ensure children’s basic safety and security (such as that each classroom has light, ventilation, and a clear exit). In some countries these requirements are called minimum standards. Then there are the kinds of things that are helpful to have in each classroom, but not absolutely essential (such as a chair for each child). Ensuring the balance between minimum and ideal standards can help ensure that providers who could provide more affordable programs of sufficient quality are not forced out of the market. Understand the early childhood education context. Before beginning the quality measurement, it is critical to understand key contextual variables. These include school location (urban/rural), the school system’s organizational structure and human resources, dates and schedules of operation, daily routines, and the monitoring and evaluation system, among other variables. Ensure that selected dimensions of quality reflect country priorities. A country’s priorities, upcoming policy decisions, and political economy issues should guide the selection of quality dimensions to measure to ensure that results will be useful for decision making. Even if this process takes additional time at the outset, it will pay off in the usefulness and relevance of the results. Time quality measurement appropriately. Start the measurement of quality about one month after the beginning of the school year, to ensure that teachers have time to adjust to new students and early kinks can get worked out. Select tools that are feasible to implement. When the main purpose of the measurement is monitoring, simpler quality measures are recommended. If financial or human resources are limited, simplified tools and adapted approaches can be used. For example, short rating scales can be used along with videotaping, which can later be used to code the more complex constructs. Adapt tools. The tools presented here can be applied in a number of settings but should be adapted to be fully responsive to local values and context. Adaptation may involve translation, modification of content, or adaptations to the process of administration. Ensure that enumerators are appropriately trained. Enumerators should have significant knowledge of child development and early childhood education settings and receive the necessary training in the specific quality measurement tool. 10 EARLY LEARNING PARTNERSHIP Measuring the quality of early learning programs 11 ANNEX 1 Selected tools to measure the quality of preschool settings TOOL DESCRIPTION DOMAINS Arnett Caregiver For children ages 36 – 60 months 4 domains: sensitivity; harshness; Interaction Scale (CIS) detachment; and permissiveness (26 items) CLASS CLASS Pre-K is an observation-based tool that assesses 10 dimensions of classroom quality across (Classroom Assessment classroom quality in settings for children ages 36 months to these 3 domains: emotional support; Scoring System) Pre-K kindergarten. All observers are required to receive prior training classroom organization; and instruction and are encouraged to use videotape footage. support Early Childhood Developed at the University of North Carolina at Chapel Hill, 7 domains: space and furnishings; personal Environment Rating ECERS is designed to assess group programs for children ages care routines (health and safety); language Scale (ECERS-R) 2 to 5. The tool is mostly used for policy development, program and reasoning; activities; interactions; evaluation, improvement advocacy, and training. ECERS-R program structure; and parents and staff. includes classroom observation and a teacher’s interview from the enumerator. The enumerator or observer should receive prior training. Acei Global Guidelines The GGA is an instrument designed to help ECCE professionals The assessment includes environment Assessment (GGA) systematically assess the quality of their programs. GGA is and physical space, curriculum mostly used for self-assessment by centers, to design new early content and pedagogy, educators and childhood programs, or to improve existing programs. caregivers, partnerships with families and communities, and children with special needs. International Step By Principles of Quality Pedagogy is a tool developed by ISSA 7 focus areas, which include: interactions; Step (ISSA) Principles to define quality in ECCE teaching practices and classroom family and community: inclusion, diversity of Quality Pedagogy environment and is primarily used for planning and improvement. and values of democracy; assessment and planning; teaching strategies; learning environment; professional development. Measure Early Learning The MELE is used to measure the quality of early learning The MELE addresses environment and Environments (MELE) environments for children ages 3 to 6. It includes a classroom materials; teacher-child interactions; observation tool, teacher/director survey, and parent survey. pedagogy and approaches to learning; family and community engagement; inclusion; and play. Stallings Classroom A tool to gather information on the interaction between teachers The focus areas are teachers’ use of Snapshot instrument and students in the classroom. instructional time; teachers’ use of (or Stanford Research materials; core pedagogical practices; and Institute Classroom teachers’ ability to keep students engaged. Observation System) 12 EARLY LEARNING PARTNERSHIP LENGTH COST COUNTRIES AND LANGUAGES CONTACT 90 minutes: 2 observation Manual and score Used in Bermuda and the United States Instructions available at http://www.eec. cycles of 45 minutes, on sheets are readily state.ma.us/docs1/qris/20110121_arnett_ separate occasions accessible and at scale.pdf no cost 80 minutes (suggested): 4 Training costs Mainly used in the United States. Also Contact: Paul H. Brookes Publishing Co, classroom observations of 20 vary from US$670 used in Chile, Finland, and Portugal www.brookespublishing.com minutes each to US$1,500 per person for 2 to 5 days of training. Manuals cost US$50 each, and a pack of 10 scoring forms costs US$25. 2 to 3 hours; contains 43 US$19.95 for the Used in Austria, Bangladesh, Canada, http://ers.fpg.unc.edu/early-childhood- items and 7 subscales manuals and US$59 Chile, England, Germany, Greece, environment-rating-scale-ecers-r for the training Hungary, Iceland, India, Italy, Korea, videos. Portugal, Russia, Singapore, Spain, Contact: Sweden, and 7 Caribbean countries. A Richard Clifford, dick.clifford@unc.edu Spanish version of the tool is available. N/A Freely available Used in more than 35 countries, For more information, visit http://acei. including Canada, Guatemala, India, org/images/stories/GGAenglish.pdf and Kenya, Mexico, Peru, Sierra Leone, and http://acei.org/acei-news/acei-global- Thailand. Available in Arabic, Chinese, guidelines-assessment.html English, French, Greek, Nepali, Russian, Contact: Belinda Hardin, bjhardin@ and Slovak. uncg.edu and Doris Bergen, bergend@ muohio.edu N/A Freely available Used in more than 27 countries, mostly For more information, visit: http://www. in Eastern Europe, including Moldova, issa.nl/content/issa-quality-principles Slovenia, and Tajikistan. Contact: Liana GHENT, lghent@issa.hu Depends on how module is Free Used in Nicaragua, Tanzania Contact: Lucy Bassett lbassett@ adapted by each country worldbank.org; Abbie Raikes araikes@ unicef.org; Kate Anderson klanderson@ brookings.edu 10 “snapshot” observations N/A Used in Brazil, Colombia, Honduras, of 15 seconds each, over the Jamaica, Mexico, Peru, and the United course of one class period. States, among others Measuring the quality of early learning programs 13 ANNEX 2 Measure of Early Learning Environments (MELE)6 The Measure of Early Learning Environments (MELE) was developed by the Measuring Early Learning Quality and Outcomes (MELQO)7 initiative led by the World Bank, UNICEF, UNESCO, and the Center for Universal Education at the Brookings Institution; it engaged experts and partners from around the world. The MELE was developed through a consultative process, drawing on the best experiences and tools for measuring the quality of early learning environments to date, and its items will share similarities with items from many of the tools profiled in Annex 2. Given the state of evidence on quality in early learning environments and the strong cultural influences on what defines “good quality”—the number of items considered universally cross-culturally relevant was small—the MELE focuses on constructs rather than on specific items. A range of items are available for countries to select from, based on their own notion of quality. The MELE includes a classroom observation tool, a teacher and director survey, and a parent survey. Quality is measured through six constructs: • Environments and materials: Safety and cleanliness of the physical space; access to clean drinking water, nutritious meals, and adequate sanitation facilities; and whether a variety of culturally relevant and meaningful learning materials are available (such as visual displays, books, art supplies, and musical instruments). • Teacher–child interactions: Children experience daily interaction with teachers and school staff who are nurturing, emotionally supportive, trained in pedagogy and early childhood development (ECD), and attuned to children’s individual needs. • Pedagogy and Approaches to learning: Child-centered teaching encourages curiosity, persistence, attentiveness, cooperation, participation and active engagement; children engage in age-appropriate play, activities and routines; curriculum addresses children’s physical, socio-emotional, linguistic, and cognitive development needs and stimulates early literacy and numeracy skills. • Family and community engagement: Programs share information, promote positive relationships, and create opportunities for parent and community engagement; families and the local community are actively involved in planning, decision making, and action to improve early childhood care and education (ECCE). • Inclusion: All children and families have access to high-quality ECCE services; teachers speak the home language of the majority of students; teachers are trained in providing ECCE to children with disabilities and special needs and in fostering age-appropriate development for all children. • Play: Children are given ample opportunity to explore and enjoy learning through play, with time for pretend play and interactions with peers. Ideally, the MELE can be used to measure a nationally representative distribution of the quality of early learning environments (usually for children aged 3-6), which could then inform policy planning and budgeting, including: resource allocation, curriculum design and teacher training programs, early learning standards, and ongoing monitoring of quality. The MELE module, along with manuals, guides, and other resources, is available upon request. 14 EARLY LEARNING PARTNERSHIP Measuring the quality of early learning programs 15 ANNEX 3 Resources For more information on instruments or tools to measure the quality of early learning settings, please consult: Bouguen, A., D. Filmer, K. Macours, and S. Naudeau. 2013. Impact Evaluation of Three Types of Early Childhood Development Interventions in Cambodia. Policy Research working paper no. WPS 6540, Impact Evaluation series no. IE 97. Washington, DC: World Bank. Britto, P. R., H. Yoshikawa, and K. Boller. 2011. “Quality of Early Childhood Development Programs in Global Contexts: Rationale for Investment, Conceptual Framework and Implications for Equity,” Social Policy Report (Society for Research in Child Development), vol. 25, no. 2 López Boo, F., M. Caridad Araujo, and R. Tomé. 2016. How Is Child Care Quality Measured? A Toolkit. Washington, DC: Inter-American Development Bank. Naudeau, S., N. Kataoka, A. Valerio, M. J. Neuman, and L. K. Elder. 2011. Investing in Young Children: An Early Childhood Development Guide for Policy Dialogue and Project Preparation. Washington, DC: World Bank. Neuman, M. J., and A. Devercelli. 2013. What Matters Most for Early Childhood Development: A Framework Paper. SABER (Systems Approach for Better Education Results) working paper no. 5. Washington, DC: World Bank. Raikes, A. 2014. Early Childhood Care and Education: Addressing Quality in Formal Pre-Primary Learning Environments. Paris: UNESCO. UNICEF. 2014. A Framework and Tool Box for Monitoring and Improving Quality (draft). ECD Framework PART II (2012). (Accessed online August 2014.) World Bank. 2015. User Guide – Conducting Classroom Observations Analyzing Classroom Dynamics and Instructional Time Using the Stallings “Classroom Snapshot” Observation System.” Washington, DC. Young, M., and L. Richardson, eds. 2007. Early Child Development: From Measurement to Action–A Priority for Growth and Equity. Washington, DC: World Bank. NOTES 1 Bouguen et al. 2013. 2 Adapted from Neuman and Devercelli 2013 and Naudeau et al. 2011. 3 These considerations were compiled based on anecdotal findings from the experience pre- piloting MELE, the MELQO tool, specifically through conversations with MELQO’s Technical Advisory Group for Child Development and Early Learning Outcomes. This section on budget and timeline was also compiled based on experience pre-piloting the 4 MELQO. 5 The example in Table 3 is from López Boo, Araujo, and Tomé 2016. 6 Information on MELE and MELQO in this annex appears thanks to the MELQO Core Team and Technical Advisory Group. 7 Note that the MELQO initiative also developed a module, called the Measure of Development of Early Learning (MODEL), to assess child development and learning. The two modules (MODEL and MELE) are complementary and are both meant to be adapted to align with national systems and standards and to be used to inform policy decisions to improve early childhood development. 16 EARLY LEARNING PARTNERSHIP