2 1626 4iUMANDJVELOPWEU Effective Schools and Teachers NP / WoR K Basic Education Cluster EDUCATkN T H E W 0 R L D B A N K Evaluating School Performance Tools and Approaches Putting self-evaluation in place Professor Kathryn Riley World Bank and University of Surrey Roehampton Professor John MacBeath University of Cambridge October 2000 Putting School Self-evaluation in Place Kathryn Riley and John MacBeath This booklet has been developed by the authors following a session at the World Bank's Human Development Week in Washington, February 2000 which focused on evaluating school performance. The session was organised by the Effective Schools and Teachers Thematic Group which is part of the Basic Education Cluster. The Cluster brings together expertise in adult and community education, girls' education, school health, teaching and leaming, school effectiveness and improvement, training and development. The booklet provides practical insights into school self-evaluation (or self-review as it is also called) and offers a number of tools and which can be adapted to a range of settings. A broader coverage of the issues discussed in this booklet, and a range of references, are included in 'The Effective Schools and Teachers' Web-site'. To visit the Effective Schools and Teachers web-site: From the Bank's external web-site: From the Bank's internal web-site: > Go to http://www.woddbank.orq > Click on Knowledge management 0 Click on Development Topics ;> Click on Education > Click on Education > Click on Effective Schools and Teachers > Click on Effective Schools and Teachers > Click on Topic Brief at the top. This will Indicate the resources available. 1 1. Putting self-evaluation in its place Education reform is usually seen as the business of Govemments. However, when we think about reforming education and improving schools, we also need to think about how to involve teachers, parents, pupils, govemors and the wider education community. A self- evaluation approach can be used to enable school communities to come to grips with some of the key issues: * How good is our school? * How can we make it better? * Are teachers' skills being put to good use? * How good is learning and teaching in our school? * How do we know? Education reforms are driven by a range of economic and social imperatives. Education has a critical part to play in reducing poverty. In many developing countries, the provision of basic education is a major challenge (particularly in the context of the spread of HIV/AIDS, and the social and economic upheaval which this generates). For countries in transition, education is closely linked to the social transformation of society. In countries in which dissent has been discouraged for many years, education can support democracy, by encouraging pupils and teachers to be questioning and challenging. Whatever the impetus for education reform, there is a growing awareness that the structural changes which have been typical of many of the reforms of recent years have had insufficient impact on what happens in the classroom, and on teaching and learning. One of the reasons for this is that many reform initiatives have been top down, often leaving those involved in a state of confusion. Many governments have sought compliance for their reforms, but the compliance mentality generated by many large-scale reforms can exclude communities. foster institutional inertia and stifle professional creativity. Another key issues (which Dakar 2000 has reminded us of) is that education expansion is not enough. It needs to be linked to quality improvement. One important way of improving the quality of schools is by capturing the 'voices' of the different groups closest to them - parents, pupils and teachers - and by bringing them intro the change and improvement process. The approach to school self-evaluation adopted in this booklet gives some practical suggestions about how to do that. School self-evaluation can help us to take into account different perspectives, and through a process of dialogue with the many stakeholders (administrators, teachers, parents and pupils) create shared understandings of what makes schools 'good'. A focus on school- evaluation enables us to think not just about how schools are organised, but also about the value of education itself. 2 Notions of what constitute a 'good' school are bound in culture and content. They change over time, and different 'stake-holders' may hold different view about what a 'good' school is. Table 1 summarises the views of students from the UK, Uzbekistan, Thailand and South Africa about what makes a good school. Table 1: What makes a good school? (How pupils see it) UK UZBEKISTAN THAILAND SOUTH AFRICA Safety and Security Enjoy School * * * Good Teacher-Pupil Relationships Support for Learning Difficulties Good Communications with * * * * Parents Things to do out of Lesson Time Support for Teaching from Senior Management Teachers Listen to Pupils' Views Variety of Teaching Methods * * No Vandalism Running Water/Toilets Materials for Learnin[ Cool Classrooms Whilst context clearly matters (children in South Africa, Thailand and Uzbekistan identified access to running water and toilets as basic requisites for leaming) children from all four countries identified the following three features as key: * Good teacher-pupil relationships; * Support for leaming difficulties; * Good communications with parents. School self-evaluation provides an opportunity to widen the dialogue about what those features are, and how they can be achieved. 3 2. The goals of self-evaluation School self-evaluation is a journey. Before starting on the self-evaluation journey, it is important to think about the destination, and how the journey will be undertaken. It is also important to take stock of the current state of affairs in the school by trying to answer such questions as: * What is it like to be a part of the life of the school? * How do teachers and pupils see it? * Are pupils really engaged in leaming? * Are they thriving in the school community? Pupils can achieve good exam results but still be unhappy about many aspects of their school life. For a number of pupils, school can be a profoundly disappointing and unhappy experience. At the heart of self-review is the belief that self-knowledge and self-efficacy are as important for schools as they are for individuals. Self-evaluating schools are less dependent, less vulnerable and more accountable for their actions than other schools. They encourage teachers and pupils to be reflective and self-critical. The evidence is that tools of self- evaluation, built into the everyday life of the school can improve the quality of leaming and teaching within a school, as well as its ethos and leadership. Engaging in school self-review is about embracing the riches, as well as the foibles of schools, in ways that will improve the lives of teachers, and the lives and life-chances of pupils. Self-evaluation provides schools with the opportunity to listen to teachers, parents and pupils: to gain their views about what the school is, and what it could be, and to use that information to make changes. School self-review is not another 'add on' but something which is fundamental to the way schools works. What we know about self-evaluation is that it cannot be imposed. It is not the Trojan horse to be used to capture teachers unawares. If it is to work, it needs to be introduced as a collaborative exercise aimed at making improvements in the daily life of the school. There are seven important steps to the self-evaluating school: 1. Promote a leaming climate (for teachers, as well as pupils); 2. Identify the green shoots of growth (the opportunities for development;) 3. Identify the barriers to change (what is getting in the way); 4. Share leadership (recognise that good schools are created by more than one person, the headteacher or principal); 5. Create intelligence from within (build on the skills and knowledge of the school community); 6. Use 'critical' friends (listen carefully to those who are close to the school e.g. govemors and parents); 7. Build resilient networks (to support the aims of the school). 4 3. Developing system capacity for self-evaluation The concept of school self-evaluation is a new one in many countries and contexts. Before starting at the school level, a capacity building activity could be introduced with the support of a development teamThis activity would have a number of linked stages: 1. The development team familiarises itself with the system. They are first and foremost listeners and leamers, following the maxim 'seek first to understand before seeking to be understood'. 2. The team works together with designated personnel (probably from the Ministry, Higher Education and the school system), to map or 'audit' the bases of self-evaluation that already exist within the system. No school or teacher can operate without some form of self-evaluation however implicit, informal and unsystematic. Recognition of this is important for developing a sense of confidence and ownership, and willingness to participate in building a more structured and formal process. 3. Through workshops, the development team engages with key personnel (Ministry/DistricVHigher Education/school levels) in a sequence of activities which help to make explicit the implicit criteria of effectiveness on which schools base their work. As they become explicit, they are open to challenge and revision. This leads to a set of key criteria, or 'indicators' of good schools and begins to identify what may be needed as an effective support infrastructure. 4. This exercise can be conducted at a number of different levels and can involve some or all of the following groups: regional administrators and advisors, school principals/management teams, teachers, parents, pupils/students. The process is not only helpful in terms of outcomes - a self-evaluation framework - but it also has a number of tangible benefits for those who are engaged in the process. For example: - modelling a process of dialogue; - making explicit implicit beliefs and values; - establishing a consensus of school and system goals; and - identifying specific expectations and concerns of different stakeholder groups. 5. The next stages in the process is to agree a cluster of schools which are interested in piloting the initiative, and to hold an introductory workshop for those who would be involved in supporting those schools. Involvement from the district level is critical here. This workshop would aim to develop the capacity of those educators to work as critical friends in supporting self-evaluation at school level. The workshop would need to: - agree a procedure for working with schools; - model a process, for example, for a 1-2 day workshop for school principals and other personnel, taking them through a process of rationale, evidence, criteria, tools, logistics and issues at school level. 5 6. Following the workshop, there would need to be piloting and testing of the approach to self-evaluation with a small group of volunteer schools, say 5-10. This would be followed by monitoring and evidence on school practice in self-evaluation. This would lead to a refining and adapting process of the criteria and tools and workshop activities; a re- affirmation of the framework for self-evaluation; and consideration of issues for a roll out of the programme on a wider basis. 4. Carrying out school self-evaluation at the school level One possible starting point for school self-evaluation is with the findings of research into effective schools. These provide us with a clear set of criteria which can be turned into indicators, benchmarks and measuring tools which are then introduced into schools by mandate or legislation. However, in its crudest form this classic top down approach is likely to engender resistance, and indeed there is much evidence to demonstrate that this is the case in practice. Coming at it from the opposite end has salient benefits which have proved themselves in practice. By involving key stakeholders (teachers, students, parents) in the developmental process, a deeply rooted sense of commitment and ownership is generated. Commitment is built up through a number of simple steps. At an individual school level these could be as follows: 1. Stakeholder groups are formed - teachers, administrators, parents, students (and any other group the school wishes to involve). Experience suggests that each group should consist of about 5-7 people who are as representative a cross-section of opinion as possible. In a small school, there might be only one parent group, one teacher group and 2-3 student groups taken from different age levels. In a large school, there may be as many as 10 student groups, 5-6 teacher groups and 3-4 parent groups. A facilitator is appointed to work with the groups. 2. The task of the groups would need to be along the following lines: Stage 1: Each individual in the group is given a few minutes to write down what for them are five key indicators of an 'effective' school (or a good teacher, or whatever the focus is). Language has to be adapted for different groups (for younger children - '5 things that make a good school'). Very young children can be given the option of drawing or painting things they like, or dislike, about their school. Stage 2: The group pools their ideas and agrees on a composite list - in the region of 5-10 key items. The quality of the discussions that lead them to their conclusions is very important, and it is here that the facilitator can play a key role, ensuring voices are heard, encouraging a supportive but critical sensitivity within the group. Staqe 3: The group is asked to consider how they might gather evidence on each of the indicators which have been developed. The facilitator may need to inject some ideas but 6 also encourage people to think of what already exists and how it might be built on or modified. The group is encouraged to identify/develop just one strategy or 'tool' for evaluation. StaQe 4: The findings of all the groups are pooled. These may be categorised to illustrate issues from different perspectives. One example of this is shown in Table 2. Table 2 What makes a good schools? (Different perspectives) Teachers Management Parents Students + Support for + Effective * A safe and * Teachers who listen teaching leadership ordery to you + Good * A stimulating ethos environment * Teachers who can information and * Good reputation in * Prompt action to control the class communication the community deal with flow problems Altematively, these items may be arranged in a Venn pattem (Diagram 1), with areas of overlap showing the common items, and where there is no overlap, identifying items that are of relevance to different stakeholders. Diagram 1 (Venn pattern) What makes a good school - common and differing views parents management teachers students Staae 5: The school's own indicator set may then be compared with other examples, from the research literature, or from other schools in the locality. Gaps may then be filled and modifications made. This smooths the path for people to engage with the literature or other schema because they already have a stake and an interest in the issues and can make meaning from what were previously alien vocabulary or concepts. Stage 6: One indicator may then be taken and worked through as is shown in Table 3. 7 Table 3 School Climate Indicators The indicator Evidence Evidence Tools (qualitative) (quantitative) + There is a climate + Quality of students' * Measures of * Achievement testing of achievement in work achievement, * Sampling of the school * Pupil, parent, progress and students' work teacher views value added * Records of * Values placed on + Attendance achievement different kinds of records by school, * Shadowing (e.g. of achievement and by pupils to see the class/subject school from their perspective) With the confidence gained from this exercise, other indicators may then be tackled, a range of indicators developed and a toolbox of strategies made available for different purposes. Throughout all this process, the local district or administration can play a helpful role, particularly by encouraging schools to work together, and by helping them leam from each other. 5. The toolbox In the next section we give some examples of classroom based and school level tools which can help the support a school self-evaluation approach. Some tools for the classroom Force field analysis As we have learned from school effectiveness research, the key influence lies at classroom level. When teachers and students are in possession of simple easy-to-use tools, they can discover a great deal about the nature, quality and progress of leaming. An example of this is the force field. It simply asks students to suggest three things that help them leam better and three things that hinder their learning. In a class of 30 this would generate 90 positive and 90 negative items in a space of a few minutes. Things that help Things that hinder -110 4 _ -----------* 4 8 As the following example illustrates, these can be very helpful to teacher and the class in addressing both common and individual difficulties. Below are just a few of the 180 items which pupils raised using this activity, following an extended project set by a secondary school Geography teacher. Things that helped-* *- Things that hindered + Working with different themes to summanse * During the first week we had no idea how written articles in one's own words to manage the whole thing * Having time for reading and questions * Some texts were too difficult * Various books were not to hand as other classes were using them * It's hard not just to copy but to sum manse in one's own words. Spot check Another instrument is the spot check which examines what pupils are doing and learning at a given moment of time in their classroom or home study. The spot check is based on research undertaken by Csikzentimihalyi (1993) and gives a snapshot at any given moment of a student's state of concentration, anxiety, boredom or 'flow'. What we have found is that when teachers use this instrument at a random moment in the class and take in the feedback (anonymously so that individual students aren't identified) the data is highly revealing. Spot Check Concentrating 1 2 3 Thinking about other things Alert 1 2 3 Drowsy Relaxed 1 2 3 Anxious Wishing to be here 1 2 3 Wishing to be somewhere else Happy 1 2 3 Sad Active 1 2 3 Passive Excited 1 2 3 Bored Time passing quickly 1 2 3 Time passing slowly Full of Energy 1 2 3 Very little energy Something at stake 1 2 3 Nothing at stake Sociable 1 2 3 Lonely Easy to concentrate 1 2 3 Difficult to concentrate Cheerful 1 2 3 Irritable Easy to be creative 1 2 3 Difficult to be creative Some schools have developed the instrument and used it in different ways, for example, completing the spot check from three perspectives - students, teacher and observer, triangulating the findings to identify new insights and new opportunities for action. The following triangulation, undertaken by Anderson High School in the Shetland Islands of Scotland revealed a number of interesting discrepancies between teacher judgements (shown in a solid colour) which assumed that, by and large, pupils were concentrating); student responses (open blank) which suggested that pupils were thinking about other things); and observer ratings (shaded) also suggested that pupils were thinking about other things). Concentrating Thinking about other things 3 2 1 Flow The spot check leads to an exploration of 'flow' (when a consistent rating of 'l's is achieved), and what classroom conditions best facilitate the 'high' which can come from challenging and rewarding learning situations. When we experience a challenging situation and have the skills to meet the challenge, the state is described as 'flow'. Flow is a powerful feeling of being at one which what you are doing, thoroughly engaged and deriving energy and satisfaction from it. It is seen at its most concentrated form in sports in which athletes set targets for themselves, above their present level of skills. As the next diagram indicates, flow occurs where high challenge and high skills coincide. 10 High challenge FLOW ANXIETY High skill Low skill BOREDOM APATHY Low challenge However, a number of things get in the way of flow. A situation can provoke anxiety when there is a high challenge but a low level of skill to meet it. When we do have the skills but are presented with something which does not convey a challenge, we are bored, and when we experience neither challenge nor use of our skills, we experience apathy. Critical incident analysis Another useful tool is the critical incident analysis, used to identify single events which can inhibit leaming and escalate into major crises, as in the following example from a London school. The pupils were asked to identify a recent incident which had inhibited learning but which could have followed a different chain of events, and had a different outcome. You're fooling about the teacher shouts at you (she overreacts). You shout back A the teacher is shocked and puts you in detention. You don't go (got to help your mum) A" school says they are going to suspend you. Your mum comes up to school and shouts at the teacher you are suspended. You come back and everyone says 'You're the girl whose mum shouted you get behind in your work and get into at the teacher more trouble. 11 The exercise helped the students to identify a number of key issues: Pressure and lack of time Authority and respect The student recognises that the teacher The student recognises that in retrospect she is stressed and under pressure. She had undermined the teacher's authority in recognises that both she and the teacher front of the whole class. She also feels that have both overreacted but they didn't her own authority has been diminished in front have the opportunity to sort it out before of her classmates. She feels greater mutual it spiralled out of control. respect may have averted the situation. Some tools at the school level All of these tools may be used and adapted at a whole school level to evaluate the organisational life of the school and to answer such questions as: * When is the school in 'flow'? * What are the force fields pushing it towards, and away from its goals? * What are the critical incidents in school life that could be looked at more closely and lead to different outcomes? A powerful instrument that was used with great success in a European project with 101 schools in 18 countries is the Self-evaluation Profile (Table 4 Table 4 School Self-evaluation Profile I 1+~~~~~~+ 1I - .I-¶-t1i Outcomes academic achievement _ 1 personal and social development = = - = - T pupil destinations = Process at classroom level ___ _ time for leaming _ _ quality of leaming and teaching _ _ support for leaming difficulties Process at school level 12 school as a learning place _ l ___ school as a social place e _ _ school as a professional place ._ .__ Environment _ .__ _ school and home school and community . school and work Each of the schools was asked to form four groups (pupils, parents, teachers and governors if they had any). The groups were asked to go through the twelve items, one by one, trying to reach an agreement on a rating for the school. The next step was to agree on whether each group saw the school as improving, declining or 'coasting'. As the group members worked through the items, they had to press for evidence - how do we know? What data do we have on that topic? Finally, each of the four stakeholder group sent two representatives to a school evaluation group which went through the exercise again with added rigour and more concerted focus on the issue of evidence. The school evaluation group then had to select one or more areas to examine in greater depth and with finer grained tools. It was a perfect illustration of what Jerome Bruners describes as the twin concepts of economy and power. The simplicity of the tool led to such generative discussion and in-depth investigation that most of the participating schools were never the same again. In many of the countries involved it has led to a large scale expansion of the pilot and in a number, new policies at national level. 6. Final thoughts In developing school self-evaluation there are no short term quick fixes. It takes time to build trust and a shared culture. School self-evaluation may well provoke some anxiety and resistance at the outset. It may need to be gently introduced, starting small and then building up. In a study for the UK's National Union of Teachers schools were apprehensive at the outset, but in every instance became enthusiastic about the process and outcomes. The same was true of the European project, a post evaluation of which showed 98 of the original 101 schools opting for further involvement. In general we have found that school self- evaluation creates energy rather than dissipates it. It lays the ground work and opens doors to self-driven school improvement. ............................................................................................................................ Some references and sources 13 MacBeath, J., Boyd, B., Rand,J. and Bell, S. (1996) Schools Speak for Themselves: Towards a Framework for Self-evaluation. Glasgow: University of Strathclyde, Quality in Education Centre. MacBeath, J. (1999) Schools Must Speak for Themselves, London: Routledge and the National Union of Teachers. Riley, K.A. (1998) Whose School is it Anyway? London: Falmer Press. Riley, K.A. (2000) 'Leadership, leaming and systemic reform', Joumal of Educational Change, Vol.l. No.l. January, pp 29-55. Scheerens, J. (1999). School Effectiveness in Developed and Developing Countries: a Review of the Research: World Bank, Effective Schools and Teachers web-site. 14