Evoke Developing Skills in Youth to Solve the World’s Most Complex Problems: Randomized Impact Evaluation Findings World Bank Education, Technology & Innovation: SABER-ICT Technical Paper Series (#19) Barbara Freeman & Robert Hawkins 2017 saber.worldbank.org To cite this publication: Freeman, B. & Hawkins, R. 2017. Evoke – Developing Skills in Youth to Solve the World’s Most Complex Problems: Randomized Impact Evaluation Findings. World Bank Education, Technology & Innovation: SABER-ICT Technical Paper Series (#19). Washington, DC: The World Bank. Available at: http://saber.worldbank.org Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: Freeman, B. & Hawkins, R. 2017. Evoke – Developing Skills in Youth to Solve the World’s Most Complex Problems: Randomized Impact Evaluation Findings. World Bank Education, Technology & Innovation: SABER- ICT Technical Paper Series (#19). Washington, DC: The World Bank. License: Creative Commons Attribution CC BY 3.0 IGO Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. © 2017 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 USA Telephone: +1-202-473-1000; Internet: www.worldbank.org Some rights reserved Disclaimer: The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. saber.worldbank.org Table of Contents Acknowledgements Executive summary About EVOKE and the “Developing Skills in Youth to Solve the World’s Most Complex Problems” papers About the authors 1. Background .............................................................................................................................................. 1 2. Research Questions, Design and Methods ............................................................................................. 2 3. Results ……………………………………………………............................................................................. 12 4. Discussion ……………………………………………………........................................................................ 15 5. Limitations and Questions …………………………………......................................................................... 18 Bibliography ................................................................................................................................................. 19 Appendices Appendix A. 21st Century and Social Emotional Skills Appendix B: Example of Activities for the Imagination Skill Appendix C: Summary of Pre-Post Scores for Research Questions 1 and 2 Tables Table 1: Summary of Study Participants by Characteristic Table 2: Statistical Technique used to address each Research Question Table 3: Summary of the Main Effects of the Multivariate Analysis Figures Figure 1. Graphic Novel on the Evoke Platform Figure 2. Evoke Global Social Learning Platform Figure 3. Theory of Change for Evoke Project in Colombia Figure 4. 21st Century and Socio-emotional Skills Post-test Results by Condition Figure 5. Peace Perceptions Post-test Results by Condition Figure 6. 16 Strengths of a Social Innovator Figure 7. Example of an Activity to Develop the Imagination saber.worldbank.org Acknowledgements The World Bank Education, Technology & Innovation: SABER-ICT Technical Paper Series explores a variety of topics and issues related to the use of information and communication technologies (ICTs) in the education sector. The Systems Approach for Better Education Results (SABER) initiative seeks to improve the global knowledge base related to education systems analyses, assessments, diagnoses, and opportunities for dialogue. SABER-ICT aims to improve the availability of policy-related data, information, and knowledge on what matters most in using ICTs to improve the quality of education. This publication series is made possible through generous support from the Government of Korea, most notably through the Korea - World Bank Partnership Facility. The Social Innovators’ Framework was created as part of the Evoke project. The authors would like to thank the Government of Sweden for its generous support of the Evoke project and this paper, Evoke – Developing Skills in Youth to Solve the World’s Most Complex Problems: Randomized Impact Evaluation Findings. They would also like to acknowledge and thank the numerous peer reviewers from the World Bank, Uniminuto Universidad Soacha and Bogotá, the International Society for Technology in Education (ISTE), and the British Council. saber.worldbank.org Evoke: Randomized Impact Evaluation Findings Executive summary This paper presents the findings of an impact evaluation designed to study an educational initiative called Evoke: Youth as Agents of Change in the Colombian Peace Process. Evoke is a project-based learning model that uses storytelling, game mechanics, and global social networks, to imbue young people with the skills they need to develop social innovations that address grand challenges (e.g., refugees, poverty, climate change) in their communities. Creating novel solutions to these complex and intractable problems requires curiosity, creativity, collaboration, aggregative thinking, empathy, plus a host of other 21st century and socio- emotional skills. The development of each skill individually is important but these skills are deeply intertwined and even more relevant when combined in a project-based framework. These transferable skills enable young people to see, listen, question, imagine, think, care, act, and reflect, in a fundamentally different and potentially transformative way (see Freeman & Hawkins, 2016 for details on the framework). The purpose of this study was to evaluate the effect of Evoke on students’ acquisition of 21st century and socio-emotional skills, and on students’ belief in their capacity to bring about a more peaceful and hopeful future for people who have been displaced as a result of the decades-long internal conflict in Colombia. The effectiveness of learning using the Evoke method was compared to the effectiveness of learning using a well-established approach to teach social responsibility, through a randomized controlled trial (RCT) with a pre-test/post-test control group design. The study was conducted over one semester, with 297 students at the Soacha campus of Uniminuto University on the outskirts of Bogotá (Uniminuto, 2017). Students conducted field work in Soacha, a community of thousands of people violently displaced in Colombia. The principal finding was that students who used Evoke demonstrated statistically significant greater learning outcomes in 21st century and socio-emotional skills than did the control groups. The effect size on skills acquisition was medium, which is strong for a brand new program when compared with two well-established programs focused on social responsibility, and especially strong when considering that this was not a small supplemental intervention but a whole program that replaced five mandatory university classes for an entire semester. Of particular importance was the finding that the treatment group’s skills and perceptions of their capacity to bring about a more hopeful and peaceful future for people who have been displaced improved equally regardless of gender, ethnicity, academic field of study, or age. This suggests that Evoke has the potential to improve students’ 21st century and socio-emotional skills and empower young people from diverse backgrounds to become forceful agents of change. Other findings of this study include: • The Evoke treatment group showed higher gains in knowledge about displacement of people than both comparison groups, although this difference was not significant. • A correlation was found between students who engaged most in activities as a creative visionary (imagination, ideation, vision, courage) and as an empathetic activist (leadership, empathy, transformation, curiosity) and students’ belief in their capacity to bring about a more hopeful and peaceful future. Investing students with new skills and capacities, as Evoke has demonstrated, is valuable in and of itself as it opens up multiple pathways for youth to successfully make their way in their world. Moreover, these capabilities are also a vital foundation for youth to be able to improve the lives of people who have been displaced, because equipped with these skills they are better placed to navigate complex social challenges and become better problem solvers and leaders of change. saber.worldbank.org Evoke: Randomized Impact Evaluation Findings saber.worldbank.org Evoke: Randomized Impact Evaluation Findings About EVOKE and the “Developing Skills in Youth to Solve the World’s Most Complex Problems” papers How can I change the world for the better? What is my life project and how can I craft it? These are questions that educators and students around the world increasingly ask themselves. Implicit in such questions is the imperative that what students learn should be relevant, empowering, and engaging. What are the types of pedagogies and curriculum that prepare students to shape their lives and make a positive impact on the world? What are the skills and knowledge that they need to develop in order to negotiate complex and ambiguous challenges and develop their world-changing ideas? These are questions that the World Bank project Evoke aims to answer. “Developing Skills in Youth to Solve the World’s Most Complex Problems” World Bank publication consists of three parts. “The Social Innovators’ Framework” is the first paper in this three-part series. In this paper, we present the Social Innovators’ Framework. In the first section of this paper, we provide an overview of the SIF and the Evoke project. In the second section, we then explain the theoretical and empirical foundation that underpins the framework and connects cognitive and socio- emotional skills development to social innovation and social change. In the final section of this paper, we outline the way in which the SIF was designed, provide the detailed framework, and describe how the framework has been developed as part of the Evoke project. This includes: defining the skills, describing how these skills are operationalized in activities and measured using concrete evidence, illustrating how skills are grouped into an integrated and iterative project-based learning process, and articulating the assessment approach. Part two of this series, “Evoke -- Developing Skills in Youth to Solve the World’s Most Complex Problems: Randomized Impact Evaluation Findings”, focuses on the pilot conducted in Colombia (August to November 2016). It shares results of the application of this framework in the context of the Evoke implementation in Colombia, directed toward the grand challenge of human trafficking and displacement of people, including the study’s methodology and findings, and its impact on the development of university students as future social innovators. It also explains how we contextualized the skills, activities, and proposed outcomes (e.g., developing attitudes associated with imagining a more peaceful future and believing in one’s ability to make a difference in creating that future) to the specific grand challenge, to the local needs of people in Soacha (a community on the outskirts of Bogotá), and to the requirements of our local university partner, Uniminuto (Corporación Universitaria Minuto de Dios). In part three, “Applications and Sustainability”, we discuss how the Social Innovators’ Framework has also been applied to different grand challenges in diverse settings and with different implementation partners globally. We will also explore other potential applications and sustainability. saber.worldbank.org Evoke: Randomized Impact Evaluation Findings About the authors Barbara Freeman, EdD, MBA is the co-creator of the Social Innovators’ Framework and Chief Evaluator of the Evoke project in Colombia. She is a visiting scholar at U.C. Berkeley Graduate School of Education. Dr. Freeman has worked extensively with government and intergovernmental agencies across four continents, using her interdisciplinary experience and academic training to research and develop interventions and solutions that help address complex educational and socio-economic issues. Robert Hawkins is a senior education specialist at the World Bank with over 20 years of experience working at the intersection of education and technology in developing countries around the world. He is the creator of the World Bank Evoke network and co-creator of the Social Innovators’ Framework. saber.worldbank.org Evoke: Randomized Impact Evaluation Findings 1. Background Global grand challenges (e.g., refugees, poverty, climate change) represent the most demanding problems we face in an increasingly complex world. The acceleration of complexity requires a fundamentally different way for young people to solve ambiguous and seemingly intractable problems and to successfully make their way in the world. Educators fervently discuss how best to transform teaching and learning pedagogies; how to integrate technology into the classroom curriculum; and how to make teaching and learning more relevant to contemporary demands—that is, transdisciplinary, innovative, collaborative, project-based, and focused on real-world problems. Likewise, the international development community seeks to get young people involved in community-based service learning, social innovation and civic engagement. The World Bank created Evoke in 2010 as a result of precisely such a request from the South African government. Evoke is a project-based learning model that uses storytelling, game mechanics, and global social networks, to drive collaborative social innovation skills development. The Evoke project is designed to support young people as they develop a first-hand understanding of the problems in their community; acquire 21st century skills (e.g., creativity, collaboration, critical reflection) and socio-emotional skills (e.g., curiosity, empathy, generosity); learn to imagine a more just and fairer world; experiment, collaborate, and create innovative solutions; and gain the confidence and courage they need to communicate and implement their plans and world-changing ideas. Since its inception, Evoke has been studied formatively in South Africa, Brazil, and Colombia, and has won numerous awards for its innovation in education. As a precursor to scaling the Evoke project, the World Bank team sought to understand the impact that it has on students’ acquisition of 21st century and socio-emotional skills and on students’ belief in their abilities to effect positive change in their local context. Toward this end, an experimental study was conducted at Uniminuto University’s Soacha campus on the outskirts of Bogotá (Uniminuto, 2017). Tens of thousands of people who have been forcibly and violently displaced by internal conflict in Colombia, and who currently live in dire conditions of poverty and economic insecurity, live in Soacha. The experiment took place over one semester, 16 weeks, from August to November 2016. The Evoke project in Colombia was focused on the ‘grand challenge’ of human trafficking and displacement of people. A randomized controlled trial (RCT) was used to compare the effectiveness of learning using the Evoke pedagogical approach with learning using a more conventional approach to teach social responsibility. Qualitative data was nested into the quantitative design to increase confidence in the evidence base with regard to skills and to obtain a more complete understanding of the impact of the Evoke intervention on the creation of social innovations. The results of the Evoke Project randomized impact evaluation are provided in this paper. Further details about the Evoke project and the study can be found in “Evoke - Developing Skills in Youth to Solve the World’s Most Complex Problems: The Social Innovators’ Framework” and a series of related blog posts. 1 1 The 21st century skills and socio-emotional skills tested in this impact evaluation form the core of the Social Innovators’ Framework. These skills, sometimes referred to as social innovators’ skills in this paper, form the skill set that Evoke seeks to develop. They are defined in Appendix A. See https://openknowledge.worldbank.org/handle/10986/26106 and http://www.worldbank.org/en/topic/edutech/brief/evoke-an-online-alternate-reality-game- supporting-social-innovation-among-young-people-around-the-world for more information. saber.worldbank.org 1 Evoke: Randomized Impact Evaluation Findings 2. Research Questions, Design and Methods The evaluation addressed three key research questions: Research Question 1: To what extent does Evoke improve participants’ social innovator 21st century and socio-emotional skills as compared to the control groups? Research Question 2: In what ways does Evoke shift participants’ perceptions toward believing they can help bring about a more hopeful and peaceful future through social innovation as compared to the control groups? Research Question 3: What are the relationships between engagement with the Evoke project and participants’ learning outcomes, perceptions and personal characteristics (gender, ethnicity, age, and major area of study)? The evaluation design and data analysis plan were peer reviewed prior to implementation. Peer reviewers included individuals from throughout the World Bank, Uniminuto University and the Uniminuto Social Innovation Science Park in Bogotá and Soacha campus, and the International Society for Technology in Education (ISTE). 2.1 About the Evoke Model and Theory of Change Evoke is a project-based learning model that uses storytelling, game mechanics, and global social networks, to drive collaborative social innovation skills development. Evoke aims to inspire youth around the world to develop a passionate curiosity for learning and a belief that their actions can make a difference in the world. It leverages technology to engage youth to better understand the complex challenges faced in their communities and to collaborate with their peers locally and globally to create innovative solutions that address these challenges. The target audience is young people, spanning the upper secondary level and university populations including continuing education students, as well as out-of-school and working youth. The initial focus is on university students. Evoke connects students with their peers and mentors through a global social learning platform that uses game mechanics, such as Evocoin (the in-game currency), badges, and a game board module. Evoke is positioned where young people tend to spend their time – in game environments, on social networks, and reading graphic novels – to engage them in both learning and global development challenges. The game mechanics are designed to engage and motivate youth to spend the time necessary to both complete activities designed to develop skills and create their final project together with their team. Evoke uses a graphic novel as a means of absorbing students into the harsh realities that define too many people’s lives. The graphic novel is set in the future and is also designed to transport the player into an alternate reality and to encourage deeper thinking about future alternatives. Too often, youth are trapped in the reality of their daily struggles and lack an optimistic future view. Evoke is designed to help youth escape this mindset and imagine their own future and a better future for their community. In the context of Colombia, the novel deals with one aspect of violence in Colombia, human trafficking. This, of course, is not the whole of the displacement tragedy. The graphic novel is meant to provide insight into the importance of looking for solutions to grand challenges by drawing students into a personal story. The novel concludes with a call to help solve the problem—an Urgent Evoke. This call to action is delivered by the character Alchemy, who is the head of the secretive group of agents in the Evoke network; you can hear it here https://www.youtube.com/watch?v=Nzueroug_90. Figure 1 shows a snapshot from the graphic novel used in the Evoke project in Colombia. saber.worldbank.org 2 Evoke: Randomized Impact Evaluation Findings Figure 1 Graphic Novel on the Evoke Platform At the core of Evoke is the Social Innovators’ Framework (SIF), a project-based learning model that is organized into eight missions, with each mission designed to advance students’ understanding of an aspect of the social innovation cycle (community, problem, solution, communication) and promote skills development. SIF specifies four key qualities (broad clusters of skills): Creative Visionary, Deep Collaborator, Systems Thinker, and Empathetic Activist. These four qualities constitute the major 21st century and socio-emotional skills that are critical for becoming a social innovator (see Appendix A). In total, the SIF aims to cultivate 16 strengths and 48 specific skills. The 48 skills are each translated into engaging and interesting activities that capture the essence of the skills and that can be demonstrated by students through their actions. In the Evoke project, a specific activity is designed to represent (operationalize) a specific skill or competency, though many activities require a mix of multiple skills to complete. The activities vary in nature; some are intended to be conducted in the real-world through field work, others can be conducted online and through research (see Appendix B for an example of activities for the imagination skill). The SIF model combines and sequences these 21st century and socio-emotional skills to support students’ development of a world-changing idea in their final team project (called an Evokation). Each of the 48 activities, across the 8 missions, creates unique content that is used to build a team’s project. Throughout the 16 weeks, students submit ‘evidence’ of their activities (video, photos, text) to the Evoke global social learning platform, and a rubric is provided that enables peers, mentors, and teachers to assess and comment on other players’ submissions. Mentors provide feedback to help students develop their skills and progress their Evokation. The Evoke platform is shown in Figure 2. saber.worldbank.org 3 Evoke: Randomized Impact Evaluation Findings Figure 2 Evoke Global Social Learning Platform The Evoke model is grounded theoretically and empirically in the Evoke ‘theory of change’, as depicted in Figure 3. It is the Evoke theory of change, that a project-based learning model that uses storytelling, game mechanics, and global social networks to address urgent global grand challenges will result in major increases in students’ 21st century and socio-emotional skills development and improve students’ perception of their abilities to bring about a more peaceful and hopeful future. This, in turn, will lead to more social entrepreneurship in the community and the development and implementation of multiple social innovations by youth who are able to imagine a more just, equitable, and fairer world, and who are empowered with the skills and confidence to make it happen. Over time, and through the use of the Evoke global social learning platform as a meeting place for the exchange of ideas and the cross-fertilization of talents, an active network of youth will emerge as change agents. Through this network, projects will synthesize and pollinate, which will result in better conditions for people who have been violently displaced by the decades-long conflict in Colombia. It will also tend to assist the young people, who have acted as change agents, to lead more successful and productive lives as dynamic and active citizens. saber.worldbank.org 4 Evoke: Randomized Impact Evaluation Findings Figure 3 Theory of Change for Evoke Project in Colombia saber.worldbank.org 5 Evoke: Randomized Impact Evaluation Findings 2.2 Setting and Participants The study was conducted at the Soacha campus of Uniminuto University. The university is the ideal partner to enable Evoke to reach its target population because of its vision of social inclusion and goal to provide technical and professional education to the most remote and vulnerable communities in Colombia. Seventy percent of students are female and 98 percent of students come from the bottom half of the Colombian government’s socioeconomic classification for households, with a high proportion of students being the first person in their family to attend college (International Finance Corporation, 2015). Additionally, Uniminuto’s Social Innovation Science Park has strong relationships with communities throughout Colombia and is dedicated to jointly developing solutions to social problems with these communities. A total of 297 students, across fourteen university classes, participated in this study. Attrition was negligible (> .01%) because the Social Responsibility and the Contemporary Social Development classes from which the treatment and control group classes were drawn are mandatory classes for graduation. A senior administrator was responsible for coordinating the study and overseeing its implementation during the semester. Eight professors participated in the study; four professors were assigned to the Evoke condition. The local coordinator at Uniminuto worked closely with a researcher on the World Bank evaluation team who was responsible for implementation. The World Bank researcher spoke with the local coordinator on a weekly basis and with the teachers on a bi-monthly basis. A summary of student participants is provided in Table 1. TABLE 1. Summary of Participants by Characteristic Variable Groups Total Treatment Control 1* Control 2* (n = 297) (n = 109) (n = 108) (n = 80) Gender Male 121 (40.7%) 55 (50.5%) 43 (39.8%) 23 (28.8%) Female 172 (57.9%) 53 (48.6%) 63 (58.3%) 56 (70.0%) No Answer 4 (1.3%) 1 (0.9%) 2 (1.9%) 1 (0.9%) Ethnicity Mestizo 168 (56.6%) 53 (48.6%) 57 (52.8%) 58 (72.5%) Blanco** 97 (32.7%) 47 (43.1%) 35 (32.4%) 15 (18.8%) Other*** 30 (10.1%) 8 (7.4%) 15 (13.9%) 7 (8.7%) No Answer 2 (0.7%) 1 (0.9%) 1 (0.9%) - Age Less than 18 3 (1.0%) 1 (0.9%) 1 (0.9%) 1 (0.9%) 18-22 194 (65.4%) 58 (53.1%) 77 (71.3%) 59 (73.7%) 23-29 77 (25.9%) 39 (35.8%) 20 (18.5%) 18 (22.5%) 30+ 22 (7.4%) 11 (10.1%) 9 (8.4%) 2 (2.5%) No Answer 1 (0.9%) - 1 (0.9%) - Major Business/ 32 (10.8%) 6 (5.5%) 21 (19.5%) 38 (47.6%) Accounting Children 22 (7.4%) 5 (4.6%) 13 (12%) 4 (5.0%) Pedagogy Psychology 67 (22.6%) 2 (1.8%) 31 (28.7%) 34 (42.5%) Social Work 86 (29.0%) 26 (23.9%) 26 (24.0%) 34 (42.5%) Related Technology 89 (29.9%) 70 (64.2%) 16 (14.9%) 3 (2.7%) Related No Answer 1 (0.3%) - 1 (0.9%) - * Control 1 is the Social Responsibility condition; Control 2 is the Contemporary Social Development condition. ** Blanco is Caucasian or White participants *** Includes Negro/Afro Colombian, Indigenous, Not stated saber.worldbank.org 6 Evoke: Randomized Impact Evaluation Findings 2.3 Procedures Prior to the start of the autumn 2016 semester, researchers worked with the Uniminuto Soacha campus coordinator on data collection, randomization, and scheduling issues. The implementation occurred over one semester. During the first two weeks of the semester, teacher training was provided to all teachers and researchers worked with the teachers to ensure equivalency in the curriculum during classroom sessions; students were trained on the platform and the Evoke experience; partners in the field were recruited to participate (e.g., foundations, organizations, families that had been displaced); and the pre-tests were administered and completed. Throughout the 16 weeks, both the treatment group and the main control condition, Control 1 group, met face-to-face for 2 hours each week to work on a social responsibility curriculum and had 7 hours of required fieldwork and self-study. The Control 2 students in the Contemporary Social Development class also met 2 hours a week in class and worked on a social responsibility curriculum but were not required to participate in fieldwork. During the last weeks of the semester, the Treatment and Control 1 groups delivered their end of year projects and all three groups completed the study’s post-tests. (The distinction between Control 1 and 2 is discussed in the next section.) 2.4 Study Design and Conditions A Randomized Control Trial (RCT) with a pre-test-post-test control group design was used to minimize threats to the validity of conclusions drawn from the study. Random assignment to either the Treatment or Control conditions occurred at the classroom level. Students, however, selected their class at random, largely based on their schedules. Researchers selected this method because random assignment at the student level could have potentially introduced a threat to internal validity in that students within the same classroom share experiences and a strong possibility exists that students in the control group would be exposed to the treatment method and curriculum. This design minimized this spill-over threat and enabled the researchers to study classes intact, meaning that all students in a class are in the same group, either treatment or control. There were two control groups. Control 1 is a Social Responsibility class and the main control group, with a comparable intervention. Control 2 is a Contemporary Social Development class, which is infused with social responsibility concepts and designed to raise social awareness but does not engage in field work with the community. Forty-three percent of the Control 2 student participants had already completed the Social Responsibility class at the start of the study. The treatment students worked on the Evoke curriculum, as described in the Evoke Program and Theory of Change section (see Freeman & Hawkins, 2016 for further detail). The Control 1 students worked on the Social Responsibility curriculum, which is a mandatory class for students to graduate. Social Responsibility requires that students perform community service in vulnerable communities and is one of the three pillars of Uniminuto’s “praxeological” education model. 2 The Evoke and Social Responsibility classes share the goals of combining theory and practice through classroom work and active learning in the field. One of the distinguishing features between the two methods is that students in the Social Responsibility class participate in a long-standing program that is offered by the university and are assigned to a foundation, organization, or family, in which the type of service that they do and the tasks required are largely prescribed; that is after students choose which type of service they wish to do (e.g., students studying Children Pedagogy as their major may wish to work in pre-schools, etc.). 2 Praxeological refers to the integration of theory and action or more precisely theory and “praxis” (i.e., the practice of knowledge and skills). The other two pillars in the model are human development and professional competencies. saber.worldbank.org 7 Evoke: Randomized Impact Evaluation Findings Whilst Evoke students also choose their area of focus, they work through a project-based learning schema that is organized into eight missions. Each mission is designed to help students understand an aspect of the social innovation cycle (community, problem, solution, communication). Within each mission, students work through a set of activities that combine and sequence cognitive and non-cognitive skills, which support the design of a team-based social innovation project that addresses the ‘grand challenge’ faced by the community; displacement of people in this case. Another important difference between the Evoke method and the control conditions is the online component of Evoke in which students post evidence of their activities on the Evoke platform. Evidence is then reviewed and voted on by their peers and evaluated by a mentor network, which includes their Uniminuto professors and local and global experts. Mentors use students’ evidence of their activities to provide guidance and help students improve their individual skills and help teams create their final project. 2.5 Instruments and Measures The research team searched for previously validated and reliable instruments and measures to use to assess the 21st century and socio-emotional skills of social innovators. Although instruments were identified that addressed aspects of the Social Innovators’ Framework used in Evoke (e.g., tested creativity, perseverance or empathy), we were unable to find any single instrument that was applicable to this study. In particular, there were no instruments that addressed the interconnectedness of skills within a project-based learning model or the application of skills to develop students as social innovators, who were working in the real world, outside of the classroom. Furthermore, most instruments were US school-based measures and looked at schools’ culture and climate or viability in a high-stakes testing environment and many were applicable to younger students. Therefore, the researchers needed to design, develop, and test new instruments to evaluate students’ learning outcomes and perceptions of their abilities. These instruments furthermore needed to be translated into Spanish and tested to see if there was a contextual fit. The various instruments used are described below. 21st Century and Socio-emotional Skills Assessment. A pre-post-test was used to measure the effect of the pedagogical approaches on students’ skills and attitudes development. This assessment was designated in advance as the primary indicator to assess research question 1 (skills improvement). The skills assessed are shown in Appendix A. The assessment includes Likert scale and multiple choice questions. A ‘test and retest’ method was used to check for reliability. To check for content validity the 24-item pre-post-test was: (a) reviewed and vetted by a set of content experts in skills development, social innovation, social responsibility, and assessment; (b) piloted by students at Uniminuto from the previous semester’s Social Responsibility class (a different set of students than those participating in the Autumn 2016 study); (c) revised: test items that failed to meet requirements were thrown out or re-written; (d) retested with a different set of students; and (e) rechecked by experts. There was an A and B version of the pre-post-tests. Those students who received Version A during the pre-test were given Version B during the post-test and vice-versa. Tests A and B contained the same items but presented both the sections of the test and the items within a section in a different order. The 21st Century and Socio-emotional Skills section was comprised of two parts, for a total possible score range of 16-343. The first part contained 16 Likert scale questions, with a range of 1-21 for each question. The second part contained eight multiple choice questions, with a score of 0-1 for each question. The two parts were integrated into a single composite score. The first part of the 21st Century and Socio-emotional Skills test was designed to assess a student’s disposition and thoughts toward a particular skill as opposed to evaluating a student’s proficiency at performing a particular task or tasks that represent the skill. For instance, the question designed to assess imagination is as follows: To what extent do you agree with the following statement? People who achieve excellence are distinguished by particular rare talents. saber.worldbank.org 8 Evoke: Randomized Impact Evaluation Findings Among these talents is the ability to paint a picture of a different future invisible to other people. The choice at the left side of the scale is: Very few people have sufficient imagination to see a different future and the choice to the right side of the scale is: All people have sufficient imagination to see a different future. Students who demonstrated increases in the skill of imagination would have scored this question lower before the intervention than after – illustrating that this skill is not exogenous to their reality but something they too could possess. Similarly for the skill of Critical Reflection, the question presented to the participants was: Thinking about your school and/or work demands, how true it is it of you that finding extra time to reflect on your work (e.g. write in a journal, post a blog) is unrealistic. The responses were then: I rarely reflect on my work or Reflection on my work is part of my routine. Positive gains in this skill would illustrate a larger number of participants indicating that reflection is part of their routine after the intervention thus illustrating a change in disposition toward this skill. The second part of the 21st Century and Socio-emotional Skills Assessment was designed to test a student’s understanding of the social innovation process and how combined sets of skills fit into the social innovation cycle. This second part was essential because it captures the importance of the interaction of skills in the context of understanding problems and designing solutions to grand challenges in the real world. For example, the question concerned with the integrated skills of collaboration and leadership was: Which of the following is the least important part of a collaborative leader’s role? The responses were: (a) Sharing stories, (b) Assigning roles, (c) Removing the weaker members of the team, and (d) Maintaining contact with the network. The correct answer is c and positive changes occurred on the post-test as students came to understand the value of the diverse strengths of team members and the role the leader plays in recognizing these various strengths and directing them toward the achievement of the team’s shared goals. Likewise, another question on the assessment was designed to prompt students to consider communication, problem solving, and analytical skills separately: Which of the following is the most important element that early and later stages of the social innovation process have in common? The responses were: (a) Concretely measuring impact, (b) Clearly communicating the solution, (c) Working closely with community members, and (d) Carefully analyzing every element of the problem. Positive gains from pre-test to post-test occurred when students recognized that it was necessary to use all of these skills at different points in the social innovation cycle yet it is essential to work closely with community members throughout the process using all of these competencies, which is answer c. Displacement Knowledge Assessment. A pre-post-test was used to measure the effect of the different curriculums on knowledge pertaining to the displacement of people. This test was created by the Uniminuto professors. It was used as a secondary indicator. The test had 9 items, with a score range of 0-9. There was an A and B version of the test. Peace Perceptions Assessment: A pre-post-test was used to assess students’ perceptions about their capacity to bring about a more hopeful and peaceful future for displaced people. The same ‘test and retest’ method used for the 21st Century and Socio-emotional Skills Assessment was used to check for reliability of the Peace Perceptions Survey. Likewise, there were two versions of the test. There were nine questions, including two Likert scale questions that were supported by seven ranking questions, which were used to illuminate the answers to these questions. The score therefore ranged from 2-14. The assessment culminated in the key question: Given your previous answers, to what extent do you agree that your voice and your actions can help make a positive difference on the issues that you choose to focus on either globally or locally? The Likert Scale ranged from Strongly Agree to Strongly Disagree. Data Capture. Instrumentation was built into the Evoke platform and stored in a central database, which enabled the monitoring of each student’s performance. Data captured by the system included: the number of activities that students engaged with, the types of activities that students did, the number of evidences submitted, the number of evidences reviewed by mentors, and whether evidence included attachments, such as photos or videos. saber.worldbank.org 9 Evoke: Randomized Impact Evaluation Findings Final Project Rubric. A 4-criteria rubric was used to assess the final projects for the Treatment and Control 1 groups. Each of the four criterion had a 4-point corresponding assessment: Level 1 – Does not comply Level 2 – Meets minimum requirements Level 3 – Good work Level 4 – Exceeds expectations. The rubric was based on the World Bank Group’s scoring guidelines for a 2016 youth competition and modified to reflect the requirements of this project. Projects were assessed by the respective Uniminuto professors, and for comparison purposes by an expert panel comprised of six reviewers; the Uniminuto coordinator plus five international experts in the fields of social innovation, social responsibility, and civic engagement. In future evaluations, we intend to compare the quality of the work between the Evoke and Control groups. During this impact evaluation, the data collected was used only for Evoke behavioural analysis to examine the relationship between Evoke students’ Evokation (final project) and data captured by the Evoke platform. It was not used for comparison purposes in the RCT. 2.6 Data Analyses Initially, data were plotted to check for normality, outliers, and data entry accuracy, and then cleaned. Descriptive statistics were calculated. Analysis of variance (ANOVA) tests were run to look at differences between the treatment and control conditions on the pre-test. In order to determine the most appropriate means of handling missing data, Little’s MCAR test was conducted to see whether values were Missing Completely At Random (MCAR) or missing in a systematic way (Little, 1988). Table 2 shows the statistical techniques used to address the three research questions (RQ). TABLE 2. Statistical technique used to address each research question Research Statistical Dependent Covariate Independent Question technique Variables Variables RQ 1: ANCOVA Post-tests: Pre-tests: Treatment vs Learning 21st Century & 21st Century & Control Outcomes Socioemotional Socioemotional Skills Skills Knowledge Knowledge RQ 2: ANCOVA Peace Peace Treatment vs Students’ Perceptions Perceptions Control Perceptions Post-test Pre-test RQ3: MANCOVA Post-tests: Pre-tests: Gender Relationship 21st Century & 21st Century & Age between Socio- Socio- Ethnicity Evoke emotional emotional Major Students’ Skills Skills Learning Perceptions & Peace Peace Characteristics Perceptions Perceptions RQ3 Pearson’s Variables include: Correlation Creative Visionary activities (one-tailed) Deep Collaborator activities Systems Thinker activities Empathetic Activist activities 21st Century & Socioemotional Skills change pre-test to post-test Peace Perceptions change pre-test to post-test saber.worldbank.org 10 Evoke: Randomized Impact Evaluation Findings In order to address research questions 1 and 2, an analysis of covariance (ANCOVA) was used to investigate the impact of the interventions on learning outcomes (RQ1) and on students’ perceptions about their capacity to bring about a more peaceful and hopeful future (RQ2). Covariates were used for all assessments since they explain a part of the variance related to the respective post-test. The use of a covariate also reduced the potential for bias, by acting as a precaution against systematic bias and within-group error of variance (Stevens, 2002). Bonferroni post hoc analyses were conducted as the test statistic, enabling us to compare non- orthogonal means (Pedhazur and Schmelkin, 1991). Miles and Shevlin’s (2001) rules of thumb were used to estimate the magnitude of the effect size. Multivariate analysis of covariance (MANCOVA) was used to examine the relationships between Evoke students learning outcomes, their beliefs in their capacity to bring about a more peaceful future, and their personal characteristics. The dependent variables were the 21st Century and Socio-emotional Assessment and the Peace Perceptions Assessment and the independent variables, including gender, age, ethnicity, and major. For the purpose of informing future development, the researchers also investigated whether there was a relationship between Evoke activities completed and the results of the 21st Century and Socio-emotional Skills and Peace Perceptions assessments. All activities in Evoke are tagged in the database with the particular skills with which they are associated. Activities data were extracted from the Evoke platform database and Pearson’s correlations analyses were conducted. A one-tailed test was specified because the researchers were testing a directional hypotheses (e.g., higher scores on activities leads to increases in students’ perceptions of their abilities in the future). saber.worldbank.org 11 Evoke: Randomized Impact Evaluation Findings 3. Results ANOVA tests were conducted to look at differences between the treatment and control conditions at pre-test for each of the assessments. The Levene’s test was not significant for the 21st Century and Socio-emotional Skills Assessment (F (2, 271) = 1.90, p > .05), the Knowledge Assessment (F(2, 269) = 0.51, p > .05), or the Peace Perceptions Assessment (F(2, 269) = 1.48, p > .05). Therefore, equality of variances were assumed and descriptive statistics were calculated. A summary of the pre-post scores for research questions 1 and 2 is provided in Appendix C. The Little’s MCAR test was performed and showed a χ2 (27, N = 297) = 23.86, p = .64, demonstrating that any data missing in this study (e.g., a missing pre-test or a participant’s gender) were missing completely at random, with no systematic relationship between the ‘missingness’ and a participant’s assignment to the treatment or control conditions or to the participant’s background characteristics (Gerber & Green, 2012). Given that data were MCAR, pairwise deletion was selected as the preferred method of addressing missing data. This method was selected for two reasons. First, pairwise and listwise deletion are acceptable techniques for making valid inferences on incomplete data when the values are MCAR (Little & Rubin, 2002; van Ginkel & Kroonenberg, 2014). Second, researchers were not concerned with loss of power. The first research question was: To what extent does Evoke improve participants’ social innovator 21st century and socio-emotional skills as compared to the control groups? A significant effect favoring the treatment over both control group’s learning outcomes was found on the 21st Century and Socio-emotional Skills post-test (F(2, 219) = 8.93, p < .01, = .08). The Bonferroni post hoc analyses showed the treatment group significantly outperformed Control 1 condition (MD = 17.16, SE = 4.32) and Control 2 condition (MD = 15.00, SE = 4.6), demonstrating a medium size effect on students’ skills development when compared to well-established programs in two control groups. The post-test results by condition are shown in Figure 4. Figure 4: 21st century and socio-emotional skills post-test results by condition saber.worldbank.org 12 Evoke: Randomized Impact Evaluation Findings In order to understand the effect of the interventions on learning about the ‘grand challenge’ and to inform the ongoing improvement of the Evoke intervention, researchers also looked at the effect of the interventions on the improvement of knowledge related to the displacement of people. The analysis showed that the treatment group scored higher than the control groups on the Displacement Knowledge Assessment post-test when controlling for the pre-test, although this was not significant (F(2, 217) = 2.62, NS). The second research question was: In what ways does Evoke shift participants’ perceptions toward believing they can help bring about a more hopeful and peaceful future through social innovation as compared to the control groups? The students in the Evoke group had a larger shift from pre-test to post-test in perceptions regarding their capacity to affect positive change than their counterparts in Control 1 (MD = 0.25) and Control 2 (MD = 0.50), although this effect was not statistically significant. Figure 5 shows the post-test results by condition. The data captured on the platform enabled researchers to analyze the relationship between when the treatment students earned points in Evoke for engaging in a particular skill-related activity and the change in their perceptions. The one-tailed Pearson correlation showed that the engagement in Creative Visionary activities (imagination, ideation, vision, courage) is positively related to improvements in student’s perceptions of their ability to bring about a more peaceful future, r = .284, p < .01. Empathetic Activist activities (leadership, empathy, transformation, curiosity) were also positively correlated with improving students’ self-beliefs, r = .251, p < .05. Figure 5 Peace perceptions post-test results by condition saber.worldbank.org 13 Evoke: Randomized Impact Evaluation Findings The third research question was: What are the relationships between engagement with the Evoke project and participants’ learning outcomes, perceptions and personal characteristics (gender, ethnicity, age, and major)? A statistically significant relationship was revealed by the MANCOVA was between the 21st Century and Socio-emotional Skills post-test and the Peace Perceptions post-test (F(1, 216) = 9.34, p < .01, n2p = 0.097). No other significant relationship between variables was found, as shown in Table 3. TABLE 3. Summary of the main effects of the multivariate analysis Grouping Variable Dependent Variable F p 21st Century Skills Post-test Peace Perceptions Post-test 9.34 .01* Gender 21st Century Skills Post-test 0.28 .60 Gender Peace Perceptions Post-test 0.05 .83 Age 21st Century Skills Post-test 1.08 .39 Age Peace Perceptions Post-test 2.46 .02 Ethnicity 21st Century Skills Post-test 0.63 .68 Ethnicity Peace Perceptions Post-test 0.22 .95 Major 21st Century Skills Post-test 1.15 .33 Major Peace Perceptions Post-test 1.40 .18 The multivariate analysis shows that female’s and male’s learning outcomes and peace perceptions improved equally in the Evoke condition. Likewise, skills development and a greater belief in one’s capacity to effect positive change were the same for students of different ethnicities, ages, and majors. This suggests that Evoke has the potential to improve students’ 21st century and socio-emotional skills and empower young people from all backgrounds to become forceful agents of change. saber.worldbank.org 14 Evoke: Randomized Impact Evaluation Findings 4. Discussion The remarkable finding of this study is that students in the Evoke treatment group not only demonstrated greater learning outcomes in 21st century and socio-emotional skills than did the control groups, but this growth was statistically significant. The effect size on skills acquisition was medium, which is strong for a brand new program when compared with two well-established programs, and especially strong when considering that this was not a small supplemental intervention but a whole program that replaced five existing university classes for an entire semester. During the semester, Evoke students improved their skills comprised of 16 skill strengths across four broad categories, which have enabled them to be more innovative, empathetic, collaborative, and better problem solvers, as depicted in Figure 6. Figure 6: 16 Strengths of a Social Innovator After participating in the experiment, the Evoke treatment group had a stronger perception of their capacity to bring about a more hopeful and peaceful future for people who have been violently displaced by conflict than did the control groups. Although this finding was not significant, a statistically significant correlation was found between engaging in Creative Visionary and Empathetic Activist activities and students’ improved belief in their abilities to effect change. In the Evoke program, students have the opportunity to apply the skills they are learning in the community, and the findings suggest that this may have been enough to start the process of changing perceptions toward students believing that their voices and actions matter. Recall that students in the Control 1 group also did fieldwork and their perceptions also improved, but not to the same extent. A key difference between the groups is that the Evoke program requires students to put their skills into action in order to complete their activities and missions. Students are expected to submit evidence of their activities to an online platform and, in return, they receive feedback from local and global mentors and experts in the Evoke social network. All activities in Evoke are intended to develop a particular skill or set of skills. For example, an activity that requires students to interview a person in the community is meant to develop curiosity, communication, and empathy skills. These same skills are also employed in other activities and are needed to complete the ultimate Evoke mission of creating a plan for a new social innovation (see About the Evoke Model and Theory of Change section). Figure 7 provides a concrete example of an activity designed to foster the imagination strength, which is defined with the following three specific skills: saber.worldbank.org 15 Evoke: Randomized Impact Evaluation Findings • Produces original and novel ideas through the willingness to take risks and try something different; • Views familiar things in a different light; • Dreams of creative ways to resolve a conflict or problem and initiates forward-looking solutions. For each of these specific skills, three examples are provided as means to make the skill concrete. For example, for the last skill – “dreams of creative ways to resolve a conflict or problem and initiates forward-looking solutions”, the three examples are: • Tells stories; • Learns experientially through small games or scenarios (e.g. Prisoner’s Dilemma); • Talks with people from different cultures, backgrounds, etc. Figure 7 illustrates an activity that employs the example of telling stories to foster the skill of dreaming of creative ways to resolve a conflict or problem and imitates forward-looking solutions as a means to develop the strength of the imagination. Figure 7 Example of an activity to develop the imagination The findings from the correlational analysis lend credence to the idea that requiring students to operationalize their skills through real-world activities may be the beginning of the process that leads to students’ increased belief in their abilities to effect change. The analysis showed a statistically significant relationship between students’ belief in their capacity to bring about a more hopeful and peaceful future and engagement with particular Evoke activities, as a creative visionary (imagination, ideation, vision, courage) and as an empathetic activist (leadership, empathy, transformation, curiosity). This is an interesting finding because it suggests a connection between engagement with instructional activities and change in perceptions. As the study enters its next phase, researchers will have an opportunity to further investigate this relationship between activities and change in perceptions and better understand its cause and effect. The results suggest, however, that it is not just the fieldwork that is responsible for the improvement in the treatment group’s 21st century and socio-emotional skills, since the Control 1 group also conducted fieldwork. Evoke is a game-based learning experience, which uses game mechanics to advance performance and injects storytelling and a modicum of fantasy into the rigorous curriculum. When saber.worldbank.org 16 Evoke: Randomized Impact Evaluation Findings Evoke students work in the community, they are both providing a service to the community (similar to the Control 1 group) and also acting as undercover agents who are seeking to understand the problems in order that they can provide an innovative solution. A substantial body of research has shown that games which capture students’ imagination and hold their interest can motivate them to learn and acquire new skills (Freeman & Higgins, 2017; Gee, 2007; Papert, 1993; Romero et al., 2015; Ulicsuk & Wright, 2010). An especially important finding was that the treatment group’s skills and perceptions improved equally regardless of gender, ethnicity, academic field of study, or age. Over the years, Evoke has iterated in a conscious way to engage both genders equally. For instance, the user interface, the colors and the language have been adapted to be more engaging to girls. Moreover, female characters have taken prominent roles in the graphic novel in the latest iterations. Taken together the results of the RCT demonstrate that the Evoke experience increases social innovators’ skills and improves the perceptions of young people from diverse backgrounds and wide-ranging academic interests; thus indicating that Evoke has the potential to empower students to develop their personal agency and become forceful agents of change in the community. As Scheffler (1985) maintained, “to enhance one’s capability to perform is to empower oneself to perform” (p. 61). Investing students with new skills and capacities, as Evoke has demonstrated, is valuable in and of itself as it opens up multiple pathways for youth to successfully make their way in their world. Moreover, these capabilities are also a vital staging post on the road to being able to change and improve the lives of people who have been displaced. saber.worldbank.org 17 Evoke: Randomized Impact Evaluation Findings 5. Limitations and Conclusions The objective of this evaluation was to study the impact of Evoke on students’ acquisition of 21st century and socio-emotional skills and on students’ belief in their capacity to effect positive change in their local context. The results provide evidence that it can. The research team has begun its implementation at the Uniminuto Soacha campus again this year (spring 2017 and fall 2017 semesters), with new cohorts of students. The Evoke developers have made modifications to the intervention based on this study. This next study will not only enable the researchers to see whether these changes can strengthen outcomes in the same setting, but will also enable them to probe the relationships between motivation, engagement, self-belief, and activity in order to better understand why Evoke had a positive impact on observed outcomes. The subsequent step is to test whether these results are generalizable to other contexts, and to evaluate the impact that Evoke has on the lives of displaced people in Soacha. In the medium-term, a range of additional research questions will be explored. Issues such as the effect of these skills on school retention, graduation and employability is an important area to determine the impact on youth livelihood. Furthermore, the impact and sustainability of the social innovations in the community and also the potential impact of this intervention on out-of- school youth are unexplored topics. In the areas of skills and skills measurement, more research is needed to understand the link between activities and skills development as well as the potential for using a multiple measures and indicators – computer, expert, peer, psychometric – to better assess skill acquisition. The instruments used to evaluate these skills also require further iteration and refinement. The impact of social capital and networks presents another area of fascinating research to better understand the role of mentors and peers in supporting students. Finally, the game mechanic of the evocoin as an exchange of value and as a motivating element in the platform provides interesting opportunities as a mechanism to better target and incentivize behaviour as an alternative means of conditional cash transfer. This too provides opportunities for further investigation. The randomized impact evaluation clearly demonstrated that Evoke is a project that offers enormous potential for youth engagement, skills development and deeper learning. Ultimately, through the interaction of storytelling, social networks, game mechanics, and real world project- based learning model, Evoke can help foster the next generation of learners and social innovators who, in turn, can improve their life prospects and change the world for the better. saber.worldbank.org 18 Evoke: Randomized Impact Evaluation Findings Bibliography Freeman, B. and Hawkins R. (2016). Evoke - Developing Skills in Youth to Solve the World’s Most Complex Problems: The Social Innovators’ Framework. Retrieved from: https://openknowledge.worldbank.org/handle/10986/26106 Freeman, B. and Higgins, K. (2017). Randomized controlled trial of a digital learning game in the context of a design-based research project. International Journal of Technology Enhanced Learning. 1(1). doi: 10.1504/IJTEL.2016.10001504. Gee, J. P. (2007). What video games teach us about learning and literacy. New York: Palgrave, Macmillan. Gerber, A. S. and Green, D. P. (2012). Field experiments: Design, analysis, and interpretation. New York: W.W. Norton & Company. International Finance Corporation. (2015). Education within reach of all. Retrieved from: https://www.ifc.org/wps/wcm/connect/f0220a004c23ab84bf9cbfd8bd2c3114/IFC%2BCase%2B Study_UNIMINUTO_FINAL_web.pdf?MOD=AJPERES Little, R. J. A. (1988). Missing-data adjustments in large surveys. Journal of Business and Economic Statistics, 6:287–296. Little, R. J. A. & Rubin, D. B. (2002). Statistical analysis with missing data. 2nd ed. New York: Wiley. Miles, J. N. V., & Shevlin, M. (2001). Applying regression and correlation: A guide for students and researchers. London: Sage. Papert, S. (1993). The children’s machine: Rethinking school in the age of the computers. New York: Basic Books. Pedhazur, E., & Schmelkin, L. (1991). Measurement, design and analysis: An integrated approach. Hillsdale, NJ: Erlbaum. Romero, M., Usart, M., & Ott, M. (2015). Can serious games contribute to developing and sustaining 21st century skills? Games and Culture, 10(2), 148-177. Scheffler, I. (1985). Of human potential. Boston: Routledge & Kegan Paul. Stevens, J. P. (2002). Applied multivariate statistics for the social sciences (4th ed.). Hillsdale, NJ: Erlbaum. Technology, Pedagogy and Education, 14(1), 107-125. Ulicsak, M. & Wright, M. (2010). Games in education: Serious games. Bristol, UK: Futurelab. Uniminuto (2017). Retrieved from: http://www.uniminuto.edu/ van Ginkel, J.R. & Kroonenberg, P.M. (2014). Analysis of variance of multiply imputed data. Multivariate Behavioral Research, 49(1): 78–91. doi:10.1080/00273171.2013.855890. saber.worldbank.org 19 Evoke: Randomized Impact Evaluation Findings Appendix A: 21st Century and Social Emotional Skills The Social Innovators’ Framework specifies 4 key qualities (Creative Visionaries, Deep Collaborators, Systems Thinkers, and Empathetic Activists) and 16 key skill strengths of the social innovator. These skill strengths establish the contours of what needs to be learned and developed to bring about solutions to global grand challenges, and are not meant to be prescriptive. The following is a list and brief explanation of each skill strength. For a complete description, see the World Bank 2017 series of linked publications, “Evoke - Developing Skills in Youth to Solve the World’s Most Complex Problems”. Social Innovators are Creative Visionaries • Imagination – Social innovators imagine a better future and a better world. They are willing to dream of original untried solutions and view familiar things in a different light. • Ideation – Social innovators generate lots of new and original ideas through research and work in the field. Then, they work with others to brainstorm and experiment to improve these ideas. • Vision – Social innovators see possibilities when others may see only barriers and failures. They inspire and motivate other people to see the opportunities and believe they can bring them to life. • Courage – Social innovators ask difficult questions, face up to complex realities, and are brave enough to experiment with their ideas even if it goes against the conventional view. Social Innovators are Deep Collaborators • Communication – Social innovators listen intently, empathetically, and respectfully to people who may see the world from a different perspective. They give their total attention to the people with whom they are speaking. They also present their ideas in a thoughtful and compelling way. • Teamwork – Social innovators understand that they cannot act alone. They are appreciative of the various skills that others bring to the team, and recognize that teams are most effective when there are members who are different and who may hold opposing world views and work hard at building trust in one another. • Networking – Social innovators seek feedback not only from their team but from a broader social saber.worldbank.org 20 Evoke: Randomized Impact Evaluation Findings network. They connect with the community and engage with a broader global social network. • Generosity of Spirit – Social innovators willingly share their time and resources and are grateful for the time and support they receive from others, on their team, in the community, and in their broader social network. They support others even if it takes extra time to help someone develop their skills. Social Innovators are Systems Thinkers • Problem Solving – Social innovators take on unfamiliar problems. They experiment with possible solutions, learn from their mistakes, and continue to explore situations, even after they reach their initial hypothesis or solution. • Analysis – Social innovators consider the problem they are addressing in light of the broader social context. They consider the interrelationship of various forces and factors within the system and use human-centered design and other structured means of visualizing the problem and making connections between cause and effect. • Aggregation – Social innovators use multiple sources of information, through research and work in the field, to gain awareness and deep understanding of the issues they are dealing with. They draw on resources and make connections from multiple domains and disciplines. • Critical Reflection – Social innovators set aside time in their busy schedules to critically reflect on what they are learning and experiencing. They question, analyze, and reconsider their ideas, both individually and with their team. Social Innovators are Empathetic Activists • Leadership – Social innovators motivate their team through their passion and commitment. They recognize the strengths of their team members and understand that different people can lead on different parts of the problem, utilizing the whole team. • Empathy – Social innovators are respectful and sensitive to other people’s lives and experiences, even if they cannot fully comprehend them. They develop a genuine understanding by listening and paying attention, recognizing that issues that led people to the situation they are in are very often ambiguous (not black and white). • Transformation – Social innovators are proactive and show their commitment. They engage the community to create change and they truly believe in the possibility of bringing about social change. • Curiosity – Social innovators are interested in other people. They seek out new knowledge and try to understand different experiences and perspectives. saber.worldbank.org 21 Evoke: Randomized Impact Evaluation Findings Appendix B: Example of Activities for the Imagination Skill Below is an example of the types of activities developed for the Human Trafficking and Displaced Persons Grand Challenge in Colombia for the skills of imagination. Skill definition and ways to Activity Basic make concrete Rubric Produces original and novel Imagine the Future: My friend Cipher thinks a lot Does this ideas through the willingness about the future. He is always drawing diagrams of evidence to take risks and try devices that don’t yet exist and telling stories about presents something different. how they would be used. I often would ask him the an original value of always thinking about what could be and novel • Engages in thought instead of focusing on the present and what is. His idea? experiments to response: “Everything begins with an idea born imaginatively speculate from imagination and eventually made real. The (plays with ideas) telephone, Internet, virtual reality...all began as • Tinkers and makes things only ideas. • Employs lateral thinking Agent: brainstorm with your team to come up with to explore taken for potential positive alternative realities for the future granted assumptions of your community. Share these ideas or tell a story about the future to the Evoke network Views familiar things in a Reshape your reality: Remember Mikaela’s Does this different light. mother? The moment she suspected her evidence husband’s betrayal was when she noticed his new views a • Challenges ingrained and expensive new purchase. To see these boots familiar assumptions from a new perspective, the horrible reality of what thing in a her husband did crashed her world. different • Reframes old problems, light? considering people’s feeling and reactions Agent: look around your community. Take a picture of an object, a place, or a person and conceive • Conceives of different how it could be transformed to improve the purposes for existing community. tools Dreams of creative ways to Dream: Almost all of the stories Cipher writes are Does this resolve a conflict or problem utopias. While most sci-fi stories imagine evidence and initiates forward-looking pessimistic future worlds, Cipher writes about what tell a solutions. is possible with descriptions full of details of new captivating technologies that will eradicate corruption and or • Tells stories injustice. I love his stories. They helped me to persuasive never relinquish my dream. story? • Learns experientially through small games or scenarios (e.g., Agent: what is your dream? What do you imagine Prisoner’s Dilemma) for your country? Tell a compelling story through text, images, or video of what a post-conflict • Talks with people from Colombia will look like. different cultures, backgrounds, etc. saber.worldbank.org 22 Evoke: Randomized Impact Evaluation Findings Appendix C: Summary of Pre-Post Scores for Research Questions 1 & 2 Below is an example of the types of activities developed for the Human Trafficking and Displaced Persons Grand Challenge in Colombia for the skills of imagination. Assessment Condition Treatment Control 1* Control 2* Pre-test Post-test Pre-test Post-test Pre-test Post-test M** M (SD) M (SD) M (SD) M (SD) M (SD) (SD**) 21st Century/ 208.52 218.94 205.93 203.22 206.25 206.88 Socio-emotional (24.40) (28.16) (44.58) (30.38) (27.44) (25.34) Skills Peace Perceptions 10.46 11.06 10.31 10.70 10.14 10.46 (2.03) (1.82) (1.86) (1.49) (1.68) (1.46) * Control 1 is the Social Responsibility class and Control 2 is the Contemporary Social Development class. ** M = Mean and SD = Standard Deviation *** 21st Century and Socioemotional Skills Assessment score range is 16-343 and Peace Perceptions score range is 2-14 saber.worldbank.org 23 Evoke: Randomized Impact Evaluation Findings saber.worldbank.org 24 Evoke: Randomized Impact Evaluation Findings World Bank Education, Technology & Innovation: SABER-ICT Technical Paper Series [1] SABER-ICT Framework Paper for Policy Analysis: Documenting national educational technology policies around the world and their evolution over time (Michael Trucano) [2] Building and sustaining national ICT/education agencies: Lessons from international experiences (Michael Trucano & Gavin Dykes) [3] Building and sustaining national ICT/education agencies: Lessons from Korea (KERIS) (Youngsun Kwon & Sanghyun Jang) [4] Building and sustaining national ICT/education agencies: Lessons from Malaysia (Smart Schools) (Molly N.N. Lee & Soon Seng Thah) [5] The Role and Status of National Research and Education Networks (NRENs) in Africa (Michael Foley) [6] Building and sustaining national ICT/education agencies: Lessons from England (Becta) (Gavin Dykes) [7] Building and sustaining national ICT/education agencies: Lessons from Chile (Enlaces) (Eugenio Severin) [8] Building and sustaining national ICT/education agencies: Lessons from Armenia (NaCET) (Edmund Gaible and Anush Shahverdyan) [9] Building and sustaining national ICT/education agencies: Lessons from Uruguay (Plan Ceibal) (Eugenio Severin) [10] Building and sustaining national ICT/education agencies: Lessons from Indonesia (PUSTEKKOM) (Neil Butcher & Petra Bodrogini) [11] Evoke -- Developing Skills in Youth to Solve the World’s Most Complex Problems: The Social Innovators’ Framework (Barbara Freeman & Robert Hawkins) [12] Technologies in education across the Americas: The promise and the peril – and some potential ways forward (Michael Trucano) [13] Building and sustaining national ICT/education agencies: Lessons from Costa Rica (The Omar Dengo Foundation) (Carla Jimenez) [14] Building and sustaining national ICT/education agencies: Lessons from Thailand (NECTEC/Schoolnet Thailand) (Saowaruj Rattanakhamfu) [15] Building and sustaining national ICT/education agencies: Lessons from the Philippines (Benjamin Vergel De Dios) [16] Building and sustaining national ICT/education agencies: Lessons from Australia (EdNA) (Gerald White & Lesley Parker) [17] ICT and the Education of Refugees: A Stocktaking of Innovative Approaches in the MENA Region. Lessons of Experience and Guiding Principles (Kent Lewis with Simon Thacker) [18] Digital teaching and learning materials: Opportunities, options and issues (Michael Trucano) [19] Evoke -- Developing Skills in Youth to Solve the World’s Most Complex Problems: Randomized Impact Evaluation Findings (Barbara Freeman & Robert Hawkins) saber.worldbank.org Evoke: Randomized Impact Evaluation Findings World Bank Education, Technology & Innovation: SABER-ICT Technical Paper Series @WBedutech saber.worldbank.org worldbank.org/education blogs.worldbank.org/edutech worldbank.org/en/topic/edutech series coordinator: Michael Trucano saber.worldbank.org