An Evaluation of Non-Formal Learning in Professional Technical Networks, 2000-2001 Sukai Prom-Jackson Maria Bina Palmisano Wema Kategile Jackson Roman Novojilov Tena Malone WBI Evaluation Studies No. EG03-61 The World Bank Institute The World Bank Washington, DC September 2002 Acknowledgements This report was prepared for the Learning Board under the overall guidance of Marlaine Lockheed, Manager, Evaluation Group. The team was led by Sukai Prom-Jackson. WBI Evaluation Studies are produced by the WBI Evaluation Group (WBIEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at http://www.worldbank.org/wbi/evaluation/puball.htm. Assistance in data collection was provided by various members of the WBIES unit to include Padma M. Karunaratne, Shobha Kumar, Heidi Zia, and Biko Sankofa. The report benefited from reviews by Kenneth M. Gwilliam (TUDTR), Donald MacDonald (HRSLB), David Ellerman (DEC), and Bruno Laporte (WBKL) Vice President, WBI Ms. Frannie Léautier Chief Learning Officer Ms. Phyllis Pomerantz Manager, WBI Evaluation Group Ms. Marlaine Lockheed Task Manager Ms. Sukai Prom-Jackson ii TABLE OF CONTENTS Acknowledgements .........................................................................................................................................................ii I. BANK STAFF LEARNING NEEDS AND ALTERNATIVES TO FORMAL TRAINING......................1 II. RATIONALE FOR STUDYING NON-FORMAL LEARNING....................................................................3 III. STUDY OBJECTIVES..........................................................................................................................................5 IV. METHODOLOGY .................................................................................................................................................7 V. RESULTS ....................................................................................................................................................................9 VI. CONCLUSIONS AND RECOMMENDATIONS .........................................................................................17 ANNEX 1. ILLUSTRATION OF LEARNING PHASES AMONG BANK STAFF AND ASSOCIATED ACTIONS FOR LEARNING ENGAGEMENT......................................................................................................18 ANNEX 2. DESCRIPTION OF LEARNING PHASES AMONG BANK STAFF........................................19 ANNEX 3. DESCRIPTION OF FORMAL AND NON-FORMAL LEARNING ACTIVITIES .................20 ANNEX 4. SEVEN PRINCIPLES FOR LEARNING BY THE INSTITUTE FOR RESEARCH ON LEARNING....................................................................................................................................................................21 ANNEX 5. SUCCESSFUL LEARNING STRATEGIES AND FACILITATION FOR LEARNING EMPOWERMENT ........................................................................................................................................................23 ANNEX 6. NON-FORMAL LEARNING ACTIVITIES INCLUDED IN THE STUDY.............................24 ANNEX 7. NON-FORMAL LEARNING OBSERVATION INSTRUMENT................................................27 ANNEX 8. QUESTIONNAIRE FOR NON-FORMAL LEARNING EVENTS .............................................31 iii iv EXECUTIVE SUMMARY Purpose of the Study In FY01, the Knowledge and Learning Council (KLC) asked WBIES to study non- formal learning events and to (a) describe their nature and guiding principles, and (b) evaluate the learning that occurs in them. Non-formal learning events, as distinguished from formal training for knowledge acquisition are organized in the Bank as brown bag lunch sessions, clinics, workshops, and work sessions for "learning groups" or for "communities of practice" engaged in: learning, sharing, and using learning for further reflections, for solutions or new perspectives or for developing improved products. In contract to Sector Weeks, they are conceptualized to go beyond collective and reflective learning to focus on actionable learning and knowledge generation1. This report presents an exploratory study of non-formal learning in the Bank in FY01. It provides descriptions of non-formal learning as well as typologies, models for conceptualizing learning, and appropriate instruments for assessing learning processes and productive learning strategies. The study also analyzes the delivery features and learning process and outcomes of non-formal events convened in FY01. It is important to note that this study does not cover the wide range of "brown bag lunches" that occur on a daily basis all over the Bank, but rather is focused on events which were identified by key stakeholders and training coordinators of the Bank's Professional Networks as applying to some degree the criteria noted above for non-formal learning. Methodology The study started with dialogue and research on communities of practice in learning organizations and on the non-formal learning events convened by the Network Thematic Groups. This was then followed by non-structured interviews with Network Learning Representatives who took a lead role in the conceptualization of the study. Informal interviews and dialogue were also conducted with Thematic Group Leaders and members, and staff of the Knowledge Management unit responsible in FY00/01 for monitoring Thematic Groups and their learning and knowledge management activities2. Preliminary questionnaires were administered to get learner ratings of the quality of the non-formal learning events in which they participated and learner indication of types of delivery modes and learning outcomes they expect of non-formal learning events. 1See WBIES study on The Analysis of Sector Week, October 2000. 2See WBIES, 2001Background Information and Prospectus for the Evaluation of Thematic Group. Non formal Learning Program. v The information collected from all these sources was then used to develop: a conceptual framework for evaluating non-formal learning; a systematic direct observation instrument to assess the learning process (i.e. the use of effective or productive learning strategies and the empowerment of learners for self-directed learning); and a questionnaire to assess learning outcomes. Because of the exploratory nature of the study, a considerable amount of data that was collected but not found valid, reliable or codeable was discarded. The study is limited to data from the following: (i) questionnaires completed by 459 learners who participated in 54 PREM and PSI events convened between October 2000 and February 2001, and who provided information on the overall quality and on what learners value from such events; (ii) direct, systematic observations of 18 events of PSI, PREM, HD, ESSD for which data are valid and reliable; and (iii) questionnaires of 7 PSI events focused on assessing what staff gained from the learning events. Quantitative data are analyzed using descriptive statistic. Qualitative data are content analyzed and responses with large frequencies of occurrence are reported. Results There is no professional agreement on the definition of the various types of non-formal learning events in the Bank. · Non-formal learning occurs in brown bag lunch sessions, clinics, seminars, workshops, and work sessions. These different modes generally have different types of learning and action objectives, but within the Bank, these terms are used interchangeably. Thus a learner expecting to be involved in an event announced as a problem-solving session could end up in a forty-five minute lecture presentation with a 15-minute period for questions and answers. · To build a common definition and understanding of the different modes, the study provides a typology that could be used as a basis for describing and planning non- formal events. This could also be used to develop a more refined basis for sampling events in future evaluations of non-formal learning. Bank staff value rigorous interactive learning and exposure of diverse viewpoints · World Bank staff reported valuing non-formal learning events that have: effective discussion sessions, intensive exchange of views, question and answer sessions, a wide range of participants and organizations with diverse experiences, the sharing of "hands-on application, and the opportunity to expose views so as to verify, confirm, adapt or reject. This adult learning need is precisely what the non- formal events are supposed to address. This however did not generally characterize the mode of operation of most of the events studied. Only 5% of the time is spent on interactive learning among learners vi · Contrary to expectation, the non-formal learning events that occurred in FY01 were primarily lecture presentations: 71% of the time was spent on lecture presentations, 24% was spent on question and answer generally of a bi-lateral mode between the presenter and one learner. Less than 5% involved interactions among learners. · Informal interviews and discussions with various organizers suggest the following as possible explanations for the high level of lecture presentations. Task leaders for the events did not have a good theoretical grounding on what the non-formal events were supposed to accomplish and how to structure them to facilitate learning. Furthermore, the resource-constrained situation with training in FY01 lead to the loading of the non-formal events with basic knowledge content that should have been delivered in regular training courses, this done at the expense of the type of interactive learning that was planned. The use of facilitation and expert coaching strategies that motivate or "empower" learners to take charge of the learning process is average · During the 30% of time spent on question and answer and interactive exchange and discussions, facilitator efforts directed at "empowering" successful learning is average: · An effective facilitation process that empowers learning includes giving learners the lead role in the learning process, encouraging the exchange of knowledge in an open and challenging way, introducing complementary knowledge and challenging situations to enhance depth of understanding or reflection, analyzing implications or application to Bank, assisting learners to develop their problem solving strategies, engaging all participants in the final solution product, and setting the stage for continuous learning and the action. · Observations indicate that during question and answer sessions, participants do not generally build on each other's questions but generally focused on their own particular question. There is little or no attempt to raise the level of discourse by either facilitators or learners to a more in depth analysis and evaluation of the issues raised. · This limitation is partially attributed to the fact that the groups that come together are generally not "learning groups". They are staff with varying frameworks and interests in the subject matter of the event and as such they do not readily engage in a direct, rigorous, unrestrained and open exchange of views. This explanation is however countered by the point made by some stakeholders, that even with disparate groups, the discussions could have been enhanced if the presenters/facilitators know how to apply effective coaching and self-directed learning strategies. vii The intellectual level of presentations and of discussions does not significantly include higher-level cognitive operations - synthesis, evaluation, risk analysis and other reflective behaviors significant for innovation · The cognitive operations or intellectual nature of presentations and of the discussions are primarily factual and secondarily analytical. Limited time is spent on higher levels of cognition functioning such as synthesis, evaluation, application, risk analysis, or development of refined idea or product. It is these levels, which when applied, would lead to innovation adoption or reflections on new ways of doing business. There are however, pockets of excellence among the learning events · Some events of the PSI Network, such as the High Speed Rail Infrastructure and Rural Electrification in Senegal, committed over 50% of the time to interactive learning, and had a clear focus on analysis, synthesis, risk analysis and the definition of new directions or refined products or ideas. The former event also involved a specific "learning group" with a clear objective of brainstorming for developing a refined product. The learners present demonstrated high levels of expertise in the subject matter of discussion. Conclusions and Recommendations · In seeking to enhance its organizational learning capability, the Bank seeks to promote various learning paradigms that range from knowledge acquisition and adaptive learning models to collective, reflective/evaluative, actionable and knowledge generation models. Non-formal learning events are theoretically very important for the development of the Bank as a learning organization with active learners engaged in collective sharing, exchanging, refining and developing new knowledge. The majority of non-formal learning events studied did not effectively engage staff in the use of successful or productive learning strategies that are significant for knowledge use and innovation. · The evidence from selected cases of excellence in the study nevertheless suggest that when events are consciously structured with clear objectives and structure, non-formal events could play a major role in the development of the Bank as a learning organization focused on action from learning and as mechanisms for changing mental paradigms and for introducing innovation in the Bank. The following are recommendations for non-formal learning events. · Non-formal events need to have clearly defined outcomes which should drive the learning processes that are put in place to engage learners. It is recommended that the typology provide in the study be refined by the newly created Learning Board Pedagogic Support Unit and used as a guide to help the various stakeholders involved in non-formal learning. It should be used to guide the design of viii announcements of the events so these become less misleading and wasteful of staff time. · In convening non-formal events, it is important to define whether the target group is a "disparate group", a "learning group", or "communities of practice". This would help to tailor the expectations of interactive learning, of open and frank dialogue with no hidden agendas from the participants and the level of learner initiative in directing the learning process. It should also help guide the facilitation of learning by presenters or conveners. The study is significant in highlighting how active learning strategies and learning empowerment promoted via non-formal learning can be assessed via direct, systematic observation. The study is also significant in highlighting the types of learning outcomes (besides knowledge acquisition) that are significant for Bank staff as "knowledge workers". These outcomes should guide the design of non-formal and work-based learning interventions. The observation instrument and questionnaires developed from the study serves as useful tools that are actually now being utilized by various stakeholders to assess learning processes, the use of productive learning strategies, and the process of empowering learning. The instruments should be particularly pertinent for the programs of the Professional Technical Networks. A Level 3 Impact3 study by WBIES indicates that the "level of active involvement of staff in the learning process" is the most significant predictor of behavior change as evidenced by staff use of the knowledge and skills acquired from formal training. This study provides some useful tools that could be used in evaluating the level of active learning in Professional Technical training programs. Equally important as "active learners" for the success of non-formal events are facilitators who can serve as expert coaches and managers of group dynamics. This role needs to be clearly communicated to all facilitators to ensure effective facilitation is applied and learners are empowered. The strategies of effective facilitation to empower learning, provided in this study, should be communicated and applied. It is recommended, based on the lessons learned by WBIES in studying non-formal learning, that for future studies of non-formal learning, the design should consider the following: (a) the direct systematic observation of a random sample of events from a well defined set of learning events; (b) the administration of an electronic questionnaire on the key outcomes identified in the study to a random sample of participants of non- formal events at the mid-points of the fiscal year as opposed to at the end of each non- 3WBI Evaluation Unit, 2001. The impact of formal training programs for world bank staff. An evaluation based on learners' self-reports of the level of Use of Knowledge and Skills Acquired from Training and Their Impact on Performance ix formal events; and (c) the conduct of a feasibility study before embarking on a study of non-formal events. x I. BANK STAFF LEARNING NEEDS AND ALTERNATIVES TO FORMAL TRAINING Over the past three years (1999-2001), a large number of "non-formal" learning events described as - brown-bag lunches, clinics, workshops, action learning sessions have taken place in the Bank. These learning events represent an effort on the part of key stakeholders of the Staff Learning Agenda to respond to; (a) the evolution of the Bank as a learning organization and the accompanying requirements for a well integrated system of learning interventions; and (b) in a salutary fashion, the variety of learning and knowledge development needs among Bank staff. WBIES analysis indicates that the learning needs of Bank staff occur in five phases - awareness, acquisition, adoption, adaption, and the integration and renewal of knowledge. The phases are illustrated in Annex 1 and they are described in greater detail in Annex 2. The first two learning phases are generally best addressed via formal training or a well structured process of teaching and learning where knowledge is efficiently packaged and transferred into people's minds. Phases three through five are closely associated with very specific on-the-job learning and professional development needs and generally require less structured forms of learning activities. These are described as non-formal learning. They could be characterized by: · the sharing and exchange of knowledge, experiences, and good practices leading sometimes to the development of refined knowledge and approaches; · analyzing and developing solutions or major modifications to ideas and practices to increase value for the Bank and for clients; · integrating efforts across disciplines and developing joint ideas and products; · evaluating and reflecting on acquired knowledge, developing alternatives to existing knowledge, and generating new knowledge. · developing common frameworks, language or knowledge sets for mutual trust and joint efforts in development; · fulfilling a social need to be generative or for self-actualization; and · increasing commitment, passion and honesty in participating in world development. It is these continuous learning and use of knowledge (or "actionable learning") modes and outcomes that define the basis of non-formal learning in the Bank. Annex 3 provides definitions of non-formal learning and its key characteristics. It also describes formal training to highlight differences between the two modes of delivery. 1 2 II. RATIONALE FOR STUDYING NON-FORMAL LEARNING Studying and reporting on the non-formal learning activities of FY01 is recognized as significant for four reasons. 1. Non-formal learning events continue to be an important part of the FY02 curriculum of the Networks and Regions. They will continue to be important particularly in the current re-structuring of the staff learning programs and the focus on action learning and intact team learning groups engaged in collective and/or actionable learning. In this context, it is important to provide information on the nature of non-formal learning events, the guiding principles for their development, and the factors that enhance their effectiveness. This information is in high demand and WBIES has already started sharing the information from this report with interested parties. 2. A study of non-formal learning will also help to highlight some of the significant differences or complementarities with other new delivery modes in the Bank such as Sector Weeks4. In fact, this study follow-up on an earlier study by WBIES which was also commissioned by the Knowledge Learning Council. Together, these studies should provide the Bank with information to guide the improvement of innovations in the Staff Learning Agenda. 3. Non-formal active learning processes are not validly assessed via the traditional measures of questionnaires and interviews. One of the best methods of assessment involves direct observation of the learning process, of group dynamics, and the facilitation of learning. This study developed such an instrument and developed procedures for enhancing the objectivity and reliability of direct observation. The study piloted the use of the instrument and provides information based on direct observation of selected non-formal events. 4. WBIES subscribes to the position that while it is important to study learning as a process, it is equally important to determine the effects of the active learning process on learning outcomes that are noted by staff to be important. As shown above, these learning outcomes go beyond knowledge acquisition to include: refining knowledge, developing solutions, developing alternative perspectives, developing meta-cognition and learning-to- learn strategies, the fulfillment of a social need to be generative in sharing knowledge. These outcomes are analyzed in this exploratory study using learner self-report. 4See WBIES Study - The Analysis of the FY00 Sector Weeks, October 2000. 3 4 III. STUDY OBJECTIVES The study describes the following. 1. The nature of non-formal activities in the Bank based on observations of the different types of events delivered in FY01 as non-formal learning, i.e. brown bag lunch presentations, clinics, action learning workshops, brainstorming workshops, etc. 2. The quality of the design and delivery of non-formal learning events with a focus on the amount of time spent on (i) presentations, (ii) questions and answer and discussions, and (iii) interactive learning involving sharing and exchange of knowledge and experiences among learners. 3. The quality of the learning process and the degree of focus on the use of productive learning strategies, and the empowerment of learning by facilitators and presenters. 4. The degree to which the events responded to learner needs for: (i) new, cutting-edge knowledge, (ii) refinement of knowledge, (iii) the development of alternative perspectives, (iv) contacts from networking or exposure to experts, (v) solutions to problems, (vi) analysis of risks and developing an understanding of consequences in the application of ideas and efforts in development. 5 6 IV. METHODOLOGY 1. Instrument Development The study utilized a variety of methods to develop understanding of the rationale and principles of non-formal learning and to develop valid instruments for data collection. There were extensive discussions with Network Learning Representatives on the nature of the non- formal events they were promoting and their goals and objectives. This was followed by an extensive literature review of learning organizations, communities of practice, active and effective learning strategies used in learning organizations by active learners and "knowledge workers", and the facilitation process that empowers learning. The literature review also included action learning, emerging theories of learning as a process and not a product, theories on the non-linear nature of learning, action/behavior, and performance change. Annex 4 provides the guiding principles about the learning process and the development of communities of practice developed by the Institute for Research on Learning. Annex 5 identifies the strategies of effective learners, and the critical elements in facilitating and empowering learning among adults. The information from all these sources was used to develop a direct observation instrument to assess the nature and quality of the learning process. Annex 6 provides the complete list of non formal learning events included in the study. Annex 7 provides the final observation instrument. Prior to its use for this study, the instrument was piloted with over twelve events and inter-rater reliability procedures were established to enhance the objectivity of the instrument. A questionnaire was also designed and administered to assess participant ratings on the type of knowledge and strategies developed from the learning events. Annex 8 provides the final questionnaire which was preceded by a version which provided some information useful for developing the final questionnaire and the observation instrument. 2. Data Collection and Sample Studied In FY02, the majority of non-formal learning events were not planned well ahead of their occurrence. They occurred as part of an evolutionary planning process characteristic of the FY01 staff learning agenda. In the absence of planned information, WBIES approach was to find out about the occurrence of the non-formal events from the Network training coordinators and to send a team of trained observers to conduct the assessment. Where feasible or acceptable to the organizers, the evaluators did one or two of the following: (i) conducted observations, (ii) administered a questionnaire, (iii) conducted non structured interviews with the learners and facilitators or presenters. Based on this approach, the following data were collected. 1. The assessment of 54 PSI and PREM learning events via questionnaire assessing the overall quality of the delivery and learner indications of the types of learning processes and outcomes which they value of non-formal learning events. The events occurred between October and February, FY01. There were 459 respondents from the 52 events. 7 2. The results from the above questionnaire were used to develop a more refined instrument which assessed among other things, what learners gained from the events. The instrument was used in 7 PSI events which occurred between April and June, FY01. Attempts to use the instrument with a substantial number of other events were unsuccessful. 3. To assess active and effective learning processes and the facilitation process, systematic direct observations were conducted for 29 events. Only 18 of the 29 events are included in this study. Data from the 11observations were dropped from the study as they were unreliable and un-codeable. 8 V. RESULTS 1. Types of Non-Formal Learning Events in the Bank Non-formal learning events are described by various stakeholders as - brown bag lunch, clinic, seminar, workshop, and work session. They averaged ninety minutes in duration, with a range of 60 to 120 minutes. Most of them are held around the lunch hour, thus the general designation ­ Brown Bag Lunch. While professionals differentiate among the terms clinics, seminars, workshops, and work sessions, these terms are used interchangeably in the Bank to refer to the content of the various non-formal events. We propose the definitions outlined in Table 1, which are consistent with professional practice and which may help staff choose learning opportunities more easily. 2. Delivery Features for Non-Formal learning Events Valued by Bank Staff Seventy-two percent of the respondents of the 459 learners who participated in non- formal learning events convened by PREM and PSI between July and March of FY01, rated the learning events as good or excellent on a scaled questionnaire. They also valued five design and delivery modes of non-formal learning events: · discussions that are open and consider diverse perspectives; · an intensive or rigorous exchange of views involving analysis and opportunity to verify, confirm, adapt or reflect on ideas and practices; · question and answer sessions to firm up or broaden knowledge; · a wide range of participants and organizations to enrich the variety of experiences to be shared; · the sharing of "hands-on" applications. 9 Table 1: Typology of Non-Formal Learning Events Type of Event Definition and Processes Outcomes Brown Bag § Aforumforlecturesandpresentationson ØAcquisitionofnewknowledge Lunch cutting edge knowledge (could involve a and refinement of old knowledge relatively unstructured exchange of ideas or discussion among members) § Aforumfortheexchangeofexperiences ØFulfillment of a social need to be and good practices to share and to get generative and to share (on the reactions from others. part of presenter) § Aforumfortheanalysisandevaluationof ØDevelopmentofalternatives, specific problems, approaches or methods practical ideas or solutions, and the development of new ways of comparative perspectives. approaching a subject. Seminar § Aforumwhereeachlearnerinthegroup is expected to be at a sufficient level of Ø Enhancement ofrefined content-competence on the seminar topic knowledge Social inclusion in to actively participate. Members take learning responsibility for the interaction during the seminar. Clinics § Aforumfortheexchangeofcommon Ø Developingcommon experiences and analysis of good practices frameworks and knowledge or of principles that could lead to the sets for mutual trust and joint development of a refined product or idea. efforts in development. § Aforumfortheanalysisandevaluationof Ø Developingsolutionsormajor specific problems, approaches or methods modifications to ideas and and the development of specific and practices to increase value for concrete solutions for the job. (Done by a the bank and for clients. well defined group to produce a tangible product) Workshop ­ § Aforumtoraiseissuesaboutatopicand Ø Arangeofpotentialideasand Brainstorming to brainstorm for ideas, solutions and solution sessions approaches for resolution. Ø Socialinclusioninsolvingreal problems Ø Aspecificsolutionorarefined product Workshop ­ § Anactionlearningforumtodevelopa ØRefinedproductsand Action Learning specific solution or to refine a product for innovations of a more use on the job.(by an intact team) integrated nature. § Anactionlearningforumtointegrate efforts across disciplines and develop modifications or alternatives to current ideas and products. (by a cross-sectoral team) 10 3. The Primary Delivery Mode The non-formal learning events did not feature the types of learning delivery modes most responsive to the staff needs noted above. Direct observation of 18 events sponsored by PSI, PREM, HD and ESSD found that the primary mode of delivery was lecture presentations. For the 18 events: · 71% of the time was spent on lecture presentations · 24 % was spent on question and answer generally a bi-lateral type involving one participant and the presenter; participants did not actively build on each other's questions but generally focused on their own particular question. · Only 6% of the time was spent on interactive learning and discourse among the learners. HD sessions denoted a higher share (85%) of the time spent to lecture presentations, while PSI events denoted the least amount of time spent on lecture presentations (average 60%). 4. The Quality of Lecture Presentations Since the primary delivery mode was lecture presentation, the observation instrument, which was designed to assess interactive learning was expanded to include an assessment of the cognitive level and delivery quality of the presentations. (See Parts III and IV of the instrument.) Analysis of the observation data revealed the following. Cognitive Level of Presentations: Presentations are primarily of a factual nature and secondarily analytical. (See Table 2.) Limited time is spent on pulling together the information provided, evaluating it for its applicability to development, analyzing the risk factors in the adoption of any innovation, and making decision on actions to be taken or products to be used. The latter high levels of cognitive operation form the core of "reflective learning", which are most significant for enhancing innovation adoption. 11 Table 2: Average Frequency of Occurrence of Cognitive Operations in Lecture Presentations Based on Observer Ratings For 18 Non-Formal Events (Low Frequency of occurrence = 1, High Frequency of occurrence =5) Cognitive Levels Mean Ratings Description: Focus on factual information and clarifications. 3.7 Analysis: Break complex information into manageable parts. 2.4 Synthesis: Synthesize the information and develop from it principles and guidelines or lessons or new modes, methods, approaches, 2.3 practices Evaluation: Evaluate the information for its relevance or appropriateness for the job or for other contexts of significance. 1.8 Application: Consider how the new information will be used in one's particular situation. 1.5 Risk Assessment: Assess potential risks to success and consider the conditions for success; consider the potential impact and the 2.0 consequences of application. Refined Product: Use feedback from group or risk analysis to re- construct ideas or for development of new knowledge or new product. 1.7 Delivery quality: Observer ratings of the professional quality of the presentations, indicated that the presenters were well prepared, are knowledgeable and could relate knowledge being presented to the development context of Bank's work. This is a significant and positive finding for the Staff Learning Agenda of the Bank given the empirical evidence from the WBIES FY99 and FY00 evaluations of presentations which indicated limited capacity on the part of presenters to link up their presentations with the practical issues of development work.. Table 3: Observer Rating of the Quality of Lecture Presentations for 18 Non-Formal Learning Events Presenter Quality Criteria Mean Ratings · Was well prepared and organized 4.2 · Had knowledge of development and could put the information in context 4.2 · Was able to answer/address the participants' questions or issues raised 3.9 · Was effective in presenting the subject 3.7 matter within the time allocated Total 4.0 12 5. Empowering Learning Table 4 provides the observer ratings of the degree to which the presenters engaged learners and used effective facilitation strategies. This would have occurred during the 30% of time focused on question and answer between presenters and learners (see 3 above), and on 6% focused on interactive learning among participants. For this 36% of the learning time, the results demonstrate average effort in empowering learning and using the criteria noted in Table 4. Table 4: Observer Ratings of Presenter Use of Learning Empowerment Strategies (on a scale: Low 1 2 3 4 5 High) Empowering Learning Criteria Mean Ratings Encouraged discussions and the exchange of knowledge and ideas 3.4 Analyzed and integrated participants ideas 3.1 Analyzed group feedback to proposed solutions in an open and challenging environment 2.9 Provided information for continuous learning and application (web-sites, resources, experts) 3.0 Managed coordination of different and conflicting roles, concepts, ideas. 3.3 Managed coordination of the socialdimensions of participation (identity, relationship building, and inclusion) 3.2 Average 3.1 6. Learners as the Key Drivers of the Learning Process While there were cases when the Bank staff took over the learning session and re- focused it to address their specific learning needs and interest for social engagement and membership, the general ethos is for staff not to interrupt the forty-five minute lecture presentations. It is unclear whether this reflects: (i) a predisposition to politeness in letting presenters complete their lectures, (ii) a lack of preparedness to engage the learning process, or (iii) the fact that the participants generally come from diverse parts of the Bank and work programs and are not "learning groups" that very readily engage in active learning. Whichever the reason, the evidence suggests that sponsors of non-formal learning need to take an active role in designing and delivering these sessions with due regard to the seven principle of learning that are described by the Institute for Research on Learning (See Annex 4.) and to their respective objectives for productive learning. The evidence on the level of discourse in the interactive learning among learners indicates that learner discussion remains primarily descriptive and at times analytical. (See Table 5.) Engagement among learners in 13 evaluating ideas and strategies, defining new directions and developing knowledge of risk associated with such new directions is very rare . These behaviors are significant for the intelligent adoption of innovations in development. Moving non-formal learning events to this level of cognitive operation is a challenge of the Staff learning agenda and the training organizers and staff as learners. Table 5 Observer Ratings of Learner Level of Cognitive or Intellectual Discourse (Scale: Low 1 2 3 4 5 High) Cognitive Levels Mean Ratings Description: Focus on factual information and clarifications. 4.0 Analysis: Break complex information into manageable parts. 3.5 Synthesis: Synthesize the information and develop from it principles and guidelines or lessons or new modes, methods, approaches, practices 2.5 Evaluation: Evaluate the information for its relevance or appropriateness for the job or for other contexts of significance. 2.6 Application: Consider how the new information will be used in one's particular situation. 2.7 Risk Assessment: Assess potential risks to success and consider the conditions for success; consider the potential impact and the consequences 2.5 of application. Refined Product: Use feedback from group or risk analysis to re-construct ideas or for development of new knowledge or new product. 1.7 7. Good Examples of Active Learning Analysis of one of the best events observed, High Speed Rail, sponsored by PSI indicates that participants take the lead in being active learners when they have a high level of knowledge and expertise in the subject matter and when the learners form part of a larger well defined community with a common conceptual framework and a specific problem or issue to be solved. This session is also exemplary in that 55% of the time is spent on question and answer and interactive learning with a focus on the analysis, synthesis, evaluation, risk analysis and definition of products or strategies that would have an impact on development. A second exemplary learning event is the highly participative seminar, the Rural Electrification in Senegal, a 120 minute seminar, involving extensive sharing and exchange of knowledge and an excellent management of the nature and level of discourse by the facilitator. 8. Learning Outcomes While this study focused on the learning process, the use of successful learning strategies and the conditions that allow for this use, it nevertheless investigated what learners gain from sharing, exchanging ideas and analyze and evaluate strategies and approaches for 14 development. This question was investigated towards the end of the fiscal year `01 and thus the empirical data provided is limited to the 7 PSI events Table 6 shows how learners of 7 PSI events rated the gains they made from participating in the events. These results can only be generalized to PSI events which, as noted above, generally have a higher level of learning engagement and membership that is not typical of the majority of the non-formal events observed in this study by the WBIES evaluators. Table 6. What Participants Gained from the Learning Events: Results Based on Ratings by Participants of 7 PSI Events (Note: findings of limited generalizability) Cognitive Levels Frequency of occurrence Rating of 4 (above average) or 5 (high) New Knowledge 75% Refinement of knowledge 83% Alternative perspective 74% Contacts for Networking 68% Solutions to problems 71% References/ Resources 71% Total: 74% 15 16 VI. CONCLUSIONS AND RECOMMENDATIONS § Non-formal learning events are theoretically very important for the development of the Bank as a learning organizations with active learners engaged in sharing, exchanging, and using knowledge to refine and developing new knowledge. The majority of non-formal learning events are primarily lecture presentations and did not effectively engage staff in the use of productive learning strategies of significance for learning organizations. Non- formal events need to have clearly defined outcomes which should drive the learning processes that are put in place to engage learners. It is recommended that the typology provided in the study be refined by the newly created pedagogic unit and used as a guide to help the various stakeholders involved in non-formal learning. § In convening non-formal events, it is important to define whether the target group is a "disparate group", a "learning group", or "communities of practice. This would help to tailor the expectations of interactive learning, of open and frank dialogue with no hidden agendas from the participants and the level of learner initiative in directing the learning process. This study has developed a set of instruments that could be modified and used by various parties interested in assessing learning processes, the use of effective learning strategies, and the process of empowering learning. This would be particularly pertinent for the Professional Network for which a Level 3 Impact study by WBIES indicates that the "level of active involvement of staff in the learning process" is the most significant predictor of staff use of knowledge and skills acquired from formal training. Analyzing and evaluating the active learning and the use of in depth or higher levels of cognitive operation via the instruments developed in this study would provide valuable information on the effectiveness of the training offered by the Professional networks. § Equally important as "active learners" for the success of non-formal events are facilitators who can serve as expert coaches and managers of group dynamics. This role needs to be clearly communicated to all facilitators. The strategies of effective facilitation to empower learning should be communicated and support provided in their application. It is recommended, based on the lessons learned by WBIES in studying non-formal learning, that for future studies of non-formal learning, the design consider the following: 1. the direct systematic observation of a random sample of events from a well defined set of learning events as defined in the typology provided in this study; and 2. the administration of a questionnaire on the key outcomes identified in the study to a random sample of participants of non-formal events. This approach is likely to increase the response rate over the one used in this study where questionnaires were requested to be completed at the end of the short 3 hour or ninety minute session. 17 ANNEX 1. ILLUSTRATION OF LEARNING PHASES AMONG BANK STAFF AND ASSOCIATED ACTIONS FOR LEARNING ENGAGEMENT Bank Staff Learning Agenda: Learning Phases and Actions Renewal Awareness of Knowledge Generation of New Orientation Knowledge Acquisition Integration Staff Training Adaption of Basic of Knowledge Knowledge Evaluation / Sharing/ Exchange Reflection Acquisition Adoption of In-depth Learning Phases of Knowledge Learning Actions Knowledge 18 ANNEX 2. DESCRIPTION OF LEARNING PHASES AMONG BANK STAFF Phase 1: Awareness of and Orientation to new knowledge In this phase, staff has needs for: · Knowledge of new general or cutting-edge information: theory, philosophy, concepts, general principles, varying perspectives, practices and lessons from users of both OECD and developing countries. · Understanding the value of knowledge for staff, the Bank and its clients: linkage with development in varying contexts; linkage with Bank business or sector priorities and challenges; the implications, requirements, and demands upon staff work programs and the institution; implementation requirements; access to experts and other support systems when adopting new ideas. Phase 2: Development of basic knowledge and skills In this phase, staff has needs for declarative and content knowledge that must include: · Enhanced knowledge and understanding of the theory, principles, concepts, perspectives and practices · Development of basic skills for applying concepts, principles, and strategies · Enhanced knowledge of goal structure and sub-routines for appropriate application: cases, samples, good practices, challenges. · Specific knowledge of key issues in application and specific support systems during application. Development of in-depth knowledge, skills and strategies: In this phase, staff has needs for procedural knowledge or the development of mental models of goal structures and task routines. · Knowledge of the goal structures and sub-routines of task performance and problem solving; · Knowledge of how to apply the concepts and principles in general and in particular to their own tasks or work programs including: (i) the goals and sub-goals of the task; (ii) procedures for performing the tasks; (iii) the rules, mental models, and metaphors important for selecting and applying procedures for achievement of task goals and sub-goals. Phase 3: Adoption of Knowledge and Skills · Staff behaviors are first characterized by a "mechanical" use of the knowledge and skills acquired. Use is disjointed and at this stage, staff require reference manuals, procedures and guidelines, web sites and knowledge banks, mentoring by senior staff, and networking with colleagues at the same learning level. · With continued use, behavior becomes "routine" and staff is able to direct more effort towards increased effectiveness for the client and to begin to make necessary "adaptions" of knowledge as noted in the next phase. Phase 4: Adaption of Knowledge - Refining of knowledge for increased impact on clients that is based on both immediate and long-term consequences for clients. In this phase, staff has needs for contextual knowledge and the development of meta-cognitive skills, perspectives, and learning-to-learn strategies including;. · Knowledge of the conditions and contexts in which knowledge is useful; · The institutional contexts for implementation (global and local); · The rules and norms of behavior that are important for application; · The assumptions and parameters or risks factors that affect success; and · The immediate and long-term consequences of use for clients. Phase 5 Integration of knowledge involves a combination of knowledge and efforts with colleagues, clients, partners, practitioners and other professionals to achieve a collective product or idea of greater impact on clients. Renewal of knowledge in which staff evaluate the quality of use of the innovation and seeks to explore or apply major modifications or alternatives to present innovation to achieve increased impact on clients, examines new developments in the field, or explores new goals for self and for the Bank 19 ANNEX 3. DESCRIPTION OF FORMAL AND NON-FORMAL LEARNING ACTIVITIES Formal learning activities or training is the structured classroom training organized for the acquisition of new and well defined knowledge and skills. § Itisthetraditional,structured,classroomtraining. § Involvesalargenumberofparticipantswithidenticalneeds. § Focusedontheacquisitionofbasicorin-depthknowledgeandskills- declarative(factual) andprocedural(goalstructures,mentalmodels, and sub-routines for tasks). § Addressalllevelsofcognitiveoperations (understanding,analysis,synthesis,evaluation). § Deliveredbyexpertinstructorswhoknowmoreaboutthesubjectmatterthantheparticipants(peoplecometolearnnewinformation). § Thesuccessoftheeventdependstoaverylargedegreeontheroleplayedbytheinstructorstostructuretheactivitiestoteachandengagethe adult learner. § Useswelldefinedprinciplesandmethodsforeffectiveinstruction,learning,andthetransferoflearning. Non-Formal learning activities are for "continuous learning" and for "enhancing performance" on the job. The objectives are to refine, integrate or renew knowledge, skills, strategies and perspectives for increased impact on the job. The prime mechanisms for delivery are knowledge sharing and the development of new ideas or strategies among knowledge workers and partners. § Arenottraining,but"continuouslearning"and"performanceenhancement"activitiesthatcomplementtraining. § TheycoverawiderangeofeventsintheBankdesignatedasclinics,action-learningworkshops,intactteamworksessions,brownbagsessions, seminars, discussion group sessions. § Focusonactionablelearning,wherethelearningiscloselytiedtospecifictaskstobecompletedbystafforteams,orproblemstoberesolved, or products and strategies to be developed. § Focusonmakinglearningempoweringbydevelopinglearning-to-learnstrategies(goodpractices,contextualizedinformation,analysisofrisks and consequences, information on resources and experts available, websites, help-desks, networking opportunities and the development of partnerships). § Thustheobjectivesofnon-formaleventsare:torefineknowledge,ortointegrateknowledgeandperspectives(acrosssectors,disciplines), to renew and generate new knowledge and strategies for use on the job or on specified tasks. § Thepredominantmodeofdeliveryisknowledgesharingandtheconstructionordevelopmentofnewideasandproductsorstrategies. § Thesuccessoftheeventdependson(a) activeandhighlycommittedlearnerswhousethestrategiesofsuccessfullearners;and(b)facilitators with the skills to empower staff, skills to manage the interactive process of sharing, and provide feedback on implications and applications (see attachment in annex on active learning); and (c) provision of follow-up learning materials and support systems to enhance performance or use of the new ideas. § Usetheprinciplesof"communitiesofpractice"inlearningorganizationswiththemembersengagedinadynamicandsocialprocessof learning and using learning for action and impact. 20 ANNEX 4. SEVEN PRINCIPLES FOR LEARNING BY THE INSTITUTE FOR RESEARCH ON LEARNING Challenging Fundamental Assumptions INTRODUCTION In the outmoded view of learning that dominates our institutions, knowledge is regarded as a substance that can be deposited directly into people's mind. Learning is seen as the Process by which knowledge is transferred into the learner's mind, and teaching is seen as the packaging of knowledge for efficient transfer. Our research at IRL leads us to offer a new view, in which learning is inseparable from engagement in the world, and intellect is inseparable from experience. This view is summed up in seven basic principles: 1. LEARNING IS FUNDAMENTALLY SOCIAL Schools and workplaces commonly require participants to choose between learning and social fulfillment. This is a choice that should never arise. An important part of what makes adult professionals successful and productive is their ability to integrate their work with their social lives. They forge their identities and connections around their work, their knowledge and their contributions to the community in which they work. Yet most students and workers in our society are currently expected to live differently ­ to maintain distance between work and social activity. 2. KNOWLEDGE IS INTEGRATED IN THE LIFE OF COMMUNITIES Knowledge, activity and social relations are closely intertwined, whether in families, scientific communities, jump rope groups, jazz bands or design teams. United by a common enterprise, people come to develop and share ways of doing things, ways of talking, beliefs, values ­ in short, practices ­ as a function of their joint involvement in mutual activity. We call such informal aggregations communities of practice, because they are defined not only by their membership, but by shared ways of doing things. Every individual belongs to, and seeks membership in, many communities of practice. In communities of practice, social relations form around activities, activities take shape through relationships, and particular kinds of knowledge and expertise become part of individuals' identities and places in the community. Because shared knowledge underlines this activity, learning is the means by which people gain membership, and participate in community activity. 3. LEARNING IS AN ACT OF MEMBERSHIP Learning is not just the activity of a sole individual, but the primary vehicle for engagement with others. It is what enables people to enter and participate in the new communities of practice. The key to enhancement and motivation in learning lies in the intimate connection between the desire for participation and the role of new knowledge in enabling that participation. 4. KNOWING DEPENDS ON ENGAGEMNET IN PRACTICE. Only in the classroom is knowledge presented in the abstract, and only in the classroom are people expected to demonstrate knowledge through abstract performance. Knowledge does not lie around in the world in some pure form; nor is there any reason to believe that it is stored in the human mind in such a form. Rather people glean knowledge from observations of, and participation in, myriad situations and activities. A productive lifelong learner - a person who can adapt and learn swiftly in new situations ­ is a person who can transform all situations into a learning situation. Learners must able to learn as they engage in new activity, and as they move into new settings. This entails figuring out what the characteristics of the situation are, what is its 21 relation to situations that they already know, what there is to learn, and what new knowledge they need in order to be able to participate productively in the situation. 5. ENGAGEMENT IS INSEPARABLE FROM EMPOWERMENT. Individuals perceive their identities in terms of their ability to contribute ­ and in terms of their contributions ­ to a community. Meaningful participation in a community involves the power to affect the life of that community. Settings and situations that provide the greatest potential for learning, therefore, will be those in which participants have meaningful and active roles ­ in which they are engaged in real action that has consequences not only for them but for their community as a whole. 6. "FAILURE" TO LEARN IS A COMMON RESULT OF EXCLUSION FROM PARTICIPATION. Learning requires access and opportunity. People have difficulty learning when they are only accorded marginal or tentative membership. Limited privileges of participation do not entail rights to contribute and make meaning, hence do not provide opportunities for engaged learning. This deeper perspective requires a more textured understanding of the means and implications of discrimination and exclusion. 7. WE ALREADY HAVE A SOCIETY OF LIFELONG LEARNERS. People are learning all the time, but what they are learning is not necessarily in their best interests or in the best interests of society. People learn what enables them to participate in communities of practice ­ not just any communities of practice, but those that appear to them to be real, to be available, and to hold possibility for meaningful participation. It is this need for meaningful participation that motivates both the gang member and honor student, the scientist and the soloist, the public servant and the entrepreneur. (IRL 1990) 22 ANNEX 5. SUCCESSFUL LEARNING STRATEGIES AND FACILITATION FOR LEARNING EMPOWERMENT Successful Learning Strategies Facilitation for Learning Empowerment Problem Clarity and Readiness The facilitation process is closely aligned with the strategies for empowering successful learner. They include · Come to a pre -determined learning the following. situation with questions, problems or issues to address or experiences to share Mind Set (of relevance to the job or to professional · Develop a mind set and readiness for learning and sharing by encouraging the exchange development). of knowledge among the community of learners; identifying resources, experts, diversity · Do not come looking for a ready-made among community members, and defining session objectives, outcomes and structure for answer to their own problems. participation. · Focus on learning-to-learn and other Knowledge and Understanding strategies that empower learning. · Introduce complementary or new knowledge and enhance understanding. Analysis Discussion · Break complex information into · Discuss and analyze implications and applications in varying contexts of Bank work. manageable parts. Application Synthesis · Facilitate exercises on application to a designed situation · Synthesize the information and develop Feedback from it principles and guidelines or · Assist learners to develop and use their own problem-solving strategies. lessons. · Analyze group feedback to proposed solutions in an open and challenging environment. Evaluation · Engage all participants in the development of final products or ideas. · Evaluate the information for its relevance Depth or appropriateness for the job or for other · Enhance depth of learning in a self-directed fashion contexts of significance. · Provide opportunity for participants to develop new methods, to apply new ideas, or plans Construction and Application for collaboration, or identify major alternatives or replacements to the current innovations · Consider how the new information will be or practices; used in one's particular situation. · Provide opportunity for participants to develop immediate and long-range plans that Risk Assessment anticipate possible and needed resources and support to achieve impact. Provide · Assess potential risks to success and information on appropriate tools and other factors for enrichment or depth consider the conditions for success · Provide information on good practices · Consider the potential impact and the · Provide challenging situations for refection consequences of application · Focus analysis on risk factors and consequences Refined Product · · Provide information for continuous learning and application (web-sites, resources, Use feedback from group or risk analysis experts) to re-construct ideas or for development of Continuous Learning and Practice new knowledge · Encourage development of continuous learning among community (virtual learning and Social Identity and Interpersonal Skills · application groups; group meetings, peer supervision methods). Share pertinent knowledge Management of Multiple Forms of Knowledge and Group Diversity · Share knowledge for the benefit of others · Manage coordination of different and conflicting roles, concepts, ideas. but not to control the learning process · · Manage coordination of multi-disciplinary and cultural dimensions. Keep the group moving forward and not · Manage coordination of social dimensions of participation, identity, relationship building. hold it back 23 ANNEX 6. NON-FORMAL LEARNING ACTIVITIES INCLUDED IN THE STUDY Evaluated through Title of the Learning Activity Sponsor Observation Evaluated through Checklist Questionnaire PRSP Macro-modeling I PREM Bangladesh: Gender Dimensions on Impacts of PREM New Agricultural Technologies Gender, Poverty and Sustainability of Project PREM Interventions: What are the Linkages Gender and Digital Divide PREM Gender and Law: Lessons and Findings PREM Social Cohesion PREM Delivery of Services to the Poor PREM Gender, Growth and Equity PREM Attacking Poverty: Summarizing the WDR PREM Bridging the Economic Divide Within Nations: A PREM scorecard on the Performance of Regional Policies in Reducing Regional Income Inequalities Commodity Price Volatility, Stabilization PREM Mechanisms, and Financial Risk Confronting the Challenge of State Capture & PREM Administrative Corruption Consultant Trust Fund Procedures PREM Corporate Responsibility, Business Ethics, and PREM Reputational Risk Management Database Management in the LDB PREM DSM+ PREM E-Governance: Can it help Governments Become PREM Smart Economic Statistics PREM Engendering Development: Findings of a New PREM Policy Research Report and Implications for the Bank's Work Evaluating the Impact of Development Projects on PREM Poverty Global and Regional Inequality: Trends, PREM Determinants, and Implications "Hear our voices" ­A New Video on Voices of the PREM Poor and the WDR Inflation Targeting PREM Key Institutional Arrangements for Better Fiscal PREM Policy Launch of Civil Service Website & Toolkit PREM Live Database PREM Managing Depoliticization of the Civil Service: PREM Options for a Second Best World New Data Management Tools and Initiatives PREM Performance Budgeting PREM PREM Quality Enhancement Support: What's in it PREM for you? Public Expenditures Guidelines PREM Regional Exchange Rate Arrangement for East PREM Asia RSMS PREM Subnational Credit Markets PREM 24 Evaluated through Title of the Learning Activity Sponsor Observation Evaluated through Checklist Questionnaire 10 Things you always wanted to know about a PREM career in PREM Tools for Integrating Gender into Bank PREM Operations E-Commerce: Status & Trends PSI Annual Road Management PSI California Deregulation Experiment PSI From Belief to Knowledge PSI Highway Design Management Standards 4 PSI Training Innovation & Project Examples Regional Exp PSI Max Development Impacts of ICTs PSI Overview of Project Financing PSI Sources of Political Risk support PSI WBG Risk Mitigation PSI Rural W&S PPP for Comm Water Supply PSI W&SL Building Partnership to Extend PSI Services Introduction to Telecoms PSI Benefits of Clean Energy in Infra Develop PSI Dev Strat and Action Plans PSI Distribution Generation Ans to California PSI Innovative Market Based Financing Methods PSI Land and Housing Part 1 PSI Land and Housing Part 2 PSI Rural Infrastructure PSI Infrastructure Forum PSI Cybercity Connections Road and Highway TG PSI Tariff and Policy Reforms in the Water Sector Water TG in China PSI Support to Micro Enterprise Development as Urban an Instrument of Poverty Allocation Economics & Strategy TG & Services to the Urban Poor TG Urban DevelopmentTimeline Urban TG Rural Electrification in Senegal E & M Family Guarantees Leveraging Private Finance in Africa Frontier Market Infrastructure Sector HNP and the Poor: Applying the Life Cycle HNP TG Framework HD Limiting Child Labor Through Behavior­ Impact based Income Transfers: An Evaluation of Evaluation TG the PETI Program in Rural Brazil The Potential Pitfalls of Partnership in Health HDNP Care (Uganda) Women and War: Gender Dimension and PREM Reintegration Program 25 Evaluated through Title of the Learning Activity Sponsor Observation Evaluated through Checklist Questionnaire CDD in ECA: Decentralization and Municipal ESSD Finance Delhi-Noida Bridge Project PSI_ Transportation & IFC Community Contracting: A practitioner's CDD TG Perspective Educational Poverty Reduction HD High Speed Rail Infrastructure DBFM- INFTD Contract CDD in ECA: Cultural Assets in Support of ECSSD Transition Performance Target Based Non­revenue PSI 26 ANNEX 7. NON-FORMAL LEARNING OBSERVATION INSTRUMENT Observation by: _________________________________ Date: _________________________________ Part 1: Context and Background Part II: Delivery Mode Part III: Facilitator Strategies for Learning Empowerment Part IV: Cognitive Level of Presentation (by Presenters) Part V: Active Learning Processes and Cognitive Levels of Discourse Among Participants Part VI: General Comments and Observations 27 PART I: CONTEXT and BACKGROUND Title of Activity: ___________________________________________________________ Network/Sector: _________________ Thematic Group: _____________________ Date: ______________________________ Duration:____________________________ Participant Profile: Total # ______ # Male _______ # Female _______ Number coming in late: _________ Number leaving before session ends: ____________ Comments about Participants:____________________________________________________ Facilitator Profile: Bank Staff: ______________ Other: ___________________ Activity Type : Clinic _____ Brown bag _____ Seminar _____ Workshop ____ Other ___________________ Objectives and Outcomes from announcement: Available:________________ Not Available: _________________ Pre-Session preparation: (e.g. Reading materials): Provided prior to the meeting? Nature? __________________ Provided during the meeting? Nature? ___________________ Special Features for Observation: ________________________________________________ ______________________________________________________________________________ Contact Name(s) for follow-up: __________________________________________________ PART II: DELIVERY MODE % of time allocated to : # Participants Male Female Involved Lecture/Presentation: ______% _________ _____ _____ Q & A , Comments, Observations: ______% _________ _____ _____ Interactive Learning (among participants) ______% _________ _____ _____ Other: ________________________________________ ______% _________ _____ _____ Total 100% 28 PART III. FACILITATOR STRATEGIES for LEARNING EMPOWERMENT (Rating of 1-5) Presenters/Facilitators Please rate the Presenters or Facilitators on a five point scale. First Second Third Fourth Use 1 for Low Degree and 5 for High Degree. Was well prepared and organized Had knowledge of development and could put the information in context Was able to answer/address the participants' questions or issues raised Was effective in presenting the subject matter within the time allocated Encouraged discussions and the exchange of knowledge and ideas Analyzed and integrated participants ideas Analyzed group feedback to proposed solutions in an open and challenging environment Provided information for continuous learning and application (web-sites, resources, experts) Managed coordination of different and conflicting roles, concepts, ideas. Managed coordination of the social dimensions of participation, (identity, relationship building. and inclusion) Comments about Presenters/Facilitators: PART IV: COGNITIVE LEVEL of LECTURE/PRESENTATION Please indicate the Percentage of time dedicated to the following cognitive activities or levels of discourse by the Presenters/Facilitators. /Or rate the frequency of occurrence using the scale : Low 1 2 3 4 5 High Description: Focus on factual information and clarifications. Analysis: Break complex information into manageable parts Synthesis: Synthesize the information and develop from it principles and guidelines or lessons or new modes, methods, approaches, practices Evaluation: Evaluate the information for its relevance or appropriateness for the job or for other contexts of significance. Application: Consider how the new information will be used in one's particular situation. Risk Assessment: Assess potential risks to success and consider the conditions for success; consider the potential impact and the consequences of application Refined Product: Use feedback from group or risk analysis to re-construct ideas or for development of new knowledge or new product Others: Total 100% Comments about Cognitive Level or Depth of Presentation: 29 PART V: ACTIVE LEARNING PROCESSES and COGNITIVE LEVELS of DISCOURSE AMONG PARTICIPANTS (To be completed if Interactive Learning occurs (i.e sharing, and exchanging knowledge among participants) For the time interval noted, please indicate the frequency of occurrence of the following cognitive activities involving First 1/3 Second 1/3 Third 1/3 Total participants. Use 1 for Low Frequency and 5 for High Frequency RATING of Occurrence Description Focus on factual information and clarifications Analysis Break complex information into manageable parts Synthesis Synthesize the information and develop from it principles and guidelines or lessons or new modes, methods, approaches, practices Evaluation Evaluate the information for its relevance or appropriateness for the job or for other contexts of significance. Construction and Application consider how the new information will be used in one's particular situation. Risk Assessment Assess potential risks to success and consider the conditions for success; consider the potential impact and the consequences of application Refined Product Use feedback from group or risk analysis to re-construct ideas or for development of new knowledge or new product Comments about the Nature of Discourse and Interactive Learning Among Participants: PART VI: GENERAL COMMENTS AND OBSERVATIONS ABOUT NON-FOMAL LEARNING 30 ANNEX 8. QUESTIONNAIRE FOR NON-FORMAL LEARNING EVENTS 5. What did you gain from participating in the activity. (Please rate all that I. Background Information apply) Low High 1. What is your primary network? n/a 1 2 3 4 5 ¡ ¡ ¡ ¡ ¡ ¡ ¡ ACS ¡ ESSD ¡ HD ¡ FSE a) New knowledge ¡ ¡ ¡ ¡ ¡ ¡ ¡ PREM ¡ PSI ¡ OCS ¡ ISN b) Refinement of knowledge ¡ ¡ ¡ ¡ ¡ ¡ ¡ Other________ c) Alternative perspectives d) Contacts from networking and sharing ¡ ¡ ¡ ¡ ¡ ¡ II. Design and Delivery experiences e) Solutions to problems and issues ¡ ¡ ¡ ¡ ¡ ¡ 2. What was the format of the learning activity? f) References/resources ¡ ¡ ¡ ¡ ¡ ¡ ¡ Primarily presentation(s) followed ¡ Primarilydiscussionand g) Other (please specify) ¡ ¡ ¡ ¡ ¡ ¡ by a few questions & answers sharing among participants __________________ ¡ Short presentation(s) followed by ¡ Other(pleasespecify) primarily questions & answers _________________ 6. What did you like best about the learning activity? 3. How would you rate the quality of the delivery learning activity? ______________________________________________________ Poor Excellent n/a 1 2 3 4 5 a) The presentation of the subject matter ¡ ¡ ¡ ¡ ¡ ¡ ______________________________________________________ b) The facilitation of the exchange among participants ¡ ¡ ¡ ¡ ¡ ¡ c) The exchange of knowledge among participants ¡ ¡ ¡ ¡ ¡ ¡ IV. Comments and Recommendations III. Outcomes 7. What changes would you suggest for improving the quality of similar 4. How would you rate theoverall usefulness of the activity? activities (content, structure, format, etc.)? Low High N/a 1 2 3 4 5 ______________________________________________________ a) For your job performance ¡ ¡ ¡ ¡ ¡ ¡ b) For your professional growth and interest ¡ ¡ ¡ ¡ ¡ ¡ ______________________________________________________ c) For future assignment ¡ ¡ ¡ ¡ ¡ ¡ d) Other ¡ ¡ ¡ ¡ ¡ ¡ 8. Comments. 31