Page 1 DRAFT The 1998 Conference of the Mediterranean Development Forum Evaluation Report _______________________ Keith Mackay EDI Evaluation Studies Number ES99-18 Economic Development Institute The World Bank Washington, D.C. 48933 Page 2 Evaluation Report _____________________________________________________________________________ Copyright © 1999 The International Bank for Reconstruction and Development/The World Bank 1818 H Street, N.W. W ashington, D.C. 20433, U.S.A. The World Bank enjoys copyright under protocol 2 of the Universal Copyright Convention. This material may nonetheless be copied for research, educational, or scholarly purposes only in the member countries of The World Bank. Material in this series is subject to revision. The findings, interpretations, and conclusions expressed in this document are entirely those of the author(s) and should not be attributed in any manner to The World Bank, to its affiliated organizations, or the members of its Board of Directors or the countries they represent. If this is reproduced or translated, EDI would appreciate a copy. Page 3 Evaluation Report _____________________________________________________________________________ iii This report was prepared by Keith Mackay, with the assistance of Jamal Al-Kibbi, Mark Bardini, Haleh Bridi, Mary Cusick, Christopher Duggan, Bill Eckert, Keshav Gaur, Padma Karunaratne, Violaine Le Rouzic, Sondra Patrick, Ray Rist, Hana Salah, and Patrice Scanlon. Page 4 Evaluation Report _____________________________________________________________________________ iv EXECUTIVE SUMMARY The 1998 conference of the Mediterranean Development Forum (MDF) was held in Marrakech, Morocco, on September 3-6, 1998. Over 500 people attended the conference, including 270 workshop participants, from over 20 countries in the region. The MDF is a partnership of ten organizations in the Middle East and North Africa (MENA) region and the Economic Development Institute of the World Bank. The 1998 conference was the second regional conference of the MDF ¾ the first was held in May 1997, and a third is planned for March 2000. In the future it is intended to hold these MDF regional conferences every 18 months or so. The 1998 conference was structured around five workshops, which constituted a core activity and were each held over two days ¾ these focused on globalization, poverty reduction, governance, human development and public-private partnerships. There were also plenary sessions, with keynote speeches, panel discussions and videoconferencing, as well as social events to encourage networking, a cyber café and optional field visits. Conference Objectives The overall theme of the conference was Participation and Development . Encompassed within these were three stated objectives: · make a substantial contribution to the MENA policy debate in key issues of regional interest; · empower civil society and the private sector to influence public policy; and · help participants develop networks with each other. The conference was clearly intended to be more than a training event ¾ its focus was on raising awareness and understanding of policy issues, and directly contributing to policy dialogue in the MENA region. The conference was also designed as part of a process of helping regional organizations to develop, and of engaging the World Bank more in the region. Evaluation Objectives and Methods There are a number of methodological difficulties in attempting to evaluate directly whether the conference was successful or not in achieving each of the three objectives (see Section 2). One approach followed in this evaluation is to report the views of participants and of the Bank’s conference and workshop task managers about the extent to which the conference objectives are likely to be met. In assessing the effectiveness of the conference it is also useful to include other lines of inquiry, such as the extent to which participants and task managers were satisfied with various aspects of the conference, including the workshops and the plenary sessions ¾ the conference is unlikely to have achieved its objectives if participant satisfaction levels were low. Similarly, it is important to find out if participants considered they had increased their understanding of topics and regional policy issues addressed at In conclusion, the success of the 1998 conference is heartening. It underscores the success of the MDF partnership and provides another platform on which future collaboration, policy dialogue and development efforts in the MENA region can build. Page 5 Evaluation Report _____________________________________________________________________________ v the conference. The evaluation also identifies options for improving future regional conferences. I n summary, the evaluation addresses the following questions: · did participants, and the Bank’s conference and workshop task managers, judge that the conference met each of its three objectives? · how satisfied were participants with various aspects of the conference, and what suggestions did they have for future improvement? · to what extent did participants consider that they have increased their depth of understanding of the core themes and topics covered by the workshop which they attended? · how satisfied were task managers with the planning and administration of the conference, and with the contribution of MENA partner organizations? · which features of the conference and workshops worked well, which did not, and why? · how can future conferences be improved? · what was the overall cost of the conference? Five broad data sources were used in conducting this evaluation. They included: a participant questionnaire; informal interviews of conference participants; interviews of Bank conference and workshop task managers, of staff responsible for conference administration, and of staff in the Bank’s MENA operational area; the published conference/workshop programs and the conference website; and the Bank’s conference databases on participants and speakers, and on costs. Various methods are used to analyze these data, including analysis of respondent ratings of satisfaction, and the comparison of these ratings with results for other conferences. A caveat when interpreting the respondent ratings reported in this evaluation is that the overall response rate is not known with certainty, and thus the extent to which respondent satisfaction levels are representative of all participants in the conference is unclear. Regression analysis is used to identify which features of the conference and the workshops were most influential on respondent ratings. Participants had been invited to make open-ended comments and suggestions on the conference ¾ the features which they judged had contributed to or limited its effectiveness, and suggestions for future improvement. These comments, and the comments of the conference and workshop task managers, were analyzed using a content analysis approach. Together, these approaches provide multiple lines of inquiry; the extent to which the evidence is consistent or contradictory is also discussed. Evaluation findings Extent of achievement of conference objectives Participants ¾ respondents ¾ were generally very positive when asked to assess the likely extent to which the conference would meet its objectives, although these assessments should be accepted with caution because of the intangible and long-term nature of the objectives. Respondents rated the conference highly for achieving its objective of contributing to MENA policy debate, and also the objective of helping participants develop networks. However, respondents felt that the impact of the conference was undercut to some extent by the failure of the workshops to treat issues Page 6 Evaluation Report _____________________________________________________________________________ vi in sufficient depth and to allow sufficient time for constructive discussions. Also, respondents did not rate the conference highly in terms of achievement of the objective of empowering civil society and the private sector. The Bank’s conference and workshop task managers were generally supportive of the extent to which the conference had met its objectives. They pointed to a number of ongoing activities in which the Bank and its MENA partners are involved. These activities reveal a substantive contribution to, and development of, MENA policy debates. Task managers were also able to point to a number of substantive activities which will increase the involvement of civil society and the Bank’s MENA think-tank partners in policy debates in the region; in pursuing this objective the conference was successful in involving a large number of eminent persons from the region as speakers at the conference. The majority of task managers also volunteered the view that the conference had encouraged the development of networks in MENA. Thus a recurrent theme expressed by task managers was that the conference activities and initiatives should be viewed as part of an ongoing process of regional involvement and dialogue ¾ one which includes the regional partners and the Bank as key players. In other words, the conference should not be viewed as a stand-alone activity but as part of an ongoing collaborative process. Comparison with other major conferences When compared with four other major conferences which the Bank has organized or with which it has been heavily involved ¾ the 1997 MDF conference, Global Knowledge ’97, the South Asia Beyond 2000 conference and the Asian Development Forum conference ¾ the 1998 MDF conference achieved the highest levels of participant satisfaction (Section 3). On a scale of one (minimum) to five (maximum), the average respondent rating in response to a question ‘How would you rate the overall effectiveness’ of the conference was 3.93. Moreover, 78% of respondents gave a score of 4 or 5 out of 5 ¾ these ratings thus reveal a high level of satisfaction with the MDF conference. Nevertheless, there is still room for improvement, especially as the Bank aims to provide or support world-class events such as conferences and workshops ¾ thus one target which could be set for such events would be for at least 85% of respondents to give a satisfaction rating of 4 or 5. The 1998 MDF conference came near to achieving such a level. ...one target which could be set for such events would be for at least 85% of respondents to give a satisfaction rating of 4 or 5 out of 5. The 1998 MDF conference came near to achieving such a level. Page 7 Evaluation Report _____________________________________________________________________________ vii The World Bank is often viewed as independent and apolitical in nature, and this gives it a special role to play in helping to focus attention, via such collaborative conferences, on topics which are important for the region but which may be politically very sensitive ¾ on issues such as girls’ education, the manner in which the Arabic language is taught, and corruption. MDF conferences provide the opportunity to raise new ideas and to provoke open dialogue and debate; such opportunities can be rare in the MENA region. In this way, the Bank can make a substantial contribution to regional policy dialogues. Well-regarded features of the conference Features of the conference with which respondents were most satisfied included the five workshops and various plenary sessions such as keynote speakers, panels and video-conferencing ¾ these features, and particularly the workshops, were also the most important conference features for respondents. The main factor which determined whether respondents were satisfied with the conference as a whole appears to have been the quality of the specific workshop which each participant attended. A number of lessons learned from the 1997 MDF conference were important in raising the levels of respondent satisfaction with the 1998 conference (Section 4). Lessons from the workshops These lessons include conducting a smaller number of workshops (five in 1998 compared with 12 in the 1997 conference), shorter workshops and a less congested conference timetable. A problem with the 1997 conference had been participant fatigue, which resulted in low levels of attendance at a number of whole-of-conference events, and it also restricted the amount of networking and interaction among participants. Another lesson from the 1997 conference was the need to avoid a ‘talking heads’ approach, which involved over-reliance on a lecture style of workshop format with little opportunity for discussion and debate among participants ¾ some workshop task managers in 1997 had appeared to have crammed more and more speakers into whatever time was available. A feature of both the 1998 and 1997 MDF conferences was the notable variation in performance ¾ as measured by respondent satisfaction ¾ across the various workshops. Using a regression analysis for the 1998 conference data, a positive relationship was found between respondent satisfaction and two specific features of each workshop: the extent to which the workshop focused on the issues which each respondent most hoped would be addressed, and the extent to which adequate time had been allocated for constructive discussions. Unfortunately, for most workshops at the 1998 conference the latter was the least successful feature. MDF conferences provide the opportunity to raise new ideas and to provoke open dialogue and debate; such opportunities can be rare in the MENA region. In this way, the Bank can make a substantial contribution to regional policy dialogues. A number of lessons learned from the 1997 MDF conference were important in raising the levels of respondent satisfaction with the 1998 conference . Page 8 Evaluation Report _____________________________________________________________________________ viii Thus it appears that some key lessons from the 1997 conference ¾ the need to avoid a talking heads approach and congested workshop agendas ¾ were n ot sufficiently heeded in planning and designing the 1998 conference. The World Bank, and especially its external training arm, the Economic Development Institute, needs to pay heed to these lessons and act on them in planning future conferences and workshops. The upside is that there exists a real opportunity to achieve considerably higher levels of participant satisfaction by avoiding these and other problems in organizing future conferences. To ensure this happens, there might need to be a stronger measure of workshop quality control by conference task managers. In addition, a number of useful suggestions for the improvement of future conferences were made by task managers and respondents (Section 5). These deserve careful consideration by the organizers of future conferences. Conference structure In designing a conference there are a number of possible tradeoffs in the balance between overall conference length, number of workshops and plenary events, number and duration of workshops, number and type of participants per workshop, and so on. Achieving an optimal mix of such choices is never easy; however, the additional experience gained from the 1998 MDF conference can aid in designing future conferences if its lessons are heeded. These tradeoffs also help to determine the overall cost-effectiveness of such conferences. The cost of the 1998 MDF conference was $878,000, or about $3,240 per workshop participant. Good practice features Good practice features of the 1998 MDF conference included: · workshop rehearsals before the conference; · the clos e involvement of the World Bank’s operational area; and · preparation of detailed descriptions of the topics to be covered by each workshop session. Changes in participant understanding Participants were asked to assess the extent of increase of their depth of understanding of the main topics covered in each workshop (Section 4). Such self-assessments can suffer from some limitations in terms of their reliability and accuracy. Nevertheless, it is interesting to note that the average increase in understanding for respondents ¾ as measured on a five point scale from one (very low) to five (very high) ¾ increased by 25%, a substantial amount. Again, there was considerable variation between the averages for different workshops, with estimated increases ranging from a high of 37% to a low of 15%. But it appears that some key lessons from the 1997 conference - the need to avoid a talking heads approach and congested workshop agendas - were not sufficiently heeded in planning and designing the 1998 conference. The World Bank, and especially its external training arm, the Economic Development Institute, needs to pay heed to these lessons and act on them in planning future conferences and workshops. Good practice features of the 1998 MDF conference: · workshop rehearsals before the conference; · close involvement of Bank’s operational area; and · preparation of detailed descriptions of the topics to be covered by each workshop session Page 9 Evaluation Report _____________________________________________________________________________ ix Contribution of MDF partner organizations The contribution of the MDF partner organizations in organizing the workshops appears t o have been of varying quality (Section 5). Task managers acknowledged the valuable contributions which several partners made to planning workshop themes and agendas, to workshop preparation via rehearsals conducted prior to the workshops, and to the selection of participants. However, the majority of task managers was highly dissatisfied with the performance of their regional partners, and felt that the World Bank had been left to shoulder an unfair share of the administrative burden. Reasons for this are not entirely clear. They might include lack of clarity on the part of some partner organizations about the expectations on them, and a possible lack of management oversight of staff within some partner organizations. A list of recommendations is provided in the report to help guide future collaboration with partner organizations in planning future conferences. Notwithstanding the problems encountered with some aspects of the 1998 MDF conference the task managers consider that close collaboration with partners has a lot to offer, both in organizing specific events such as conferences, and also as a means to foster and support the further development of partner organizations’ own capabilities. Conclusion In conclusion, the success of the 1998 conference is heartening. It underscores the success of the MDF partnership and provides another platform on which future collaboration, policy dialogue and development efforts in the MENA region can build. Page 10 Evaluation Report _____________________________________________________________________________ 1: INTRODUCTION The Mediterranean Development Forum (MDF) is a partnership of ten organizations in t he Middle East and North Africa (MENA) and the Economic Development Institute (EDI) of the World Bank. The MDF held its second regional conference on September 3-6, 1998, in Marrakech, Morocco. The conference attracted over 500 people, including 270 workshop participants from over 20 countries in the region ¾ other attendees included speakers, the media and World Bank staff. A. Stated Objectives The overall theme of the conference was Participation and Development . Under this banner, the broad conference objectives were: MDF 1998 aims to empower thinkers, encourage private sector involvement, and mobilize civil society for greater participation in policy dialogue and to assert more influence over policy outcomes. Workshop discussions are intended to generate a better understanding of the challenges facing societies in the region, and to discuss strategies that can involve the various parts of society in a joint effort to boost economic growth, improve integration with the world economy, and reduce poverty and inequality . 1 Within these broad objectives were three more specific, stated objectives: 2 · make a substantial contribution to the MENA policy debate in key issues of regional interest; · empower civil society and the private sector to influence public policy; and · help participants develop networks with each other. The emphasis of the conference was thus on raising awareness and understanding of policy issues, and directly contributing to the policy dialogue in the MENA region. The conference should be viewed as more than a single, stand-alone event. It was designed to continue and develop the World Bank’s collaboration and partnerships with organizations in the MENA region; an earlier step in this process had been the first MDF conference, held in May 1997; the 1998 conference focused on a number of themes and issues raised in the earlier conference. A number of these themes are likely to continue to be examined and discussed at the third MDF conference, planned for March 2000, and in subsequent conferences ¾ which in the future will be held every 18 months or so. 1 MDF 1998 conference brochure. 2 As articulated by the conference coordinator, Ms. Haleh Bridi. Page 11 Evaluation Report _____________________________________________________________________________ 2 B. Conference Structure and Content The conference was structured around five workshops, which constituted a core activity; e ach workshop was held over two days. Also scheduled throughout the conference were: plenary sessions with keynote speeches, panel discussions and video- conferencing; social events to encourage participants to network with each other; a cyber cafe to allow participants to post electronic discussions with each other ¾ this was complementary to the MDF website; and an optional field visit to a site event on the final day of the conference. Special events prior to the conference included a two-day journalists’ symposium and a four-day workshop on microcredit challenges in the MENA region. There was also a special roundtable discussion on poverty issues, on the final day of the conference. The overall structure of the conference is shown in Table 1. Table 1: Conference Structure  Thursday 3 to Sunday 6 September, 1998 Thursday Friday Saturday Sunday Morning (Arrival and Registration) ·Plenary Panel ·Workshops ·Plenary ·Plenary Panel ·Workshops ·Optional Activities (field visit and roundtable) Afternoon (Arrival and Registration) ·Press Briefing ·Workshops ·Workshops ·Optional Activities (field visit and roundtable) Evening ·Opening Ceremony ·Keynote Address ·Dinner ·Plenary ·Dinner ·Closing Ceremony ·Dinner The five two-day workshops addressed the following issues: · Benefiting from Globalization · Economic Development and Poverty Reduction · Governance: Efficiency and Participation · Human Development: Moving Forward · Public-Private Partnerships in the MENA region. C. Participants Table 2 shows the numbers of participants and others who attended the conference. The conference organizers had hoped that one-third of conference participants would be from academia and non-government organizations (NGOs), one-third from regional governments ¾ Ministers and officials, and one-third from the private sector. Women account for only low proportions of senior positions in MENA governments, academia, NGOs and the private sector, and so it was hoped that the role of women in the region could be furthered by inviting a relatively high number of prominent women to the conference. Attendance at the conference was by invitation only, and quotas were set for each country in the region in order to ensure a balanced representation across the region. Page 12 Evaluation Report _____________________________________________________________________________ 3 Table 2: Participants and Others at the MDF Conference 1998 Conference 1997 Conference (a) Participants - of whom: academics/NGOs officials/Ministers private sector others 271 89 98 50 34 497 226 213 58 (c) Media 66 (b) 32 Speakers 118 (c) SUB-TOTAL 455 529 - of whom: female 22% 17% Others (d) 60 129 TOTAL 515 (b) 658 (a) Numbers are based on percentage disaggregations and are approximate only. Source: EDI, The Mediterranean Development Forum: Looking Back and Planning Ahead , draft report dated 23 September 1997. (b) Does not include local media of 40-70 per day. (c) Included und er ‘others’. (d) Includes World Bank staff, etc. The MDF partners took the decision to reduce overall participant numbers at the 1998 conference compared with the preceding year, from 500 in 1997 to just over half that number in 1998. The intention was to ensure the participation of only the most prominent representatives from a number of sectors. Large and similar numbers of academics/NGOs and officials/Ministers attended the 1998 conference ¾ almost 100 people in each category. However, the number of private sector participants fell short of the level hoped for, and was even slightly lower than in the preceding year. Almost 70 media representatives attended the conference overall, in addition to between 40 and 70 local media per day ¾ a significant increase on the level in 1997. Given the very low representation of women at senior levels in the MENA region, the high proportion of participants who were women (22%) was a particular achievement ¾ and was higher than the preceding year (17%). Page 13 Evaluation Report _____________________________________________________________________________ 4 Page 14 Evaluation Report _____________________________________________________________________________ 5 2: EVALUATION APPROACH A. Desired Impact, and Objectives of the Conference The ultimate impact which the World Bank seeks from its activities is the alleviation of poverty, and this is pursued via the provision of loans, grants and other assistance to support various activities and projects. Activities such as conferences and workshops can be evaluated, in principle, according to the extent to which they have an impact on countries, their policies and capacities, and the development effectiveness of their activities. Put another way, the MDF conference can be seen as one link in a desired ‘results chain’ which it is hoped will lead to the ultimate goal of poverty alleviation. A methodological difficulty in attempting to evaluate these links, however, is that the line of causality between a single conference and the ultimate goal of poverty alleviation is too long to be identified with any confidence ¾ it is simply infeasible to expect that a single 3-4 day conference would have any measurable impact on poverty in developing countries. Instead, when conducting an evaluation of such a conference, the focus is typically on intermediate measures of ‘success’. These would include the extent to which both the specific and the broad objectives identified for the MDF conference are achieved. Even here, however, the identification of causal links is difficult, as can be seen from an examination of the MDF conference objectives mentioned earlier ¾ i.e., to contribute to MENA policy debate; to empower civil society and the private sector to influence public policy; and to help participants to develop networks with each other. For example, the influence of a single conference on MENA region policy debates can be difficult ¾ though not impossible ¾ to identify, simply because there would typically exist a large number of factors which influence the policy debates and actual policies of any individual government. Also, the influence which many of these factors exert might only develop over a protracted period of time, or after a period of delay once a country’s policy environment has become conducive to change, and this introduces a further measurement difficulty. It would be possible to attempt to measure the influence of a conference by surveying partner organizations and former conference participants at some time in the future to ask them whether the knowledge and understanding they gained at the conference helped them in advising or seeking to influence their government ¾ or, in the case of government ministers, actually deciding ¾ on desirable policy choices. But again, it could be unrealistic to ask them to try to attribute any particular policy advice or decision to the knowledge and understanding gained at a particular conference. There might well exist some instances where it is possible to observe a policy shift and to infer that the conference played some role in changing opinions and judgments, but such instances can be expected to be very much the exception rather than the rule. One example has been reported by a Bank officer who was a workshop task manager at both the 1997 and 1998 MDF conferences. He believes that the advice by the World Bank, via the two conferences and by other means, which has been provided to MENA region governments concerning the European Union free trade agreement, has been influential. He considers that this advice has strongly influenced at least two governments in the region to resist signing that agreement; he also acknowledges the difficulties in proving a causal relationship. Page 15 Evaluation Report _____________________________________________________________________________ 6 The approach take n in this evaluation is to ask participants and also the Bank’s conference and workshop task managers for their judgments concerning the extent to which the conference met each of the three stated objectives. A particular issue investigated is the views of Bank task managers and operational staff concerning the development and support for partner organizations in the MENA region. This is part of the conference objective to empower civil society. It is also important to the Bank’s wish to move from being a ‘retailer’ of learning/training activities such as conferences and seminars, to being more of a ‘wholesaler’, and thus encourage regional partners to assume more responsibility for planning, design and delivery of such activities. Thus the partners’ performance in helping to plan and organize the conference is relevant to this objective. An issue outside the scope of this evaluation is the Bank’s overall objectives and activities in the MENA region, and the effectiveness and efficiency of the MDF partnerships overall. EDI intends to evaluate its regional partnerships at a later date. B. Other Evaluation Issues A standard framework to evaluate training or learning events such as seminars, workshops and many conferences is provided by Kirkpatrick (1998). 3 He identifies four sets of issues which can be evaluated, and these correspond to a number of links in the results chain: (1) the reaction of participants at the end of the conference ¾ high participant ratings would be expected to be necessary but not sufficient to indicate that a conference has been a ‘success’; (2) the extent to which skills and knowledge have been transferred to participants; (3) changes in the work or the workplace of participants as a result of skills or knowledge acquired; (4) the impact on the participants’ organizations. A version of this model can be used in evaluating the MDF conference, as it broadly corresponds to parts of the results chain discussed earlier. One difference is that the conference was not concerned with training per se ¾ its focus was on increasing participants’ depth of understanding of the policy issues discussed at the conference. This emphasis was reflected in the workshops at the conference ¾ they were intended to encourage debate and discussion among participants and speakers, rather than to be a training event. An additional difference relates to the previous discussion concerning the various objectives and different types of impact sought from such conferences. This focus is much broader than a concern simply with participants and their organizations. It would encompass an examination of the impact on regional and national policy dialogues and government decision-making within the MENA region. Another issue, which receives some ¾ albeit limited ¾ attention in this evaluation is the planning, development and administration of the conference itself. These issues are also the subject of a separate review by the Bank’s conference task manager, and so are not addressed comprehensively in this evaluation. Some of the material collected as part of the evaluation ¾ the comments of workshop task managers and of 3 Kirkpatrick, D. (1998). Evaluating Training Programs: the Four Levels (2 nd Edition). Berrett- Koehler Publications: San Francisco, CA. Page 16 Evaluation Report _____________________________________________________________________________ 7 participants ¾ has already been provided as an input to the separate administrative review. Finally, the total cost of the conference and the average cost per participant are relevant in assessing the overall cost-effectiveness of the conference. C. Evaluation Methods and Questions As noted above, there are a number of methodological difficulties in attempting to evaluate directly whether the conference was successful or not in achieving each of the three objectives. This evaluation reports the views of participants and of the Bank’s conference and workshop task managers about the extent to which the conference objectives are likely to be met. Other lines of inquiry which are pursued here are to assess the effectiveness of the conference in raising participants’ awareness and understanding of regional policy issues. Thus, there is a focus on the extent of participant and task manager satisfaction with various aspects of the conference ¾ such as the workshops and the plenary sessions ¾ and on whether participants consider they have increased their understanding of topics and issues addressed at the conference. The evaluation also identifies options for improving future regional conferences. The evaluation therefore addresses the following seven questions (the methodology is discussed in more detail in Annex A): · did participants, and the Bank’s conference and workshop task managers, judge that the conference met each of its three objectives?; · how satisfied were participants with various aspects of the conference, and what suggestions did they have for future improvement?; · to what extent did participants consider that they had increased their depth of understanding of the core themes and topics covered by the workshop which they attended?; · how satisfied were task managers with the planning and administration of the conference, and with the contribution of MENA partner organizations?; · which features of the conference and workshops worked well, which did not, and why?; · how can future conferences be improved?; and · what was the overall cost of the conference? In undertaking this evaluation five broad data sources were used: · a participant questionnaire (reproduced in Annex A) which sought their reactions, suggestions for future improvement, and self-assessments concerning the extent to which they had increased their understanding of workshop topics; · informal interviews of a number of participants during the conference; · interviews of Bank conference and workshop task managers, of EDI staff responsible for various administrative aspects of the conference, and of staff in the Bank’s MENA region operational area; · an examination of the published conference and workshop program and the conference Web site, which contains background papers as well as a number of those presented at the conference; and · EDI’s conference databases on participants and others who attended the conference, and on conference costs. Page 17 Evaluation Report _____________________________________________________________________________ 8 Various methods were used to analyze these data. Participants had been asked to rate their satisfaction with a range of aspects of the conference overall and of the particular workshop which they attended. These aspects included: achievement of conference o bjectives; the effectiveness of the main types of forum activity (workshops, plenary panels and speakers, the cyber café, and interaction with other participants during free time); and the extent to which participants considered the conference to have been a worthwhile use of their time and its overall effectiveness. A caveat when interpreting respondent ratings is that the overall response rate is not known with any certainty, and thus the extent to which respondent satisfaction levels are representative of all participants in the conference is unclear. 4 Ratings for respondent satisfaction levels are compared with those for four other major conferences which EDI has organized either on its own or with others. The relationship between self-ratings of respondent satisfaction and respondent ratings of the effectiveness of specific parts of the conference and the workshop attended are investigated using regression analysis. This approach helps to reveal which features of the conference and the workshops were most influential on respondent ratings. A regression analysis can provide evidence about which aspects of the conference and workshops are considered most important by participants, and about their relative importance. This type of information is valuable because it can be used to design more effective conferences and workshops in the future. The questionnaire also asked participants to identify specific features of the conference which they considered contributed to its effectiveness, as well as the features which limited its effectiveness. Participants were also invited to make comments or suggestions for improving future conferences. These open-ended responses were analyzed using a content analysis approach ¾ this technique enables key or recurrent themes to be identified. A content analysis was also applied to the interview responses of the conference and workshop task managers. Together, these various approaches provide multiple lines of inquiry. The extent of consistency (or ‘triangulation’) in these different types of information is also discussed. 4 Only 25% of all conference registrants completed an evaluation questionnaire, although the effective response rate is believed to have been considerably higher ¾ see Annex A. Page 18 Evaluation Report _____________________________________________________________________________ 9 3: OVERALL EFFECTIVENESS OF THE CONFERENCE A. Achievement of Conference Objectives The extent to which conference objectives ¾ i.e., to contribute to MENA policy debate; to empower civil society and the private sector to influence public policy; and to help participants to develop networks with each other ¾ have been achieved is an important measure of the effectiveness of a conference. As discussed earlier, it is extremely difficult to measure this. The approach followed here is to report the views of respondents and of the conference and workshop task managers about the extent to which conference objectives are likely to be met. Participants were asked the question ‘To what extent did the forum meet each objective?’ Given the lofty nature of these objectives ¾ especially the one which relates to the empowerment of civil society and the private sector, given the likelihood that the achievement of the objectives would probably occur only over a protracted period of time, and given the problem of identifying causality, respondent and task manager judgments should be interpreted with some caution. Participant Views The judgments of participants who responded to the conference questionnaire are shown in Table 3. In absolute terms the ratings appear to be reasonably high ¾ other than for the second objective, on empowerment. These ratings might even be regarded as surprisingly high, given the difficulty in achieving such objectives at a single conference of short duration. Table 3: Respondent Ratings of Achievement of Conference Objectives (a) To what extent did the forum achieve each objective?: Average rating % who rated 4 or 5 N (a) make a substantial contribution to the policy debate in key areas of regional interest 3.77 68.9 106 (b) empower civil society and the private sector to influence public policy 3.25 38.5 104 (c) help you develop networks with other participants 3.74 61.0 105 (a) Relates to respondents who are MENA region residents. Respondents are those who had registered for a particular workshop at the conference and who completed a questionnaire at the end of that workshop. Networking is an important vehicle for future collaboration, as well as providing an ongoing means by which lessons learned, expertise and insights can be shared. The respondent rating for the conference (3.74) is somewhat higher than for the recent Asian Page 19 Evaluation Report _____________________________________________________________________________ 10 Development Forum conference (3.60) 5 and for the recent South Asia Beyond 2000 conference (3.49) 6 . O f the 96 participants who provided additional comments on the conference, ten nominated the opportunities for networking as a specific feature of the conference which contributed to its effectiveness. Task Manager Views The difficulty in measuring the achievement of conference objectives was highlighted in the interviews of the five workshop task managers. Each was asked whether it would be feasible to attempt to undertake a tracer study some six months or so after the conference ¾ such a study would involve contacting participants and asking them if the conference had helped increase their involvement in policy debates in the MENA region, and also whether it had helped them develop contacts and networks within the region. However, the workshop task managers ¾ and also the conference task manager ¾ noted the difficulty in identifying the separate effect of any single conference on MENA region policy debates. (1) Contributing to MENA policy debates The conference task manager and four of the five workshop task managers volunteered the judgment that the 1998 conference should best be viewed as part of a long and continuing process of World Bank engagement in the region. The conference task manager noted the close involvement of MENA partner organizations in conference preparation ¾ in tasks such as preparation of conference and workshop agendas, preparation of speakers, and networking with regional speakers. She argued that this close involvement has both strengthened and empowered these partner organizations. 5 The wording of the question asked of participants at the ADF conference was similar though not identical: ‘To what extent did the forum meet its objective to strengthen networks among participants?’ The ‘raw’ rating was 4.62 out of a maximum of 6; this translates to a rating of 3.60 out of a maximum of 5. 6 The wording of the question was: ‘To what extent have networks of individuals been created or fostered by the seminar?’ Features of the conference which contributed to its effectiveness were, for example: · ‘the informal networking of women from the region was a brilliant idea’ (participant from a university/research institute, Iran) · ‘opportunities for participants to establish a dialogue with specialists’ (private sector participant, Lebanon) · ‘ability to interact with policy-makers and grass-roots activists simultaneously’ (NGO participant, Morocco) · ‘primarily, the interaction and the contacts established among the various partners’ (media representative, Tunisia) Page 20 Evaluation Report _____________________________________________________________________________ 11 It has done this in part by deepening their understanding of and involvement in key regional issues. The conference also provided an opportunity to interact with Ministers, s enior government officials and other influential persons ¾ it thus lifted the profile and perhaps even the credibility of partner organizations within their country. Four task managers said that the conferences should be both viewed and designed as part of a sequence ¾ they argued that the 1998 conference did a good job of building on the work of the 1997 conference, and that the MDF conference planned for 2000 should likewise follow up in some way on a number of the issues raised earlier. Task managers reported specific plans and suggestions for helping to achieve continuity, and also for helping to deepen the Bank’s regional involvement. These include, for example: · additional, small, stand-alone regional workshops which would focus on training and detailed technical issues in areas such as education or social service provision; · a national conference (for Turkey) on governance issues such as public sector budgeting and corruption, involving also participants from the 1998 MDF conference; · creation of a network of experts in social funds, to exchange information and expertise ¾ this network plans a follow-up meeting to the 1998 MDF conference, in Cairo; · inclusion of a workshop on governance issues in the forthcoming 2000 MDF conference, but with the discussions being focused on technical issues in order to provide participants with more of the ‘how to’ ¾ by contrast, the 1998 conference focused on broader policy issues; · a number of planned research projects and training (for practitioners and also trainer-training), on issues such as the environment and water policy, and involving the MDF partners. These activities will be partly a follow-on from the 1998 conference, and partly a precursor to the 2000 conference; and · regular meetings of the MDF partner organizations. Such activities help to involve World Bank staff and task managers more fully in the MENA region ¾ the need to achieve this was noted in the evaluation of the 1997 conference (EDI, 1998, p.14). It is interesting to note that three of the five workshop task managers in 1998 had also been workshop task managers at the 1997 conference; the remaining two task managers in 1998 have had a long-term involvement in the MENA region. One of the task managers noted that, for his area of expertise, most of the World Bank’s agenda for the region had been based on discussions and conclusions emerging from the MDF conferences. One task manager argued that only the Bank could run sessions on topics which are politically very sensitive ¾ such as girls’ education, the manner in which the Arabic language is taught, and corruption ¾ and get away with it. He also argued that the Bank is viewed as independent, apolitical and objective, and it can raise new ideas, provoke dialogue and debate. He noted that the MDF conferences provide the opportunity to participants to debate these issues openly, and that in some MENA countries information and open debate are heavily constrained. At the same time, he argued, such debates also enable Bank staff to understand better the range and complexity of these regional issues. Another workshop task manager argued that the MDF conferences provide the only MENA-wide forum for debate of policy issues. Page 21 Evaluation Report _____________________________________________________________________________ 12 (2) Empowerment of civil society and the private sector Three of the workshop task managers thought it would be very difficult to expect a single c onference to achieve such a lofty and intangible objective. Indeed, one argued that it would be ‘pretentious’ to argue that the conference could ever be a major driver in the empowerment of civil society and the private sector; however, he did acknowledge that the conference had been successful in helping civil society and NGOs, including via encouraging them to participate in the conference, and thus involving them in the issues discussed at the conference, and in raising their profile among the influential participants there. Another task manager also argued that the 1998 conference had made a lot of progress in engaging NGOs and in raising their profile with government ministers. A number of NGO representatives were specifically invited to participate in the conference ¾ attendance was by invitation only ¾ and it is noteworthy that 146 NGO representatives participated in the conference. 7 The conference task manager reported several initiatives which would strengthen the role of NGOs in the region: · a network of lawyers was created during the conference to review, harmonize and improve national laws relating to NGOs ¾ a meeting in Cairo is planned in order to pursue these objectives; · a second initiative at the conference was to set up a network of women’s NGOs, with the objective of helping the Bank to evaluate some of its social, agricultural and rural infrastructure projects ¾ a meeting in Washington D.C. is planned; · agreement at the conference to set up a Meet Civil Society Initiative , through which the Bank could communicate its programs to civil society in various MENA countries ¾ this would provide NGOs and partner institutions the opportunity to provide feedback to the Bank about its programs and projects within individual countries; · agreement that the Bank would undertake consultations with MENA governments and partner organizations with a view to improving their access to data, in order to support their policy advising capacities; and finally, · another initiative, which is only tentative at this stage, is that a MENA microcredit institute ¾ involving a number of NGOs ¾ might be set up as a result of participant discussions at the conference. A large number of regional speakers were each invited to make a presentation at the conference ¾ 105 regional speakers, some of whom were Ministers and government officials. This growing reliance on regional speakers helps to support individuals and organizations in MENA, and by increasing the regional focus of the conference it is also a means to ensure that such conferences are not overly ‘Bank-centered’. This is consistent with a recommendation from the evaluation of the 1997 conference, that ‘it would be opportune for the next conference to focus on a smaller set of the most topical issues in the area and to draw more heavily on regional speakers’ (EDI, 1998, p.13). 7 This figure also includes participants from universities, research institutes and partner organizations. Page 22 Evaluation Report _____________________________________________________________________________ 13 (3) Development of networks Three workshop task managers responded positively on this issue (the other two did not r espond in a definite manner). One of the three opined that the conference ‘will meet this objective beautifully’. Another argued that the conference will have helped develop networks through cementing existing networks and fostering nascent ones. The third argued that the MDF conferences provide solid networking to support regional policy dialogues. One workshop task manager noted that there already exists an informal network of regional speakers in his area of expertise, and added that there is an opportunity and a need to involve these individuals in future events (such as MDF conferences) ¾ this would help to maintain the group’s existence, and he argued that it would also be worthwhile to broaden the membership of this network. B. Overall Satisfaction with the Conference Participant Views The reactions of respondents provide a measure of the usefulness of a conference ¾ the conference is unlikely to have achieved its objectives if participant satisfaction levels were low. Two similar questions were asked of participants at the MDF conference: ‘To what extent has the forum been a worthwhile use of your time?’ and ‘How would you rate the overall effectiveness of the forum?’ Both questions were asked in order to enable comparisons to be made between the levels of respondents’ satisfaction with the MDF conference and the levels achieved at similar large conferences in the past. Table 4 provides a comparison of satisfaction levels at five major conferences with which the World Bank has recently been involved. It is evident that for each question where comparisons can be made, the 1998 MDF conference achieved the highest respondent rating of the five major conferences. On a scale of 1 (minimum) to 5 (maximum), the 1998 MDF conference achieved an average rating of 3.93 in response to the question ‘How would you rate the overall effectiveness of the forum?’; moreover, 78% rated the 1998 MDF highly ¾ i.e., a rating of 4 or 5. In response to the question ‘To what extent has the forum been a worthwhile use of your time?’ the average rating was 4.14; 83% gave a rating of 4 or 5. These ratings reveal a high level of satisfaction with the 1998 MDF conference; nevertheless, there is still room for improvement, especially as the Bank aims to provide or support world-class events such as conferences and workshops. Thus one target which could be set for such events would be for at least 85% of respondents to give a rating of 4 or 5. The 1998 MDF conference comes near to achieving such a level. Page 23 Evaluation Report _____________________________________________________________________________ 14 Table 4: Respondent Reactions to Major Conferences Number of Registrants ‘How would you rate the overall effectiveness of the forum?’ ‘To what extent has the forum been a worthwhile use of your time?’ Average rating (a) % who rated 4 or 5 Average rating (a) % who rated 4 or 5 N Mediterranean Development Forum #2 (Marrakech, September 1998) (b) 515 3.93 78 4.14 83 107 Mediterranean Development Forum #1 (Marrakech, May 1997) 658 n.a. n.a. 3.75 (c) n.a. 193 Global Knowledge ‘97 (Toronto, June 1997) 1752 3.20 n.a. n.a. n.a. 408 South Asia Beyond 2000 Conference (Colombo, March 1998) 76 3.66 60 n.a. n.a. 47 Asian Development Forum (Manila, March 1998) 400 (d) 3.87 n.a. n.a. n.a. 108 (a) The questionnaire for the MDF2 and the South Asia Beyond 2000 conferences used a five-point ratings scale of 5=maximum, and 1=minimum. The other three conferences used a six-point scale (6=maximum, 1=minimum). An advantage of a five-point scale is that it allows respondents to provide a mid-point rating. To achieve comparability with average ratings for the other three conferences shown here, the ratings for the latter were converted to five-point equivalent scores by applying a conversion factor of 0.78. The choice of this conversion factor is based on empirical evidence on rating differences with five-point and six-point scales (EDI, Evaluating EDI Participant Reactions via Different Response Scales: A Technical Review , 1999.) (b) These data relate to MENA region residents only. (c) This rating related to the workshop attended by each respondent, rather than to the conference as a whole. (d) Approximate number only. Page 24 Evaluation Report _____________________________________________________________________________ 15 The ratings for different types of respondent at the 1998 conference are shown in Table 5 ¾ this shows the ratings given by respondents from the MENA region who were from the following sectors: non-government organizations (NGOs); Ministers/ Parliamentarians; government ministry/organization; media; private sector; and university/ research/training institute. Table 5: Conference Ratings by Different Groups of Respondent ( a) Group (organization/type of function) ‘How would you rate the overall effectiveness of the forum?’ N Average rating % who rated 4 or 5 Non-government organization 3.89 72.2 18 Ministers/Parliamentarians 4.20 100.0 5 Government ministry/organization 3.93 78.6 28 Media 3.88 75.0 8 Private sector 4.36 100.0 5 University/research/training institute 3.87 73.7 38 Total (b) 3.91 76.4 133 (a) All groups relate to MENA region residents, except for World Bank/IMF staff. Respondents are those who had registered for a particular workshop at the conference and who completed a questionnaire at the end of that workshop. (b) Includes others, including various categories of non-MENA resident. Although some of the numbers in Table 5 are small, there are interesting differences in the ratings of the different types of groups which attended the conference. The highest ratings were given by respondents who were from the private sector (4.36) or who were Ministers/Parliamentarians (4.20) ¾ the latter category of participant is a particularly important target group for a conference to assist and influence. All respondents in both groups gave the conference a high rating for its effectiveness (a score of 4 or 5 out of 5). The four other main groups of respondents ¾ NGO representatives, government officials, media and university/research institutes ¾ gave very similar, but somewhat lower scores on average; the average scores ranged from 3.87 to 3.93, and the proportions of each group giving a high rating ranged from 72% to 79%. Task Manager Views Four of the five workshop task managers, and also the conference task manager, believed that the conference was a worthwhile activity, and one suggested the desirability of MDF conferences becoming annual events. Two task managers noted that the conference also helped to engage World Bank staff more firmly in the region by helping them better understand regional perspectives. The conference task manager reported the close invol vement of the Bank’s operational area in the planning and preparation of the conference, and she considers that this resulted in the operational area’s being a full partner in the conference. This viewpoint was shared by Bank operational staff. One workshop task manager also noted the high level of collaboration Page 25 Evaluation Report _____________________________________________________________________________ 16 between EDI and the Bank operational area concerning the conference, although he believed that this collaboration was very much the exception rather than the rule. 8 O ne workshop task manager did not agree that the conference as a whole was worthwhile. He argued that there was too much ‘show and glitter’ involving events such as plenary sessions and video-conferences, and that the overall conference theme ¾ Participation and Development ¾ was too broad and detracted from the depth of coverage of issues, thus diffusing both the messages and the possibilities for influencing policy. C. Participant Satisfaction with Individual Conference Activities and Features Participants were asked to rate the effectiveness of each of the main types of conference activity: workshops; plenary panels and speakers; the cyber cafe; and interaction during free time. Respondent ratings are shown in Table 6. Table 6: Respondent Ratings of Conference Activities (a) How would you rate the effectiveness of the following types of forum activities? Average Rating %4 or 5 N a) Workshops 3.88 72 108 b) Plenary panels and speakers 3.81 72 108 c) cyber cafe 3.46 50 68 d) Interaction during free time (including meals, breaks, evenings, etc.) 3.57 57 102 Overall effectiveness of the forum 3.93 78 107 (a) Relates to respondents who are MENA region residents. Respondents are those who had registered for a particular workshop at the conference and who completed a questionnaire at the end of that workshop. The ratings indicate that the workshops, closely followed by the plenary panels and speakers, were the most popular parts of the conference, with average ratings of 3.88 and 3.81 respectively. Participant interaction during free time (3.57) and the cyber cafe (3.46) had lower average ratings and much lower proportions of respondents who rated them highly (57% and 50% respectively). A regression analysis was conducted to investigate the relationship between overall respondent satisfaction with the conference and the four different types of conference activity identified above. This analysis tells us about the extent to which respondent satisfaction was influenced by different conference activities ¾ i.e., it provides insights into the importance of different activities as perceived by respondents. In the equation shown in the box below, respondent satisfaction (i.e., the rating they gave in judging the overall effectiveness of the conference) is expressed as dependent on how they rated each of the four main conference activities ¾ the workshops, plenary sessions, cybercafe and opportunities for participant interaction. The size of the coefficients shown in front of each of these four variables 9 indicates the relative 8 Since the 1998 MDF conference, and in recognition of the opportunity and need to strengthen the momentum of MENA region activities following the conference, a closer and formal relationship has been established between the MENA operational area of the Bank and EDI. 9 The other (first) variable shown is the constant term. Page 26 Evaluation Report _____________________________________________________________________________ 17 importance of each variable. The numbers in parentheses are t -statistics of significance; asterisks indicate that the coefficients are statistically significant, and thus that the activity rating is a determinant of the level of respondent satisfaction with the conference a s a whole. (a) i.e., respondent rating of the overall effectiveness of the forum. t -statistics are in parentheses. ** indicates p-value £ .01. The results indicate that respondents’ overall level of satisfaction with the conference was influenced by their views of the workshops and the plenary sessions. Of the two types of conference activity, workshops were viewed as much more important than plenary sessions ¾ this result is perhaps unsurprising, as the MDF partner organizations had selected potential participants on the basis of the workshops which were planned, and then invited those individuals to attend the workshop judged to be most suitable for them. The results also suggest that the key to participant satisfaction with the conference as a whole was the quality of the specific workshop which participants attended ¾ this issue is explored in more depth in Section 4. The results also indicate that participant interaction during free time did not influence their level of satisfaction with the conference as a whole. Respondent comments on factors which contributed to or limited the effectiveness of the conference (Section 4) also do not reveal a lack of opportunities for interaction as being particularly important to them. However, these observations are not consistent with a finding from the evaluation of the 1997 MDF conference, which concluded that participants in that conference valued highly the opportunities for networking with others. Future evaluations of such conferences might usefully examine more closely the extent to which participants regard such opportunities for interaction as being important. Among the features of the conference which were nominated by individual respondents as contributing to its effectiveness, the main ones were: · the caliber, diversity and expertise of presenters ¾ nominated by 44 of the 96 respondents who provided additional comments in their questionnaire responses 10 ; · organization, structure or adherence to time schedules (mentioned by 32 respondents); · choice of topics and themes addressed (31 respondents); · the quality and diversity of participants (14 respondents); · the workshops (11 respondents); and · the plenary sessions (9 respondents). The main features nominated as limiting the conference’s effectiveness included: 10 The 96 respondents constituted 35% of workshop participants. Respondent 1.263 + 0.496 Workshops + 0.258 Plenaries - 0.031 Cybercafe + 0.068 Interaction satisfaction (a) = (2.64)** (4.95)** (2.65)** (0.31) (0.66) Adjusted R 2 = .33 Page 27 Evaluation Report _____________________________________________________________________________ 18 · lack of time for discussion or for dealing with issues in sufficient depth (36 respondents) ¾ this was mainly a problem with the workshops, and is discussed in t he next Section; · the organization and structure of the conference, such as overlapping topics in different sessions (15 respondents); · caliber of presenters or of their presentations (14 respondents); and · quality of conference facilities (6 respondents). Overall, the 1998 conference was rated much higher than its 1997 counterpart (refer Table 4 above). The 1997 conference overall was rated much lower than the ratings of the content and relevance of the workshops at that conference; a key reason for the low satisfaction with the 1997 conference overall was the heavy agenda that scheduled numerous whole-of-conference activities and the consequent fatigue experienced by many participants. The lack of free time in the 1997 conference agenda was also an impediment to interaction and networking among participants. One recommendation from the evaluation of the 1997 conference was that a conference structure involving shorter and fewer workshops and fewer whole-of-conference events should be tried (EDI, 1998, p. vii) ¾ this approach was followed at the 1998 conference, with evident success as measured by respondent feedback (see also Section 4 below). Page 28 Evaluation Report _____________________________________________________________________________ 19 4: EFFECTIVENESS OF WORKSHOPS The five workshops formed the core of the conference. The workshops focused on the f ollowing topics: · Benefiting from Globalization · Economic Development and Poverty Reduction · Governance: Efficiency and Participation · Human Development: Moving Forward · Public-Private Partnerships in the MENA region. Two of the workshops each had two separate themes: the one on globalization had themes on trade and finance; the one on human development had themes on health and education. Each of the five workshops lasted for two days, and the number of participants and speakers registered for each ranged from over 70 to almost 100. By contrast, the 1997 MDF conference included 12 workshops, each ranging from one and a half to five days in duration and consisting of between 20 and 60 participants. An important lesson learned from the 1997 conference was the need for fewer workshops, and for each to be of shorter duration (EDI, 1998, p.19). To some extent, the length and crowded agendas of the workshops in that conference undercut the usefulness of the conference as a whole ¾ participant fatigue was a problem, and it resulted in low levels of attendance at a number of the whole-of-conference events, and also restricted the amount of networking and interaction among participants. Another lesson learned from the 1997 conference was the need to avoid a ‘talking heads’ approach, which involves over-reliance on a lecture style of workshop format with little opportunity for discussion and debate among participants. One reason why some of the 1997 workshops fell into this trap was the crowded workshop agendas ¾ the evaluation of the conference argued that ‘it is hard to escape the conclusion that some task managers tended to cram more and more speakers into whatever time was available’ (EDI, 1998, p.17). By contrast, the philosophy pursued in the 1998 conference was more of a dialogue among equals ¾ one task manager argued that the 17 speakers in his workshop could also be considered to be participants, and some of the other participants were equally experienced and expert in the workshop topics; the task manager judged that this encouraged a dialogue among equals as distinct from a ‘talking heads’ approach. In designing the evaluation questionnaire for the 1998 conference particular attention was paid to the difficulties identified by respondents at the 1997 conference. Responses to the questionnaires are examined below. A. Participant Satisfaction As usual for a conference with a number of different workshops, there was a range of levels of respondent satisfaction with the workshops ¾ the scores for workshops with the highest ratings are shown shaded in Table 7. One caveat in interpreting the respondent responses is the low response rates for some workshops ¾ reasons for this are reported in Annex A; however, it is possible that these low response rates have resulted in some of the average ratings for individual workshops being unrepresentative of all participants in the workshop. Page 29 Evaluation Report _____________________________________________________________________________ 20 Participants were asked their views about the effectiveness of the conference’s workshops; respondent ratings are shown here on the basis of the workshop for which they registered. 11 The Public-Private Partnerships workshop had the highest rating ¾ 4.05 out of a maximum of 5 ¾ for effectiveness of workshops. Moreover, 82% of respondents gave a high rating of 4 or 5 ¾ this is close to a target level of 85% of respondents with a high level of satisfaction, which is suggested earlier in this report as a target level of respondent satisfaction for world-class events such as workshops or conferences. The Economic Development and Poverty workshop had the lowest rating among the five workshops ¾ a rating of 3.55 ¾ with only 64% of respondents giving a 4 or 5 rating. The spread between the highest (4.05) and lowest rating (3.55) was 0.50. A direct comparison with the workshop ratings for the 1997 conference cannot be made because of differences in the questions asked of participants ¾ in 1997 participants were asked ‘To what extent has the workshop been relevant to your job?’. Nevertheless, it is interesting that the satisfaction scores for the 12 workshops in 1997 ranged from 4.21 to 3.36, a spread of 0.85. 12 Respondents in the Human Development workshop gave the highest rating among any workshop for the conference as a whole, in terms of it being a worthwhile use of participants’ time ¾ a rating of 4.28, with 92% rating the conference a 4 or 5. The alternative measure of respondent satisfaction ¾ i.e., the overall effectiveness of the forum ¾ was rated most highly by respondents in the Public-Private Partnerships workshop, with a rating of 4.05 and with 77% of respondents rating the conference a 4 or a 5. 11 As each respondent was registered for only one workshop, it is highly likely that r espondents’ views relate solely to the particular workshop for which they registered (although some participants moved between workshops during the conference). 12 To ensure comparability, these 1997 ratings have been adjusted from a 6-point scale to a 5- point scale. Page 30 T h e 1 9 9 8 C o n f e r e n c e o f t h e M e d i t e r r a n e a n D e v e l o p m e n t F o r u m _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2 1 T a b l e 7 : R e s p o n d e n t S a t i s f a c t i o n w i t h e a c h W o r k s h o p ( a ) N u m b e r o f r e g i s t r a n t s ( b ) N u m b e r o f s p e a k e r s , d i s c u s s a n t s a n d c h a i r - p e r s o n s E f f e c t i v e n e s s o f w o r k s h o p s W o r t h w h i l e u s e o f y o u r t i m e O v e r a l l e f f e c t i v e n e s s o f f o r u m N R e s p o n s e r a t e ( % ) ( c ) A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 G l o b a l i z a t i o n 9 6 2 5 3 . 7 7 6 8 4 . 1 3 8 4 3 . 9 4 8 1 3 1 3 2 E c o n o m i c D e v ’ t / P o v e r t y 9 6 3 4 3 . 5 5 6 4 3 . 8 2 6 4 3 . 6 4 6 4 1 1 1 1 G o v e r n a n c e 7 8 2 3 3 . 9 0 6 5 4 . 0 6 7 8 3 . 8 4 7 9 1 9 2 4 H u m a n D e v e l o p m e n t 8 9 3 5 4 . 0 0 7 9 4 . 2 8 9 2 4 . 0 0 7 8 2 4 2 7 P u b l i c - P r i v a t e P a r t n e r s h i p s 7 3 3 0 4 . 0 5 8 2 4 . 2 3 8 6 4 . 0 5 7 7 2 2 3 0 C o n f e r e n c e t o t a l 4 3 2 - 3 . 8 8 7 2 4 . 1 4 8 3 3 . 9 3 7 8 1 0 7 2 5 ( a ) R e l a t e s t o r e s p o n d e n t s w h o a r e M E N A r e g i o n r e s i d e n t s . R e s p o n d e n t s a r e t h o s e w h o h a d r e g i s t e r e d f o r a p a r t i c u l a r w o r k s h o p a t t h e c o n f e r e n c e a n d w h o c o m p l e t e d a q u e s t i o n n a i r e a t t h e e n d o f t h a t w o r k s h o p . T h e s h a d e d c e l l s r e l a t e t o t h e w o r k s h o p s w i t h t h e h i g h e s t r a t i n g i n e a c h d i r e c t i o n . ( b ) I n c l u d e s w o r k s h o p p a r t i c i p a n t s a n d a l s o s p e a k e r s , d i s c u s s a n t s a n d c h a i r p e r s o n s . ( c ) R e s p o n s e r a t e s s h o w n h e r e a r e u n d e r e s t i m a t e s o w i n g t o i n c l u s i o n o f n o n - M E N A r e s i d e n t s i n t h e d e n o m i n a t o r Page 31 Evaluation Report _____________________________________________________________________________ 22 More detailed respondent reactions to key features of each workshop are shown in Table 8. The five features listed in the table are ones which respondents in the 1997 conference highlighted as being important and requiring some attention in designing f uture MDF conferences. They include the extent to which the workshop: · focused on the issues the respondent most hoped would be addressed · allocated adequate time for constructive discussions · treated issues in sufficient depth · included suitable presenters and resource persons · provided useful reading and resource materials . An examination of the average workshop ratings (the second last row in Table 8) reveals that, overall, respondents judged the two most successful features of workshops to have been the suitability of presenters and resource persons (3.96) and the provision of reading/resource materials (3.71). The workshop focus on issues which respondents most hoped would be addressed received an average rating of 3.61. The two least successful features of workshops were whether issues had been treated in sufficient depth (3.08) and whether adequate time had been available for constructive discussions (2.92); these two workshop features are related to each other. It is also striking that for the latter two features, only very low proportions of workshop respondents ¾ 34% and 33% respectively ¾ rated workshops a 4 or a 5. An additional source of information is a regression analysis conducted to investigate the relationship between respondent satisfaction with workshops and the five key workshop features identified. This type of analysis can tell us about the extent to which respondent satisfaction is related to each of the workshop features ¾ i.e., it provides some insights into the importance of different activities as perceived by respondents. (a) i.e., respondent rating of the overall effectiveness of the workshops. The independent variables in this regression relate to the workshop for which the respondent had registered. t -statistics are in parentheses. ** indicates p-value £ .01; * p-value £ .05. The regression analysis suggests 13 that the two workshop features which were most important in influencing respondent satisfaction were whether the workshop had focused 13 There existed moderate collinearity between several of the explanatory variables in this analysis. As a result, the estimates of the relative importance of each variable (i.e., the relative magnitude of the coefficients) might not be accurate, and thus the results shown here should be viewed as tentative only. A further caveat is that there is some lack of alignment between the dependent variable (respondent rating of the overall effectiveness of the (conference) workshops and the explanatory variables (which related solely to the particular workshop for which the respondent had registered). However, the importance of this misalignment is unclear ¾ it could be argued that the statistical significance of the regression results suggest that the misalignment is not particularly important. Respondent = 1.633 + 0.198 Issues + 0.222 Time + 0.196 Depth + 0.155 Presenters + 0.006 Materials satisfaction (a) (4.56)** (2.08)* (2.66)** (1.92) (1.54) (0.06) Adjusted R 2 = .30 Page 32 T h e 1 9 9 8 C o n f e r e n c e o f t h e M e d i t e r r a n e a n D e v e l o p m e n t F o r u m _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2 2 T a b l e 8 : R e s p o n d e n t R e s p o n s e s t o K e y W o r k s h o p F e a t u r e s ( a ) T o w h a t e x t e n t d i d t h e w o r k s h o p : f o c u s o n t h e i s s u e s y o u m o s t h o p e d w o u l d b e a d d r e s s e d a l l o c a t e a d e q u a t e t i m e f o r c o n s t r u c t i v e d i s c u s s i o n s t r e a t i s s u e s i n s u f f i c i e n t d e p t h i n c l u d e s u i t a b l e p r e s e n t e r s a n d r e s o u r c e p e r s o n s p r o v i d e u s e f u l r e a d i n g a n d r e s o u r c e m a t e r i a l s O v e r a l l e f f e c t i v e n e s s o f w o r k s h o p s A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 A v e r a g e r a t i n g % w h o r a t e d 4 o r 5 G l o b a l i z a t i o n 3 . 5 8 5 5 2 . 5 0 1 6 2 . 8 3 2 0 3 . 6 3 5 6 3 . 6 5 6 1 3 . 7 7 6 8 E c o n o m i c D e v ’ t / P o v e r t y 3 . 1 8 4 6 2 . 8 2 2 7 2 . 7 3 1 8 3 . 8 2 6 4 3 . 5 5 5 5 3 . 5 5 6 4 G o v e r n a n c e 3 . 7 0 6 0 3 . 3 5 5 0 3 . 4 0 5 0 3 . 9 5 7 0 3 . 7 5 5 5 3 . 9 0 6 5 H u m a n D e v e l o p m e n t 3 . 5 0 5 5 2 . 8 8 2 9 2 . 9 2 2 5 4 . 2 2 9 1 3 . 9 2 7 2 4 . 0 0 7 9 P u b l i c - P r i v a t e P a r t n e r s h i p s 3 . 9 1 7 3 3 . 2 4 5 2 3 . 5 0 5 5 4 . 2 7 8 6 3 . 6 2 6 2 4 . 0 5 8 2 W o r k s h o p A V E R A G E 3 . 6 1 5 9 2 . 9 2 3 3 3 . 0 8 3 4 3 . 9 6 7 3 3 . 7 1 6 2 3 . 8 8 7 2 R a n g e i n r a t i n g s ( h i g h e s t v e r s u s l o w e s t w ’ k s h o p s ) 0 . 7 3 0 . 8 5 0 . 7 7 0 . 6 4 0 . 3 7 0 . 5 0 ( a ) R e l a t e s t o r e s p o n d e n t s w h o a r e M E N A r e g i o n r e s i d e n t s . R e s p o n d e n t s a r e t h o s e w h o h a d r e g i s t e r e d f o r a p a r t i c u l a r w o r k s h o p a t t h e c o n f e r e n c e a n d w h o c o m p l e t e d a q u e s t i o n n a i r e a t t h e e n d o f t h a t w o r k s h o p . T h e s h a d e d c e l l s r e l a t e t o t h e w o r k s h o p s w i t h t h e h i g h e s t r a t i n g a g a i n s t e a c h c r i t e r i o n . Page 33 Evaluation Report _____________________________________________________________________________ 24 on issues respondents had hoped would be addressed, and the availability of sufficient time for discussions ¾ however, the latter was the least successful feature of the workshops. In a nutshell, the workshops overall appear to have been reasonably successful in including good presenters, in having good resource materials available, and in covering the most pertinent issues. Where the workshops fell down substantially in their performance was in failing to treat issues in sufficient depth and to allow sufficient time for constructive discussions. Unfortunately, this conclusion echoes one from the evaluation of the 1997 conference, quoted earlier: ‘It is hard to escape the conclusion that some task managers tended to cram more and more speakers into whatever time was available’. This lesson from the 1997 conference was evidently not sufficiently heeded. This failure is particularly important because a focus of the conference was on increasing participants’ depth of understanding of issues ¾ as noted earlier, workshops at the conference were intended to encourage debate and discussion among participants and speakers, rather than a traditional, ‘talking heads’ lecture style of presentation. Table 7 also shows the number of speakers, discussants and chairpersons in each workshop. The number of these presenters ranged from 23 to 35 ¾ all of these presenters were accommodated within the 11 hours and 15 minutes total time available for each workshop. There is no clear inverse correlation between the number of these persons and levels of respondent satisfaction with the adequacy of time for constructive discussions (Table 8). But it seems surprising that so many speakers, discussants and chairpersons were crammed into the two days available for each workshop ¾ especially as a number of plenary sessions were also included in the schedules for the two days. One workshop task manager has suggested that participants could be given more time to discuss issues, to participate more, and to ensure more of a policy focus, by a complete redesign of workshop approach. He suggested a workshop format involving a formal presentation and panel discussion at the start of the day, with the remainder of the day devoted to participant discussion and debate. Alternative workshop formats of this type merit serious consideration by conference task managers, by partners and by EDI management. The average figures discussed to this point mask considerable variation between the five workshops (last row in Table 8). The workshop with the highest respondent ratings for three of the five features was the one on Public-Private Partnerships. That workshop also had the highest-rated feature of any workshop ¾ a rating of 4.27 for the suitability of the presenters and resource persons (and with 86% of respondents giving the workshop a 4 or 5 rating for this feature). However, even that workshop had only low percentages of respondents giving a 4 or 5 rating for the two workshop features which were weak in ‘Because of the large number of presenters (as many as five in some cases) and the limited time available, exchanges and debates were sometimes too short, a situation that is highly detrimental to the workshop format’ (participant feedback from a government Minister from Morocco) Page 34 Evaluation Report _____________________________________________________________________________ 25 every workshop ¾ the treatment of issues in depth (55% of respondents) and the adequacy of time for constructive discussions (52%). The lowest ratings for individual workshop features were received by the workshops on Globalization and on Economic Development/Poverty Reduction. The former workshop had the lowest rating for any workshop feature ¾ 2.50 for the adequacy of time for constructive discussions (with only 16% of respondents giving a rating of 4 or 5 for this feature). B. Gains in Understanding The conference and its component workshops were intende d to increase participants’ understanding of key topics of importance to the MENA region. Each workshop task manager prepared an outline of the topics and issues to be addressed in the workshop ¾ these outlines were available to potential participants to help them decide whether to attend or not. The evaluation questionnaires (Annex A) included a set of questions which asked participants to self-assess their depth of understanding of the four or five main topics of the workshop which they attended. For each topic, participants were asked at the conclusion of the workshop to assess what their level of understanding of each topic was at the start of the workshop and at the end, using a scale of 1 (very low) to 5 (very high). Table 9 shows the results of these self-assessments averaged over the main topics for each workshop. The detailed results for each topic are shown in Annex B, Table B1. One aspect of these assessments relevant in interpreting the responses is that they are self-assessments , and may or may not accurately measure individuals’ actual depth of understanding. Evaluations of the extent of learning from some World Bank courses have found that self-assessments can vary considerably from independent assessments, measured via cognitive-based testing for example. 14 There were large differences between the workshops in the extent of increase of respondents’ depth of understanding of the main topics covered ¾ the largest increase (37%) was reported by participants in the workshop on Public-Private Partnerships; the smallest increase (15%) was reported by respondents in the Economic Development and Poverty workshop. The average increase over the five workshops was 25%. Bearing in mind the caveat regarding self-assessments, a 25% increase is substantial. 15 14 The focus in the present study is depth of understanding , rather than learning per se. For an example of the differences between self-assessments and cognitive-based tests of learning, see EDI (1998). Core Course on Social Security and Pension Reform ¾ Exemplary Evaluation. World Bank, Washington D.C. 15 Respondents at the MDF2 conference who assessed themselves as having a low level of understanding before the workshop tended to have the greatest increases in their depth of understanding by the end of the workshop. Page 35 Evaluation Report _____________________________________________________________________________ 26 Table 9: Average Levels of Respondents’ Depth of Understanding, by Workshop (a) Workshop Understanding before workshop Understanding after workshop % Change Average rating of effectiveness of workshops Globalization 3.18 3.81 20 3.77 Economic Dev’t/Poverty 3.47 4.00 15 3.55 Governance 2.98 3.92 32 3.90 Human Development 2.83 3.54 25 4.00 Public-Private Partnerships 2.93 4.01 37 4.05 AVERAGE for all workshops 3.08 3.86 25 3.88 (a) Relates to respondents who are MENA region residents. Respondents are those who had registered for a particular workshop at the conference and who completed a questionnaire at the end of that workshop . The shaded cell shows the workshop with the highest percentage change. Unsurprisingly, there was a significant and high correlation (0.42) between respondents’ satisfaction with the workshop attended and their self-assessment of their increase in understanding of workshop topics. C. Workshop Synergies There are several possible synergies which can arise from a conference such as the 1998 MDF. The first arises from the opportunity to include conference sessions and prominent speakers on cross-cutting or high-profile issues of interest to the MENA region as a whole, such as the Asian financial crisis or the role of civil society in development. A large conference thus offers possibilities for economies of scale which would not be available to an individual, stand-alone workshop. Related to this is that the size and breadth of a large conference can provide additional credibility to each individual workshop. A second possible synergy is the opportunity for workshops on related topics to share speakers and even participants. Having related workshops at the same conference may also provide economies of scope by fostering debate and networking in relation to common or complementary issues. When interviewed, task managers were asked about possible synergies from having the five workshops at the same conference; and the proposition was put to them that it might have been just as effective to have had five separate, stand-alone workshops. The conference task manager and the five workshop task managers all argued that the effectiveness of the conference as a whole was greater than the sum of the parts ¾ although there were different views as to why this was the case. One workshop task manager stressed that the five workshops were designed to fit together in a complementary manner within the overall conference theme of Participation and Development . The conference task manager and two workshop task managers believed that the focus of the plenary sessions on cross-cutting issues was of Page 36 Evaluation Report _____________________________________________________________________________ 27 considerable interest and excitement to participants; one task manager argued that ‘plenary sessions provided the glue which held the workshops together’. T wo task managers noted synergies because participants were able to migrate between workshops ¾ this was particularly noticeable on the last afternoon of the workshops, when the microcredit session of the workshop on Economic Development and Poverty Reduction was crowded with participants from other workshops who were keenly interested in that session. Two task managers also appreciated the sharing between workshops of speakers and chairpersons ¾ such sharing appears to have occurred to only a minor extent, but the ability to share was regarded by these task managers as providing a safety net in cases of last-minute unavailability of speakers or chairpersons. One of the task managers considered that some of these speakers and chairpersons, including government ministers and former ministers, only attended the workshop because of the overall size and importance of the conference as a whole. The other task manager argued that the plenary sessions provided a ‘speaker bank’ from which workshop speakers could be chosen. Two other task managers did not draw on plenary speakers for their workshops, but acknowledged the potential for doing so. Page 37 Evaluation Report _____________________________________________________________________________ 28 Page 38 Evaluation Report _____________________________________________________________________________ 29 5: CONFERENCE PLANNING AND ADMINISTRATION The planning, development and administration of a major conference such as the 1998 M DF is a complex and difficult undertaking. A comprehensive review of these organizational aspects of the conference is outside the scope of this evaluation. Instead, the issues identified here are those raised by task managers and respondents as being highly relevant for the future organization of large conferences. Because relatively few respondents commented on conference planning and administration, particular attention is paid to what task managers noted as having worked well or not well, and to suggestions for improvement in the future. However, a number of useful respondent suggestions for the improvement of future MDF conferences are also noted. A. Task Manager Views (1) Partner collaboration An important feature of the conference was the participation of MDF partner organizations from the MENA region. These regional think-tanks played a key role, together with the Economic Development Institute of the World Bank, in planning for the conference. Several planning meetings were held between the first MDF conference in May 1997 and the second conference in September 1998. These meetings decided the overall theme for the second conference, as well as the number of workshops and the sub-themes for the five workshops chosen. There are ten MDF partner organizations from the MENA region. Two are international and their work covers most or all of the countries in the region. The remaining eight are national organizations based in the following countries/areas: Egypt, Jordan, Lebanon, Morocco, Palestine, Syria, Tunisia and Turkey. Only five of the ten MENA partner institutions were in existence five years ago, and the MDF itself only came into existence at the May 1997 conference. An objective of the MDF is the development and encouragement of regional organizations ¾ indeed, it emphasizes an approach of ‘participatory capacity-building’, and this objective has been pursued in MDF’s planning and administration of its conferences. Partner organizations were invited to collaborate closely with the World Bank in the planning and administration of each of the five workshops at the 1998 conference. Between one and four regional partner organizations agreed to work closely with the World Bank in the organization of individual workshops. The intended division of labor between the regional partners and the World Bank was that there would be joint responsibility for selecting workshop themes, speakers and chairpersons. The regional partners, with their greater knowledge of MENA countries, would also be responsible for identifying potential participants who would be invited to attend the conference. The World Bank would also be responsible for logistics and administrative tasks such as booking of the conference venue, travel arrangements, and so on ¾ it was felt that there would be economies of scale in logistics/administration, and that the World Bank with its experience in organizing major conferences would be well placed to achieve these economies. The World Bank’s task managers viewed the contribution of the partner organizations in organizing the workshops to have been varied. They acknowledged the valuable contributions which several partners made to planning workshop themes and agendas, Page 39 Evaluation Report _____________________________________________________________________________ 30 to workshop preparation via rehearsals conducted prior to the workshops, and to the selection of participants. However, the majority of task managers were highly dissatisfied with the performance of their regional partners, and felt that the World Bank h ad been left to shoulder an unfair share of the administrative burden. Only one of the five workshop task managers was highly satisfied with the performance of his regional partner. A second was reasonably satisfied, and valued in particular the work of the partner in planning for the workshop at an earlier partners’ meeting, in selecting workshop speakers and in a workshop rehearsal. The remaining three workshop task managers were highly dissatisfied and considered that their partner organizations had contributed little if anything to the workshops. As a result, it was left to these task managers to assume responsibility for all of the planning of their workshops, and this meant that much of the workshop preparation was done late and under considerable pressure. In several cases, the partner representatives did not even attend the workshop for which they had agreed to be jointly responsible. There appear to have been several reasons for this failure. In the case of one workshop, the task manager believes that the partner organization was unaware of the expectations on it. However, repeated requests to the partner for assistance proved to be fruitless. In the case of the two other workshops, the task managers are firm in their belief that expectations had been made clear to the partners on several occasions, and that the partners had agreed to make substantive contributions to the workshops. One task manager attributed the partner’s failure to deliver to a lack of management oversight within the partner organization ¾ the top management of the organization appeared to be fully committed to the workshop, but the staff to whom the task had been delegated had not followed through on this commitment. There are a number of lessons from this experience to help guide future collaboration with partner organizations in planning future conferences : · there should be a clear and agreed understanding of the respective roles, tasks and expectations of partner organizations including the World Bank; · those partner organizations wishing to be selected as joint organizers of workshops need to be able to demonstrate some substantive experience in organizing similar events in the past; · it might be desirable to select only one partner organization to work with the World Bank on each workshop, in order to simplify communication and to clarify expectations; · a need for top management within selected partners to convey their expectations clearly to those staff within the partner organization who are tasked to collaborate in workshop planning and preparation, and to adequately supervise their staff; · there is a similar need to ensure that workshop task managers in the World Bank, and their supervisors, are fully aware of and committed to the expectations on them; · staff tasked within partner organizations should be sufficiently experienced and should be allocated sufficient time to undertake the agreed tasks; and · progress reports on workshop planning and preparation should be sent regularly both to the conference task manager and to the top management of partner organizations. Any emerging problems should be identified, communicated and addressed early. The conference and workshop task managers recognize and value the benefits from close collaboration with partners. Notwithstanding the problems encountered with some Page 40 Evaluation Report _____________________________________________________________________________ 31 aspects of the 1998 conference they consider that the close participation of partners has a lot to offer, both in organizing specific events such as conferences, and also as a means to foster and support the further development of partner organizations’ own c apabilities. For the more immediate future, there remains a clear role for the World Bank to play in helping to coordinate MDF conferences and in acting as the secretariat to help organize them. (2) Other planning and administration issues The conference and workshop task managers nominated a number of good practice features of the workshops which would be worth adopting in other conferences. These included: · having rehearsals of all workshops immediately before the conference. One task manager also organized a rehearsal of part of the workshop two months before the conference; of the eight papers presented at the rehearsal, four were selected for inclusion in the conference. Rehearsals provided valuable feedback to speakers on the content, delivery and length of their presentations, and allowed clarification and agreement among speakers concerning what each would or would not cover in their presentations. They also helped to build the workshop presenters into a team and encouraged subsequent networking; · the close involvement of the World Bank’s operational area for the MENA region, which provided one of the workshop task managers and was involved in conference planning and in the provision of plenary and workshop speakers; · preparation of a detailed (paragraph-long) description of the topics to be covered by each session of a workshop. This ensured that speakers had a common understanding and ownership of the topics they would cover, and also helped to lock in speakers to adhering to these topics. The descriptions also clarified to participants what they could expect from each session, and assisted in any preparation they might wish to undertake; · one task manager arranged a number of papers to be prepared on different countries, and asked each speaker to prepare a paper on their country using a standard set of terms of reference ¾ this ensured consistency in coverage of issues and comparability of the experience of different countries. This approach was also highly appreciated by workshop participants. Finally, the conference task manager and all workshop task managers noted the problems encountered by participants and speakers in traveling to Marrakech, and difficulties with the conference facilities themselves. These problems were also identified by respondents and speakers. For example, one of the speakers at the conference had traveled from Egypt, but experienced delays and long travel times because of difficulties in airline connections, late finalization of bookings by the conference travel agent and airline over-booking. These problems led her to conclude, ironically, that ‘Washington is closer (to Egypt) than is Marrakech’. The first two MDF conferences have been held in Marrakech, and it would seem appropriate to hold the next MDF conference at an alternative location. B. Some Participant Suggestions Page 41 Evaluation Report _____________________________________________________________________________ 32 · opt for less quantity of sessions and more quality speakers. Three sessions per day with quality speakers can be much more effective. Limit papers to two per session followed by deeper discussions (government ministry official, Lebanon) · decrease the number of workshops; cover more topics in plenary meetings (private sector participant, Morocco) · would have been more useful for the conference to have focused on fewer issues and give them more in depth treatment (university/research institute participant, Turkey) · schedule fewer sessions and allow more time for each one (private sector representative, Morocco) · spread the workshops over a longer period of time; hold an annual meeting to summarize the results of the conference (media representative, Morocco) · would have been useful to have had a brief statement (1-2 pages) of the findings/recommendations of each workshop discussed at a final plenary session (donor participant, Italy) · prepare a report on the first and second MDF conferences, summarizing the papers and topics; ensure its wide dissemination (private sector participant, Tunisia) · publish summaries of conference presentations (Minister/Parliamentarian, France) · as part of the conference structure arrange contacts and meetings among institutional representatives to encourage and support the formation of networks (university/research institute participant, Turkey · give more attention to the issue of participation and civil society ¾ e.g., via a workshop on the legal framework of NGOs, etc (NGO representative, Jordan) · at the next conference, have a greater focus on issues relating to GCC countries - their problems and issues are very different from those of other areas in the MENA region (government ministry official, Oman) · to ensure the declarations of support for women’s empowerment are treated fully, organize a special day or workshop on this topic (NGO representative, Belgium) Page 42 Evaluation Report _____________________________________________________________________________ 33 C. Cost of the Conference The direct cost of the conference, excluding in-kind costs such as staff time of Bank and partner organizations, and of donations of services, was $878,000. This is equivalent to $3,240 per workshop participant, or $1705 per person who attended the conference ¾ workshop participants, speakers, media and others. 16 The Bank’s share of this amount was around $550,000, or 63% of the total. Trust funds, including from Japan and Sweden, contributed $150,000, and the remainder (around $50,000) came from the private sector. 16 A comparison with other conferences would provide useful information on relative cost- effectiveness. However, this would require a detailed analysis of costs, and has not been conducted for this evaluation. · expand the pool of MDF partners to include additional key players from the business community (private sector representative, Lebanon) · organize a genuine dialogue between NGOs and national think-tanks to ensure the preparatory process for the conference is truly a shared and coordinated effort (university/research institute participant, Morocco) · send invitation letters to participants in previous MDF conferences; ask prospective participants to indicate their interests, so that meetings of interest groups can be arranged at the conference (university/research institute participant, Morocco) · the main objective (post-conference) should be the action and the follow- up in participating countries (government ministry official, Morocco) · establish a MENA region network to exist in-between the MDF conferences (government ministry official, Lebanon) Page 43 Evaluation Report _____________________________________________________________________________ 34 Page 44 Evaluation Report _____________________________________________________________________________ 35 6: CONCLUSIONS The 1998 MDF conference was a success. It appears to have achieved a number of its o bjectives, and it succeeded in attracting a large number of participants from key target groups in the MENA region, including Ministers and senior officials, representatives of NGOs, and from academia and regional think-tanks. A number of private sector representatives also attended the conference, although fewer than the conference organizers had hoped. The cost of the conference was $878,000, or about $3,240 per participant. The conference had three main objectives ¾ to contribute to MENA policy debate; to empower civil society and the private sector to influence public policy; and to help participants to develop networks with each other. While it is difficult to evaluate the extent to which these objectives were fully met, this evaluation has found that participants ¾ respondents ¾ rated the conference highly for achieving its objective of contributing to MENA policy debate, and also the objective of helping participants develop networks. However, respondents also felt that the impact of the conference was undercut to some extent by the failure of the workshops to treat issues in sufficient depth and to allow sufficient time for constructive discussions. Also, respondents did not rate the conference highly in terms of achievement of the objective of empowering civil society and the private sector. The Bank’s conference and workshop task managers were generally supportive of the extent to which the conference had met its objectives. They pointed to a number of ongoing activities in which the Bank and its MENA partners are involved. These activities reveal a substantive contribution to, and development of, MENA policy debates. Task managers were also able to point to a number of substantive activities which will increase the involvement of civil society and the Bank’s MENA think-tank partners in policy debates in the region; in pursuing this objective the conference was successful in involving a large number of eminent persons from the region as speakers at the conference. The majority of task managers also volunteered the view that the conference had encouraged the development of networks in MENA. Thus a recurrent theme expressed by task managers was that the conference activities and initiatives should be viewed as part of an ongoing process of regional involvement and dialogue ¾ one which includes the regional partners and the Bank as key players. In other words, the conference should not be viewed as a stand-alone activity but as part of an ongoing collaborative process. In assessing the effectiveness of the conference it is also useful to focus on the extent of participant and task manager satisfaction with various aspects of the conference, such as the workshops and the plenary sessions ¾ the conference is unlikely to have achieved its objectives if participant satisfaction levels were low. Similarly, it is important to find out if participants considered they had increased their understanding of topics and regional policy issues addressed at the conference. The evaluation finds that respondents expressed high levels of satisfaction with the conference overall. Levels of respondent satisfaction with the 1998 MDF conference were higher than for other major conferences ¾ including the 1997 MDF conference ¾ which the World Bank has organized on its own or with others. These higher levels of satisfaction reflect a number of lessons learned from previous conferences, such as a less congested conference agenda, a smaller number and Page 45 Evaluation Report _____________________________________________________________________________ 36 shorter duration of workshops, and the need for targeted invitations to potential participants. Respondent feedback also affirmed the appropriateness of the topics and themes selected for detailed coverage in the five workshops held at the conference. R espondents reported that the workshops helped them to increase substantially their depth of understanding of key issues and themes which were addressed. Nevertheless, there is still room for improvement, and this is indicated by two observations. The first is that there existed considerable variation in respondent satisfaction between the different workshops ¾ this suggests there are good practice features in workshop design and participant selection which could and should be adopted more widely. Good practice features of the 1998 MDF conference included: · workshop rehearsals before the conference; · the close involvement of the World Bank’s operational area; and · preparation of detailed descriptions of the topics to be covered by each workshop session. The second observation is that respondents were dissatisfied with the lack of time in the workshops for constructive discussion and debate. Workshop agendas were crammed with an excessive number of speakers, panelists and chairpersons ¾ a case of ‘speaker congestion’. This is not a new phenomenon, and was a feature of the 1997 MDF conference. Yet despite its existence having been highlighted in the evaluation of that earlier conference, and despite its existence in a large number of World Bank workshops and conferences, the lesson was evidently not heeded in the planning and design of the 1998 conference. The World Bank  and especially its external training arm, the Economic Development Institute  needs to hear this message and act on it. In designing a conference there is a number of possible tradeoffs in the balance between overall conference length, number of workshops and plenary events, number and duration of workshops, number and type of participants per workshop, and so on. Achieving an optimal mix of such choices is never easy; however, the additional experience gained from the 1998 MDF conference can aid in designing future conferences if its lessons are heeded. These tradeoffs also help to determine the overall cost-effectiveness of such conferences. The cost of the 1998 MDF conference was $878,000, or about $3,240 per workshop participant. The Bank’s conference and workshop task managers are highly supportive of the key role of the MENA partner organizations in developing and supporting the agenda and ongoing development of the Mediterranean Development Forum (MDF). Part of this role is their support for the MDF conferences ¾ the two held so far, and also future conferences. Their support has taken many forms, including advocacy, support and encouragement for the proposal to have MENA region conferences in the first place, advice on the key policy issues in MENA and their inclusion in the MDF conferences, identification of potential speakers for the conferences and workshops, identification of potential participants at the conferences, and assistance in organizing the conference workshops. A number of the MDF partner organizations had agreed to jointly organize the five workshops at the 1998 conference, but the performance of a number of the partners was Page 46 Evaluation Report _____________________________________________________________________________ 37 disappointing. Reasons for this are not entirely clear but might include lack of clarity on the part of some partner organizations about the expectations on them, and a possible lack of management oversight of staff within some partner organizations. This report c ontains a number of recommendations to help improve future collaboration with partner organizations in planning future conferences. Despite these administrative problems, the success of the 1998 conference is heartening. It underscores the success of the MDF partnership and provides another platform on which future collaboration, policy dialogue and development efforts in the MENA region can build. Page 47 Evaluation Report _____________________________________________________________________________ 38 Page 48 39 ANNEX A: EVALUATION APPROACH The approach adopted in this evaluation is outlined in Section 2 of the report. This Annex contains further discussion of particular issues, and also contains the evaluation design matrix ¾ this shows the evaluation questions asked, information required and information sources, design strategy, data collection and analysis methods, limitations, and what the analysis allows us to say. Evaluation Questionnaire The questionnaire given to workshop participants is reproduced at the end of this annex. The main focus of the questionnaire is on respondent satisfaction with a range of features of the conference and with the particular workshop for which the respondent had registered. Participants were also asked to list features of the conference which had contributed to or detracted from its effectiveness, and were asked for any other comments or suggestions for improvement of future MDF conferences. The response rate was low (25%) when calculated as a percentage of conference registrants (from the MENA region). However, the evaluation questionnaires were distributed to workshop participants during the last afternoon of the workshop sessions, a time when a number of participants had evidently chosen either not to attend any workshop or had decided to attend a workshop different to the one for which they had registered. The latter was particularly evident for the workshop on Economic Development and Poverty Reduction, for example, which attracted as many as 100 persons on the final afternoon; the workshop task manager judged that many of these persons had not registered for that workshop, and evaluation questionnaires were not given to most of these persons. The effective response rate for each workshop ¾ i.e., the proportion of workshop registrants who were given an evaluation questionnaire and who completed a questionnaire ¾ is not known with any precision, but was estimated by the individual rapporteurs for each workshop to have been well over 50%; one rapporteur estimated almost 100% for her workshop. Future conference evaluations should ensure that a careful count is made of the number of registrants attending the session at which evaluation questionnaires are completed, so that an accurate estimate of response rates can be made. A related issue which merits investigation is the reasons for the fall-off in participant numbers towards the end of conferences ¾ this phenomenon has been observed for other major conferences in which the Bank has been involved. One possible explanation is that the fall-off reflects participant fatigue after a lengthy or congested agenda (there is evidence to support this possibility for both the 1998 and 1887 MDF conferences). Another possibility is that some participants were dissatisfied with the workshops or the conference as a whole and ‘voted with their feet’. Either of these explanations ¾ and there are no doubt others ¾ would suggest that questionnaire respondents were unrepresentative of all conference registrants, and thus that the questionnaire results reported in this evaluation (and other conference evaluations) are not representative of the views of all conference registrants. Also included in the questionnaire is a question on participants’ depth of understanding of each of the four or five key themes or topics covered in the particular workshop attended (a list of the main topics covered in each workshop, and of respondents’ detailed self-assessments, is shown in Table B.1 in Annex B). Participants were given the questionnaire during the last session of the workshop they attended and were asked to rate their own understanding of the workshop themes ¾ they were asked to rate their understanding both before and after the workshop. This approach was chosen because it was felt that participants would only be in a good position to Page 49 40 a ssess their pre-existing level of understanding after t hey had been exposed to the workshop themes. 17 Regression Analysis The regression analyses which are referred to earlier in this report were based on the ratings g iven by respondents against a range of different aspects of the conference. For example, participants were asked to rate the effectiveness of the workshops ( Question 6a in the questionnaire below) and this was correlated against other respondent ratings, such as the extent to which the workshop had treated issues in sufficient depth ( Question 4a ). Correlation matrices corresponding to the equations reported in Sections 3.C and 4.A are shown below. Moderate collinearity exists between some of the explanatory variables (nos 2-6) shown in Table A.2 ¾ see footnote 13 above. Participants were asked to make their ratings on the basis of a five-point Likert type ratings scale ¾ with a scale of 1 (minimum) to 5 (maximum). This meant that the independent variables used in the regression had little variation compared to a situation where the scale ranged, say, from 1 to 100. The adjusted R 2 of 0.30 also indicates the presence of other factors, not included in the model, which help to explain participant satisfaction. Multiple regression analysis assumes interval (continuous) data on the independent (explanatory) and dependent variables. Strictly speaking, the use of Likert type data is contrary to this assumption: thus a satisfaction rating of ‘4’ does not imply that the respondent is four times as satisfied as s/he would have been if s/he had given a rating of ‘1’. The distortions caused by the use of Likert type data are greater, the more the data are skewed towards one side of the measurement scale. Some of the data used in this evaluation do display some moderate skewing (towards the upper end of the scale). Nevertheless, this does not mean that the regression analyses in this evaluation are invalid ¾ it simply means that the findings from them should best be regarded as tentative. The qualitative (open-ended) responses provided by respondents ¾ concerning the features of the conference which they did and did not like ¾ are consistent with the regression findings. These two types of finding thus support each other and provide greater confidence in the evaluation’s findings. 17 Participants typically have some pre-existing knowledge of workshop issues, but can find it difficult to gauge the extent of their understanding if they have not been recently exposed to the issues. Page 50 41 T able A.1: Correlation Matrix for Conference Structure Variable 1 2 3 4 5 1 How would you rate the overall effectiveness of the forum? 1.000 .547** .271** .081 .185* 2 How would you rate the effectiveness of the … workshops? .547** 1.000 .061 -.009 .186* 3 How would you rate the effectiveness of the … plenary panels and speakers? .271** .061 1.000 .264* -.015 4 How would you rate the effectiveness of the … cyber café? .081 -.009 .264* 1.000 .237* 5 How would you rate the effectiveness of the … interaction during free time (including meals, breaks, evenings, etc)? .185* .186* -.015 .237* 1.000 ** indicates p-value £ .01; *p-value £ .05. Table A.2: Correlation Matrix for Workshop Features Variable 1 2 3 4 5 6 1 How would you rate the effectiveness of the … workshops? 1.000 .442** .413** .487** .421** .245** 2 To what extent did the workshop focus on issues you most hoped would be addressed? .442** 1.000 .252** .573** .508** .261** 3 To what extent did the workshop allocate adequate time for constructive discussions? .413** .252** 1.000 .413** .267** .184* 4 To what extent did the workshop treat issues in sufficient depth? .487** .573** .413** 1.000 .553** .350** 5 To what extent did the workshop include suitable presenters and resource persons? .421** .508** .267** .553** 1.000 .424** 6 To what extent did the workshop provide useful reading and resource materials? .245** .261** .184* .350** .424** 1.000 **indicates p-value £ .01; *p-value £ .05. Task Manager Interviews As noted in Section 2, the five workshop task managers were formally interviewed in Washington D.C.; each interview lasted at least one hour. The list of interview questions is as follows: · how satisfied were you with the collaboration of MDF partners in the organization of your workshop? · how did you decide the content/focus of the workshop - was it possible to do a needs analysis beforehand? · how satisfied were you with the planning, development and administration of the workshop, including the choice of speakers? · how happy were you with the number and caliber of participants at the workshop? · what features of the workshop do you consider to be best practice and would like to share with others? · what other features of the workshop worked well? · what did not? Page 51 42 · to what extent do you think it likely the conference will meet its 3 objectives: (a) to develop networks among participants? (b) to contribute substantially to the MENA policy debate on key issues? ( c) to empower civil society and the private sector to influence policy? · were there any synergies in having the 5 workshops at the same event?; conversely, would 5 separate, stand-alone workshops have been just as effective? · with the benefit of hindsight, was the conference a worthwhile activity?; was it valuable for you - if so, in what ways? · do you have any final suggestions about how future MDF conferences might be improved? · was the evaluation report for MDF1 of any use to you in planning for MDF2? Page 52 4 3 E V A L U A T I O N D E S I G N M A T R I X E v a l u a t i o n q u e s t i o n I n f o r m a t i o n r e q u i r e d I n f o r m a t i o n s o u r c e s D e s i g n s t r a t e g y D a t a c o l l e c t i o n m e t h o d s D a t a a n a l y s i s m e t h o d s L i m i t a t i o n s W h a t t h e a n a l y s i s a l l o w s u s t o s a y 1 . W e r e p a r t i c i p a n t s s a t i s f i e d w i t h t h e c o n f e r e n c e ? : ( a ) i t s o v e r a l l w o r t h a n d e f f e c t i v e n e s s ? ( b ) e f f e c t i v e n e s s o f d i f f e r e n t t y p e s o f c o n f e r e n c e a c t i v i t i e s ? ( c ) e x t e n t t o w h i c h t h e f i v e w o r k s h o p s m e t t h e i r o b j e c t i v e s ? ( d ) s a t i s f a c t i o n w i t h k e y f e a t u r e s o f w o r k s h o p s ( e . g . p r e s e n t e r s , d i s c u s s i o n s , m a t e r i a l s ) p a r t i c i p a n t r e a c t i o n s ( i ) c o n f e r e n c e q u e s t i o n n a i r e ; ( i i ) r e s p o n d e n t c o m m e n t s ( i ) s u r v e y o f a l l p a r t i c i p a n t s ; ( i i ) i n f o r m a l s a m p l e o f p a r t i c i p a n t s ( i ) q u e s t i o n - n a i r e ; ( i i ) i n f o r m a l i n t e r v i e w s ( i ) c a l c u l a t i o n o f r e s p o n d e n t r a t i n g s ; ( i i ) c o n t e n t a n a l y s i s ; ( i i i ) r e g r e s s i o n a n a l y s i s ( i ) p o s s i b l e r e s p o n s e r a t e b i a s ; ( i i ) p o s s i b l e c u l t u r a l f a c t o r s r e l a c k o f w i l l i n g n e s s t o c o m p l a i n e x t e n t o f r e s p o n d e n t s a t i s f a c t i o n w i t h c o n f e r e n c e o v e r a l l , a n d w i t h d i f f e r e n t t y p e s o f c o n f e r e n c e a c t i v i t y , a n d w i t h t h e w o r k s h o p s 2 . R e p r e s e n t - a t i v e n e s s o f p a r t i c i p a n t s ? ( i ) m i x o f d i f f e r e n t t y p e s o f p a r t i c i p a n t – e . g . , g o v e r n m e n t , N G O , a c a d e m i a ( i i ) g e n d e r a n d c o u n t r y s p l i t s p a r t i c i p a n t r e g i s t r a t i o n d a t a c o n s t r u c t d a t a b a s e c o n f e r e n c e d a t a b a s e i n t e r r o g a t i o n d e s c r i p t i v e a n a l y s i s d a t a b a s e e n t r y a n d e x t r a c t i o n e r r o r s q u a l i t a t i v e a s s e s s m e n t o f p a r t i c i p a n t ‘ b a l a n c e ’ ; c o m p a r i s o n s o f p a r t i c i p a n t p r o f i l e s w i t h o t h e r c o n f e r e n c e s 3 . G r o w t h i n p a r t i c i p a n t s ’ m e a s u r e m e n t o f p a r t i c i p a n t p a r t i c i p a n t s e l f - a s s e s s m e n t s s u r v e y o f a l l p a r t i c i p a n t s p r e / p o s t s e l f - a s s e s s m e n t c a l c u l a t i o n o f c h a n g e s i n p o s s i b l e r e s p o n s e r a t e e x t e n t t o w h i c h r e s p o n d e n t s j u d g e Page 53 4 4 E v a l u a t i o n q u e s t i o n I n f o r m a t i o n r e q u i r e d I n f o r m a t i o n s o u r c e s D e s i g n s t r a t e g y D a t a c o l l e c t i o n m e t h o d s D a t a a n a l y s i s m e t h o d s L i m i t a t i o n s W h a t t h e a n a l y s i s a l l o w s u s t o s a y u n d e r s t a n d i n g o f t h e k e y i s s u e s a d d r e s s e d a t t h e c o n f e r e n c e u n d e r s t a n d i n g q u e s t i o n n a i r e ( c o m p l e t e d e x p o s t ) r e s p o n d e n t s e l f - r a t i n g s p e r w o r k s h o p b i a s ; s e l f - r e p o r t i n g - - l a c k o f i n d e p e n d e n t o r o b j e c t i v e a s s e s s m e n t t h e y h a v e i n c r e a s e d t h e i r u n d e r s t a n d i n g o f k e y w o r k s h o p i s s u e s 4 . E x t e n t t o w h i c h c o n f e r e n c e m e t i t s o b j e c t i v e s ? : ( a ) d i d t h e c o n f e r e n c e f o s t e r n e t w o r k i n g ? ( b ) i m p a c t o f c o n f e r e n c e o n M E N A p o l i c y d e b a t e s ? ( c ) e m p o w e r - m e n t o f c i v i l s o c i e t y a n d p r i v a t e s e c t o r t o i n f l u e n c e p u b l i c p o l i c y ? j u d g m e n t s o f k e y s t a k e h o l d e r s ( i ) c o n f e r e n c e q u e s t i o n n a i r e ; ( i i ) p a r t i c i p a n t c o m m e n t s ; ( i i i ) c o n f e r e n c e a n d w o r k s h o p t a s k m a n a g e r c o m m e n t s ( i ) s u r v e y o f a l l p a r t i c i p a n t s ; ( i i ) i n f o r m a l s a m p l e o f p a r t i c i p a n t s ; ( i i i ) s u r v e y o f a l l t a s k m a n a g e r s ; ( i v ) t r a c e r s t u d i e s ( i ) p a r t i c i p a n t q u e s t i o n n a i r e ; ( i i ) i n f o r m a l i n t e r v i e w s o f p a r t i c i p a n t s a n d t a s k m a n a g e r s a t c o n f e r e n c e ; ( i i i ) s t r u c t u r e d i n t e r v i e w s o f t h e 2 c o n f e r - e n c e t a s k m a n a g e r s a n d 5 l e a d w ’ s h o p t a s k m a n a g e r s l a t e r ; ( i v ) t r a c e r s t u d i e s o f p a r t i c i p a n t s a n d t a s k m a n a g e r s 3 / 9 m o n t h s a f t e r c o n f e r e n c e ( y e t t o b e a g r e e d w i t h c o n f e r e n c e T M s ) ( i ) c a l c u l a t i o n o f r e s p o n d e n t r a t i n g s ; ( i i ) c o n t e n t a n a l y s i s s u b j e c t i v i t y o f c o m m e n t s ; c o m m e n t s a t c o n f e r e n c e t o o e a r l y t o r e v e a l l o n g - t e r m e f f e c t s r e s p o n d e n t a n d t a s k m a n a g e r j u d g m e n t s c o n c e r n i n g e x t e n t o f s u c c e s s i n a c h i e v i n g c o n f e r e n c e o b j e c t i v e s 5 . H o w c a n s u c h c o n f e r e n c e s b e i m p r o v e d i n f u t u r e ? ( i ) p a r t i c i p a n t s u g g e s t i o n s ; ( i i ) c o n f e r e n c e a n d w o r k s h o p t a s k ( i ) c o n f e r e n c e q u e s t i o n n a i r e ; ( i i ) p a r t i c i p a n t c o m m e n t s ; s u r v e y o f a l l p a r t i c i p a n t s a n d t a s k m a n a g e r s ( i ) p a r t i c i p a n t q u e s t i o n n a i r e ; ( i i ) i n f o r m a l i n t e r v i e w s o f c o n t e n t a n a l y s i s u n c e r t a i n w h a t w a s t h e q u a l i t y o f c o n f e r e n c e a n d w o r k s h o p p l a n n i n g Page 54 4 5 E v a l u a t i o n q u e s t i o n I n f o r m a t i o n r e q u i r e d I n f o r m a t i o n s o u r c e s D e s i g n s t r a t e g y D a t a c o l l e c t i o n m e t h o d s D a t a a n a l y s i s m e t h o d s L i m i t a t i o n s W h a t t h e a n a l y s i s a l l o w s u s t o s a y m a n a g e r s u g g e s t i o n s ( i i i ) t a s k m a n a g e r c o m m e n t s p a r t i c i p a n t s a n d t a s k m a n a g e r s a t c o n f e r e n c e ; ( i i i ) s t r u c t u r e d i n t e r v i e w s o f t h e 2 c o n f e r e n c e t a s k m a n a g e r s a n d 5 l e a d w o r k s h o p T M s l a t e r a n d o r g a n i z a t i o n ; i d e n t i f y g o o d / b e s t p r a c t i c e s ; i d e n t i f y m a j o r o p p o r t u n i t i e s f o r i m p r o v e m e n t ; i n v e s t i g a t e e x t e n t t o w h i c h o t h e r c o n f e r e n c e s h a v e e n c o u n t e r e d s i m i l a r p r o b l e m s ; i d e n t i f y e x t e n t t o w h i c h M D F 2 t a s k m a n a g e r s h a v e l e a r n e d f r o m p a s t e x p e r i e n c e ( i n c l u d i n g p r e v i o u s E D I e v a l u a t i o n s ) Page 55 46